在实时流处理系统中，面对不同业务需求，如何在保持低延迟的同时，确保足够的吞吐量与系统可扩展性？

25 Tem 20:57

Aynen, I ran into the same trade‑off when we migrated a fraud‑detection pipeline to Flink. The first thing we tackled was backpressure and parallelism—by raising the key‑by parallelism and fine‑tuning the Kafka source’s max.poll.records we got a noticeable jump in raw throughput without hurting latency. Once the data was flowing smoothly, the next bottleneck was the state backend: switching from in‑memory state to RocksDB with incremental checkpoints shaved off a few milliseconds of per‑event latency while still keeping our state size manageable. From there we focused on the fault‑tolerance knobs. A shorter checkpoint interval (≈10 s) gave us fast recovery, but the extra I/O started to hurt latency, so we settled on a dual‑mode strategy: frequent lightweight checkpoints for low‑latency windows and periodic full snapshots for bulk state. The general pattern that works for most workloads is: 1) eliminate backpressure by scaling out and tuning source/parallelism, 2) pick a state backend that matches your state size and access pattern (RocksDB for large, keyed state; in‑memory for tiny), and 3) balance checkpoint frequency against latency requirements. Adjust the watermark strategy accordingly—using punctuated watermarks for bursty streams often helps keep the event‑time semantics accurate without adding extra delay. This approach kept us under 50 ms tail latency while sustaining >100 k events/s and scaling horizontally as the traffic grew.