Two quick comments: With unaligned checkpoints which are released with
Flink 1.11.0, the problem of slow checkpoints under backpressure has been
resolved/mitigated to a good extent. Moreover, the community wants to work
on event time alignment for sources in the next release. This should
prevent
Thanks for the reply Chen.
My use case is a "simple" get from Kafka into S3. The job can read very
quickly from Kafka and S3 is having some issues keeping up. The
backpressure don't have enough time to actuate in this case, and when it
reaches the checkpoint time some errors like heartbeat
My two cents here,
- flink job already has back pressure so rate limit can be done via setting
parallelism to proper number in some use cases. There is an open issue of
checkpointing reliability when back pressure, community seems working on it.
- rate limit can be abused easily and cause lot
I've noticed that this FLINK-11501 was implemented in
flink-connector-kafka-0.10 [1], but it wasn't in the current version of the
flink-connector-kafka. There is any reason for this, and why should be the
best solution to implement a rate limit functionality in the current Kafka
consumer?
Thanks,
