Hi,
In spark one can handle back pressure by setting the spark conf parameter:
sparkConf.set("spark.streaming.backpressure.enabled","true")
With backpressure you make Spark Streaming application stable, i.e.
receives data only as fast as it can process it. In general one needs to
ensure that your microbatching processing time is less that your batch
interval, i.e the rate that your producer sends data into Kafka. For
example this is shown in Spark GUI below for batch interval = 2 seconds
[image: image.png]
Is there such procedure in Flink please?
Thanks
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
