Thank you for your feedback Mich. In general how can one optimise the cloud data warehouses (the sink part), to handle streaming Spark data efficiently, avoiding bottlenecks that discussed.
AK On Monday, 9 October 2023 at 11:04:41 BST, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: Hi, Please see my responses below: 1) In Spark Structured Streaming does commit mean streaming data has been delivered to the sink like Snowflake? No. a commit does not refer to data being delivered to a sink like Snowflake or bigQuery. The term commit refers to Spark Structured Streaming (SS) internals. Specifically it means that a micro-batch of data has been processed by SSS. In the checkpoint directory there is a subdirectory called commits that marks the micro-batch process as completed. 2) if sinks like Snowflake cannot absorb or digest streaming data in a timely manner, will there be an impact on spark streaming itself? Yes, it can potentially impact SSS. If the sink cannot absorb data in a timely manner, the batches will start to back up in SSS. This can cause Spark to run out of memory and the streaming job to fail. As I understand, Spark will use a combination of memory and disk storage (checkpointing). This can also happen if the network interface between Spark and the sink is disrupted. On the other hand Spark may slow down, as it tries to process the backed-up batches of data. You want to avoid these scenarios. HTH Mich Talebzadeh,Distinguished Technologist, Solutions Architect & Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destructionof data or any other property which may arise from relying on this email's technical content is explicitly disclaimed.The author will in no case be liable for any monetary damages arising from suchloss, damage or destruction. On Sun, 8 Oct 2023 at 19:50, ashok34...@yahoo.com.INVALID <ashok34...@yahoo.com.invalid> wrote: Hello team 1) In Spark Structured Streaming does commit mean streaming data has been delivered to the sink like Snowflake? 2) if sinks like Snowflake cannot absorb or digest streaming data in a timely manner, will there be an impact on spark streaming itself? Thanks AK
