Unsubscribe
> Em 9 de out. de 2023, à(s) 07:03, Mich Talebzadeh <mich.talebza...@gmail.com>
> escreveu:
>
> Hi,
>
> Please see my responses below:
>
> 1) In Spark Structured Streaming does commit mean streaming data has been
> delivered to the sink like Snowflake?
>
> No. a commit does not refer to data being delivered to a sink like Snowflake
> or bigQuery. The term commit refers to Spark Structured Streaming (SS)
> internals. Specifically it means that a micro-batch of data has been
> processed by SSS. In the checkpoint directory there is a subdirectory called
> commits that marks the micro-batch process as completed.
>
> 2) if sinks like Snowflake cannot absorb or digest streaming data in a
> timely manner, will there be an impact on spark streaming itself?
>
> Yes, it can potentially impact SSS. If the sink cannot absorb data in a
> timely manner, the batches will start to back up in SSS. This can cause Spark
> to run out of memory and the streaming job to fail. As I understand, Spark
> will use a combination of memory and disk storage (checkpointing). This can
> also happen if the network interface between Spark and the sink is disrupted.
> On the other hand Spark may slow down, as it tries to process the backed-up
> batches of data. You want to avoid these scenarios.
>
> HTH
>
> Mich Talebzadeh,
> Distinguished Technologist, Solutions Architect & Engineer
> London
> United Kingdom
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss,
> damage or destruction of data or any other property which may arise from
> relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>
>
> On Sun, 8 Oct 2023 at 19:50, ashok34...@yahoo.com.INVALID
> <ashok34...@yahoo.com.invalid> wrote:
>> Hello team
>>
>> 1) In Spark Structured Streaming does commit mean streaming data has been
>> delivered to the sink like Snowflake?
>>
>> 2) if sinks like Snowflake cannot absorb or digest streaming data in a
>> timely manner, will there be an impact on spark streaming itself?
>>
>> Thanks
>>
>> AK
