Unsubscribe
> Em 9 de out. de 2023, à(s) 07:03, Mich Talebzadeh
> escreveu:
>
> Hi,
>
> Please see my responses below:
>
> 1) In Spark Structured Streaming does commit mean streaming data has been
> delivered to the sink like Snowflake?
>
> No. a commit does not refer to data being
Your mileage varies. Often there is a flavour of Cloud Data warehouse
already there. CDWs like BigQuery, Redshift, Snowflake and so forth. They
can all do a good job for various degrees
- Use efficient data types. Choose data types that are efficient for
Spark to process. For example, use
Thank you for your feedback Mich.
In general how can one optimise the cloud data warehouses (the sink part), to
handle streaming Spark data efficiently, avoiding bottlenecks that discussed.
AKOn Monday, 9 October 2023 at 11:04:41 BST, Mich Talebzadeh
wrote:
Hi,
Please see my
Hi,
Please see my responses below:
1) In Spark Structured Streaming does commit mean streaming data has been
delivered to the sink like Snowflake?
No. a commit does not refer to data being delivered to a sink like
Snowflake or bigQuery. The term commit refers to Spark Structured Streaming
(SS)
Hello team
1) In Spark Structured Streaming does commit mean streaming data has been
delivered to the sink like Snowflake?
2) if sinks like Snowflake cannot absorb or digest streaming data in a timely
manner, will there be an impact on spark streaming itself?
Thanks
AK
