Thank you.
From: Peyman Mohajerian [mailto:mohaj...@gmail.com]
Sent: Thursday, February 24, 2022 9:00 AM
To: Michael Williams (SSI)
Cc: user@spark.apache.org
Subject: Re: Consuming from Kafka to delta table - stream or batch mode?
If you want to batch consume from Kafka, trigger-once config
If you want to batch consume from Kafka, trigger-once config would work
with structured streaming and you get the benefit of the checkpointing.
On Thu, Feb 24, 2022 at 6:07 AM Michael Williams (SSI) <
michael.willi...@ssigroup.com> wrote:
> Hello,
>
>
>
> Our team is working with Spark (for the
Hello,
Our team is working with Spark (for the first time) and one of the sources we
need to consume is Kafka (multiple topics). Are there any practical or
operational issues to be aware of when deciding whether to a) consume in
batches until all messages are consumed then shut down the spark
