RE: Consuming from Kafka to delta table - stream or batch mode?

2022-02-24 Thread Michael Williams (SSI)
Thank you. From: Peyman Mohajerian [mailto:mohaj...@gmail.com] Sent: Thursday, February 24, 2022 9:00 AM To: Michael Williams (SSI) Cc: user@spark.apache.org Subject: Re: Consuming from Kafka to delta table - stream or batch mode? If you want to batch consume from Kafka, trigger-once config

Re: Consuming from Kafka to delta table - stream or batch mode?

2022-02-24 Thread Peyman Mohajerian
If you want to batch consume from Kafka, trigger-once config would work with structured streaming and you get the benefit of the checkpointing. On Thu, Feb 24, 2022 at 6:07 AM Michael Williams (SSI) < michael.willi...@ssigroup.com> wrote: > Hello, > > > > Our team is working with Spark (for the