If you don't set rate limiting through `maxOffsetsPerTrigger`, Structured Streaming will always process until the end of the stream. So number of records waiting to be processed should be 0 at the start of each trigger.
On Mon, Jul 30, 2018 at 8:03 AM, Kailash Kalahasti < kailash.kalaha...@gmail.com> wrote: > Is there any way to find out backlog on kafka topic while using spark > structured streaming ? I checked few consumer apis but that requires to > enable groupid for streaming, but seems it is not allowed. > > Basically i want to know number of records waiting to be processed. > > Any suggestions ? >
