Re: [Spark Streaming] [DISCUSS] Clear metadata method and Generate Batches using same Event Loop

2021-03-10 Thread Karthikeyan Ravi
Bring it up. On Mon, Feb 8, 2021 at 12:19 PM Karthikeyan Ravi wrote: > Hello, > > Our system observed this behaviour of Batches getting delayed for > generation in spark streaming and thereby creating a very big batch and > followed by few zero record batches. I read the code and added logs to

[Spark Streaming] [DISCUSS] Clear metadata method and Generate Batches using same Event Loop

2021-02-08 Thread Karthikeyan Ravi
Hello, Our system observed this behaviour of Batches getting delayed for generation in spark streaming and thereby creating a very big batch and followed by few zero record batches. I read the code and added logs to confirm this root cause details below 1. GenerateBatch(Recurring Timer) and