Hi, This has to do with your batch duration and processing time, as a rule, the batch duration should be lower than the processing time of your data. As I can see from your screenshots, your batch duration is 10 seconds but your processing time is more than a minute mostly, this adds up and you will end up having a lot of scheduling delay.
Maybe see, why does it take 1 min to process 100 records and fix the logic. Also, I see you have higher number of events which takes some time lower amount of processing time. Fix the code logic and this should be fixed. Thanks & Regards Biplob Biswas On Wed, Nov 7, 2018 at 11:08 AM bsikander <behro...@gmail.com> wrote: > We are facing an issue with very long scheduling delays in Spark (upto 1+ > hours). > We are using Spark-standalone. The data is being pulled from Kafka. > > Any help would be much appreciated. > > I have attached the screenshots. > < > http://apache-spark-user-list.1001560.n3.nabble.com/file/t8018/1-stats.png> > > <http://apache-spark-user-list.1001560.n3.nabble.com/file/t8018/4.png> > <http://apache-spark-user-list.1001560.n3.nabble.com/file/t8018/3.png> > <http://apache-spark-user-list.1001560.n3.nabble.com/file/t8018/2.png> > > > > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
