Re: [Spark-Core] Long scheduling delays (1+ hour)

Forgot to add the link https://jira.apache.org/jira/browse/KAFKA-5649 -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: [Spark-Core] Long scheduling delays (1+ hour)

Could you please give some feedback. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: [Spark-Core] Long scheduling delays (1+ hour)

Actually, our job runs fine for 17-18 hours and this behavior just suddenly starts happening after that. We found the following ticket which is exactly what is happening in our Kafka cluster also. WARN Failed to send SSL Close message (org.apache.kafka.common.network.SslTransportLayer) You

Re: [Spark-Core] Long scheduling delays (1+ hour)

Hi, This has to do with your batch duration and processing time, as a rule, the batch duration should be lower than the processing time of your data. As I can see from your screenshots, your batch duration is 10 seconds but your processing time is more than a minute mostly, this adds up and you

[Spark-Core] Long scheduling delays (1+ hour)

We are facing an issue with very long scheduling delays in Spark (upto 1+ hours). We are using Spark-standalone. The data is being pulled from Kafka. Any help would be much appreciated. I have attached the screenshots.