Can you look in the worker logs and see whats going on? It may happen that you ran out of diskspace etc.
Thanks Best Regards On Thu, Jun 25, 2015 at 12:08 PM, barmaley <o...@solver.com> wrote: > I'm running Spark 1.3.1 on AWS... Having long-running application (spark > context) which accepts and completes jobs fine. However, it crashes at as > it > seems random times (anywhere from 1 hour and up to 6 days)... At a latter > case, context run and finished hundreds of jobs without an issue and then > suddenly crashed with the following 2 lines on executors' logs: > > /15/06/24 10:35:44 ERROR executor.CoarseGrainedExecutorBackend: Driver > Disassociated > [akka.tcp://sparkExecutor@ip- > ***-**-36-70.us-west-2.compute.internal:59891] > -> > [akka.tcp://sparkDriver@ip-***-**-42-150.us-west-2.compute.internal:56572] > disassociated! Shutting down. > 15/06/24 10:35:44 WARN remote.ReliableDeliverySupervisor: Association with > remote system > [akka.tcp://sparkDriver@ip-***-**-42-150.us-west-2.compute.internal:56572] > has failed, address is now gated for [5000] ms. Reason is: > [Disassociated]./ > > Following advises on the forums, I've removed SPARK_PUBLIC_DNS setting and > increased the following Akka configs: > > /spark.akka.failure-detector.threshold 30000 > spark.akka.heartbeat.interval 100000 > spark.akka.heartbeat.pauses 600000/ > > This resulted in context crash after 2 hours with different warnings/errors > *during* the operation: > > [2015-06-25 04:50:16,769] WARN ient.AppClient$ClientActor [] > [akka://JobServer/user/context-supervisor/spark-sql-context] - Connection > to > akka.tcp://sparkMaster@ec2-***.us-west-2.compute.amazonaws.com:7077 > failed; > waiting for master to reconnect... > [2015-06-25 04:50:17,400] ERROR cheduler.TaskSchedulerImpl [] > [akka://JobServer/user/context-supervisor/spark-sql-context] - Lost > executor > 0 on ip-***.us-west-2.compute.internal: remote Akka client disassociated > > Despite of these, the log continues and shows couple jobs even done after > these... But then the end of story, context silently died... > > Help with understanding and dealing with this would be greatly appreciated! > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Akka-failures-Driver-Disassociated-tp23486.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
