Thanks for the comments. Any ideas how to search for the problem? As even a smallish data set (something which I can run locally) fails in a cluster I don't expect this to be a memory problem, for the same reason data/code related problems don't feel reasonable. Could it be something with Spark/Mesos cluster communication? How to debug that?
-Perttu On Sun, May 25, 2014 at 10:42 PM, Mark Hamstra <[email protected]>wrote: > The end of your example is the same as SPARK-1749. When a Mesos job > causes an exception to be thrown in the DAGScheduler, that causes the > DAGScheduler to need to shutdown the system. As part of that shutdown > procedure, the DAGScheduler tries to kill any running jobs; but Mesos > doesn't support that, so that's why you see the failure of doCancelAllJobs > in your stack trace. > > I don't believe that any of that is the root cause of your problem though, > which has to be found in whatever is causing the DAGScheduler to need to > shutdown in the first place. > > > On Sun, May 25, 2014 at 12:10 PM, Perttu Ranta-aho < > [email protected]> wrote: > >> Hi, >> >> We have a small Mesos (0.18.1) cluster with 4 nodes. Upgraded to Spark >> 1.0.0-rc9, to overcome some PySpark bugs. But now we are experiencing >> random crashes with almost every job. Local jobs run fine, but same code >> with same data set in Mesos cluster leads to errors like: >> >> 14/05/22 15:03:34 ERROR DAGSchedulerActorSupervisor: eventProcesserActor >> failed due to the error EOF reached before Python server acknowledged; >> shutting down SparkContext >> 14/05/22 15:03:34 INFO DAGScheduler: Failed to run saveAsTextFile at >> NativeMethodAccessorImpl.java:-2 >> Traceback (most recent call last): >> File "tag_prefixes.py", line 58, in <module> >> tag_prefix_counts.saveAsTextFile('tag_prefix_counts.data') >> File "/srv/spark/spark-1.0.0-bin-2.0.5-alpha/python/pyspark/rdd.py", >> line 910, in saveAsTextFile >> keyed._jrdd.map(self.ctx._jvm.BytesToString()).saveAsTextFile(path) >> File >> "/srv/spark/spark-1.0.0-bin-2.0.5-alpha/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", >> line 537, in __call__ >> File >> "/srv/spark/spark-1.0.0-bin-2.0.5-alpha/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", >> line 300, in get_return_value >> py4j.protocol.Py4JJavaError14/05/22 15:03:34 INFO TaskSchedulerImpl: >> Cancelling stage 0 >> : An error occurred while calling o44.saveAsTextFile. >> : org.apache.spark.SparkException: Job 0 cancelled as part of >> cancellation of all jobs >> at org.apache.spark.scheduler.DAGScheduler.org >> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033) >> at >> org.apache.spark.scheduler.DAGScheduler.handleJobCancellation(DAGScheduler.scala:998) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$doCancelAllJobs$1.apply$mcVI$sp(DAGScheduler.scala:499) >> >> >> Which looks similar to https://issues.apache.org/jira/browse/SPARK-1749, >> with the exception that the code isn't "bad". Furthermore we are seeing >> lots of Mesos(?) warnings like this: >> >> W0522 14:51:19.045565 10497 sched.cpp:901] Attempting to launch task 869 >> with an unknown offer 20140516-155535-170164746-5050-22001-112345 >> >> Which we didn't see with previous Mesos&Spark versions. There aren't any >> related errors in Mesos slave logs, instead they report jobs done without >> problems. Scala code seems to run without problems, so I suppose this isn't >> issue with out Mesos instalation >> >> Any ideas what might be wrong? Or is this bug in Spark? >> >> >> -Perttu >> > >
