Brad Willard created SPARK-4779: ----------------------------------- Summary: PySpark Shuffle Fails Looking for Files that Don't Exist when low on Memory Key: SPARK-4779 URL: https://issues.apache.org/jira/browse/SPARK-4779 Project: Spark Issue Type: Bug Components: PySpark, Shuffle Affects Versions: 1.1.0 Environment: ec2 launched cluster with scripts 6 Nodes c3.2xlarge
Reporter: Brad Willard When Spark is tight on memory it starts saying files don't exist during shuffle causing tasks to fail and be rebuilt destroying performance. The same code works flawlessly with smaller datasets with less memory pressure I assume. 14/12/06 18:39:37 WARN scheduler.TaskSetManager: Lost task 292.0 in stage 3.0 (TID 1099, ip-10-13-192-209.ec2.internal): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/root/spark/python/pyspark/worker.py", line 79, in main serializer.dump_stream(func(split_index, iterator), outfile) File "/root/spark/python/pyspark/serializers.py", line 196, in dump_stream self.serializer.dump_stream(self._batched(iterator), stream) File "/root/spark/python/pyspark/serializers.py", line 127, in dump_stream for obj in iterator: File "/root/spark/python/pyspark/serializers.py", line 185, in _batched for item in iterator: File "/root/spark/python/pyspark/shuffle.py", line 370, in _external_items self.mergeCombiners(self.serializer.load_stream(open(p)), IOError: [Errno 2] No such file or directory: '/mnt/spark/spark-local-20141206182702-8748/python/16070/66618000/1/18' org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:124) org.apache.spark.api.python.PythonRDD$$anon$1.next(PythonRDD.scala:91) org.apache.spark.api.python.PythonRDD$$anon$1.next(PythonRDD.scala:87) org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) scala.collection.Iterator$$anon$12.next(Iterator.scala:357) org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) scala.collection.Iterator$$anon$12.next(Iterator.scala:357) scala.collection.Iterator$class.foreach(Iterator.scala:727) scala.collection.AbstractIterator.foreach(Iterator.scala:1157) org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:335) org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$1.apply$mcV$sp(PythonRDD.scala:209) org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$1.apply(PythonRDD.scala:184) org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$1.apply(PythonRDD.scala:184) org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1311) org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:183) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org