Hi Eric, I just wanted to do a sanity check, do you know what paths it is trying to write to? I ask because even without spilling, shuffles always write to disk first before transferring data across the network. I had at one point encountered this myself where we accidentally had /tmp mounted on a tiny disk and kept running out of disk on shuffles even though we also don't spill. You may have already considered or ruled this out though.
On Thu, Sep 3, 2015 at 12:56 PM, Eric Walker <eric.wal...@gmail.com> wrote: > Hi, > > I am using Spark 1.3.1 on EMR with lots of memory. I have attempted to > run a large pyspark job several times, specifying > `spark.shuffle.spill=false` in different ways. It seems that the setting > is ignored, at least partially, and some of the tasks start spilling large > amounts of data to disk. The job has been fast enough in the past, but > once it starts spilling to disk it lands on Miller's planet [1]. > > Is this expected behavior? Is it a misconfiguration on my part, e.g., > could there be an incompatible setting that is overriding > `spark.shuffle.spill=false`? Is it something that goes back to Spark > 1.3.1? Is it something that goes back to EMR? When I've allowed the job > to continue on for a while, I've started to see Kryo stack traces in the > tasks that are spilling to disk. The stack traces mention there not being > enough disk space, although a `df` shows plenty of space (perhaps after the > fact, when temporary files have been cleaned up). > > Has anyone run into something like this before? I would be happy to see > OOM errors, because that would be consistent with one understanding of what > might be going on, but I haven't yet. > > Eric > > > [1] https://www.youtube.com/watch?v=v7OVqXm7_Pk&safe=active > -- *Richard Marscher* Software Engineer Localytics Localytics.com <http://localytics.com/> | Our Blog <http://localytics.com/blog> | Twitter <http://twitter.com/localytics> | Facebook <http://facebook.com/localytics> | LinkedIn <http://www.linkedin.com/company/1148792?trk=tyah>
