"java.io.IOException: No space left on device" while doing repartitioning in Spark

2015-05-04 Thread shahab
Hi, I am getting "No space left on device" exception when doing repartitioning of approx. 285 MB of data while these is still 2 GB space left ?? does it mean that repartitioning needs more space (more than 2 GB) for repartitioning of 285 MB of data ?? best, /Shahab java.io.IOException: No spa

Re: "java.io.IOException: No space left on device" while doing repartitioning in Spark

2015-05-04 Thread Ted Yu
See https://wiki.gentoo.org/wiki/Knowledge_Base:No_space_left_on_device_while_there_is_plenty_of_space_available What's the value for spark.local.dir property ? Cheers On Mon, May 4, 2015 at 6:57 AM, shahab wrote: > Hi, > > I am getting "No space left on device" exception when doing repartitio

Re: "java.io.IOException: No space left on device" while doing repartitioning in Spark

2015-05-04 Thread Akhil Das
It could be filling up your /tmp directory. You need to set your spark.local.dir or you can also specify SPARK_WORKER_DIR to another location which has sufficient space. Thanks Best Regards On Mon, May 4, 2015 at 7:27 PM, shahab wrote: > Hi, > > I am getting "No space left on device" exception