RE: FileNotFoundException in appcache shuffle files

I have similar issues... Exception only with very large data. And I tried to double the memory or partition as suggested by some google search, but in vain.. any idea? -- View this message in context:

Re: FileNotFoundException in appcache shuffle files

Usually there is another error or log message before FileNotFoundException. Try to check your logs for something like that. 2015-12-10 10:47 GMT+01:00 kendal : > I have similar issues... Exception only with very large data. > And I tried to double the memory or partition as

Re: FileNotFoundException in appcache shuffle files

Hey, I am having a similar issue, did you manage to find a solution yet? Please check my post below for reference: http://apache-spark-user-list.1001560.n3.nabble.com/IOError-Errno-2-No-such-file-or-directory-tmp-spark-9e23f17e-2e23-4c26-9621-3cb4d8b832da-tmp3i3xno-td21076.html Thank you,

Re: FileNotFoundException in appcache shuffle files

As Jerry said, this is not related to shuffle file consolidation. The unique thing about this problem is that it's failing to find a file while trying to _write_ to it, in append mode. The simplest explanation for this would be that the file is deleted in between some check for existence and

Re: FileNotFoundException in appcache shuffle files

Hi, Ryan We have met similar errors and increasing executor memory solved it. Though I am not sure about the detailed reason, it might be worth a try. On Wed, Oct 29, 2014 at 1:34 PM, Ryan Williams [via Apache Spark User List] ml-node+s1001560n17605...@n3.nabble.com wrote: My job is failing

RE: FileNotFoundException in appcache shuffle files

-+-MSXVcohWEhjiAEKaCccOKr_N5e0HPXcNgnxZd=h...@mail.gmail.com -Original Message- From: Shao, Saisai [saisai.s...@intel.commailto:saisai.s...@intel.com] Sent: Wednesday, October 29, 2014 01:46 AM Eastern Standard Time To: Ryan Williams Cc: user Subject: RE: FileNotFoundException in appcache

RE: FileNotFoundException in appcache shuffle files

Hi Ryan, This is an issue from sort-based shuffle, not consolidated hash-based shuffle. I guess mostly this issue occurs when Spark cluster is in abnormal situation, maybe long time of GC pause or some others, you can check the system status or if there’s any other exceptions beside this one.