Swapnil,
What do you think might be the size of the file that’s not found? For spark
version below 2.0.0 there might be issues with blocks of size 2g.
Is the file actually on a file system?
I’d try to increase default parallelism to make sure partitions got smaller.
Hope, this helps.
> On
Hello All
I am facing FileNotFoundException for shuffle index file when running
job with large data. Same job runs fine with smaller datasets. These our my
cluster specifications -
No of nodes - 19
Total cores - 380
Memory per executor - 32G
Spark 1.6 mapr version
