I think you can disable it with spark.shuffle.spill=false Thanks Best Regards
On Wed, Mar 18, 2015 at 3:39 PM, Darren Hoo <darren....@gmail.com> wrote: > Thanks, Shao > > On Wed, Mar 18, 2015 at 3:34 PM, Shao, Saisai <saisai.s...@intel.com> > wrote: > >> Yeah, as I said your job processing time is much larger than the >> sliding window, and streaming job is executed one by one in sequence, so >> the next job will wait until the first job is finished, so the total >> latency will be accumulated. >> >> >> >> I think you need to identify the bottleneck of your job at first. If the >> shuffle is so slow, you could enlarge the shuffle fraction of memory to >> reduce the spill, but finally the shuffle data will be written to disk, >> this cannot be disabled, unless you mount your spark.tmp.dir on ramdisk. >> >> >> > I have increased spark.shuffle.memoryFraction to 0.8 which I can see > from SparKUI's environment variables > > But spill always happens even from start when latency is less than slide > window(I changed it to 10 seconds), > the shuflle data disk written is really a snow ball effect, it slows down > eventually. > > I noticed that the files spilled to disk are all very small in size but > huge in numbers: > > total 344K > > drwxr-xr-x 2 root root 4.0K Mar 18 16:55 . > > drwxr-xr-x 66 root root 4.0K Mar 18 16:39 .. > > -rw-r--r-- 1 root root 80K Mar 18 16:54 shuffle_47_519_0.data > > -rw-r--r-- 1 root root 75K Mar 18 16:54 shuffle_48_419_0.data > > -rw-r--r-- 1 root root 36K Mar 18 16:54 shuffle_48_518_0.data > > -rw-r--r-- 1 root root 69K Mar 18 16:55 shuffle_49_319_0.data > > -rw-r--r-- 1 root root 330 Mar 18 16:55 shuffle_49_418_0.data > > -rw-r--r-- 1 root root 65K Mar 18 16:55 shuffle_49_517_0.data > > MemStore says: > > 15/03/18 17:59:43 WARN MemoryStore: Failed to reserve initial memory > threshold of 1024.0 KB for computing block rdd_1338_2 in memory. > 15/03/18 17:59:43 WARN MemoryStore: Not enough space to cache rdd_1338_2 in > memory! (computed 512.0 B so far) > 15/03/18 17:59:43 INFO MemoryStore: Memory use = 529.0 MB (blocks) + 0.0 B > (scratch space shared across 0 thread(s)) = 529.0 MB. Storage limit = 529.9 > MB. > > Not enough space even for 512 byte?? > > > The executors still has plenty free memory: > 0 slave1:40778 0 0.0 B / 529.9 MB 0.0 B 16 0 15047 15063 2.17 > h 0.0 B 402.3 MB 768.0 B > 1 slave2:50452 0 0.0 B / 529.9 MB 0.0 B 16 0 14447 14463 2.17 h 0.0 B > 388.8 MB 1248.0 B > > 1 lvs02:47325 116 27.6 MB / 529.9 MB 0.0 B 8 0 58169 58177 3.16 > h 893.5 MB 624.0 B 1189.9 MB > > <driver> lvs02:47041 0 0.0 B / 529.9 MB 0.0 B 0 0 0 0 0 ms 0.0 B > 0.0 B 0.0 B > > > Besides if CPU or network is the bottleneck, you might need to add more >> resources to your cluster. >> >> >> > 3 dedicated servers each with CPU 16 cores + 16GB memory and Gigabyte > network. > CPU load is quite low , about 1~3 from top, and network usage is far > from saturated. > > I don't even do any usefull complex calculations in this small Simple > App yet. > > >
