Is it possible to re-run your job with spark.eventLog.enabled to true, and
send the resulting logs to the list? Those have more per-task information
that can help diagnose this.
-Kay
On Wed, Jan 21, 2015 at 1:57 AM, Fengyun RAO raofeng...@gmail.com wrote:
btw: Shuffle Write(11 GB) mean 11 GB
Hi Andrew and Gary,
I've done some experimentation with this and had similar results. I can't
explain the speedup in write performance, but I dug into the read slowdown
and found that enabling short-circuit reads results in Hadoop not doing
read-ahead in the same way. At a high level, when SCR
