Thanks, Shao
On Wed, Mar 18, 2015 at 3:34 PM, Shao, Saisai saisai.s...@intel.com wrote:
Yeah, as I said your job processing time is much larger than the sliding
window, and streaming job is executed one by one in sequence, so the next
job will wait until the first job is finished, so the
I think you can disable it with spark.shuffle.spill=false
Thanks
Best Regards
On Wed, Mar 18, 2015 at 3:39 PM, Darren Hoo darren@gmail.com wrote:
Thanks, Shao
On Wed, Mar 18, 2015 at 3:34 PM, Shao, Saisai saisai.s...@intel.com
wrote:
Yeah, as I said your job processing time is much
I've already done that:
From SparkUI Environment Spark properties has:
spark.shuffle.spillfalse
On Wed, Mar 18, 2015 at 6:34 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:
I think you can disable it with spark.shuffle.spill=false
Thanks
Best Regards
On Wed, Mar 18, 2015 at 3:39 PM,
: [spark-streaming] can shuffle write to disk be disabled?
I've already done that:
From SparkUI Environment Spark properties has:
spark.shuffle.spill
false
On Wed, Mar 18, 2015 at 6:34 PM, Akhil Das
ak...@sigmoidanalytics.commailto:ak...@sigmoidanalytics.com wrote:
I think you can disable
On Wed, Mar 18, 2015 at 8:31 PM, Shao, Saisai saisai.s...@intel.com wrote:
From the log you pasted I think this (-rw-r--r-- 1 root root 80K Mar
18 16:54 shuffle_47_519_0.data) is not shuffle spilled data, but the
final shuffle result.
why the shuffle result is written to disk?
As I
Hi, Saisai
Here is the duration of one of the jobs, 22 seconds in total, it is longer
than the sliding window.
Stage Id Description Submitted Duration Tasks:
Succeeded/Total Input Output Shuffle Read Shuffle Write
342foreach at SimpleApp.scala:58 2015/03/18
to add more
resources to your cluster.
Thanks
Jerry
From: Darren Hoo [mailto:darren@gmail.com]
Sent: Wednesday, March 18, 2015 3:24 PM
To: Shao, Saisai
Cc: user@spark.apache.org
Subject: Re: [spark-streaming] can shuffle write to disk be disabled?
Hi, Saisai
Here is the duration of one
Please see the inline comments.
Thanks
Jerry
From: Darren Hoo [mailto:darren@gmail.com]
Sent: Wednesday, March 18, 2015 9:30 PM
To: Shao, Saisai
Cc: user@spark.apache.org; Akhil Das
Subject: Re: [spark-streaming] can shuffle write to disk be disabled?
On Wed, Mar 18, 2015 at 8:31 PM, Shao
Would you please check your driver log or streaming web UI to see each job's
latency, including processing latency and total latency.
Seems from your code, sliding window is just 3 seconds, so you will process
each 60 second's data in 3 seconds, if processing latency is larger than the
sliding
