I have a DStream receiving data from a socket. I'm using local mode.
I set spark.streaming.unpersist to false and leave
spark.cleaner.ttl to be infinite.
I can see files for input and shuffle blocks under spark.local.dir
folder and the size of folder keeps increasing, although JVM's memory
usage
Hi Haopu,
Please see the inline comments.
Thanks
Jerry
-Original Message-
From: Haopu Wang [mailto:hw...@qilinsoft.com]
Sent: Wednesday, July 23, 2014 3:00 PM
To: user@spark.apache.org
Subject: spark.streaming.unpersist and spark.cleaner.ttl
I have a DStream receiving data from
a look at DStream.scala although I have no Scala experience.
-Original Message-
From: Shao, Saisai [mailto:saisai.s...@intel.com]
Sent: 2014年7月23日 15:13
To: user@spark.apache.org
Subject: RE: spark.streaming.unpersist and spark.cleaner.ttl
Hi Haopu,
Please see the inline comments
will take a look at DStream.scala although I have no Scala experience.
-Original Message-
From: Shao, Saisai [mailto:saisai.s...@intel.com]
Sent: 2014年7月23日 15:13
To: user@spark.apache.org
Subject: RE: spark.streaming.unpersist and spark.cleaner.ttl
Hi Haopu,
Please see the inline
