I somehow missed that parameter when I was reviewing the documentation,
that should do the trick! Thank you!
2014-09-10 2:10 GMT+01:00 Shao, Saisai saisai.s...@intel.com:
Hi Luis,
The parameter “spark.cleaner.ttl” and “spark.streaming.unpersist” can be
used to remove useless timeout
Sánchez
langel.gro...@gmail.com wrote:
I somehow missed that parameter when I was reviewing the documentation,
that should do the trick! Thank you!
2014-09-10 2:10 GMT+01:00 Shao, Saisai saisai.s...@intel.com:
Hi Luis,
The parameter “spark.cleaner.ttl” and “spark.streaming.unpersist” can
,
The parameter “spark.cleaner.ttl” and “spark.streaming.unpersist” can be
used to remove useless timeout streaming data, the difference is that
“spark.cleaner.ttl” is time-based cleaner, it does not only clean streaming
input data, but also Spark’s useless metadata; while
“spark.streaming.unpersist
langel.gro...@gmail.com wrote:
I somehow missed that parameter when I was reviewing the documentation,
that should do the trick! Thank you!
2014-09-10 2:10 GMT+01:00 Shao, Saisai saisai.s...@intel.com:
Hi Luis,
The parameter “spark.cleaner.ttl” and “spark.streaming.unpersist” can be
used
to use spark.cleaner.ttl and
spark.streaming.unpersist together to mitigate that problem. And I also
wonder if new RDD are being batched while a RDD is being processed.
Regards,
Luis