Re: How to clear the temp files that gets created by shuffle in Spark Streaming

OK. We have a long running streaming job. I was thinking that may be we should have a cron to clear files that are older than 2 days. What would be an appropriate way to do that? On Wed, Nov 18, 2015 at 7:43 PM, Ted Yu wrote: > Have you seen SPARK-5836 ? > Note TD's comment

How to clear the temp files that gets created by shuffle in Spark Streaming

Hi, We have a lot of temp files that gets created due to shuffles caused by group by. How to clear the files that gets created due to intermediate operations in group by? Thanks, Swetha -- View this message in context:

Re: How to clear the temp files that gets created by shuffle in Spark Streaming

Have you seen SPARK-5836 ? Note TD's comment at the end. Cheers On Wed, Nov 18, 2015 at 7:28 PM, swetha wrote: > Hi, > > We have a lot of temp files that gets created due to shuffles caused by > group by. How to clear the files that gets created due to intermediate >