RE: Cleaning up spark.local.dir automatically

That’s really useful, thanks. From: Andrew Ash [mailto:and...@andrewash.com] Sent: 09 January 2015 22:42 To: England, Michael (IT/UK) Cc: raghavendra.pan...@gmail.com; user Subject: Re: Cleaning up spark.local.dir automatically That's a worker setting which cleans up the files left behind

RE: Cleaning up spark.local.dir automatically

Thanks, I imagine this will kill any cached RDDs if their files are beyond the ttl? Thanks From: Raghavendra Pandey [mailto:raghavendra.pan...@gmail.com] Sent: 09 January 2015 15:29 To: England, Michael (IT/UK); user@spark.apache.org Subject: Re: Cleaning up spark.local.dir automatically You

Re: Cleaning up spark.local.dir automatically

You may like to look at spark.cleaner.ttl configuration which is infinite by default. Spark has that configuration to delete temp files time to time. On Fri Jan 09 2015 at 8:34:10 PM michael.engl...@nomura.com wrote: Hi, Is there a way of automatically cleaning up the spark.local.dir after

Re: Cleaning up spark.local.dir automatically

...@nomura.com wrote: Thanks, I imagine this will kill any cached RDDs if their files are beyond the ttl? Thanks *From:* Raghavendra Pandey [mailto:raghavendra.pan...@gmail.com] *Sent:* 09 January 2015 15:29 *To:* England, Michael (IT/UK); user@spark.apache.org *Subject:* Re: Cleaning up

Cleaning up spark.local.dir automatically

Hi, Is there a way of automatically cleaning up the spark.local.dir after a job has been run? I have noticed a large number of temporary files have been stored here and are not cleaned up. The only solution I can think of is to run some sort of cron job to delete files older than a few days. I