Re: Worker's BlockManager Folder not getting cleared

Looking for answer to this. Is it safe to delete the older files using find . -type f -cmin +200 -name "shuffle*" -exec rm -rf {} \; For a window duration of 2 hours how older files can we delete ? Thanks. On Sun, Feb 14, 2016 at 12:34 PM, Abhishek Anand wrote: > Hi

Re: Worker's BlockManager Folder not getting cleared

Hi All, Any ideas on this one ? The size of this directory keeps on growing. I can see there are many files from a day earlier too. Cheers !! Abhi On Tue, Jan 26, 2016 at 7:13 PM, Abhishek Anand wrote: > Hi Adrian, > > I am running spark in standalone mode. > > The

Re: Worker's BlockManager Folder not getting cleared

Hi Abhi - are you running on Mesos perchance? If so then with spark <1.6 you will be hitting https://issues.apache.org/jira/browse/SPARK-10975 With spark >= 1.6: https://issues.apache.org/jira/browse/SPARK-12430 and also be aware of: https://issues.apache.org/jira/browse/SPARK-12583 On

Re: Worker's BlockManager Folder not getting cleared

Hi Adrian, I am running spark in standalone mode. The spark version that I am using is 1.4.0 Thanks, Abhi On Tue, Jan 26, 2016 at 4:10 PM, Adrian Bridgett wrote: > Hi Abhi - are you running on Mesos perchance? > > If so then with spark <1.6 you will be hitting >

Worker's BlockManager Folder not getting cleared

Hi All, How long the shuffle files and data files are stored on the block manager folder of the workers. I have a spark streaming job with window duration of 2 hours and slide interval of 15 minutes. When I execute the following command in my block manager path find . -type f -cmin +150 -name