Hi! Yes these files are for shuffle blocks however they need to be cleaned as well right? I had been running a streaming application for 2 days. The third day my disk fills up with all .index and .data files and my assumption is that these files had been there since the start of my streaming application I should have checked the time stamp before doing rm -rf. Please let me know if I am wrong
Sent from my iPhone > On Jan 26, 2017, at 4:24 PM, Takeshi Yamamuro <linguin....@gmail.com> wrote: > > Yea, I think so and they are the intermediate files for shuffling. Probably, > kant checked the configuration here > (http://spark.apache.org/docs/latest/spark-standalone.html) though, this is > not related to the issue. > > // maropu > >> On Fri, Jan 27, 2017 at 7:46 AM, Jacek Laskowski <ja...@japila.pl> wrote: >> Hi, >> >> The files are for shuffle blocks. Where did you find the docs about them? >> >> Jacek >> >> On 25 Jan 2017 8:41 p.m., "kant kodali" <kanth...@gmail.com> wrote: >> oh sorry its actually in the documentation. I should just set >> spark.worker.cleanup.enabled = true >> >> On Wed, Jan 25, 2017 at 11:30 AM, kant kodali <kanth...@gmail.com> wrote: >>> I have bunch of .index and .data files like that fills up my disk. I am not >>> sure what the fix is? I am running spark 2.0.2 in stand alone mode >>> >>> Thanks! >>> >>> >>> >>> >>> >> >> > > > > -- > --- > Takeshi Yamamuro
