Can you try setting spark.executor.extraJavaOptions to have -D java.io.tmpdir=someValue
Regards, Keith. http://keith-chapman.com On Mon, Mar 19, 2018 at 10:29 AM, Michael Shtelma <mshte...@gmail.com> wrote: > Hi Keith, > > Thank you for your answer! > I have done this, and it is working for spark driver. > I would like to make something like this for the executors as well, so > that the setting will be used on all the nodes, where I have executors > running. > > Best, > Michael > > > On Mon, Mar 19, 2018 at 6:07 PM, Keith Chapman <keithgchap...@gmail.com> > wrote: > > Hi Michael, > > > > You could either set spark.local.dir through spark conf or java.io.tmpdir > > system property. > > > > Regards, > > Keith. > > > > http://keith-chapman.com > > > > On Mon, Mar 19, 2018 at 9:59 AM, Michael Shtelma <mshte...@gmail.com> > wrote: > >> > >> Hi everybody, > >> > >> I am running spark job on yarn, and my problem is that the blockmgr-* > >> folders are being created under > >> /tmp/hadoop-msh/nm-local-dir/usercache/msh/appcache/application_id/* > >> The size of this folder can grow to a significant size and does not > >> really fit into /tmp file system for one job, which makes a real > >> problem for my installation. > >> I have redefined hadoop.tmp.dir in core-site.xml and > >> yarn.nodemanager.local-dirs in yarn-site.xml pointing to other > >> location and expected that the block manager will create the files > >> there and not under /tmp, but this is not the case. The files are > >> created under /tmp. > >> > >> I am wondering if there is a way to make spark not use /tmp at all and > >> configure it to create all the files somewhere else ? > >> > >> Any assistance would be greatly appreciated! > >> > >> Best, > >> Michael > >> > >> --------------------------------------------------------------------- > >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >> > > >
