Re: Sparklyr and idle executors

2018-03-16 Thread Florian Dewes
dynamicAllocation.enabled=true > ? > > On Thu, Mar 15, 2018 at 1:47 PM, Florian Dewes <fde...@gmail.com > <mailto:fde...@gmail.com>> wrote: > Hi all, > > I am currently trying to enable dynamic resource allocation for a little yarn > managed spark cluster. &

Sparklyr and idle executors

2018-03-15 Thread Florian Dewes
Hi all, I am currently trying to enable dynamic resource allocation for a little yarn managed spark cluster. We are using sparklyr to access spark from R and have multiple jobs which should run in parallel, because some of them take several days to complete or are in development. Everything