Yeah, specifically the docker issue was related to volumes not being removed with `docker rm` but that's a separate issue.
So right now mesos won't remove older work directories to make room for new ones (old ones that have already been scheduled for removal in a few days time)? This means when the disk gets quite full, newer work directories will be removed much faster than older ones. Is that correct? On 31 July 2014 23:56, Benjamin Mahler <benjamin.mah...@gmail.com> wrote: > Apologies for the lack of documentation, in the default setup, the slave > will schedule the work directories for garbage collection when: > > (1) Executors terminate. > (2) The slave recovers and discovers work directories for terminal > executors. > > Sounds like the docker integration code you're using has a bug in this > respect, either by not scheduling docker directories for garbage collection > during (1) and/or (2). > > > On Thu, Jul 31, 2014 at 3:40 PM, Tom Arnfeld <t...@duedil.com> wrote: > >> I don't have them to hand now, but I recall it saying something in the >> high 90's and 0ns for the max allowed age. I actually found the root cause >> of the probably, docker related and out of mesos's control... though i'm >> still curious about the expected behaviour of the GC process. It doesn't >> seem to be well documented anywhere. >> >> Tom. >> >> >> On 31 July 2014 23:33, Benjamin Mahler <benjamin.mah...@gmail.com> wrote: >> >>> What do the slave logs say? >>> >>> E.g. >>> >>> I0731 22:22:17.851347 23525 slave.cpp:2879] Current usage 7.84%. Max >>> allowed age: 5.751197441470081days >>> >>> >>> On Wed, Jul 30, 2014 at 8:55 AM, Tom Arnfeld <t...@duedil.com> wrote: >>> >>>> I'm not sure if this is something already supported by mesos, and if so >>>> it'd be great if someone could point me in the right direction. >>>> >>>> Is there a way of asking a slave to garbage collect old executors >>>> manually? >>>> >>>> Maybe i'm misunderstanding things, but as each executor does (insert >>>> knowledge gap) mesos works out how long it is able to keep the sandbox for >>>> and schedules it for garbage collection appropriately, also taking into >>>> account the command line >>>> >>>> The disk on one of my slaves is getting quite full (98%) and i'm >>>> curious how mesos is going to behave in this situation. Should it start >>>> clearing things up, given a task could launch that needs to use an amount >>>> of disk space, but that disk is being eaten up by old executor sandboxes. >>>> >>>> It may be worth noting i'm not specifying --gc_delay on any slave right >>>> now, perhaps I should be? >>>> >>>> Any input would be much appreciated. >>>> >>>> Cheers, >>>> >>>> Tom. >>>> >>> >>> >> >
