Re: Fetcher cache: caching even more while an executor is alive

Just a random thought, have you tried something like bittorrent based deployments. They are really efficient when you have to distribute big artifacts across cluster of machines. Following two projects might be helpful in achieving that: 1. http://erdgeist.org/arts/software/opentracker/ 2. http:/

Re: Fetcher cache: caching even more while an executor is alive

I'd try the Docker image approach. We've done this in the past and used our CM tool to 'seed' all slaves by running 'docker pull foo:v1' across them all in advance, saved a lot of startup time (although we were only dealing with a Gb or so of dependencies). On 5 July 2016 at 11:23, Kota UENISHI

Re: Fetcher cache: caching even more while an executor is alive

Thanks, it looks promising to me - I was aware of persistent volumes because I thought the use case was different, like for databases. I'll try it on. As the document says > persistent volumes are associated with roles, this makes failure handling a little bit difficult - As my framework is not

RE: Fetcher cache: caching even more while an executor is alive

As you're writing the framework, have you looked at reserving persistent volumes? I think it might help in your use case: http://mesos.apache.org/documentation/latest/persistent-volume/ Aaron -- Aaron Carey Production Engineer - Cloud Pipeline Industrial Light & Magic London 020 3751 9150 ___