Just a random thought, have you tried something like bittorrent based deployments. They are really efficient when you have to distribute big artifacts across cluster of machines. Following two projects might be helpful in achieving that:
1. http://erdgeist.org/arts/software/opentracker/ 2. http://sourceforge.net/projects/ctorrent/ On Jul 7, 2016 10:58 PM, "Dick Davies" <d...@hellooperator.net> wrote: > I'd try the Docker image approach. > We've done this in the past and used our CM tool to 'seed' all slaves > by running 'docker pull foo:v1' across them all in advance, saved a > lot of startup time (although we were only dealing with a Gb or so of > dependencies). > > On 5 July 2016 at 11:23, Kota UENISHI <ueni...@nautilus-technologies.com> > wrote: > > Thanks, it looks promising to me - I was aware of persistent volumes > > because I thought the use case was different, like for databases. I'll > > try it on. > > > > As the document says > > > >> persistent volumes are associated with roles, > > > > this makes failure handling a little bit difficult - As my framework > > is not handling failure well enough, those volume IDs must be > > remembered during framework restart or failover, or must get recovered > > after. Restarted framework must reuse or collect already reserved > > volumes or those volumes just gets leaking. > > > > Kota UENISHI > > > > > > On Tue, Jul 5, 2016 at 6:03 PM, Aaron Carey <aca...@ilm.com> wrote: > >> As you're writing the framework, have you looked at reserving > persistent volumes? I think it might help in your use case: > >> > >> http://mesos.apache.org/documentation/latest/persistent-volume/ > >> > >> Aaron > >> > >> -- > >> > >> Aaron Carey > >> Production Engineer - Cloud Pipeline > >> Industrial Light & Magic > >> London > >> 020 3751 9150 > >> > >> ________________________________________ > >> From: 上西康太 [ueni...@nautilus-technologies.com] > >> Sent: 05 July 2016 08:24 > >> To: user@mesos.apache.org > >> Subject: Fetcher cache: caching even more while an executor is alive > >> > >> Hi, > >> I'm developing my own framework - that distributes >100 independent > >> tasks across the cluster and just run them arbitrarily. My problem is, > >> each task execution environment is a bit large tarball (2~6GB, mostly > >> application jar files) and task itself finishes within 1~200 seconds, > >> while tarball extraction takes like tens of seconds every time. > >> Extracting the same tarball again and again in all tasks is a wasteful > >> overhead that cannot be ignored. > >> > >> Fetcher cache is great, but in my case, fetcher cache isn't even > >> enough and I want to preserve all files extracted from the tarball > >> while my executor is alive. If Mesos could cache all files extracted > >> from the tarball by omitting not only download but extraction, I could > >> save more time. > >> > >> In "Fetcher Cache Internals" [1] or in "Fetcher Cache" [2] section in > >> the official document, such issues or future work is not mentioned - > >> how do you solve this kind of extraction overhead problem, when you > >> have rather large resource ? > >> > >> An option would be setting up an internal docker registry and let > >> slaves cache the docker image that includes our jar files and save > >> tarball extraction. But, I want to prevent our system from additional > >> moving parts as much as I can. > >> > >> Another option might be let fetcher fetch all jar files independently > >> in slaves, but I think it feasible, but I don't think it manageable in > >> production in an easy way. > >> > >> PS Mesos is great; it is helping us a lot - I want to appreciate all > >> the efforts by the community. Thank you so much! > >> > >> [1] > http://mesos.apache.org/documentation/latest/fetcher-cache-internals/ > >> [2] http://mesos.apache.org/documentation/latest/fetcher/ > >> > >> Kota UENISHI >
