----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21316/#review44301 -----------------------------------------------------------
include/mesos/mesos.proto <https://reviews.apache.org/r/21316/#comment78653> What's the reason you're only caching on a per-framework basis? Surely if the URI is identical the content would be... If I have multiple Hadoop or Spark clients running but using the same executor, it's a shame they wouldn't share a cache entry. include/mesos/mesos.proto <https://reviews.apache.org/r/21316/#comment78654> Why not cache by default? src/launcher/fetcher.cpp <https://reviews.apache.org/r/21316/#comment78655> Is this really needed? src/slave/flags.hpp <https://reviews.apache.org/r/21316/#comment78656> The naming of this is a little confusing, it's actually a cache rather than the directory URIs are "fetched" to.. the final outcome of "fetching" is actually the executor sandbox, no? - Tom Arnfeld On May 27, 2014, 11:02 a.m., Bernd Mathiske wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/21316/ > ----------------------------------------------------------- > > (Updated May 27, 2014, 11:02 a.m.) > > > Review request for mesos and Benjamin Hindman. > > > Bugs: MESOS-336 > https://issues.apache.org/jira/browse/MESOS-336 > > > Repository: mesos-git > > > Description > ------- > > Preview of the first cut at fetcher caching. See MESOS-336 JIRA for > explanation for this approach: keep the cache info in the > MesosContainerizerProcess in the save, leverage actor single-threadedness to > deal with concurrency issues without head ache. > > Features so far: > - If URI flag "fetched_externally" (default: false) is set, the fetcher does > what it did in Mesos 0.18 and before. > - If URI flag "cached" (default: false) is not set, the fetcher also fetches > every time as in Mesos 0.18 and before. > - If URI flag "cached" is set, the UIR is only fetched once and all > subsequent fetch attempts copy from the cache file. > - URIs are cached separately per framework (ID). > - Recovery is implemented by simply wiping the entire cache. > - GC for cache files. Global flag sets lifetime after last use. Default is 1 > hour. > > Potential future features: > - symlinks instead of copying > - extraction directly from URI, without cache file > - combine that with symlinks > - Refreshing, explicit cache invalidation > - ... > > > Diffs > ----- > > include/mesos/mesos.proto ce780ca > src/Makefile.am ae576c5 > src/launcher/fetcher.cpp c4425eb > src/local/local.cpp 5d26aff > src/slave/constants.hpp ace4590 > src/slave/constants.cpp 51f65bb > src/slave/containerizer/mesos_containerizer.hpp 1f5908a > src/slave/containerizer/mesos_containerizer.cpp d01d443 > src/slave/containerizer/mesos_fetcher.hpp PRE-CREATION > src/slave/containerizer/mesos_fetcher.cpp PRE-CREATION > src/slave/flags.hpp 15e5b64 > src/slave/slave.hpp 769bd00 > src/slave/slave.cpp a4b9570 > src/tests/containerizer_tests.cpp 2f4888d > src/tests/fetcher_tests.cpp PRE-CREATION > src/tests/slave_tests.cpp 85ca5c4 > > Diff: https://reviews.apache.org/r/21316/diff/ > > > Testing > ------- > > Tests need to be written, i.e. this is not commit-ready, just a preview that > shows everybody how it works. Tested with Mesosaurus, with and without > caching switched on. Seems to work for the easy cases, have not tested all > corner cases yet (e.g. extraction on/off, intermittent gc, ...). > > > Thanks, > > Bernd Mathiske > >
