> On May 29, 2014, 3:05 p.m., Tom Arnfeld wrote: > > include/mesos/mesos.proto, line 198 > > <https://reviews.apache.org/r/21316/diff/6/?file=594885#file594885line198> > > > > What's the reason you're only caching on a per-framework basis? Surely > > if the URI is identical the content would be... If I have multiple Hadoop > > or Spark clients running but using the same executor, it's a shame they > > wouldn't share a cache entry.
Security concerns. I don't know to what extent what users trust the code behind a URI to remain the same. I can change it to per slave if there is consensus. Or we could have a flag that decides between the two options. > On May 29, 2014, 3:05 p.m., Tom Arnfeld wrote: > > include/mesos/mesos.proto, line 218 > > <https://reviews.apache.org/r/21316/diff/6/?file=594885#file594885line218> > > > > Why not cache by default? Backward-compatibility. If you set no flags, Mesos behaves as before. > On May 29, 2014, 3:05 p.m., Tom Arnfeld wrote: > > src/slave/flags.hpp, line 76 > > <https://reviews.apache.org/r/21316/diff/6/?file=594895#file594895line76> > > > > The naming of this is a little confusing, it's actually a cache rather > > than the directory URIs are "fetched" to.. the final outcome of "fetching" > > is actually the executor sandbox, no? Good point. "fetcher_cache_dir"? > On May 29, 2014, 3:05 p.m., Tom Arnfeld wrote: > > src/launcher/fetcher.cpp, line 369 > > <https://reviews.apache.org/r/21316/diff/6/?file=594887#file594887line369> > > > > Is this really needed? No, will delete it, thx! - Bernd ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21316/#review44301 ----------------------------------------------------------- On May 27, 2014, 4:02 a.m., Bernd Mathiske wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/21316/ > ----------------------------------------------------------- > > (Updated May 27, 2014, 4:02 a.m.) > > > Review request for mesos and Benjamin Hindman. > > > Bugs: MESOS-336 > https://issues.apache.org/jira/browse/MESOS-336 > > > Repository: mesos-git > > > Description > ------- > > Preview of the first cut at fetcher caching. See MESOS-336 JIRA for > explanation for this approach: keep the cache info in the > MesosContainerizerProcess in the save, leverage actor single-threadedness to > deal with concurrency issues without head ache. > > Features so far: > - If URI flag "fetched_externally" (default: false) is set, the fetcher does > what it did in Mesos 0.18 and before. > - If URI flag "cached" (default: false) is not set, the fetcher also fetches > every time as in Mesos 0.18 and before. > - If URI flag "cached" is set, the UIR is only fetched once and all > subsequent fetch attempts copy from the cache file. > - URIs are cached separately per framework (ID). > - Recovery is implemented by simply wiping the entire cache. > - GC for cache files. Global flag sets lifetime after last use. Default is 1 > hour. > > Potential future features: > - symlinks instead of copying > - extraction directly from URI, without cache file > - combine that with symlinks > - Refreshing, explicit cache invalidation > - ... > > > Diffs > ----- > > include/mesos/mesos.proto ce780ca > src/Makefile.am ae576c5 > src/launcher/fetcher.cpp c4425eb > src/local/local.cpp 5d26aff > src/slave/constants.hpp ace4590 > src/slave/constants.cpp 51f65bb > src/slave/containerizer/mesos_containerizer.hpp 1f5908a > src/slave/containerizer/mesos_containerizer.cpp d01d443 > src/slave/containerizer/mesos_fetcher.hpp PRE-CREATION > src/slave/containerizer/mesos_fetcher.cpp PRE-CREATION > src/slave/flags.hpp 15e5b64 > src/slave/slave.hpp 769bd00 > src/slave/slave.cpp a4b9570 > src/tests/containerizer_tests.cpp 2f4888d > src/tests/fetcher_tests.cpp PRE-CREATION > src/tests/slave_tests.cpp 85ca5c4 > > Diff: https://reviews.apache.org/r/21316/diff/ > > > Testing > ------- > > Tests need to be written, i.e. this is not commit-ready, just a preview that > shows everybody how it works. Tested with Mesosaurus, with and without > caching switched on. Seems to work for the easy cases, have not tested all > corner cases yet (e.g. extraction on/off, intermittent gc, ...). > > > Thanks, > > Bernd Mathiske > >
