Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review73991
---



docs/fetcher.md


loosely



docs/fetcher.md


The framework provides a list of CommandInfo::URI values as part of the 
TaskInfo message that becomes the input to the Mesos fetcher.



docs/fetcher.md


same URI leads to downloading another copy of the same resource.



docs/fetcher.md


What's file size queries to the cache?



docs/fetcher.md


Not sure if putting the struct here is a good idea, as it's most likely 
going to be changed in the future.



docs/fetcher.md


the affected task



docs/fetcher.md


per-user basis


- Timothy Chen


On Feb. 23, 2015, 11:33 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 23, 2015, 11:33 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fet

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 24, 2015, 11:26 p.m., Timothy Chen wrote:
> > docs/fetcher.md, line 16
> > 
> >
> > The framework provides a list of CommandInfo::URI values as part of the 
> > TaskInfo message that becomes the input to the Mesos fetcher.

Even though you do not mean this, your suggested text could be misread that the 
entire TaskInfo message becomes input to the fetcher. Now proposing: 

As part of a TaskInfo message, the framework ordering the task's execution 
provides a list of
CommandInfo::URI protobuf values, which becomes the input to the Mesos fetcher.


> On Feb. 24, 2015, 11:26 p.m., Timothy Chen wrote:
> > docs/fetcher.md, line 68
> > 
> >
> > Not sure if putting the struct here is a good idea, as it's most likely 
> > going to be changed in the future.

When the struct changes, we change the doc.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review73991
---


On Feb. 23, 2015, 3:33 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 23, 2015, 3:33 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 25, 2015, 2:19 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Addressed Tim's review.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
  src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
074a2d82dcd882e52f8cd62ed7493295596acb26 
  src/slave/containerizer/mesos/containerizer.cpp 
d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
  src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
  src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
  src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reuse of download 

Re: Review Request 30774: Fetcher Cache

[Mesos ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74010
---


Bad patch!

Reviews applied: [30609, 30606, 30774]

Failed command: ./support/apply-review.sh -n -r 30774

Error:
 2015-02-25 10:26:35 URL:https://reviews.apache.org/r/30774/diff/raw/ 
[159918/159918] -> "30774.patch" [1]
error: patch failed: src/Makefile.am:1301
error: src/Makefile.am: patch does not apply
error: patch failed: src/launcher/fetcher.cpp:34
error: src/launcher/fetcher.cpp: patch does not apply
error: patch failed: src/slave/containerizer/fetcher.hpp:20
error: src/slave/containerizer/fetcher.hpp: patch does not apply
error: patch failed: src/slave/containerizer/fetcher.cpp:16
error: src/slave/containerizer/fetcher.cpp: patch does not apply
error: patch failed: src/tests/fetcher_tests.cpp:56
error: src/tests/fetcher_tests.cpp: patch does not apply
Failed to apply patch

- Mesos ReviewBot


On Feb. 25, 2015, 10:19 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 10:19 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 25, 2015, 9:54 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Refactored cache eviction. The only remaining planned changes are: 1. Add 
eviction tests, 2. rebase.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
  src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
074a2d82dcd882e52f8cd62ed7493295596acb26 
  src/slave/containerizer/mesos/containerizer.cpp 
d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
  src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
  src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
  src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74040
---



docs/fetcher.md


Doesn't cache items have TTLs? Then frameworks should never get into 
content that is stale for a long time?



docs/fetcher.md


Probably worth mentioning you're only considering the size of the symlink.



docs/fetcher.md


Can you provide a quick way to verify if the URI content size can be 
queried up front? And this only applies to remote URIs right?

Since you provided a big warning, I think it's best to also tell the users 
how to quickly verify this instead of going around and figuring out what it 
means to verify.



docs/fetcher.md


What's the policy for cache eviction. Is this configurable?



docs/fetcher.md


somehwere -> somewhere.
And if you're going to put the default value here, why not put the actual 
directory name instead of  "somewhere in the work dir"?

I would also suggest removing the defaults as the flags should tell the 
users and we don't want to keep two sources of truth.



docs/fetcher.md


How about just mention you can also disable the fetcher cache by specifying 
zero bytes in the cache size flag?



src/hdfs/hdfs.hpp


Remove the extra space in the beginning



src/launcher/fetcher.cpp


I don't think this is normal LOG(INFO), we usually log actual commands with 
VLOG(1).



src/launcher/fetcher.cpp


Why not put both log in one line? They might get seperated out by other log 
output if you don't concat them.



src/launcher/fetcher.cpp


I don't know if it makes sense to log this as INFO.
INFO is the default and it shows up on every task's stdout. What's your 
rationale doing INFO here?



src/slave/containerizer/fetcher.cpp


Why set space every time we fetch? Shouldn't this gets called just once?

And I remember you mentioned setting zero space will disable the cache 
right? Where is that happening?



src/slave/containerizer/fetcher.cpp


If we don't want people to ever change the size, why not put this in a one 
time initialization method?


- Timothy Chen


On Feb. 25, 2015, 5:54 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 5:54 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b81407

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

> On Feb. 25, 2015, 7:26 a.m., Timothy Chen wrote:
> > docs/fetcher.md, line 16
> > 
> >
> > The framework provides a list of CommandInfo::URI values as part of the 
> > TaskInfo message that becomes the input to the Mesos fetcher.
> 
> Bernd Mathiske wrote:
> Even though you do not mean this, your suggested text could be misread 
> that the entire TaskInfo message becomes input to the fetcher. Now proposing: 
> 
> As part of a TaskInfo message, the framework ordering the task's 
> execution provides a list of
> CommandInfo::URI protobuf values, which becomes the input to the Mesos 
> fetcher.

Sure, just wanted to make the wording more understandable


> On Feb. 25, 2015, 7:26 a.m., Timothy Chen wrote:
> > docs/fetcher.md, line 68
> > 
> >
> > Not sure if putting the struct here is a good idea, as it's most likely 
> > going to be changed in the future.
> 
> Bernd Mathiske wrote:
> When the struct changes, we change the doc.

We've avoided doing so in all our docs, and we also suggested removing from 
another review for the serice discovery docs.
No committer is going to remember this and help enforce keeping two places as 
the source of truth. If you really like to provide it leave a reference where 
the actual proto lives and comment and refer folks to there for the true 
defintions.


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review73991
---


On Feb. 25, 2015, 5:54 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 5:54 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testi

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74043
---



src/slave/containerizer/fetcher.cpp


How about introducing a Cache namespace, so then we can just do 
Fetcher::Cache::recover, Fetcher::Cache::clean, etc?



src/slave/containerizer/fetcher.cpp


Shouldn't you also log the error too?



src/slave/flags.hpp


So the default value is actually not a recommended practice? Why set the 
default to that then?


- Timothy Chen


On Feb. 25, 2015, 5:54 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 5:54 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf wit

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74048
---



src/slave/containerizer/fetcher.cpp


Seems like lookupEntry is only used here, and it's always coupled with a 
reference call.

How about making the API less error prone, we can increment the reference 
directly in lookupEntry (also we should call it getEntry, as don't usually use 
lookup in our code base).

so getEntry becomes ->
- get cache key
- get cache item
- if cache item is present, increment ref
- return item

This way no one needs to call reference().


- Timothy Chen


On Feb. 25, 2015, 5:54 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 5:54 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: 

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74053
---



src/slave/containerizer/fetcher.cpp


return size.error();



src/slave/containerizer/fetcher.cpp


Can you leave a high level comments on the top, talking about how reserve, 
set space, victims, eviction is designed to work together?


- Timothy Chen


On Feb. 25, 2015, 5:54 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 5:54 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then propagates

Re: Review Request 30774: Fetcher Cache

[Mesos ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74069
---


Bad patch!

Reviews applied: [30609, 30606, 30774]

Failed command: ./support/apply-review.sh -n -r 30774

Error:
 2015-02-25 19:43:38 URL:https://reviews.apache.org/r/30774/diff/raw/ 
[160364/160364] -> "30774.patch" [1]
error: patch failed: src/Makefile.am:1301
error: src/Makefile.am: patch does not apply
error: patch failed: src/launcher/fetcher.cpp:34
error: src/launcher/fetcher.cpp: patch does not apply
error: patch failed: src/slave/containerizer/fetcher.hpp:20
error: src/slave/containerizer/fetcher.hpp: patch does not apply
error: patch failed: src/slave/containerizer/fetcher.cpp:16
error: src/slave/containerizer/fetcher.cpp: patch does not apply
error: patch failed: src/tests/fetcher_tests.cpp:56
error: src/tests/fetcher_tests.cpp: patch does not apply
Failed to apply patch

- Mesos ReviewBot


On Feb. 25, 2015, 5:54 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 5:54 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes a

Re: Review Request 30774: Fetcher Cache

[Till Toenshoff]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review73462
---


This really is an impressive project Bernd, tipping my hat here!

I did not get all the way through, had to ignore the tests in this review (to 
be continued!).

For playing a bit with this and for testing the functionality, it would be 
great if this was rebased to the current master.

Some high level comments; 
I kind of like the idea of wrapping synchronous calls into async's, thus 
allowing error cases to fall through then-cases and hence enabling an error 
handling that isnt scattered all over. While you are now able to concentrate 
the failure handling, there is a cost involved; 1. it becomes harder to read, 
2. it possibly becomes more prone to unforseen concurrency issues, 3. this 
launches one-off processes a lot lot lot :).


docs/fetcher.md


s/reponsible/responsible/



docs/fetcher.md


s/specifiy/specify/



docs/fetcher.md


s/somehwere/somewhere/



include/mesos/mesos.proto


This should be part of the URI message definition as it is not used by the 
CommandInfo message directly.



src/Makefile.am


Alphabetize please.



src/hdfs/hdfs.hpp


For debugging purposes, it might be nice to see the actual result from this 
HDFS callup. Maybe something like:
```
return Error("HDFS du returned unexpected number of results: HDFS returned 
'" << output.str() << "'");
```



src/launcher/fetcher.cpp


Why are you mentioning `os::net` to the user?



src/launcher/fetcher.cpp


Not yours but this sounds weird, what do you think?



src/launcher/fetcher.cpp


It should be enough to return the error and let the upper layers decide on 
the logging, no?



src/launcher/fetcher.cpp


The extract method itself does not explicitely return the source path in 
the error message. It will however contain that information by the shell call 
of `extract` (lines 73 + 79):
```
command += " '" + sourcePath + "'";
```

```
return Error("Failed to extract: command " + command +
 " exited with status: " + stringify(status));
```

For consistency, I would suggest to either getting rid of his very 
extension of the error message or to adding the same in line 237.



src/launcher/fetcher.cpp


How about moving this up and thereby getting rid of the brackets around the 
"expected" case?



src/launcher/fetcher.cpp


Is this part of the comment an artifact of older revisions? The exit code 
does not seem to correlate with the number of fetched items.



src/slave/containerizer/fetcher.hpp


I know this particular issue has not yet gotten a final vote, but I would 
suggest using 
`__SLAVE_CONTAINERIZER_FETCHER_HPP__` instead.



src/slave/containerizer/fetcher.hpp


Why are we exposing the FetchProcess definition within this header? In 
other words, does anyone outside the Fetcher implementation need to know about 
this process' symbols?



src/slave/containerizer/fetcher.cpp


Why not making this a `std::string` as you never use it in its raw, C-style 
format anyway?



src/slave/containerizer/fetcher.cpp


See above.



src/slave/containerizer/fetcher.cpp


Schemes should not be limited to a length (<= 5), the standard does not 
mention such restriction (see http://tools.ietf.org/html/std66#section-3.1).



src/slave/containerizer/fetcher.cpp


Can we get rid of this validation here or within `validateUri`?

In any case, please make sure the resulting Error messages are consistent.



src/slave/containerizer/fetcher.cpp


Are these URIs canonicalized towards lower case already at some point?

According to the standard, a mixed case URI / scheme is perfectly valid: 
`Http://example.org/example.tar.gz`.

See http://tools.ietf.org/html/std66#section-3.1 again :).



src/slave/container

Re: Review Request 30774: Fetcher Cache

[Till Toenshoff]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74155
---



src/slave/containerizer/fetcher.cpp


Why not including the error message received by `chown`?

```
return Error("Failed to chown directory " + directory + ": " + 
chown.error());
```


- Till Toenshoff


On Feb. 25, 2015, 5:54 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 5:54 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then propagates these 
> additions throughout the rest of the code base where applicable. This 
> inclu

Re: Review Request 30774: Fetcher Cache

[Till Toenshoff]

> On Feb. 25, 2015, 10:20 p.m., Till Toenshoff wrote:
> > src/slave/containerizer/fetcher.cpp, lines 459-460
> > 
> >
> > So a force removes from cache -- can you explain why this is needed?

Ow I got it :) ... a force is a "remove -> re/download -> cache -> extract / 
copy"


- Till


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review73462
---


On Feb. 25, 2015, 5:54 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 5:54 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then propagat

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > docs/fetcher.md, line 165
> > 
> >
> > Doesn't cache items have TTLs? Then frameworks should never get into 
> > content that is stale for a long time?

TTLs are not implemented. (Regardless I would still give the same advice if 
they were.)


> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > docs/fetcher.md, line 212
> > 
> >
> > What's the policy for cache eviction. Is this configurable?

It is intentionally unspecified for users so we can change it.


> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > docs/fetcher.md, line 183
> > 
> >
> > Probably worth mentioning you're only considering the size of the 
> > symlink.

Turns out I do want to follow links after all.


> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > docs/fetcher.md, line 195
> > 
> >
> > Can you provide a quick way to verify if the URI content size can be 
> > queried up front? And this only applies to remote URIs right?
> > 
> > Since you provided a big warning, I think it's best to also tell the 
> > users how to quickly verify this instead of going around and figuring out 
> > what it means to verify.

I state a couple of paragrahps above exactly how sizes are determined. This 
applies to all supported URIs except ftp.


> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > docs/fetcher.md, line 229
> > 
> >
> > somehwere -> somewhere.
> > And if you're going to put the default value here, why not put the 
> > actual directory name instead of  "somewhere in the work dir"?
> > 
> > I would also suggest removing the defaults as the flags should tell the 
> > users and we don't want to keep two sources of truth.

If we don't have any defaults, we cannot run tests. "somewhere in the work dir" 
is important info, because it clarifies that tests will run, but it is not 
ideal for production. I will be vague about the cache size, too, though, to 
remove one place of truth.


> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > src/launcher/fetcher.cpp, line 80
> > 
> >
> > I don't think this is normal LOG(INFO), we usually log actual commands 
> > with VLOG(1).

I have never seen VLOG in launcher/fetcher.cpp, only LOG(INFO).


> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > src/launcher/fetcher.cpp, line 136
> > 
> >
> > Why not put both log in one line? They might get seperated out by other 
> > log output if you don't concat them.

We need only one of these.


> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > src/launcher/fetcher.cpp, line 443
> > 
> >
> > I don't know if it makes sense to log this as INFO.
> > INFO is the default and it shows up on every task's stdout. What's your 
> > rationale doing INFO here?

I believe it goes into stderr. If we get a call that the fetcher cache is 
broken somewhere, we can inspect with what parameters mesos-fetcher has been 
called for a specific task, without rerunning it.


> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 330
> > 
> >
> > Why set space every time we fetch? Shouldn't this gets called just once?
> > 
> > And I remember you mentioned setting zero space will disable the cache 
> > right? Where is that happening?

This is an artefact of flags not being injected. This will be refactored in a 
later ticket. Putting a TODO comment.

fetcher_cache_size is a slave cmd line flag.


> On Feb. 25, 2015, 10:06 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 1138
> > 
> >
> > If we don't want people to ever change the size, why not put this in a 
> > one time initialization method?

Let's correct this later. It will also change once I introduce eviction tests, 
which require dynamic cache size updates.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74040
---


On Feb. 25, 2015, 9:54 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://r

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 25, 2015, 10:25 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 103
> > 
> >
> > How about introducing a Cache namespace, so then we can just do 
> > Fetcher::Cache::recover, Fetcher::Cache::clean, etc?

There is already a cache class. The shape of this code is an artefact of not 
injecting slaveId and flags. We will clean this up when we refactor for the 
latter.


> On Feb. 25, 2015, 10:25 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 516
> > 
> >
> > Shouldn't you also log the error too?

This probably does not interrupt or change any user-driven action. It might, 
but it also might not.


> On Feb. 25, 2015, 10:25 a.m., Timothy Chen wrote:
> > src/slave/flags.hpp, line 108
> > 
> >
> > So the default value is actually not a recommended practice? Why set 
> > the default to that then?

So we can run tests. I have adjusted the doc as indicated above.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74043
---


On Feb. 25, 2015, 9:54 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 9:54 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > This really is an impressive project Bernd, tipping my hat here!
> > 
> > I did not get all the way through, had to ignore the tests in this review 
> > (to be continued!).
> > 
> > For playing a bit with this and for testing the functionality, it would be 
> > great if this was rebased to the current master.
> > 
> > Some high level comments; 
> > I kind of like the idea of wrapping synchronous calls into async's, thus 
> > allowing error cases to fall through then-cases and hence enabling an error 
> > handling that isnt scattered all over. While you are now able to 
> > concentrate the failure handling, there is a cost involved; 1. it becomes 
> > harder to read, 2. it possibly becomes more prone to unforseen concurrency 
> > issues, 3. this launches one-off processes a lot lot lot :).

1. Compared to what alternative?
2. Compared to what alternative?
3. "Launching" one-off processes should not be a problem if we have our thread 
pool implemented well, which I am assuming. Are you referring to the many 
async() calls? This stuff is supposed to be cheap and available :-)


> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > src/slave/containerizer/fetcher.hpp, line 122
> > 
> >
> > Why are we exposing the FetchProcess definition within this header? In 
> > other words, does anyone outside the Fetcher implementation need to know 
> > about this process' symbols?

Yes: tests.


> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > src/slave/containerizer/fetcher.cpp, line 1073
> > 
> >
> > Why using `auto` here - is this sanctioned by our styleguide?

I have 2 ship its for the style guide change. Please commit it :-)

https://reviews.apache.org/r/30792/


> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > src/slave/containerizer/fetcher.cpp, line 1084
> > 
> >
> > Clang-format seems to prefer not to indend a function name after its 
> > return value at all. I have never encountered such case before, hence I am 
> > entirely unsure about the mesos (aka __right__) way.
> > 
> > This appears to be triggered mostly by the fact that you are using 
> > nested class definitions - I would try to avoid that unless your 
> > implementation and the definitions are in one place as the resulting 
> > symbols are really hard to identify on a single glance.

If I don't nest them they need longer names. That saves me "::" per class name 
and leaves more room for interpretation. We are not using Clang-format, so this 
is a non-issue. (I am convinced we should move to an automatic formatter 
yesterday, for the record).


> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > src/launcher/fetcher.cpp, line 138
> > 
> >
> > Why are you mentioning `os::net` to the user?

Carried this over from old fetcher code. Removed now.


> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > src/launcher/fetcher.cpp, line 236
> > 
> >
> > Not yours but this sounds weird, what do you think?

It now says:

  // We regard as local: "file://" or the absense of any URI scheme.


> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > src/launcher/fetcher.cpp, lines 425-426
> > 
> >
> > Is this part of the comment an artifact of older revisions? The exit 
> > code does not seem to correlate with the number of fetched items.

Yep. I will bring this back later.


> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > src/slave/containerizer/fetcher.hpp, lines 19-20
> > 
> >
> > I know this particular issue has not yet gotten a final vote, but I 
> > would suggest using 
> > `__SLAVE_CONTAINERIZER_FETCHER_HPP__` instead.

Done.


> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > src/slave/containerizer/fetcher.cpp, lines 148-151
> > 
> >
> > Can we get rid of this validation here or within `validateUri`?
> > 
> > In any case, please make sure the resulting Error messages are 
> > consistent.

Carried over from old fetcher code. Now compressed into one place. :-)


> On Feb. 25, 2015, 2:20 p.m., Till Toenshoff wrote:
> > src/slave/containerizer/fetcher.cpp, lines 236-239
> > 
> >
> > Are these URIs canonicalized towards lower case already at some point?
> > 
> > According to the standard, a mixed case URI / scheme is perfectly 
> > valid

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 25, 2015, 10:30 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 686
> > 
> >
> > Seems like lookupEntry is only used here, and it's always coupled with 
> > a reference call.
> > 
> > How about making the API less error prone, we can increment the 
> > reference directly in lookupEntry (also we should call it getEntry, as 
> > don't usually use lookup in our code base).
> > 
> > so getEntry becomes ->
> > - get cache key
> > - get cache item
> > - if cache item is present, increment ref
> > - return item
> > 
> > This way no one needs to call reference().

Renamed to getEntry(). 

I disagree with the plan to combine looking up entries and referencing them. I 
think this would make the code more error-prone, not less. It is too easy then 
to use getEntry() and inadvertently also reference() when someone later changes 
the code base. 

We would then better rename it to getAndReferenceEntry(). However, this does 
not increase code readability at all.

BTW, getEntry is also used in one other location, but I could inline it there, 
so we are back to one place.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74048
---


On Feb. 25, 2015, 9:54 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 9:54 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 25, 2015, 10:56 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 528
> > 
> >
> > return size.error();

That does not compile.


> On Feb. 25, 2015, 10:56 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 832
> > 
> >
> > Can you leave a high level comments on the top, talking about how 
> > reserve, set space, victims, eviction is designed to work together?

I added an overview of how eviction works at the top of reserveCacheSpace.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74053
---


On Feb. 25, 2015, 9:54 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 25, 2015, 9:54 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
>   src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> 074a2d82dcd882e52f8cd62ed7493295596acb26 
>   src/slave/containerizer/mesos/containerizer.cpp 
> d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
>   src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
>   src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
>   src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes i

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 26, 2015, 6:03 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Rebased to current master. Addressed all issues except 1. Removed Hdfs fetch 
test, which will be reinstated by MESOS-2390, which is reviewable.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am 17d0d7aa7361c3a373f6863d36b0a4767f5c05c4 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp d290f95251def3952c5ee34f600e1d71467f6293 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp 9f31fa46304398e8f87b41b55d8f4cfd4aba10b9 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 2438620e2db94c3c56336fa5d8e69a18fe8f3bac 
  src/tests/mesos.hpp f7a0d057edea1a7ec7ae3bb9bc729230bf7dd46d 
  src/tests/mesos.cpp 23f790cbb289f6483dcdfa6ecccd462360ce02f1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74271
---



src/slave/containerizer/fetcher.cpp


Ah, I found this now. Was looking at the wrong revision. I am programming 
defensively here. There is no reason to yell error just because a file we want 
to delete does not exist. os::rm() would complain. How could the file 
disappear? Maybe by a human operator wiping the cache from the outside. The 
fetcher cache should be able to survive this!


- Bernd Mathiske


On Feb. 26, 2015, 6:03 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 26, 2015, 6:03 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am 17d0d7aa7361c3a373f6863d36b0a4767f5c05c4 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> d290f95251def3952c5ee34f600e1d71467f6293 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp 9f31fa46304398e8f87b41b55d8f4cfd4aba10b9 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 2438620e2db94c3c56336fa5d8e69a18fe8f3bac 
>   src/tests/mesos.hpp f7a0d057edea1a7ec7ae3bb9bc729230bf7dd46d 
>   src/tests/mesos.cpp 23f790cbb289f6483dcdfa6ecccd462360ce02f1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache dir

Re: Review Request 30774: Fetcher Cache

[Mesos ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74278
---


Patch looks great!

Reviews applied: [30606, 30609, 30774]

All tests passed.

- Mesos ReviewBot


On Feb. 26, 2015, 2:03 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 26, 2015, 2:03 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am 17d0d7aa7361c3a373f6863d36b0a4767f5c05c4 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> d290f95251def3952c5ee34f600e1d71467f6293 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp 9f31fa46304398e8f87b41b55d8f4cfd4aba10b9 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 2438620e2db94c3c56336fa5d8e69a18fe8f3bac 
>   src/tests/mesos.hpp f7a0d057edea1a7ec7ae3bb9bc729230bf7dd46d 
>   src/tests/mesos.cpp 23f790cbb289f6483dcdfa6ecccd462360ce02f1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then propagates these 
> additions throughout the rest of the code base where applicable. This 
> includes passing the slave ID all the way down to the place where the cache 
> dir name is constructed.
> 
> 30037: Extends the fetcher info protobuf with "actions" (fetch directly 
> bypassing the cache, fetch through the cac

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

> On Feb. 25, 2015, 6:56 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 528
> > 
> >
> > return size.error();
> 
> Bernd Mathiske wrote:
> That does not compile.

It doesn't? How about wrapping in Error again?
And why is returning size correct here?


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74053
---


On Feb. 26, 2015, 2:03 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 26, 2015, 2:03 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am 17d0d7aa7361c3a373f6863d36b0a4767f5c05c4 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> d290f95251def3952c5ee34f600e1d71467f6293 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp 9f31fa46304398e8f87b41b55d8f4cfd4aba10b9 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 2438620e2db94c3c56336fa5d8e69a18fe8f3bac 
>   src/tests/mesos.hpp f7a0d057edea1a7ec7ae3bb9bc729230bf7dd46d 
>   src/tests/mesos.cpp 23f790cbb289f6483dcdfa6ecccd462360ce02f1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then propagates these 
> additions throughout the 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 26, 2015, 7:58 p.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Rebased to current master. Fixed a bug in fetching from local files.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am 17d0d7aa7361c3a373f6863d36b0a4767f5c05c4 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp 9f31fa46304398e8f87b41b55d8f4cfd4aba10b9 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 0ae0a10b41a2c0f7459c771b31c76bbc0c02df4f 
  src/tests/mesos.hpp f7a0d057edea1a7ec7ae3bb9bc729230bf7dd46d 
  src/tests/mesos.cpp 23f790cbb289f6483dcdfa6ecccd462360ce02f1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reuse of downl

Re: Review Request 30774: Fetcher Cache

[Mesos ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74446
---


Patch looks great!

Reviews applied: [30606, 30609, 30774]

All tests passed.

- Mesos ReviewBot


On Feb. 27, 2015, 3:58 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 27, 2015, 3:58 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am 17d0d7aa7361c3a373f6863d36b0a4767f5c05c4 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp 9f31fa46304398e8f87b41b55d8f4cfd4aba10b9 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 0ae0a10b41a2c0f7459c771b31c76bbc0c02df4f 
>   src/tests/mesos.hpp f7a0d057edea1a7ec7ae3bb9bc729230bf7dd46d 
>   src/tests/mesos.cpp 23f790cbb289f6483dcdfa6ecccd462360ce02f1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then propagates these 
> additions throughout the rest of the code base where applicable. This 
> includes passing the slave ID all the way down to the place where the cache 
> dir name is constructed.
> 
> 30037: Extends the fetcher info protobuf with "actions" (fetch directly 
> bypassing the cache, fetch through the cac

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 1, 2015, 2:35 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Improved fetcher cache tests, introduced two cache eviction tests. One tests 
whether eviction succeeds, the other also tests what happens if it fails.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am 17d0d7aa7361c3a373f6863d36b0a4767f5c05c4 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp 9f31fa46304398e8f87b41b55d8f4cfd4aba10b9 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 0ae0a10b41a2c0f7459c771b31c76bbc0c02df4f 
  src/tests/mesos.hpp f7a0d057edea1a7ec7ae3bb9bc729230bf7dd46d 
  src/tests/mesos.cpp 23f790cbb289f6483dcdfa6ecccd462360ce02f1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher pro

Re: Review Request 30774: Fetcher Cache

[Mesos ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74695
---


Bad patch!

Reviews applied: [30606, 30609, 30774]

Failed command: ./support/apply-review.sh -n -r 30774

Error:
 2015-03-01 10:54:39 URL:https://reviews.apache.org/r/30774/diff/raw/ 
[169279/169279] -> "30774.patch" [1]
error: patch failed: src/tests/fetcher_tests.cpp:48
error: src/tests/fetcher_tests.cpp: patch does not apply
error: patch failed: src/tests/mesos.cpp:47
error: src/tests/mesos.cpp: patch does not apply
Failed to apply patch

- Mesos ReviewBot


On March 1, 2015, 10:35 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 1, 2015, 10:35 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am 17d0d7aa7361c3a373f6863d36b0a4767f5c05c4 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp 9f31fa46304398e8f87b41b55d8f4cfd4aba10b9 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 0ae0a10b41a2c0f7459c771b31c76bbc0c02df4f 
>   src/tests/mesos.hpp f7a0d057edea1a7ec7ae3bb9bc729230bf7dd46d 
>   src/tests/mesos.cpp 23f790cbb289f6483dcdfa6ecccd462360ce02f1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cach

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 1, 2015, 7:42 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Rebased to current master, changed nothing in the outcome, just adjusted the 
diff files.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher 

Re: Review Request 30774: Fetcher Cache

[Mesos ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74703
---


Patch looks great!

Reviews applied: [30606, 30609, 30774]

All tests passed.

- Mesos ReviewBot


On March 1, 2015, 3:42 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 1, 2015, 3:42 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then propagates these 
> additions throughout the rest of the code base where applicable. This 
> includes passing the slave ID all the way down to the place where the cache 
> dir name is constructed.
> 
> 30037: Extends the fetcher info protobuf with "actions" (fetch directly 
> bypassing the cache, fetch through the cac

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 1, 2015, 12:27 p.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Added a test for bypassing the cache when upfront size fetching fails.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reuse of do

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 2, 2015, 10:27 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Fetcher cache internals doc with overview diagram, control flow diagram, cache 
entry state diagram.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  docs/images/fetch_components.jpg PRE-CREATION 
  docs/images/fetch_flow.jpg PRE-CREATION 
  docs/images/fetch_state.jpg PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hash

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74885
---



src/slave/containerizer/fetcher.cpp


If you're incrementing all the time just to count, why not just get the 
size from list?



src/slave/containerizer/fetcher.cpp


Why is the check entries necessary? Seems like if this for test only we 
should do the validations in test?


- Timothy Chen


On March 2, 2015, 6:27 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 2, 2015, 6:27 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   docs/images/fetch_components.jpg PRE-CREATION 
>   docs/images/fetch_flow.jpg PRE-CREATION 
>   docs/images/fetch_state.jpg PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::UR

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 3, 2015, 5:01 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

More fetcher cache internals documentation with diagrams.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  docs/images/fetch_cache.jpg PRE-CREATION 
  docs/images/fetch_components.jpg PRE-CREATION 
  docs/images/fetch_flow.jpg PRE-CREATION 
  docs/images/fetch_force1.jpg PRE-CREATION 
  docs/images/fetch_force2.jpg PRE-CREATION 
  docs/images/fetch_state.jpg PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
the

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 2, 2015, 8:13 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 759
> > 
> >
> > If you're incrementing all the time just to count, why not just get the 
> > size from list?

I am not incrementing to count anything. I am incrementing to hit the right 
index in a vector that parallels the list I am iterating over. Is there a C++ 
or Boost construct that can do this without indices?


> On March 2, 2015, 8:13 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 831
> > 
> >
> > Why is the check entries necessary? Seems like if this for test only we 
> > should do the validations in test?

This is "in tests". This method is for testing. It says so in its header file 
comment.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74885
---


On March 3, 2015, 5:01 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 3, 2015, 5:01 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   docs/images/fetch_cache.jpg PRE-CREATION 
>   docs/images/fetch_components.jpg PRE-CREATION 
>   docs/images/fetch_flow.jpg PRE-CREATION 
>   docs/images/fetch_force1.jpg PRE-CREATION 
>   docs/images/fetch_force2.jpg PRE-CREATION 
>   docs/images/fetch_state.jpg PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 3, 2015, 9:30 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Patched in suggested patches from Benh. Thanks, Ben!


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  docs/images/fetch_cache.jpg PRE-CREATION 
  docs/images/fetch_components.jpg PRE-CREATION 
  docs/images/fetch_flow.jpg PRE-CREATION 
  docs/images/fetch_force1.jpg PRE-CREATION 
  docs/images/fetch_force2.jpg PRE-CREATION 
  docs/images/fetch_state.jpg PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 3, 2015, 9:53 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Removed binary files.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reuse of down

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 3, 2015, 11:40 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Removed a major piece of both functionality and complexity: no more async() for 
file operations. This is deemed good enough for MVP. The legacy fetcher 
behavior is not made worse by this.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and b

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 4, 2015, 4:54 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Refactored fetch() and its continuations. Now separating cached and uncached 
URIs at the beginning. Simplified a lot of control flow after removing async 
file operations. All tests run.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basi

Re: Review Request 30774: Fetcher Cache

[Jay Buffington]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75266
---


Hey Bernd,

I'm really looking forward to this feature.  There's a lot here, so I was 
hoping you could help me understand by responding to some of these questions:

Why do you need the cache table data structure?  Just use the filesystem?
Why are the expanded files cached as well?  
There shouldn’t be different behavior if we’re using the cache.  My understaing 
is that with this patch, if we use the cache the tar doesn’t exist in the 
sandbox.  Isn't this a regression?
What’s the point of segregating the cache by user?
Why not respect http caching headers?
Why does the framework need to even know if the cache is in use or not?
The images referenced in the fetcher docs aren’t part of the review.  Where can 
I find them?

Thanks!
Jay


docs/fetcher-cache-internals.md


s/eactly/exactly/


- Jay Buffington


On March 4, 2015, 12:54 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 4, 2015, 12:54 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run metho

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 5, 2015, 3:15 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Removed Fetcher::clearCache().


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reus

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 4, 2015, 4:39 p.m., Jay Buffington wrote:
> > Hey Bernd,
> > 
> > I'm really looking forward to this feature.  There's a lot here, so I was 
> > hoping you could help me understand by responding to some of these 
> > questions:
> > 
> > Why do you need the cache table data structure?  Just use the filesystem?
> > Why are the expanded files cached as well?  
> > There shouldn’t be different behavior if we’re using the cache.  My 
> > understaing is that with this patch, if we use the cache the tar doesn’t 
> > exist in the sandbox.  Isn't this a regression?
> > What’s the point of segregating the cache by user?
> > Why not respect http caching headers?
> > Why does the framework need to even know if the cache is in use or not?
> > The images referenced in the fetcher docs aren’t part of the review.  Where 
> > can I find them?
> > 
> > Thanks!
> > Jay

Hi Jay,

thanks for these great questions! In summary, everything you are asking for 
feature-wise can be offered later (soonish) by relatively simple to implement 
feature additions. 

Answers to your questions in order as follows.

- If I just used the file system to implement the cache without a libprocess 
actor as complement, I would need to persist state about cache contents, use 
file locks, coordinate multiple instances of running mesos-fetcher programs, 
etc. There is a possible alternative architecture for this that would also 
work. See the JIRA commoents on MESOS-336 for an earlier discussion on this. My 
personal preference would be to perhaps further develop what is now 
FetcherProcess into an external program (with fail-over) rather than trying to 
beef up mesos-fetcher, which would lead to a lot of IPC for coordination.
- I am not aware of caching expanded files. We only cache the archive file 
itself.
- Not having a tar file in the sandbox is not a regression if you see using the 
cache at all as a new feature. But I can copy it over optionally if so desired 
in an add-on patch. This is just MVP and it seems more likely that people would 
rather not have the tar file copy.
- I would not want to have a framework for one user plant a cache file that a 
framework of another user then picks up. This file could be lying around for a 
long time, from way before the second framework starts. We can later make this 
optional as an extra feature. I am erring on the side of caution in this MVP.
- Excellent suggestion. But this is for later. Extra feature that I also find 
important.
- We can have another URI.cache value that makes it so.
- Sorry for having removed the images for now. I had trouble applying the patch 
with pictures in it. Advice on what git/RB supports here is welcome! For now, 
you can git clone https://github.com/bernd-mesos/MesosFetcherDocs and then open 
the md files locally or you can look at the PDFs which I also uploaded.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75266
---


On March 5, 2015, 3:15 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 5, 2015, 3:15 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 6, 2015, 5:46 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Added outlook on future features and made other small improvements to 
documentation. Based on feedback from Robert Lacroix and Jay Buffington.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetche

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75491
---



include/mesos/fetcher/fetcher.proto


It's harder to make a optional field required, but it's much easier the 
other way around.

If we always want it to be required, I think we should make the sandbox a 
required field.



src/hdfs/hdfs.hpp


IMO this should be in another patch and we can get this commited right away.



src/launcher/fetcher.cpp


Why not log the error if it's there? available.error()



src/launcher/fetcher.cpp


You log the extraction command but in this case don't log the copy command.

I think to be consistent, let's not log the command, and like you do here 
only log when the command fails.

What you think?



src/launcher/fetcher.cpp


Failed to chmod executable



src/launcher/fetcher.cpp


CHECK_EQ(item.action(), FetcherInfo::Item::DOWNLOAD_AND_CACHE)



src/slave/containerizer/fetcher.hpp


Are these static methods going to be used somewhere else? Quite a lot of 
static methods now in the header and perhaps we just need to put the 
implementation in the cpp file, BenH also mentioned this last time in the 
review meeting.



src/slave/containerizer/fetcher.hpp


I think there is a bit of complication with this interface, where a user 
has to call two things at the end of a downloading from an Entry:

- unreference
- complete/fail

And I don't see how one ever wants to use them without each other.

Why not hide unreference all together, and decrement the reference count in 
complete or fail?

This way it's a lot less error prone, and harder to make mistakes with 
future changes.



src/slave/containerizer/fetcher.hpp


Why not just store the Path and return that?



src/slave/containerizer/fetcher.hpp


Where is this being used?



src/slave/containerizer/fetcher.cpp


Let's use strings::contains instead of find to be consistent here.



src/slave/containerizer/fetcher.cpp


What's the point of this empty branch?



src/slave/containerizer/fetcher.cpp


This seems to be very easy to make mistakes with setSpace, especially when 
it's not even expected to be called more than once? 

I thought you said you're going to change it so it's only set once during 
initialization?



src/slave/containerizer/fetcher.cpp


We should be more explicit here, that we're claiming more than the space 
we've set to.



src/slave/containerizer/fetcher.cpp


Add some message?



src/slave/containerizer/fetcher.cpp


Add some message?



src/slave/slave.cpp


Why is this just a Failure but the other recover is a LOG(FATAL)? Shouldn't 
we exit here too if unable to recover cache is a critical event?



src/tests/mesos.hpp


We want to have a consistent naming style across all the tests and files, 
and we usually in all the tests in Mesos, the "unmocked" methods tend to just 
have a prefix of "_", so run -> _run


- Timothy Chen


On March 6, 2015, 1:46 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 6, 2015, 1:46 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionalit

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > include/mesos/fetcher/fetcher.proto, line 58
> > 
> >
> > It's harder to make a optional field required, but it's much easier the 
> > other way around.
> > 
> > If we always want it to be required, I think we should make the sandbox 
> > a required field.

There was some discussion about whether this field should be required or not. 
The general idea here is that a task might be able to run without fetching 
anything into its sandbox. In this case, the framework may get away without 
naming the sandbox. But since a task always has one, we could also make it 
required. I am impartial in this choice, but I see that your argument that 
required->optional is easier has pull.


> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/hdfs/hdfs.hpp, line 82
> > 
> >
> > IMO this should be in another patch and we can get this commited right 
> > away.

I WAS in another patch: 30616. BenH advised to put all fetcher cache related 
patches that are not for stout or libprocess together in one patch.


> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.hpp, line 76
> > 
> >
> > Are these static methods going to be used somewhere else? Quite a lot 
> > of static methods now in the header and perhaps we just need to put the 
> > implementation in the cpp file, BenH also mentioned this last time in the 
> > review meeting.

Yes, these are all used both in launcher/fetcher.cpp as well as 
containerizer/fetcher.cpp. They are factored out for consistency and easier 
maintenance. In previous fetcher implementations they were not, enjoying a 
duplicitous existence.


> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.hpp, line 191
> > 
> >
> > I think there is a bit of complication with this interface, where a 
> > user has to call two things at the end of a downloading from an Entry:
> > 
> > - unreference
> > - complete/fail
> > 
> > And I don't see how one ever wants to use them without each other.
> > 
> > Why not hide unreference all together, and decrement the reference 
> > count in complete or fail?
> > 
> > This way it's a lot less error prone, and harder to make mistakes with 
> > future changes.

unreference is not only used for new cache entries that find closure in 
completion or failure. It is primarily used for pre-existing entries that have 
downloads by concurrent fetch runs. In both cases, we need to call unreference 
at the very end of fetching in our current run. If we hid unreference in only 
one of the two cases, we'd have a bug.

How can anything be less error-prone than calling unreference on everything 
that got referenced? By hard-coupling "reference" with storing the unreference 
action in a collection. I had a version exactly like this but it was turned 
down by a reviewer, because it introduced an extra class.


> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 749
> > 
> >
> > What's the point of this empty branch?

I like putting comments about what happens in the code path / case not taken in 
an empty branch instead of placing them in a less directly related place. This 
is much more clear IMHO.


> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 1087
> > 
> >
> > This seems to be very easy to make mistakes with setSpace, especially 
> > when it's not even expected to be called more than once? 
> > 
> > I thought you said you're going to change it so it's only set once 
> > during initialization?

BenH and I decided to postpone this refactoring. It would lead to a lot of 
additional code changes for ni current semantic or general architecture changes.


> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/tests/mesos.hpp, line 703
> > 
> >
> > We want to have a consistent naming style across all the tests and 
> > files, and we usually in all the tests in Mesos, the "unmocked" methods 
> > tend to just have a prefix of "_", so run -> _run

If you use "_run" how do you distinguish this from a continuation of the same 
name? We cannot possibly use this naming scheme. Please either convert to mine 
or come up with a better one. I think that unmocked-something makes it very 
clear what is going on without making first readers guess or having to put 
extra comments. So I'd prefer

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/launcher/fetcher.cpp, line 178
> > 
> >
> > You log the extraction command but in this case don't log the copy 
> > command.
> > 
> > I think to be consistent, let's not log the command, and like you do 
> > here only log when the command fails.
> > 
> > What you think?

Logging the command in both cases now.


> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.hpp, line 195
> > 
> >
> > Why not just store the Path and return that?

"directory" is a temporary artefact that will disappear once we refactor so 
that flags gets injected into the fetcher. I added a comment syaing that.


> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 175
> > 
> >
> > Let's use strings::contains instead of find to be consistent here.

Also fixed all other occurences.


> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > src/slave/slave.cpp, line 3710
> > 
> >
> > Why is this just a Failure but the other recover is a LOG(FATAL)? 
> > Shouldn't we exit here too if unable to recover cache is a critical event?

The method we are in returns a future, so we can return a Failure here. This 
leads to exiting ventually. At the other site, the method we are in only 
returns void. Suggestions for that?


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75491
---


On March 6, 2015, 5:46 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 6, 2015, 5:46 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 7, 2015, 7:21 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Addressed Tim's latest review.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reus

Re: Review Request 30774: Fetcher Cache

[Joerg Schad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75700
---



docs/fetcher.md


Could we add an explicit Note about the current behavior that the cache 
does not consider changed/updated URIs (It is mentioned in the above paragraph 
but having an explicit note about this would be helpful)?



include/mesos/mesos.proto


Could you add a comment (i.e. backlink to the documention) reminding 
developers to update docs/fetcher.md when the protobuf is changed?



src/slave/containerizer/fetcher.cpp


size_t position?



src/slave/containerizer/fetcher.cpp


Should we add a Check that items and uris have the same length in order to 
document this assumption explicitly?



src/slave/flags.hpp


Could you add a description of these flags to configuration.md (i.e. 
http://mesos.apache.org/documentation/latest/configuration/)?



src/tests/fetcher_cache_tests.cpp


Can't we simulate SERIALIZED_TASK externally (as discussed)? In this we 
would not have several modes...


- Joerg Schad


On March 7, 2015, 3:21 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 7, 2015, 3:21 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 9, 2015, 8:37 a.m., Joerg Schad wrote:
> > src/tests/fetcher_cache_tests.cpp, line 134
> > 
> >
> > Can't we simulate SERIALIZED_TASK externally (as discussed)? In this we 
> > would not have several modes...

The whole ExecutionMode enum should go. We should use executeTask inside the 
loop that creates TaskInfos in each test and then wait explicitly inide or 
outside the loop as needed. I'll refactor accordingly in the next iteration. 
Also, we don't need enum value FAIL_TO_FETCH. It's not used anywhere any more.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75700
---


On March 7, 2015, 7:21 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 7, 2015, 7:21 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicat

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 2, 2015, 8:13 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 759
> > 
> >
> > If you're incrementing all the time just to count, why not just get the 
> > size from list?
> 
> Bernd Mathiske wrote:
> I am not incrementing to count anything. I am incrementing to hit the 
> right index in a vector that parallels the list I am iterating over. Is there 
> a C++ or Boost construct that can do this without indices?

Switched to using a const_iterator for this. This should be more obviously 
paralleling the foreach.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74885
---


On March 7, 2015, 7:21 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 7, 2015, 7:21 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two var

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 6, 2015, 2:15 p.m., Timothy Chen wrote:
> > include/mesos/fetcher/fetcher.proto, line 58
> > 
> >
> > It's harder to make a optional field required, but it's much easier the 
> > other way around.
> > 
> > If we always want it to be required, I think we should make the sandbox 
> > a required field.
> 
> Bernd Mathiske wrote:
> There was some discussion about whether this field should be required or 
> not. The general idea here is that a task might be able to run without 
> fetching anything into its sandbox. In this case, the framework may get away 
> without naming the sandbox. But since a task always has one, we could also 
> make it required. I am impartial in this choice, but I see that your argument 
> that required->optional is easier has pull.

I have heard good arguments both ways. Here is how I see it. 

For the recipient of a message, "optional" is the preferred choice. Then any 
legacy recipient's code is always prepared for everything and robust wrt. 
changing to "required". Not the other way around.

But for the sender, "required" is the better choice, making sender code more 
robust. If legacy senders still provide the field when it has become optional, 
that's OK. Not the other way around.

So which side are we on in this case? As much as this is an internal protocol, 
we are on neither side and we can change this in arbitrary ways. 

This is an external protocol if someone else than a Mesos slave uses 
mesos-fetcher. (Maybe a special external containerizer.) Then we are providing 
the message recipient and we have to be on that side. Therefore I am voting for 
"optional".


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75491
---


On March 7, 2015, 7:21 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 7, 2015, 7:21 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 9, 2015, 8:37 a.m., Joerg Schad wrote:
> > include/mesos/mesos.proto, line 208
> > 
> >
> > Could you add a comment (i.e. backlink to the documention) reminding 
> > developers to update docs/fetcher.md when the protobuf is changed?

Since we are dropping the enum, there will be no such comment. There is one 
next to the remaining "cache" filed, though.


> On March 9, 2015, 8:37 a.m., Joerg Schad wrote:
> > src/slave/containerizer/fetcher.cpp, line 450
> > 
> >
> > size_t position?

Using an iterator now.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75700
---


On March 7, 2015, 7:21 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 7, 2015, 7:21 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
>   src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> ec4626f903d44c0911093ff763ef16ad27c418a9 
>   src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
>   src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fet

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 10, 2015, 8:35 a.m., Benjamin Hindman wrote:
> > src/launcher/fetcher.cpp, line 321
> > 
> >
> > Can you comment the relationship between the FetcherInfo::Item and the 
> > FetcherInfo here? Is the FetcherInfo::Item within the FetcherInfo but 
> > FetcherInfo is included because you just want to get the 
> > 'sandbox_directory' and 'cache_directory' and rather than pulling those out 
> > explicitly you just passed the entire FetcherInfo?

There are more items in the FetcherInfo than just the one we are working in 
here. That's why this one is called out explicitly. I changed this to passing 
both directories in.


> On March 10, 2015, 8:35 a.m., Benjamin Hindman wrote:
> > src/launcher/fetcher.cpp, lines 364-366
> > 
> >
> > Why are these not CHECKs? Since you're the one setting up the 
> > FetcherInfo it seems like you should know explicitly whether or not the 
> > cache_filename was set!
> > 
> > Same for the cache_directory below as well.

What if somebody else uses mesos-fetcher?


> On March 10, 2015, 8:35 a.m., Benjamin Hindman wrote:
> > src/launcher/fetcher.cpp, lines 403-404
> > 
> >
> > As mentioned above, it would be great to really capture the 
> > relationship between the FetcherInfo and the FetcherInfo::Item. If The 
> > FetcherInfo encapsulates the FetcherInfo::Item I would also suggest 
> > switching the order of the parameters to signify that.

The main purpose here is to fetch this one particular item, not everything 
FetcherInfo carries. FetcherInfo is a secondary parameter that provides extra 
parameters like cache_directory, sandbox_directory, and framework_home. Putting 
it second makes this relationship clear IMHO. Do you suggest adding all these 
as individual parameters?

Yes, the item is included in the list of items in FetcherInfo. Shall we break 
up FetcherInfo into several shells, the inner one without items?


> On March 10, 2015, 8:35 a.m., Benjamin Hindman wrote:
> > src/slave/flags.hpp, line 487
> > 
> >
> > Can we make this a Path to start?

Then it would be the only one. Confusing. I'd rather have a wholesale sweep 
over the whole code base to introduce Path - as a separate ticket.


> On March 10, 2015, 8:35 a.m., Benjamin Hindman wrote:
> > src/slave/slave.cpp, line 796
> > 
> >
> > We should do recovery on the fetcher itself:
> > 
> > Try recover = fetcher->recover(flags, slaveId);
> > 
> > It seems very weird to have a static generic Fetcher recover 
> > functionality that implies that we can't have multiple Fetchers running at 
> > the same time. How do we start multiple slaves at the same time?

This is an artefact of the lack of injection of slaveId and flags. It should be 
cleaned up when we refactor those. The slave does not have access to the 
fetcher instance as it is right now. It would cause a lot of collateral changes 
if it did. I advise to refrain for now. I have put a comment at the static 
method to explain this. That's the best fix for now IMHO.

There is no problem starting multiple slaves, because they all have a different 
slaveID that gets passed into this call.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75754
---


On March 7, 2015, 7:21 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 7, 2015, 7:21 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evi

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 11, 2015, 5:47 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Addressed almost all issues. Simplified fetcher cache test source code.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/configuration.md fc3afec248b534b1d5eb625eb66de5f90cd8cd33 
  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cac

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 11, 2015, 10:50 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Rebased.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/configuration.md fc3afec248b534b1d5eb625eb66de5f90cd8cd33 
  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am 3059818231c46484039d179cd6916932eff6cd68 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp 71ae84bbfcef208cc2ee603f3c8a79225e48a7d5 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concu

Re: Review Request 30774: Fetcher Cache

[Benjamin Hindman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review75754
---



include/mesos/fetcher/fetcher.proto


First, the "preloading" feature does not exist so let's not comment as 
much. Second, if you perceive this being optional in the future we should make 
it be optional now.



include/mesos/mesos.proto


Let's remove the 'force' functionality for now. Why? First, as a user this 
is a tedious primitive to use: it requires the scheduler to set the field when 
it wants a new version of the URI but there is no way for the scheduler to know 
after the new version has been properly cached on a particular machine. If a 
user always wants the latest version they should just set 'cache' to false. A 
better approach here would be to introduce a 'sha' for the URI that the fetcher 
can compare against and if the SHAs are different force the download then (but 
let's not do the SHA right now, we can do that as a follow up, and keep the MVP 
simple).

Given that then we'll have a basic enum for 'Cache' and folks have already 
been confused by the values 'FETCH' and 'FORCE' I think we should revert to the 
simpler: 'optional bool cache'. What would be other potential 'enum Cache' 
values going forward? If we can think of good examples then the right thing to 
do here would probably be to introduce a 'message CacheInfo' that we embed in 
URI.



src/launcher/fetcher.cpp


Can we put a newline above this to make it easier to read? The code is very 
compressed here with the line continuation from 'sourcePath ='.



src/launcher/fetcher.cpp


Why not 'else if'? The value of doing 'else if' here is that this block is 
comparing the status of the same variable so continuing with an 'else if' helps 
to capture that semantics.



src/launcher/fetcher.cpp


Can we please add a TODO here to refactor this into stout so that people 
can more easily chmod an exectuable? For example, we could define some static 
flags so that someone can do:

os::chmod(path, EXECUTABLE_CHMOD_FLAGS);



src/launcher/fetcher.cpp


Can we improve the error message here (and below), e.g.:

return Error("Failed to determine the basename of the URI: " + 
basename.error());



src/launcher/fetcher.cpp


Can you comment the relationship between the FetcherInfo::Item and the 
FetcherInfo here? Is the FetcherInfo::Item within the FetcherInfo but 
FetcherInfo is included because you just want to get the 'sandbox_directory' 
and 'cache_directory' and rather than pulling those out explicitly you just 
passed the entire FetcherInfo?



src/launcher/fetcher.cpp


Understanding the logic here is confusing:

(1) Why do we copy the file if extracting failed? It's clear to me that 
this is because this is the way we've always done this, but with the new code 
pulled into this function we should capture this with comments for a future 
reader.

(2) Why do we return the sandbox directory when extraction was sucessful? 
Why do we return the value from 'chmodExecutable'? I would have expected to 
test whether 'chmodExecuable' was successful and then returned whatever the 
semantics of this function should return. Let's comment these functions return 
semantics.



src/launcher/fetcher.cpp


Why are these not CHECKs? Since you're the one setting up the FetcherInfo 
it seems like you should know explicitly whether or not the cache_filename was 
set!

Same for the cache_directory below as well.



src/launcher/fetcher.cpp


Newline above here to break this up from the complicated line above.



src/launcher/fetcher.cpp


As mentioned above, it would be great to really capture the relationship 
between the FetcherInfo and the FetcherInfo::Item. If The FetcherInfo 
encapsulates the FetcherInfo::Item I would also suggest switching the order of 
the parameters to signify that.



src/slave/flags.hpp


Can we make this a Path to start?



src/slave/slave.cpp


We should do recovery on the fetcher itself:

Try recover = fetcher->recover(flags, slaveId);

It seems very weird to have a static generic Fetcher recover functionality 
that implies that we can't

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 17, 2015, 4:52 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Added more rational for the code structure in fetch() to 
docs/fetcher-cache-internals. Refactored fetch() somewhat: replaced 
cleanupNewCacheEntries() with ___fetch(), inlined what was in makeCacheItems() 
and applyFallbacks(). There should now be a more straightforward  "flow" to the 
whole process. However, I did not find a suitable substitute for FetcherItem as 
phase 1 future parameter. Sticking with that.

(I know there is a slight regression wrt. os::stat::size, so this is not 
rebased. Will fix that in the next patch. Apart from this, this patch is fully 
functional and runs all tests on an older version of master.)


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/configuration.md fc3afec248b534b1d5eb625eb66de5f90cd8cd33 
  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 9df972d750ce1e4a81d2e96cc508d6f83cad2fc8 
  src/Makefile.am d299f07d865080676ca8a550cf6005c6ab32839f 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
ec4626f903d44c0911093ff763ef16ad27c418a9 
  src/slave/flags.hpp 56b25caf3901b38bdecb50310e8bcae0b114efa8 
  src/slave/slave.cpp a06d68032f26ccb3f786b6ea7c3a6c3c52449bd2 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp e91e5e484eea4587ac8f2eb9cefeab4acc9f4615 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 17, 2015, 6:59 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Rebased to current master.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
  src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
  src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
  src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

300

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review76902
---



src/slave/containerizer/fetcher.cpp


What's the intention of this helper method? 
Seems odd to only have one for HDFS and not local or Net. If none of this 
is shared, I say we can lump this all in fetchSize.



src/slave/containerizer/fetcher.cpp


Do we call fetch even if we don't have anything to fetch? I think it will 
be a good idea to have a fast return if there is nothing to be fetched.



src/slave/containerizer/fetcher.cpp


Why not just mock _fetch and do a barrier on it by giving it a promise in 
test?



src/slave/containerizer/fetcher.cpp


Since this is only called in one place, how about put this in ___fetch, 
pass it the future and check if it failed log it there?



src/slave/containerizer/fetcher.cpp


In what scenario should a cache entry not exist?
If it doesn't somehow we won't be able to use it too?



src/slave/containerizer/fetcher.cpp


Feel like this can be in a infintie loop, where if we can expire one item 
then forever other fetch items will get stuck?
I wonder if we should have some remedy action, or simply crash too?



src/slave/containerizer/fetcher.cpp


Why ignore error?


- Timothy Chen


On March 17, 2015, 1:59 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 17, 2015, 1:59 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/test

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 18, 2015, 11:05 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 406
> > 
> >
> > Do we call fetch even if we don't have anything to fetch? I think it 
> > will be a good idea to have a fast return if there is nothing to be fetched.

There is a check for this in Fetcher::fetch(). No need to even dispatch the 
call to the process either if there is nothing to fetch.


> On March 18, 2015, 11:05 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 491
> > 
> >
> > Why not just mock _fetch and do a barrier on it by giving it a promise 
> > in test?

"just mock _fetch" is more work and harder to understand.

It would also function, but then you would need to touch test code every time 
you change _fetch(). Furthermore, it would not be as clear why we wait for this 
particular call.


> On March 18, 2015, 11:05 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 503
> > 
> >
> > Since this is only called in one place, how about put this in ___fetch, 
> > pass it the future and check if it failed log it there?

How would this be simpler and more readable?

What is wrong with abstracting functions that are called only once? Doing so 
saves a comment / pulls what would have been a comment into code!


> On March 18, 2015, 11:05 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 518
> > 
> >
> > In what scenario should a cache entry not exist?
> > If it doesn't somehow we won't be able to use it too?

As you can see at the call sites, this method is used in scenarios where 
fetching succeeded, where it failed, and incidentally where it left a partial 
download lying around. I added this comment:

  // We may or may not have started downloading. The download may or may
  // not have been partial. In any case, clean up whatever is there.
  
If there is no file, that's fine. Then we tried fetching and failed before 
starting to write the file. 

In any case, we remove the cache entry and the space amount it had 
reserved/claimed is released for later use.


> On March 18, 2015, 11:05 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 521
> > 
> >
> > Feel like this can be in a infintie loop, where if we can expire one 
> > item then forever other fetch items will get stuck?
> > I wonder if we should have some remedy action, or simply crash too?

This is not a loop, because the cache entry gets removed BEFORE we attempt to 
delete the file. See line 500 just above.

However, just in case future changed code were ever to call this method several 
times on the same entry, I added a line that sets the entry's size field to 
zero. This way, accounted cache space is only released once.


> On March 18, 2015, 11:05 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 726
> > 
> >
> > Why ignore error?

The code that follows this line as of line 712 handles the error case.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review76902
---


On March 17, 2015, 6:59 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 17, 2015, 6:59 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to c

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review77019
---



src/slave/containerizer/fetcher.cpp


The error case is handled right after line 711, which closes the branch for 
non-error.


- Bernd Mathiske


On March 17, 2015, 6:59 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 17, 2015, 6:59 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then propagates these 
> additions throughout the rest of the code base where applicable. This 
> in

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 18, 2015, 11:43 p.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Addressed most of Tim's review, left comments for the rest. Tim, Ben, many 
thanks for the reviews! Please revisit your issues and update or close the open 
ones, given my reponses.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
  src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
  src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
  src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
incl

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review77025
---



src/slave/containerizer/fetcher.cpp


I'm not sure I understand, the error is never logged and in the end we 
simply return 0 if os::find returns a error. To me that looks like we're 
ignoring if Try has an error right?


- Timothy Chen


On March 19, 2015, 6:43 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 19, 2015, 6:43 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then pro

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 18, 2015, 11:48 p.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 726
> > 
> >
> > I'm not sure I understand, the error is never logged and in the end we 
> > simply return 0 if os::find returns a error. To me that looks like we're 
> > ignoring if Try has an error right?

No problem, I'll rewrite it and put a comment: When there is an error then the 
cache directory does not exist, which means the number of files in the cache is 
zero.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review77025
---


On March 18, 2015, 11:43 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 18, 2015, 11:43 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, w

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review77057
---



src/slave/containerizer/fetcher.cpp


benh, what do you think of Bernd's contentionBarrier injection? commonly we 
always just mock the callback (_fetch in this case) in tests to block, but 
Bernd wanted to introduce a specific empty method for tests. I told him this is 
not a pattern we use in Mesos, but like to see what you think.


- Timothy Chen


On March 19, 2015, 6:43 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 19, 2015, 6:43 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause f

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 19, 2015, 9:40 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 379
> > 
> >
> > benh, what do you think of Bernd's contentionBarrier injection? 
> > commonly we always just mock the callback (_fetch in this case) in tests to 
> > block, but Bernd wanted to introduce a specific empty method for tests. I 
> > told him this is not a pattern we use in Mesos, but like to see what you 
> > think.

Of course I will stick to the prevalent patterns unless you start liking this 
one :-)


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review77057
---


On March 18, 2015, 11:43 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 18, 2015, 11:43 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 18, 2015, 11:05 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 503
> > 
> >
> > Since this is only called in one place, how about put this in ___fetch, 
> > pass it the future and check if it failed log it there?
> 
> Bernd Mathiske wrote:
> How would this be simpler and more readable?
> 
> What is wrong with abstracting functions that are called only once? Doing 
> so saves a comment / pulls what would have been a comment into code!

Since we probably don't need a comment here, I'll fix it.


> On March 18, 2015, 11:05 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 726
> > 
> >
> > Why ignore error?
> 
> Bernd Mathiske wrote:
> The code that follows this line as of line 712 handles the error case.

See issue below for resolution.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review76902
---


On March 18, 2015, 11:43 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 18, 2015, 11:43 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a para

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 24, 2015, 11:26 p.m., Timothy Chen wrote:
> > docs/fetcher.md, line 68
> > 
> >
> > Not sure if putting the struct here is a good idea, as it's most likely 
> > going to be changed in the future.
> 
> Bernd Mathiske wrote:
> When the struct changes, we change the doc.
> 
> Timothy Chen wrote:
> We've avoided doing so in all our docs, and we also suggested removing 
> from another review for the serice discovery docs.
> No committer is going to remember this and help enforce keeping two 
> places as the source of truth. If you really like to provide it leave a 
> reference where the actual proto lives and comment and refer folks to there 
> for the true defintions.

Yes, just putting the proto there is not good if we tend to avoid that, which I 
was not aware of. Thanks! In this case, it is very hard to explain what is 
going on without proto, so I would very much like to opt for your way out, to 
put a reference in place that points from the proto to the doc and vice-versa.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review73991
---


On March 18, 2015, 11:43 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 18, 2015, 11:43 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the f

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 25, 2015, 10:56 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 528
> > 
> >
> > return size.error();
> 
> Bernd Mathiske wrote:
> That does not compile.
> 
> Timothy Chen wrote:
> It doesn't? How about wrapping in Error again?
> And why is returning size correct here?

Yes, we can wrap it in Error again. If we don't it does not compile because a 
string does not match a Try. I will wrap it so it does not stand out.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74053
---


On March 18, 2015, 11:43 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 18, 2015, 11:43 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which com

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 25, 2015, 10:30 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 686
> > 
> >
> > Seems like lookupEntry is only used here, and it's always coupled with 
> > a reference call.
> > 
> > How about making the API less error prone, we can increment the 
> > reference directly in lookupEntry (also we should call it getEntry, as 
> > don't usually use lookup in our code base).
> > 
> > so getEntry becomes ->
> > - get cache key
> > - get cache item
> > - if cache item is present, increment ref
> > - return item
> > 
> > This way no one needs to call reference().
> 
> Bernd Mathiske wrote:
> Renamed to getEntry(). 
> 
> I disagree with the plan to combine looking up entries and referencing 
> them. I think this would make the code more error-prone, not less. It is too 
> easy then to use getEntry() and inadvertently also reference() when someone 
> later changes the code base. 
> 
> We would then better rename it to getAndReferenceEntry(). However, this 
> does not increase code readability at all.
> 
> BTW, getEntry is also used in one other location, but I could inline it 
> there, so we are back to one place.

In the latest revision I changed it to two different methods. One is called 
getEntry() and does not reference and the other is called referenceEntry() and 
does. So we can have it both ways.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review74048
---


On March 18, 2015, 11:43 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated March 18, 2015, 11:43 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
>   src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
>   src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
>   src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
>   src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
>   src/tests/docker_containerizer_tests.cpp 
> 06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
>   src/tests/mesos.cpp c8f43d21b214e75eaa

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 19, 2015, 10:48 p.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Fixed 3 more review issues.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
  src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
  src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
  src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

3

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 21, 2015, 11:54 p.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Cleanups. Thanks, Till!


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
  src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
  src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
  src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated March 24, 2015, 6:57 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

New structure for the core algorithm that attacks invariants more locally. 
Thanks to Ben H for advising and trail blazing how to do this!

Todo: remove contentionBarrier() and mock _fetch() instead, rebase.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/configuration.md 7119b1421ac1506fa118e9f91d07e027dec3d92e 
  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto ec8efaec13f54a56d82411f6cdbdb8ad8b103748 
  include/mesos/type_utils.hpp cdf5864389a72002b538c263d70bcade2bdffa45 
  src/Makefile.am 7a06c7028eca8164b1f5fdea6a7ecd37ee6826bb 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp b7bf54ac65d6c61622e485ac253513eaac2e4f88 
  src/slave/containerizer/docker.cpp 5f4b4ce49a9523e4743e5c79da4050e6f9e29ed7 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
fbd1c0a0e5f4f227adb022f0baaa6d2c7e3ad748 
  src/slave/flags.hpp dbaf5f532d0bc65a6d16856b8ffcc2c06a98f1fa 
  src/slave/slave.cpp 0f99e4efb8fa2b96f120a3e49191158ca0364c06 
  src/tests/docker_containerizer_tests.cpp 
06cd3d89ecbaaac17ae6970604b21fbe29f6e887 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp 45e35204d1aa876fa0c871acf0f21afcd5ababe8 
  src/tests/mesos.cpp c8f43d21b214e75eaac2870cbdf4f03fd18707d1 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch(

Re: Review Request 30774: Fetcher Cache

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review79838
---



src/tests/fetcher_cache_tests.cpp


Not sure why you picked an arbitrary number 5 here, why not let it be 
passed in?



src/tests/fetcher_cache_tests.cpp


Always one file expected in the cache


- Timothy Chen


On April 10, 2015, 11:33 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated April 10, 2015, 11:33 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 54c4e31ed6dfed3c23d492c19a301ce119a0519b 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 3a8e8bf303e0576c212951f6028af77e54d93537 
>   include/mesos/type_utils.hpp cdf5864389a72002b538c263d70bcade2bdffa45 
>   src/Makefile.am fa609da08e23d6595a3f6d2efddd3e333b6c78f1 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp 6893684e6d199a5d69fc8bba8e60c4acaae9c3c9 
>   src/slave/containerizer/docker.cpp f9fb07806e3b7d7d2afc1be3b8756eac23b32dcd 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> e4136095fca55637864f495098189ab3ad8d8fe7 
>   src/slave/flags.hpp d3b1ce117fbb4e0b97852ef150b63f35cc991032 
>   src/slave/flags.cpp 35f56252cfda5011d21aa188f33cc3e68a694968 
>   src/slave/slave.cpp 9fec023b643d410f4d511fa6f80e9835bab95b7e 
>   src/tests/docker_containerizer_tests.cpp 
> c772d4c836de18b0e87636cb42200356d24ec73d 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 0e98572a62ae05437bd2bc800c370ad1a0c43751 
>   src/tests/mesos.cpp 02cbb4b8cf1206d0f32d160addc91d7e0f1ab28b 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On April 12, 2015, 11:11 p.m., Timothy Chen wrote:
> > src/tests/fetcher_cache_tests.cpp, line 308
> > 
> >
> > Not sure why you picked an arbitrary number 5 here, why not let it be 
> > passed in?

OK, I will add an explanation in a comment. Two requirements need to be met by 
this constant.
- It needs to be larger than the expected number of status updates. We might 
choose something much larger than 5, but all tests run just fine with 5.
- It needs to be finite. Otherwise we will keep waiting for updates when none 
arrive due to a bug.

However, if we passed this constant in, then we would need to explain it at all 
the call sites, i.e. multiple times instead of only once. But the situation is 
exactly the same every time. So I will refrain from that.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review79838
---


On April 10, 2015, 4:33 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated April 10, 2015, 4:33 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 54c4e31ed6dfed3c23d492c19a301ce119a0519b 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 3a8e8bf303e0576c212951f6028af77e54d93537 
>   include/mesos/type_utils.hpp cdf5864389a72002b538c263d70bcade2bdffa45 
>   src/Makefile.am fa609da08e23d6595a3f6d2efddd3e333b6c78f1 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp 6893684e6d199a5d69fc8bba8e60c4acaae9c3c9 
>   src/slave/containerizer/docker.cpp f9fb07806e3b7d7d2afc1be3b8756eac23b32dcd 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> e4136095fca55637864f495098189ab3ad8d8fe7 
>   src/slave/flags.hpp d3b1ce117fbb4e0b97852ef150b63f35cc991032 
>   src/slave/flags.cpp 35f56252cfda5011d21aa188f33cc3e68a694968 
>   src/slave/slave.cpp 9fec023b643d410f4d511fa6f80e9835bab95b7e 
>   src/tests/docker_containerizer_tests.cpp 
> c772d4c836de18b0e87636cb42200356d24ec73d 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 0e98572a62ae05437bd2bc800c370ad1a0c43751 
>   src/tests/mesos.cpp 02cbb4b8cf1206d0f32d160addc91d7e0f1ab28b 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On March 18, 2015, 11:05 a.m., Timothy Chen wrote:
> > src/slave/containerizer/fetcher.cpp, line 491
> > 
> >
> > Why not just mock _fetch and do a barrier on it by giving it a promise 
> > in test?
> 
> Bernd Mathiske wrote:
> "just mock _fetch" is more work and harder to understand.
> 
> It would also function, but then you would need to touch test code every 
> time you change _fetch(). Furthermore, it would not be as clear why we wait 
> for this particular call.

Meanwhile I tried mocking _fetch, but it does not work. See the 
related/duplicate issue below. Let's drop this one here now so we can keep the 
comments on the same topic and code region in one place going forward, OK?


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review76902
---


On April 10, 2015, 4:33 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated April 10, 2015, 4:33 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/configuration.md 54c4e31ed6dfed3c23d492c19a301ce119a0519b 
>   docs/fetcher-cache-internals.md PRE-CREATION 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
>   include/mesos/mesos.proto 3a8e8bf303e0576c212951f6028af77e54d93537 
>   include/mesos/type_utils.hpp cdf5864389a72002b538c263d70bcade2bdffa45 
>   src/Makefile.am fa609da08e23d6595a3f6d2efddd3e333b6c78f1 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
>   src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
>   src/slave/containerizer/docker.hpp 6893684e6d199a5d69fc8bba8e60c4acaae9c3c9 
>   src/slave/containerizer/docker.cpp f9fb07806e3b7d7d2afc1be3b8756eac23b32dcd 
>   src/slave/containerizer/fetcher.hpp 
> 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp 
> 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
>   src/slave/containerizer/mesos/containerizer.hpp 
> ae61a0fcd19f2ba808624312401f020121baf5d4 
>   src/slave/containerizer/mesos/containerizer.cpp 
> e4136095fca55637864f495098189ab3ad8d8fe7 
>   src/slave/flags.hpp d3b1ce117fbb4e0b97852ef150b63f35cc991032 
>   src/slave/flags.cpp 35f56252cfda5011d21aa188f33cc3e68a694968 
>   src/slave/slave.cpp 9fec023b643d410f4d511fa6f80e9835bab95b7e 
>   src/tests/docker_containerizer_tests.cpp 
> c772d4c836de18b0e87636cb42200356d24ec73d 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
>   src/tests/mesos.hpp 0e98572a62ae05437bd2bc800c370ad1a0c43751 
>   src/tests/mesos.cpp 02cbb4b8cf1206d0f32d160addc91d7e0f1ab28b 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated April 13, 2015, 5:45 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Addressed the latest 2 issues.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/configuration.md 54c4e31ed6dfed3c23d492c19a301ce119a0519b 
  docs/fetcher-cache-internals.md PRE-CREATION 
  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto 311af9aebc6a85dadba9dbeffcf7036b70896bcc 
  include/mesos/mesos.proto 3a8e8bf303e0576c212951f6028af77e54d93537 
  include/mesos/type_utils.hpp cdf5864389a72002b538c263d70bcade2bdffa45 
  src/Makefile.am d15a37365bcdd5c3906160b46b389635b38b1673 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 796526f59c25898ef6db2b828b0e2bb7b172ba25 
  src/slave/constants.hpp fd1c1aba0aa62372ab399bee5709ce81b8e92cec 
  src/slave/containerizer/docker.hpp 6893684e6d199a5d69fc8bba8e60c4acaae9c3c9 
  src/slave/containerizer/docker.cpp f9fb07806e3b7d7d2afc1be3b8756eac23b32dcd 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 9e9e9d0eb6b0801d53dec3baea32a4cd4acdd5e2 
  src/slave/containerizer/mesos/containerizer.hpp 
ae61a0fcd19f2ba808624312401f020121baf5d4 
  src/slave/containerizer/mesos/containerizer.cpp 
e4136095fca55637864f495098189ab3ad8d8fe7 
  src/slave/flags.hpp d3b1ce117fbb4e0b97852ef150b63f35cc991032 
  src/slave/flags.cpp 35f56252cfda5011d21aa188f33cc3e68a694968 
  src/slave/slave.cpp a0595f93ce4720f5b9926326d01210460ccb0667 
  src/tests/docker_containerizer_tests.cpp 
c772d4c836de18b0e87636cb42200356d24ec73d 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp 4549e6a631e2c17cec3766efaa556593eeac9a1e 
  src/tests/mesos.hpp 42e42ac425a448fcc5e93db1cef1112cbf5e67c4 
  src/tests/mesos.cpp fc534e9febed1e293076e00e0f5c3879a78df90f 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 8, 2015, 11:34 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Changed title. Now an official request relating to JIRAs.


Summary (updated)
-

Fetcher Cache


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description (updated)
---

Replaces all other reviews for the fetcher cache except those related to stout: 
30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 30618, 
30621, 30626. See descriptions of those.


Diffs
-

  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing (updated)
---

make check


Thanks,

Bernd Mathiske

Re: Review Request 30774: Fetcher Cache

[Benjamin Hindman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review71575
---



src/launcher/fetcher.cpp


Separate with newline please.



src/launcher/fetcher.cpp


'{' on previous line please.



src/launcher/fetcher.cpp


Kill '} else {'.



src/slave/containerizer/fetcher.hpp


Alphabetical ordering please (move 'stout' down below 'process').



src/slave/containerizer/fetcher.hpp


If we change the name here we should change the name in the Fetch class too 
please



src/slave/containerizer/fetcher.cpp


This error message is a bit little wierd since it's not that it can't be 
removed, it's that we can determine it's 'realpath'. Also, why not use that for 
the actual os::rmdir below?



src/slave/containerizer/fetcher.cpp


Fix indentation.



src/slave/containerizer/fetcher.cpp


auto downloads = make_shared>();

Also, I think using 'auto' here makes sense because the type information is 
redundant. Let's send an email to the mailing list with this as a proposal to 
use 'auto' in these contexts because the type information is redundant, where 
as the type information when we get something returned from a function is 
invaluable!



src/slave/containerizer/fetcher.cpp


Were you explicitly trying to do this asynchronously? Since it could be an 
expensive operation (filesystem) I'm totally for it but just wanted to confirm.



src/slave/containerizer/fetcher.cpp


Let's make HDFS::du() do more work for future people using it by moving 
this code into HDFS::du and having that return a Try.



src/tests/fetcher_cache_tests.cpp


Is this supported by gcc 4.4? That's still the required compiler until 
after 0.22 is cut. How about we just make these be globals for now. same for 
the ones below too.



src/tests/fetcher_cache_tests.cpp


CHECK_SOME(os::mkdir(...));



src/tests/fetcher_cache_tests.cpp


We just name the enum directly rather than with the typedef:

enum ExecutionMode { ... };



src/tests/fetcher_cache_tests.cpp


There is only "checkpointing" mode now, so don't bother trying to test both 
cases.



src/tests/fetcher_cache_tests.cpp


Please structure the test using the 'SetUp' 'TearDown' method overloads 
rather than handrolling this stuff yourself; it's far easier to follow when 
using 'SetUp' and 'TearDown' and they let you embed ASSERT_* and EXPECT_* 
cleanly.

Note that when you need a slightly different setup/teardown flow then we 
either parameterize the test case or create more subclasses (e.g., subclasses 
of FetcherCacheTest such as ArchiveAssetFetcherCacheTest and 
ExecutableAssetFetcherCacheTest).



src/tests/fetcher_cache_tests.cpp


'{' on newline please.



src/tests/fetcher_cache_tests.cpp


This should occur in the HttpServer but then you should have an 
HttpServerProcess. When does process terminate? How are you sure that 
libprocess is not calling something like 'visit' after the memory for this 
class has been cleaned up!!!



src/tests/fetcher_cache_tests.cpp


s/node/address/



src/tests/fetcher_cache_tests.cpp


I don't like the mixture of actors and threads here. Even if this is only 
in the tests, it sets a bad precedent that we're willing compromise on the 
integrity of the assumptions you have when executing within an actor.

There is also a lot of expectations shared between the implementation here 
and the tests themselves. For example, 'pause' is meant to be used in order to 
enable queuing up multiple pending cache entry downloads.

The abstraction I think you want here is something that lets you control 
what happens whenever we get a 'visit' callback, and to do that we really need 
to mock 'visit'. If you had a mocked 'visit' then you can explicitly control 
"how", and most importantly, "when" each 'visit' call occurs.



src/tests/fetcher_cach

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 8, 2015, 2:06 p.m., Benjamin Hindman wrote:
> > src/launcher/fetcher.cpp, line 388
> > 
> >
> > Kill '} else {'.

That would be wrong. Then the CHECK would fire every time.


> On Feb. 8, 2015, 2:06 p.m., Benjamin Hindman wrote:
> > src/slave/containerizer/fetcher.cpp, line 75
> > 
> >
> > This error message is a bit little wierd since it's not that it can't 
> > be removed, it's that we can determine it's 'realpath'. Also, why not use 
> > that for the actual os::rmdir below?

This was accidental merge mess when I combined and rebased patches. Sorry! 
Cleaned up now.


> On Feb. 8, 2015, 2:06 p.m., Benjamin Hindman wrote:
> > src/slave/containerizer/fetcher.cpp, line 323
> > 
> >
> > Were you explicitly trying to do this asynchronously? Since it could be 
> > an expensive operation (filesystem) I'm totally for it but just wanted to 
> > confirm.

Yes, any file operation needs to be async! I have sometimes encountered bad 
blocks on hard disks and experienced that when that happens any file operation 
that touches one may stall for minutes. Besides, we could be using a 
network-attached/mounted file system with arbitrarily long timeouts. Etc.


> On Feb. 8, 2015, 2:06 p.m., Benjamin Hindman wrote:
> > src/tests/fetcher_cache_tests.cpp, line 220
> > 
> >
> > This should occur in the HttpServer but then you should have an 
> > HttpServerProcess. When does process terminate? How are you sure that 
> > libprocess is not calling something like 'visit' after the memory for this 
> > class has been cleaned up!!!

I added this comment to the class to clarify what the intent is here:
// A minimal HTTP server, not intended as an actor, just reusing what
// is already implemented somewhere to serve some HTTP requests for
// file downloads, plus counting how many requests are made.

There is an explicit terminate() call before Shutdown() in every test that uses 
this. I will add this to the fixture when creating one for this kind of test.

BTW, this is similar to this stretch of code in process_tests.cpp:

class FileServer : public Process
{
public:
  explicit FileServer(const string& _path)
: path(_path) {}

  virtual void initialize()
  {
provide("", path);
  }

  const string path;
};
...
  FileServer server(path);
  PID pid = spawn(server);

...
  terminate(server);
  wait(server);
---


Should I add wait() after terminate() in my tests?


> On Feb. 8, 2015, 2:06 p.m., Benjamin Hindman wrote:
> > src/tests/fetcher_cache_tests.cpp, line 242
> > 
> >
> > I don't like the mixture of actors and threads here. Even if this is 
> > only in the tests, it sets a bad precedent that we're willing compromise on 
> > the integrity of the assumptions you have when executing within an actor.
> > 
> > There is also a lot of expectations shared between the implementation 
> > here and the tests themselves. For example, 'pause' is meant to be used in 
> > order to enable queuing up multiple pending cache entry downloads.
> > 
> > The abstraction I think you want here is something that lets you 
> > control what happens whenever we get a 'visit' callback, and to do that we 
> > really need to mock 'visit'. If you had a mocked 'visit' then you can 
> > explicitly control "how", and most importantly, "when" each 'visit' call 
> > occurs.

The HttpServer here is not an "actor". I am just reusing some code to produce 
an HttpServer as quickly as possible.

I think the problem I did miss is to establish that tasks have in fact queued 
up. Rather than just observing this when debugging. I may still need pausing 
the HTTP request handling then. Or at least some HTTP server to test the 
net::-related code paths in mesos-fetcher.

I don't see how a mocked method would have more control over when the fetcher 
does anything than an overridden method.


> On Feb. 8, 2015, 2:06 p.m., Benjamin Hindman wrote:
> > src/tests/fetcher_cache_tests.cpp, line 438
> > 
> >
> > There is a ton of state, expectations, control flow requirements, etc, 
> > throughout these tests. This makes them very hard to follow, and thus hard 
> > to understand if you're really testing the things you want to test (i.e., 
> > concurrent requests, etc). While it's probably more code, I'd rather have 
> > seen each test case testing exactly what you wanted to test rather than a 
> > bunch of functions pulled out to cover "commonality". It makes it very 
> > difficult for me to determine if certain test situations are covered!
> > 
> > For example, in this function you await 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 10, 2015, 1:14 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Addressed most issues. Only left those that require refactoring for later.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Replaces all other reviews for the fetcher cache except those related to stout: 
30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 30618, 
30621, 30626. See descriptions of those.


Diffs (updated)
-

  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check


Thanks,

Bernd Mathiske

Re: Review Request 30774: Fetcher Cache

[Adam B]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review71770
---


Please update the description with a descriptive commit message.
Also explain any testing you did for HDFS fetching (since it's half-broken 
without the HADOOP_HOME fix).

- Adam B


On Feb. 10, 2015, 1:14 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 10, 2015, 1:14 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those.
> 
> 
> Diffs
> -
> 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
>   src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
>   src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
>   src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
>   src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Bernd Mathiske
> 
>

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 14, 2015, 1:41 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Rewrite of fetcher.hpp and fetcher.cpp. Now individual code paths per cache 
file leading to a single invocation of mesos-fetcher. More error situations 
covered (trying to cover all known/possible ones, actually).


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Replaces all other reviews for the fetcher cache except those related to stout: 
30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 30618, 
30621, 30626. See descriptions of those.


Diffs (updated)
-

  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check


Thanks,

Bernd Mathiske

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 14, 2015, 2:38 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Fixed minor formatting issues.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Replaces all other reviews for the fetcher cache except those related to stout: 
30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 30618, 
30621, 30626. See descriptions of those.


Diffs (updated)
-

  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check


Thanks,

Bernd Mathiske

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 14, 2015, 3 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Extended description.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description (updated)
---

Replaces all other reviews for the fetcher cache except those related to stout: 
30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 30618, 
30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reuse of download 
results in the cache when multiple fetcher runs occur concurrently. 

30614: This is to ensure that all this refactoring of fetcher code has not 
broken HDFS fetching. Adds a test that exercises the C++ code paths in Mesos 
and mesos-fetcher related to fetching from HDFS. Uses a mock HDFS client 
written in bash that acts just like a real "hadoop" command if used in the 
right limited way.

30124: Inserted fetcher cache zap upon slave startup, recovery and shutdown. 
This implements recovery in an acceptable, yet most simple way.

30173: Created fetcher cache tests. Adds a new test source file containing a 
test fixture and tests to find out if the fetcher cache works with a variety of 
settings.

30616: Adds hdfs::du() which calls "hadoop fs -du -h" and returns a string that 
contains the file size for the URI passed as argument. This is needed to 
determine the size of a file on HDFS before downloading it to the fetcher cache 
(to ensure there is enough space).

30621: Refactored URI type separation in mesos-fetcher. Moved the URI type 
separation code (distinguishes http, hdfs, local copying, etc.) from 
mesos-fetcher to the fetcher process/actor, since it is going to be reused by 
download size queries when we introduce fetcher cache management. Also factored 
out URI validation, which will be used the same way by mesos-fetcher and the 
fetcher process/actor.

30626: Fetcher cache eviction. This happens when the cache does not have enough 
space to accomodate upcoming downloads to the cache. Necessary provisions 
included here:
- mesos-fetcher does not run until evictions have been successful
- Cache space is reserved while (async) waiting for eviction to succeed. If it 
fails, the reservation gets undone.
- Reservations can be partly from available space, partly from evictions. All 
math included :-)
- To find out how much space is needed, downloading has a prelude in which we 
query the download size from the URI. This works for all URI types that 
mesos-fetcher currently supports, including http and hdfs.
- Size-determination requests are now synchronized, too. Only one per URI in 
play happens.
- There is cleanup code for all kinds of error situations. At the very end of 
the fetch attempt, each list is processed for undoing things like space 
reservations and eviction disabling.
- Eviction gets disabled for URIs that are currently in use, i.e. the 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 10, 2015, 1:26 a.m., Adam B wrote:
> > Please update the description with a descriptive commit message.
> > Also explain any testing you did for HDFS fetching (since it's half-broken 
> > without the HADOOP_HOME fix).

Done. The description of the HDFS testing is under 30614.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review71770
---


On Feb. 14, 2015, 2:38 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 14, 2015, 2:38 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those.
> 
> 
> Diffs
> -
> 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
>   src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
>   src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
>   src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
>   src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Bernd Mathiske
> 
>

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 15, 2015, 2:29 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Now using process::Queue to wait for task status updates in fetcher cache tests.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reuse of download 
results in the cache when multiple fetc

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 14, 2015, 10:34 p.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Shorter description that could be used as commit message, offloaded extra 
detail to the testing section.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description (updated)
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs
-

  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing (updated)
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reuse of download 
res

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 8, 2015, 2:06 p.m., Benjamin Hindman wrote:
> > src/tests/fetcher_cache_tests.cpp, lines 332-334
> > 
> >
> > We've got a concurrent queue in libprocess called 'Queue'. See 
> > scheduler_tests.cpp for how it's used. I'd love to see the 'Enqueue' gmock 
> > action be made more generic so that you can use it here too (I'm confident 
> > it can, and can help if need be, and would rather use that then introduce 
> > another concurrent queuing like data structure, even if it's only for the 
> > tests).

Now using Queue.

Let's discuss and elaborate on the Enqueue part later.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review71575
---


On Feb. 14, 2015, 10:34 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 14, 2015, 10:34 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
>   src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
>   src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
>   src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
>   src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Exten

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 8, 2015, 2:06 p.m., Benjamin Hindman wrote:
> > src/tests/fetcher_cache_tests.cpp, lines 132-135
> > 
> >
> > There is only "checkpointing" mode now, so don't bother trying to test 
> > both cases.
> 
> Bernd Mathiske wrote:
> I tried removing this distinction and tests failed, both the recovery 
> test and the others that don't test it. Keeping the distinction for now.

Found a bug, works now without distinction.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review71575
---


On Feb. 15, 2015, 2:29 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 15, 2015, 2:29 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
>   src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
>   src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
>   src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
>   src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache direc

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 15, 2015, 7:28 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

In fetcher cache tests: working towards Setup/Teardown to include more code, 
removed checkpointing mode distinction.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reuse of download 
re

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 17, 2015, 1:35 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Put code into Setup() and TearDown() as much as possible. Now testing in HTTP 
tests whether fetch contention occurs, i.e. whether FetcherProcess::_fetch() 
gets called for all tasks before they can proceed with downloading anything. By 
implication this exercises the code that sorts out what to do with each URI and 
determines that some of them need to simply wait for another task to download 
what they requested. This is further confirmed by checking the number of 
requests to the HTTP server, which are now differentiated by URI.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", w

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 8, 2015, 2:06 p.m., Benjamin Hindman wrote:
> > src/tests/fetcher_cache_tests.cpp, line 438
> > 
> >
> > There is a ton of state, expectations, control flow requirements, etc, 
> > throughout these tests. This makes them very hard to follow, and thus hard 
> > to understand if you're really testing the things you want to test (i.e., 
> > concurrent requests, etc). While it's probably more code, I'd rather have 
> > seen each test case testing exactly what you wanted to test rather than a 
> > bunch of functions pulled out to cover "commonality". It makes it very 
> > difficult for me to determine if certain test situations are covered!
> > 
> > For example, in this function you await tasks if in "concurrent" mode. 
> > Yet, I'm pretty sure tasks aren't going to run in parallel unless there are 
> > enough resources to do so, and tracking down the size of the tasks and how 
> > many resources were launched with the slave is so non-local that it 
> > requires looking all over this source file and after doing so I'm not 
> > convinced that anything can actually run in parallel since the default 
> > slave has 2 CPUs and each task takes 1 CPU (so maybe 2 tasks run in 
> > parallel but not all of them?).
> > 
> > Why not be more explicit in the tests, even if it's more code, so that 
> > we're convinced we're testing the things we want to test? My guess is that 
> > there will still be things we can pull out and abstract after having done 
> > that.
> 
> Bernd Mathiske wrote:
>

Indeed, we are not testing everything that needs to be tested yet. More tests 
will be added. 

Each test does state in a comment atop the test method what it is exactly that 
it tests. I have removed some the commonality functions and put their content 
into Setup() and TearDown(). What remains is "executeTasks()" and 
"await()". Inlining these would not make anything more readable 
IMHO.

I think that in the latest update I have now established that the code in 
_fetch() that is responsible for sorting out concurrent fetching is 
well-exercised. (See RR update message and new comments in the code.) Hardware 
parallelism is not relevant here, nor is that any tasks execute in parallel. In 
fact, I am trying hard to let no task run at all before fetching has become 
concurrent.

There are always enough resources to run "concurrently" in the sense needed 
here. See Setup() and CommandInfo usage. We never use them up.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review71575
---


On Feb. 17, 2015, 1:35 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 17, 2015, 1:35 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
>   src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/sl

Re: Review Request 30774: Fetcher Cache

[Alexander Rukletsov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review72694
---



src/tests/fetcher_cache_tests.cpp


Don't you need to explicitly call `MesosTest::SetUp()`?



src/tests/fetcher_cache_tests.cpp


Same here, don't you need to explicitly call `MesosTest::TearDown()` to 
make sure things are cleaned up?


- Alexander Rukletsov


On Feb. 17, 2015, 9:35 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 17, 2015, 9:35 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, and MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
>   src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
>   src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
>   src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
>   src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache directory to the fetcher info protobuf. And then propagates these 
> additions throughout the rest of the code base where applicable. This 
> includes pas

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 17, 2015, 8:40 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Added some features for MESOS-2073: forced cache updating, prefetching. Added 
fetcher documentation file in "docs", to be expanded.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 17, 2015, 3:02 a.m., Alexander Rukletsov wrote:
> > src/tests/fetcher_cache_tests.cpp, line 117
> > 
> >
> > Don't you need to explicitly call `MesosTest::SetUp()`?

Nope.


> On Feb. 17, 2015, 3:02 a.m., Alexander Rukletsov wrote:
> > src/tests/fetcher_cache_tests.cpp, line 165
> > 
> >
> > Same here, don't you need to explicitly call `MesosTest::TearDown()` to 
> > make sure things are cleaned up?

Nope.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review72694
---


On Feb. 17, 2015, 8:40 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 17, 2015, 8:40 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
>   src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
>   src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
>   src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
>   src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 

Re: Review Request 30774: Fetcher Cache

[Mesos ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review72822
---


Bad patch!

Reviews applied: [30609, 30606]

Failed command: ./support/apply-review.sh -n -r 30606

Error:
 2015-02-17 23:14:27 URL:https://reviews.apache.org/r/30606/diff/raw/ 
[1463/1463] -> "30606.patch" [1]
error: patch failed: 3rdparty/libprocess/3rdparty/stout/include/stout/net.hpp:83
error: 3rdparty/libprocess/3rdparty/stout/include/stout/net.hpp: patch does not 
apply
Failed to apply patch

- Mesos ReviewBot


On Feb. 17, 2015, 4:40 p.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 17, 2015, 4:40 p.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
>   src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
>   src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
>   src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
>   src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that won't be covered by other tests anyway.
> 
> 30034: Makes the code structure of all fetcher tests the same. Instead of 
> calling the run method of the fetcher directly, calling through fetch(). Also 
> removes all uses of I/O redirection, which is not really needed for 
> debugging, and thus the next patch can refactor fetch() and run(). (The 
> latter comes in two varieties, which complicates matters without much 
> benefit.)
> 
> 30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
> that will later cause fetcher cache actions. Also introduces the notion of a 
> cache di

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

> On Feb. 17, 2015, 3:02 a.m., Alexander Rukletsov wrote:
> > src/tests/fetcher_cache_tests.cpp, line 117
> > 
> >
> > Don't you need to explicitly call `MesosTest::SetUp()`?
> 
> Bernd Mathiske wrote:
> Nope.

Revisited. I inferred incorrectly from this never being used elsewhere and from 
the fact that MesosTest::SetUp() is private that it should not be used by 
someone's intention. I am calling this wrong now and will make 
MesosTest::SetUp() protected in my next patch, then call it.


> On Feb. 17, 2015, 3:02 a.m., Alexander Rukletsov wrote:
> > src/tests/fetcher_cache_tests.cpp, line 165
> > 
> >
> > Same here, don't you need to explicitly call `MesosTest::TearDown()` to 
> > make sure things are cleaned up?
> 
> Bernd Mathiske wrote:
> Nope.

This failed without the impossible-to-call MesosTest::SetUp(). Will call it 
from now on.


- Bernd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/#review72694
---


On Feb. 17, 2015, 8:40 a.m., Bernd Mathiske wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30774/
> ---
> 
> (Updated Feb. 17, 2015, 8:40 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and 
> Timothy Chen.
> 
> 
> Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and 
> MESOS-2074
> https://issues.apache.org/jira/browse/MESOS-2057
> https://issues.apache.org/jira/browse/MESOS-2069
> https://issues.apache.org/jira/browse/MESOS-2070
> https://issues.apache.org/jira/browse/MESOS-2072
> https://issues.apache.org/jira/browse/MESOS-2073
> https://issues.apache.org/jira/browse/MESOS-2074
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Almost all of the functionality in epic MESOS-336. Downloaded files from 
> CommandInfo::URIs can now be cached in a cache directory designated by a 
> slave flag. This only happens when asked for by an extra flag in the URI and 
> is thus backwards-compatible. The cache has a size limit also given by a new 
> slave flag. Cache-resident files are evicted as necessary to make space for 
> newly fetched ones. Concurrent attempts to cache the same URI leads to only 
> one download. The fetcher program remains external for safety reasons, but is 
> now augmented with more elaborate parameters packed into a JSON object to 
> implement specific fetch actions for all of the above. Additional testing 
> includes fetching from (mock) HDFS and coverage of the new features.
> 
> 
> Diffs
> -
> 
>   docs/fetcher.md PRE-CREATION 
>   include/mesos/fetcher/fetcher.proto 
> facb87b92bf3194516f636dcc348e136af537721 
>   include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
>   src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
>   src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
>   src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
>   src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
>   src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
>   src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
>   src/slave/containerizer/fetcher.hpp 
> bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
>   src/slave/containerizer/fetcher.cpp 
> 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
>   src/slave/containerizer/mesos/containerizer.hpp 
> b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
>   src/slave/containerizer/mesos/containerizer.cpp 
> fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
>   src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
>   src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
>   src/tests/docker_containerizer_tests.cpp 
> 8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
>   src/tests/fetcher_cache_tests.cpp PRE-CREATION 
>   src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
>   src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
>   src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
>   src/tests/mock_hadoop.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30774/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> --- longer Description: ---
> 
> -Replaces all other reviews for the fetcher cache except those related to 
> stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
> 30618, 30621, 30626. See descriptions of those. In dependency order:
> 
> 30033: Removes the fetcher env tests since these won't be needed any more 
> when the fetcher uses JSON in a single env var as a parameter. They never 
> tested anything that 

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 18, 2015, 12:07 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Added MesosTest::SetUp and made it protected. Now using MesosTest::Setup and 
TearDown in fetcher cache tests. Test touchups and more comments.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 3a2921dff856d37455593dcbf7340aa537997d35 
  src/Makefile.am 0fe4809542a7d23785a221901947771320b43d52 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 7717abd4f9a3bd6fdca6af2364864e374ce2e056 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
b3aafe4efa9f3469d7a3fd39243ad66b46d6a54d 
  src/slave/containerizer/mesos/containerizer.cpp 
fa40d47aee7803833bcde6cce1e86a21d7cf27d0 
  src/slave/flags.hpp f6033355d129f0013d39dd053455c936596bf159 
  src/slave/slave.cpp fff2d725fe49eee984d9151cfb2131202c47994f 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 83a369968ab2403fa341829ac5d11f7243095190 
  src/tests/mesos.cpp 21a405366f56c963611324076efe775f85b9d9f7 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cach

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 18, 2015, 12:28 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Rebased.


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
  src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
074a2d82dcd882e52f8cd62ed7493295596acb26 
  src/slave/containerizer/mesos/containerizer.cpp 
d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
  src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
  src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
  src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: Enables concurrent downloading into the fetcher cache. Reuse of download 
results in th

Re: Review Request 30774: Fetcher Cache

[Bernd Mathiske]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30774/
---

(Updated Feb. 18, 2015, 4:43 a.m.)


Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy 
Chen.


Changes
---

Complete description of the fetcher and the fetcher cache in "docs/fetcher.md". 
Read this first!


Bugs: MESOS-2057, MESOS-2069, MESOS-2070, MESOS-2072, MESOS-2073, and MESOS-2074
https://issues.apache.org/jira/browse/MESOS-2057
https://issues.apache.org/jira/browse/MESOS-2069
https://issues.apache.org/jira/browse/MESOS-2070
https://issues.apache.org/jira/browse/MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2073
https://issues.apache.org/jira/browse/MESOS-2074


Repository: mesos


Description
---

Almost all of the functionality in epic MESOS-336. Downloaded files from 
CommandInfo::URIs can now be cached in a cache directory designated by a slave 
flag. This only happens when asked for by an extra flag in the URI and is thus 
backwards-compatible. The cache has a size limit also given by a new slave 
flag. Cache-resident files are evicted as necessary to make space for newly 
fetched ones. Concurrent attempts to cache the same URI leads to only one 
download. The fetcher program remains external for safety reasons, but is now 
augmented with more elaborate parameters packed into a JSON object to implement 
specific fetch actions for all of the above. Additional testing includes 
fetching from (mock) HDFS and coverage of the new features.


Diffs (updated)
-

  docs/fetcher.md PRE-CREATION 
  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  include/mesos/mesos.proto 507845c493f65e154214fc7e562206e452990469 
  src/Makefile.am d372404d84c9ac0bc49da7407ad366701b9586a6 
  src/hdfs/hdfs.hpp 968545d9af896f3e72e156484cc58135405cef6b 
  src/launcher/fetcher.cpp 5b2d86d1867b25bf71461fd73df91b089002325b 
  src/slave/constants.hpp 12d6e92b814076fd423a7738e030dc6ce6954761 
  src/slave/containerizer/docker.hpp 70114dcef7a416ae904e95eb9ce365bcd866aebc 
  src/slave/containerizer/docker.cpp 813ddf8c98622748b2da1b739bf122387abc4c79 
  src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446 
  src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d 
  src/slave/containerizer/mesos/containerizer.hpp 
074a2d82dcd882e52f8cd62ed7493295596acb26 
  src/slave/containerizer/mesos/containerizer.cpp 
d5b90d12d63becfeb4c3efa9c6f5d826417f582c 
  src/slave/flags.hpp ddb32593566b71bcf0b66adeeb889e43ef911084 
  src/slave/slave.cpp ec7ec1356e745bb07484ae1755c9183b038043b3 
  src/tests/docker_containerizer_tests.cpp 
8b212d4e6ed5a179ebadce1bdbbf3edde87d706c 
  src/tests/fetcher_cache_tests.cpp PRE-CREATION 
  src/tests/fetcher_tests.cpp fcbf7ad912f0b1b3d106a75a9dcb8213a0c69c07 
  src/tests/mesos.hpp 60c70043c8266a422ffa03ab5a949da0bc822124 
  src/tests/mesos.cpp d76ed8c837da162d782de84e935fe8e11e01d6b0 
  src/tests/mock_hadoop.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/30774/diff/


Testing
---

make check

--- longer Description: ---

-Replaces all other reviews for the fetcher cache except those related to 
stout: 30006, 30033, 30034, 30036, 30037, 30039, 30124, 30173, 30614, 30616, 
30618, 30621, 30626. See descriptions of those. In dependency order:

30033: Removes the fetcher env tests since these won't be needed any more when 
the fetcher uses JSON in a single env var as a parameter. They never tested 
anything that won't be covered by other tests anyway.

30034: Makes the code structure of all fetcher tests the same. Instead of 
calling the run method of the fetcher directly, calling through fetch(). Also 
removes all uses of I/O redirection, which is not really needed for debugging, 
and thus the next patch can refactor fetch() and run(). (The latter comes in 
two varieties, which complicates matters without much benefit.)

30036: Extends the CommandInfo::URI protobuf with a boolean "caching" field 
that will later cause fetcher cache actions. Also introduces the notion of a 
cache directory to the fetcher info protobuf. And then propagates these 
additions throughout the rest of the code base where applicable. This includes 
passing the slave ID all the way down to the place where the cache dir name is 
constructed.

30037: Extends the fetcher info protobuf with "actions" (fetch directly 
bypassing the cache, fetch through the cache, retrieve from the cache). 
Switches the basis for dealing with uris to "items", which contain the uri, the 
action, and potentially a cache file name. Refactors fetch() and run(), so 
there is only one of each. Introduces about half of the actual cache logic, 
including a hashmap of cache file objects for bookkeeping and basic operations 
on it. 

30039: Enables fetcher cache actions in the mesos fetcher program.

30006: