Re: Review Request 66502: Update python virtualenv

2018-04-10 Thread se choi 


> On April 10, 2018, 6:37 p.m., Stephan Erb wrote:
> > Thanks for the patch! We also have another virtulenv version specified 
> > here: 
> > https://github.com/apache/aurora/blob/master/build-support/virtualenv#L17. 
> > Do you think you can bump that one as well?

Homebrew has 5.2.0, wondering if it will work in 5.1.0?


- se


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66502/#review200803
---


On April 9, 2018, 6:10 p.m., se choi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66502/
> ---
> 
> (Updated April 9, 2018, 6:10 p.m.)
> 
> 
> Review request for Aurora.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Update python virtualenv
> 
> 
> Diffs
> -
> 
>   pants 312dd2035a5ad2e65a1fb3f52d1c36693c2624f0 
> 
> 
> Diff: https://reviews.apache.org/r/66502/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> se choi
> 
>

Re: Review Request 66537: Adding enhancements to Docker functionality and client support for FetcherURIs

2018-04-10 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review200873
---


Ship it!




Master (2d6108b) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On April 11, 2018, 2:17 a.m., Steve Salevan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> ---
> 
> (Updated April 11, 2018, 2:17 a.m.)
> 
> 
> Review request for Aurora, Renan DelValle and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Hey there!
> 
> Over here at Spine we've added a few enhancements to Aurora's Docker 
> functionality to support several of our use cases. We'd like to submit these 
> back up to the upstream to support the great work y'all are doing in this 
> space.
> 
> Here's what's included in this RB:
> 
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
> 
> We currently use Aurora to schedule Docker containers without the use of 
> Thermos, so we've added support for server-side templating of common Thermos 
> variables into Docker executor's parameters for this purpose.
> 
> This change modifies Aurora's api.thrift with several new optional fields, 
> and all added code handles their absence gracefully, so no backfills have 
> been added. We've threaded these schema changes through to the Python Aurora 
> client alongside support for the Mesos Fetcher URIs already supported 
> server-side.
> 
> Let me know what you think and thanks!
> 
> 
> Diffs
> -
> 
>   .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> ef754e32172e7490a47a13e7b526f243ffa3efeb 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> bcb2bbf882f43d813dd26c746d806e78bae6bcf3 
>   src/main/python/apache/aurora/config/schema/base.py 
> a629bcd1261e5959da0a8458a55545d4e2c2a7a5 
>   src/main/python/apache/aurora/config/thrift.py 
> 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431 
>   src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 778148a7c033cba9004954cabc33a2b1d003dccf 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 686087ef858b8a5a8e956d82a7bd692f7be28b12 
>   src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
> 40851c419e4d62e6545959eebc0ce144fdecc697 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> e4f43d0573c7862adc9bc679f4cea40cc76eac38 
>   src/test/python/apache/aurora/config/test_thrift.py 
> 8e1d0e177959af12b97bdd1cd47845b72bc12fe1 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob
>  88e1c36a1aa2d192b95963f7aa36e243a447e4af 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  32fdcdacde58345cdd6c4b449b82c0c90c2b2aae 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks
>  4323031ec6bd128576c2a43ebc11f04a9f046e2f 
> 
> 
> Diff: https://reviews.apache.org/r/66537/diff/3/
> 
> 
> Testing
> ---
> 
> Unit tests for all new functionality have been added and run cleanly 
> alongside checkstyle/static analysis tooling.
> 
> Operational/functional testing completed within Spine infrastructure.
> 
> 
> Thanks,
> 
> Steve Salevan
> 
>

Re: Review Request 66537: Adding enhancements to Docker functionality and client support for FetcherURIs

2018-04-10 Thread Steve Salevan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/
---

(Updated April 11, 2018, 2:17 a.m.)


Review request for Aurora, Renan DelValle and Stephan Erb.


Changes
---

Corrects failing unit test

@ReviewBot retry


Repository: aurora


Description
---

Hey there!

Over here at Spine we've added a few enhancements to Aurora's Docker 
functionality to support several of our use cases. We'd like to submit these 
back up to the upstream to support the great work y'all are doing in this space.

Here's what's included in this RB:

* Support for the force_pull flag to help ensure container freshness
* Overrides for a Docker --entrypoint which can be specified on the job
* Support for alternative Docker networks (defaults to HOST)
* Support for user Docker networks

We currently use Aurora to schedule Docker containers without the use of 
Thermos, so we've added support for server-side templating of common Thermos 
variables into Docker executor's parameters for this purpose.

This change modifies Aurora's api.thrift with several new optional fields, and 
all added code handles their absence gracefully, so no backfills have been 
added. We've threaded these schema changes through to the Python Aurora client 
alongside support for the Mesos Fetcher URIs already supported server-side.

Let me know what you think and thanks!


Diffs (updated)
-

  .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ef754e32172e7490a47a13e7b526f243ffa3efeb 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
bcb2bbf882f43d813dd26c746d806e78bae6bcf3 
  src/main/python/apache/aurora/config/schema/base.py 
a629bcd1261e5959da0a8458a55545d4e2c2a7a5 
  src/main/python/apache/aurora/config/thrift.py 
6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431 
  src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
778148a7c033cba9004954cabc33a2b1d003dccf 
  src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
686087ef858b8a5a8e956d82a7bd692f7be28b12 
  src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
40851c419e4d62e6545959eebc0ce144fdecc697 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
e4f43d0573c7862adc9bc679f4cea40cc76eac38 
  src/test/python/apache/aurora/config/test_thrift.py 
8e1d0e177959af12b97bdd1cd47845b72bc12fe1 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob
 88e1c36a1aa2d192b95963f7aa36e243a447e4af 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks
 4323031ec6bd128576c2a43ebc11f04a9f046e2f 


Diff: https://reviews.apache.org/r/66537/diff/3/

Changes: https://reviews.apache.org/r/66537/diff/2-3/


Testing
---

Unit tests for all new functionality have been added and run cleanly alongside 
checkstyle/static analysis tooling.

Operational/functional testing completed within Spine infrastructure.


Thanks,

Steve Salevan

Re: Review Request 66537: Adding enhancements to Docker functionality and client support for FetcherURIs

2018-04-10 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review200869
---



Master (2d6108b) is red with this patch.
  ./build-support/jenkins/build.sh

 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_exist[restart]
 <- 
.pants.d/pyprep/sources/cde396d796bf1ad9c5117f22ed98b0463301bbfa/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 46%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_exist[start_cronjob]
 <- 
.pants.d/pyprep/sources/cde396d796bf1ad9c5117f22ed98b0463301bbfa/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 53%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_exist[start_job_update]
 <- 
.pants.d/pyprep/sources/cde396d796bf1ad9c5117f22ed98b0463301bbfa/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 60%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[add_instances]
 <- 
.pants.d/pyprep/sources/cde396d796bf1ad9c5117f22ed98b0463301bbfa/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 66%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[create_job]
 <- 
.pants.d/pyprep/sources/cde396d796bf1ad9c5117f22ed98b0463301bbfa/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 73%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[kill_job]
 <- 
.pants.d/pyprep/sources/cde396d796bf1ad9c5117f22ed98b0463301bbfa/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 80%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[restart]
 <- 
.pants.d/pyprep/sources/cde396d796bf1ad9c5117f22ed98b0463301bbfa/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 86%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[start_cronjob]
 <- 
.pants.d/pyprep/sources/cde396d796bf1ad9c5117f22ed98b0463301bbfa/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 93%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[start_job_update]
 <- 
.pants.d/pyprep/sources/cde396d796bf1ad9c5117f22ed98b0463301bbfa/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [100%]
 
  generated xml file: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/.pants.d/test/pytest/src.test.python.apache.aurora.client.hooks.hooks/junitxml/TEST-src.test.python.apache.aurora.client.hooks.hooks.xml
 
 === 15 passed in 0.33 seconds 
 
   src.test.python.apache.aurora.admin.admin
   .   SUCCESS
   src.test.python.apache.aurora.client.client  
   .   SUCCESS
   src.test.python.apache.aurora.client.api.api 
   .   SUCCESS
   src.test.python.apache.aurora.client.cli.cli 
   .   FAILURE
   src.test.python.apache.aurora.client.docker.docker   
   .   SUCCESS
   src.test.python.apache.aurora.client.hooks.hooks 
   .   SUCCESS
   src.test.python.apache.aurora.common.common  
   .   SUCCESS
   
src.test.python.apache.aurora.common.health_check.health_check  
.   SUCCESS
   src.test.python.apache.aurora.config.config  
   .   SUCCESS
   src.test.python.apache.aurora.executor.executor  
   .   SUCCESS
   src.test.python.apache.aurora.executor.bin.bin   
   .   SUCCESS
   src.test.python.apache.aurora.executor.common.common 
   .   SUCCESS
   src.test.python.apache.aurora.tools.tools
   .   SUCCESS
   src.test.python.apache.thermos.cli.cli   
   .   SUCCESS
   src.test.python.apache.thermos.cli.commands.commands 
   .   SUCCESS
   src.test.python.apache.thermos.common.common 
   .   SUCCESS
   src.test.python.apache.thermos.config.config 
   .   SUCCESS
   src.test.python.apache.thermos.core.core

Re: Review Request 66537: Adding enhancements to Docker functionality and client support for FetcherURIs

2018-04-10 Thread Steve Salevan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/
---

(Updated April 11, 2018, 1:45 a.m.)


Review request for Aurora, Renan DelValle and Stephan Erb.


Changes
---

Corrects ReviewBot's Python checkstyle issues

@ReviewBot retry


Repository: aurora


Description
---

Hey there!

Over here at Spine we've added a few enhancements to Aurora's Docker 
functionality to support several of our use cases. We'd like to submit these 
back up to the upstream to support the great work y'all are doing in this space.

Here's what's included in this RB:

* Support for the force_pull flag to help ensure container freshness
* Overrides for a Docker --entrypoint which can be specified on the job
* Support for alternative Docker networks (defaults to HOST)
* Support for user Docker networks

We currently use Aurora to schedule Docker containers without the use of 
Thermos, so we've added support for server-side templating of common Thermos 
variables into Docker executor's parameters for this purpose.

This change modifies Aurora's api.thrift with several new optional fields, and 
all added code handles their absence gracefully, so no backfills have been 
added. We've threaded these schema changes through to the Python Aurora client 
alongside support for the Mesos Fetcher URIs already supported server-side.

Let me know what you think and thanks!


Diffs (updated)
-

  .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ef754e32172e7490a47a13e7b526f243ffa3efeb 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
bcb2bbf882f43d813dd26c746d806e78bae6bcf3 
  src/main/python/apache/aurora/config/schema/base.py 
a629bcd1261e5959da0a8458a55545d4e2c2a7a5 
  src/main/python/apache/aurora/config/thrift.py 
6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431 
  src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
778148a7c033cba9004954cabc33a2b1d003dccf 
  src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
686087ef858b8a5a8e956d82a7bd692f7be28b12 
  src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
40851c419e4d62e6545959eebc0ce144fdecc697 
  src/test/python/apache/aurora/config/test_thrift.py 
8e1d0e177959af12b97bdd1cd47845b72bc12fe1 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob
 88e1c36a1aa2d192b95963f7aa36e243a447e4af 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks
 4323031ec6bd128576c2a43ebc11f04a9f046e2f 


Diff: https://reviews.apache.org/r/66537/diff/2/

Changes: https://reviews.apache.org/r/66537/diff/1-2/


Testing
---

Unit tests for all new functionality have been added and run cleanly alongside 
checkstyle/static analysis tooling.

Operational/functional testing completed within Spine infrastructure.


Thanks,

Steve Salevan

Re: Review Request 66490: Implement Mesos docker/volume isolator support for aurora

2018-04-10 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66490/#review200867
---


Ship it!




Master (2d6108b) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On April 11, 2018, 12:42 a.m., Justin Venus wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66490/
> ---
> 
> (Updated April 11, 2018, 12:42 a.m.)
> 
> 
> Review request for Aurora, Renan DelValle and Stephan Erb.
> 
> 
> Bugs: AURORA-1983
> https://issues.apache.org/jira/browse/AURORA-1983
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This change implements the docker/volume isolator as described at 
> http://mesos.apache.org/documentation/latest/isolators/docker-volume/.
> 
> * Use string for hostPath instead of struct
> * update config thrift tests to check for docker/volume isolator support
> * update mesos task factory to support docker/volume isolator
> * update documentation for docker/volume isolator support
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> ef754e32172e7490a47a13e7b526f243ffa3efeb 
>   docs/reference/configuration.md d4b869b938105ba301fc88d41019af2f1707f6f4 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> bcb2bbf882f43d813dd26c746d806e78bae6bcf3 
>   src/main/python/apache/aurora/config/schema/base.py 
> a629bcd1261e5959da0a8458a55545d4e2c2a7a5 
>   src/main/python/apache/aurora/config/thrift.py 
> 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431 
>   src/test/python/apache/aurora/config/test_thrift.py 
> 8e1d0e177959af12b97bdd1cd47845b72bc12fe1 
> 
> 
> Diff: https://reviews.apache.org/r/66490/diff/5/
> 
> 
> Testing
> ---
> 
> Tests pass locally
> ```sh
> ./gradlew test
> ./pants test src/test/python::
> ```
> 
> I've deployed this code to a test cluster with rexray, dvdcli, and the agents 
> have `docker/volume` isolation enabled.  I am able to exercise mounting an 
> EBS volume as I would expect into a container.
> 
> 
> Thanks,
> 
> Justin Venus
> 
>

Re: Review Request 66490: Implement Mesos docker/volume isolator support for aurora

2018-04-10 Thread Justin Venus 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66490/
---

(Updated April 11, 2018, 12:42 a.m.)


Review request for Aurora, Renan DelValle and Stephan Erb.


Changes
---

Fix: Use string for hostPath instead of struct "@ReviewBot retry"


Bugs: AURORA-1983
https://issues.apache.org/jira/browse/AURORA-1983


Repository: aurora


Description (updated)
---

This change implements the docker/volume isolator as described at 
http://mesos.apache.org/documentation/latest/isolators/docker-volume/.

* Use string for hostPath instead of struct
* update config thrift tests to check for docker/volume isolator support
* update mesos task factory to support docker/volume isolator
* update documentation for docker/volume isolator support


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ef754e32172e7490a47a13e7b526f243ffa3efeb 
  docs/reference/configuration.md d4b869b938105ba301fc88d41019af2f1707f6f4 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
bcb2bbf882f43d813dd26c746d806e78bae6bcf3 
  src/main/python/apache/aurora/config/schema/base.py 
a629bcd1261e5959da0a8458a55545d4e2c2a7a5 
  src/main/python/apache/aurora/config/thrift.py 
6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431 
  src/test/python/apache/aurora/config/test_thrift.py 
8e1d0e177959af12b97bdd1cd47845b72bc12fe1 


Diff: https://reviews.apache.org/r/66490/diff/5/

Changes: https://reviews.apache.org/r/66490/diff/4-5/


Testing
---

Tests pass locally
```sh
./gradlew test
./pants test src/test/python::
```

I've deployed this code to a test cluster with rexray, dvdcli, and the agents 
have `docker/volume` isolation enabled.  I am able to exercise mounting an EBS 
volume as I would expect into a container.


Thanks,

Justin Venus

Re: Review Request 66537: Adding enhancements to Docker functionality and client support for FetcherURIs

2018-04-10 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review200861
---



Master (2d6108b) is red with this patch.
  ./build-support/jenkins/build.sh

@@ -22,9 +22,9 @@
 from apache.aurora.config.schema.base import Job
 from apache.thermos.config.schema_base import MB, Process, Resources, Task
 
+from .util import AuroraClientCommandTest
+
 from gen.apache.aurora.api.constants import AURORA_EXECUTOR_NAME
-
-from .util import AuroraClientCommandTest
 
 
 class TestInspectCommand(AuroraClientCommandTest):

00:34:34 00:04   [complete]
   SUCCESS
+ ./build-support/python/checkstyle-check

00:34:37 00:00 [main]
   (To run a reporting server: ./pants server)
00:34:37 00:00   [setup]
00:34:38 00:01 [parse]
   Executing tasks in goals: lint
00:34:38 00:01   [lint]
00:34:38 00:01 [python-eval]
00:34:38 00:01 [pythonstyle]
00:34:38 00:01   [cache]
   No cached artifacts for 48 targets.
   Invalidated 48 targets.
E123:ERROR   
PythonFile(src/test/python/apache/aurora/config/test_thrift.py):244 closing 
bracket does not match indentation of opening bracket's line
 |  )

E123:ERROR   
PythonFile(src/test/python/apache/aurora/config/test_thrift.py):245 closing 
bracket does not match indentation of opening bracket's line
 |)


FAILURE: 2 Python Style issues found. You may try `./pants fmt `


00:34:58 00:21   [complete]
   FAILURE


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On April 11, 2018, 12:05 a.m., Steve Salevan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> ---
> 
> (Updated April 11, 2018, 12:05 a.m.)
> 
> 
> Review request for Aurora, Renan DelValle and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Hey there!
> 
> Over here at Spine we've added a few enhancements to Aurora's Docker 
> functionality to support several of our use cases. We'd like to submit these 
> back up to the upstream to support the great work y'all are doing in this 
> space.
> 
> Here's what's included in this RB:
> 
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
> 
> We currently use Aurora to schedule Docker containers without the use of 
> Thermos, so we've added support for server-side templating of common Thermos 
> variables into Docker executor's parameters for this purpose.
> 
> This change modifies Aurora's api.thrift with several new optional fields, 
> and all added code handles their absence gracefully, so no backfills have 
> been added. We've threaded these schema changes through to the Python Aurora 
> client alongside support for the Mesos Fetcher URIs already supported 
> server-side.
> 
> Let me know what you think and thanks!
> 
> 
> Diffs
> -
> 
>   .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> ef754e32172e7490a47a13e7b526f243ffa3efeb 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> bcb2bbf882f43d813dd26c746d806e78bae6bcf3 
>   src/main/python/apache/aurora/config/schema/base.py 
> a629bcd1261e5959da0a8458a55545d4e2c2a7a5 
>   src/main/python/apache/aurora/config/thrift.py 
> 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431 
>   src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 778148a7c033cba9004954cabc33a2b1d003dccf 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 686087ef858b8a5a8e956d82a7bd692f7be28b12 
>   src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
> 40851c419e4d62e6545959eebc0ce144fdecc697 
>   src/test/python/apache/aurora/config/test_thrift.py 
> 8e1d0e177959af12b97bdd1cd47845b72bc12fe1 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob
>  88e1c36a1aa2d192b95963f7aa36e243a447e4af 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  32fdcdacde58345cdd6c4b449b82c0c90c2b2aae 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks
>  4323031ec6bd128576c2a43ebc11f04a9f046e2f 
> 
> 
> Diff: https://reviews.apache.org/r/66537/diff/1/
> 
> 
> Testing
> ---
> 
> Unit tests for all new functionality have been added and run cleanly 
> alongside checkstyle/static analysis tooling.
> 
> Operational/functional testing completed wi

Re: Review Request 66536: Add more preemption metrics (jobs preempted, preemptors) and logging statements

2018-04-10 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66536/#review200858
---


Ship it!




Master (2d6108b) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On April 10, 2018, 10:47 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66536/
> ---
> 
> (Updated April 10, 2018, 10:47 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Added additional metrics:
> ```
> 1. preemptor_tasks_preempted_[JOB_NAME] - The number of times [JOB_NAME] has 
> been preempted for another task.
> 2. preemptor_tasks_preemptor_[JOB_NAME] - The number of times [JOB_NAME] has 
> preempted another task.
> 3. preemptor_slot_search_[success|failed]_for_[JOB_NAME] - The number of 
> times [JOB_NAME] has or hasn't found a slot for preemption.
> 4. preemptor_slot_validation_[success|failed]_for_[JOB_NAME] - The number of 
> times [JOB_NAME] succeeded to or failed to validate a slot before preemption.
> ```
> 
> Additionally, added some `LOG.info` statements for better visibility into 
> preemption/preemption slot finding.
> 
> Did a little bit of code refactoring as well.
> 
> 
> Diffs
> -
> 
>   
> src/main/java/org/apache/aurora/scheduler/preemptor/PendingTaskProcessor.java 
> ef06471d007b1d36300eea30cdea059c1ba231b0 
>   
> src/main/java/org/apache/aurora/scheduler/preemptor/PreemptionVictimFilter.java
>  569cfe6b04e6b7bf0dca7625b00698e9d8e47daf 
>   src/main/java/org/apache/aurora/scheduler/preemptor/Preemptor.java 
> 293d106eee383dd5352a629780b897d58c9dd439 
>   src/main/java/org/apache/aurora/scheduler/preemptor/PreemptorMetrics.java 
> 87305774db0ce6fb7ebed060ab4dc99be6c2df4c 
>   src/main/java/org/apache/aurora/scheduler/scheduling/TaskSchedulerImpl.java 
> edab03dfd7fdbb24891565ba755212f03d6ea3b8 
>   
> src/test/java/org/apache/aurora/scheduler/preemptor/PendingTaskProcessorTest.java
>  ba775f4688dc57504e2def0dc4b5dcd00da448e1 
>   
> src/test/java/org/apache/aurora/scheduler/preemptor/PreemptionVictimFilterTest.java
>  b3ffb0d4fc9b9b52bb49225765bd14fb8105169a 
>   src/test/java/org/apache/aurora/scheduler/preemptor/PreemptorImplTest.java 
> 0ef29d598784ce529bcaac7017dc0f2cc5055938 
> 
> 
> Diff: https://reviews.apache.org/r/66536/diff/1/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test` passes.
> Manually ensured new metrics are exported.
> Tested at scale.
> 
> 
> Thanks,
> 
> Jordan Ly
> 
>

Review Request 66537: Adding enhancements to Docker functionality and client support for FetcherURIs

2018-04-10 Thread Steve Salevan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/
---

Review request for Aurora.


Repository: aurora


Description
---

Hey there!

Over here at Spine we've added a few enhancements to Aurora's Docker 
functionality to support several of our use cases. We'd like to submit these 
back up to the upstream to support the great work y'all are doing in this space.

Here's what's included in this RB:

* Support for the force_pull flag to help ensure container freshness
* Overrides for a Docker --entrypoint which can be specified on the job
* Support for alternative Docker networks (defaults to HOST)
* Support for user Docker networks

We currently use Aurora to schedule Docker containers without the use of 
Thermos, so we've added support for server-side templating of common Thermos 
variables into Docker executor's parameters for this purpose.

This change modifies Aurora's api.thrift with several new optional fields, and 
all added code handles their absence gracefully, so no backfills have been 
added. We've threaded these schema changes through to the Python Aurora client 
alongside support for the Mesos Fetcher URIs already supported server-side.

Let me know what you think and thanks!


Diffs
-

  .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ef754e32172e7490a47a13e7b526f243ffa3efeb 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
bcb2bbf882f43d813dd26c746d806e78bae6bcf3 
  src/main/python/apache/aurora/config/schema/base.py 
a629bcd1261e5959da0a8458a55545d4e2c2a7a5 
  src/main/python/apache/aurora/config/thrift.py 
6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431 
  src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
778148a7c033cba9004954cabc33a2b1d003dccf 
  src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
686087ef858b8a5a8e956d82a7bd692f7be28b12 
  src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
40851c419e4d62e6545959eebc0ce144fdecc697 
  src/test/python/apache/aurora/config/test_thrift.py 
8e1d0e177959af12b97bdd1cd47845b72bc12fe1 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob
 88e1c36a1aa2d192b95963f7aa36e243a447e4af 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks
 4323031ec6bd128576c2a43ebc11f04a9f046e2f 


Diff: https://reviews.apache.org/r/66537/diff/1/


Testing
---

Unit tests for all new functionality have been added and run cleanly alongside 
checkstyle/static analysis tooling.

Operational/functional testing completed within Spine infrastructure.


Thanks,

Steve Salevan

Re: Review Request 66490: Implement Mesos docker/volume isolator support for aurora

2018-04-10 Thread Renan DelValle 


> On April 9, 2018, 3:33 p.m., Renan DelValle wrote:
> > api/src/main/thrift/org/apache/aurora/gen/api.thrift
> > Lines 1263 (patched)
> > 
> >
> > I think the better solution here would be to make Volume.hostPath 
> > optional and handle hostPath not being set at the Java server side. Maybe 
> > have hostPath mirror to VolumeSource.hostpath while it is being deprecated.
> > 
> > This can be done somewhere in 
> > `src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java`
> 
> Justin Venus wrote:
> Will it break anything to make Volume.hostPath optional?   It's my 
> unstanding that this is implicitly required now.

Yup, I didn't think this one through, sorry about that!

We can leave `Volume.hostpath` as required while making `VolumeSource.hostpath` 
as optional for now. Then, on the scheduler side, if `VolumeSource` is set, we 
use that. If `VolumeSource` is not set, we create a new `VolumeSource` using 
`Volume.hostpath`. And finally, when we create the job, we rely on 
`VolumeSource` as the source of truth.

That way, we can move to deprecate `Volume.hostpath` down the line since it 
becomes a proxy for what `VolumeSource` contains.


- Renan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66490/#review200768
---


On April 6, 2018, 5:04 p.m., Justin Venus wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66490/
> ---
> 
> (Updated April 6, 2018, 5:04 p.m.)
> 
> 
> Review request for Aurora, Renan DelValle and Stephan Erb.
> 
> 
> Bugs: AURORA-1983
> https://issues.apache.org/jira/browse/AURORA-1983
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This change implements the docker/volume isolator as described at 
> http://mesos.apache.org/documentation/latest/isolators/docker-volume/.
>  
> * update config thrift tests to check for docker/volume isolator support
> * update mesos task factory to support docker/volume isolator
> * update documentation for docker/volume isolator support
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> ef754e32172e7490a47a13e7b526f243ffa3efeb 
>   docs/reference/configuration.md d4b869b938105ba301fc88d41019af2f1707f6f4 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> bcb2bbf882f43d813dd26c746d806e78bae6bcf3 
>   src/main/python/apache/aurora/config/schema/base.py 
> a629bcd1261e5959da0a8458a55545d4e2c2a7a5 
>   src/main/python/apache/aurora/config/thrift.py 
> 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431 
>   src/test/python/apache/aurora/config/test_thrift.py 
> 8e1d0e177959af12b97bdd1cd47845b72bc12fe1 
> 
> 
> Diff: https://reviews.apache.org/r/66490/diff/4/
> 
> 
> Testing
> ---
> 
> Tests pass locally
> ```sh
> ./gradlew test
> ./pants test src/test/python::
> ```
> 
> I've deployed this code to a test cluster with rexray, dvdcli, and the agents 
> have `docker/volume` isolation enabled.  I am able to exercise mounting an 
> EBS volume as I would expect into a container.
> 
> 
> Thanks,
> 
> Justin Venus
> 
>

Review Request 66536: Add more preemption metrics (jobs preempted, preemptors) and logging statements

2018-04-10 Thread Jordan Ly 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66536/
---

Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, and 
Stephan Erb.


Repository: aurora


Description
---

Added additional metrics:
```
1. preemptor_tasks_preempted_[JOB_NAME] - The number of times [JOB_NAME] has 
been preempted for another task.
2. preemptor_tasks_preemptor_[JOB_NAME] - The number of times [JOB_NAME] has 
preempted another task.
3. preemptor_slot_search_[success|failed]_for_[JOB_NAME] - The number of times 
[JOB_NAME] has or hasn't found a slot for preemption.
4. preemptor_slot_validation_[success|failed]_for_[JOB_NAME] - The number of 
times [JOB_NAME] succeeded to or failed to validate a slot before preemption.
```

Additionally, added some `LOG.info` statements for better visibility into 
preemption/preemption slot finding.

Did a little bit of code refactoring as well.


Diffs
-

  src/main/java/org/apache/aurora/scheduler/preemptor/PendingTaskProcessor.java 
ef06471d007b1d36300eea30cdea059c1ba231b0 
  
src/main/java/org/apache/aurora/scheduler/preemptor/PreemptionVictimFilter.java 
569cfe6b04e6b7bf0dca7625b00698e9d8e47daf 
  src/main/java/org/apache/aurora/scheduler/preemptor/Preemptor.java 
293d106eee383dd5352a629780b897d58c9dd439 
  src/main/java/org/apache/aurora/scheduler/preemptor/PreemptorMetrics.java 
87305774db0ce6fb7ebed060ab4dc99be6c2df4c 
  src/main/java/org/apache/aurora/scheduler/scheduling/TaskSchedulerImpl.java 
edab03dfd7fdbb24891565ba755212f03d6ea3b8 
  
src/test/java/org/apache/aurora/scheduler/preemptor/PendingTaskProcessorTest.java
 ba775f4688dc57504e2def0dc4b5dcd00da448e1 
  
src/test/java/org/apache/aurora/scheduler/preemptor/PreemptionVictimFilterTest.java
 b3ffb0d4fc9b9b52bb49225765bd14fb8105169a 
  src/test/java/org/apache/aurora/scheduler/preemptor/PreemptorImplTest.java 
0ef29d598784ce529bcaac7017dc0f2cc5055938 


Diff: https://reviews.apache.org/r/66536/diff/1/


Testing
---

Added unit tests, `./gradlew test` passes.
Manually ensured new metrics are exported.
Tested at scale.


Thanks,

Jordan Ly

Re: Review Request 66502: Update python virtualenv

2018-04-10 Thread Stephan Erb 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66502/#review200803
---



Thanks for the patch! We also have another virtulenv version specified here: 
https://github.com/apache/aurora/blob/master/build-support/virtualenv#L17. Do 
you think you can bump that one as well?

- Stephan Erb


On April 9, 2018, 11:10 a.m., se choi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66502/
> ---
> 
> (Updated April 9, 2018, 11:10 a.m.)
> 
> 
> Review request for Aurora.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Update python virtualenv
> 
> 
> Diffs
> -
> 
>   pants 312dd2035a5ad2e65a1fb3f52d1c36693c2624f0 
> 
> 
> Diff: https://reviews.apache.org/r/66502/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> se choi
> 
>