date:20170214

Re: Review Request 55951: Use --launch_info when invoking MesosContainerizer.

2017-02-14 Thread Santhosh Kumar Shanmugham


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55951/#review165660
---



@ReviewBot retry

- Santhosh Kumar Shanmugham


On Feb. 14, 2017, 9:40 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55951/
> ---
> 
> (Updated Feb. 14, 2017, 9:40 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Stephan Erb, and Karthik Anantha 
> Padmanabhan.
> 
> 
> Bugs: AURORA-1882
> https://issues.apache.org/jira/browse/AURORA-1882
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MesosContainerizer has updated the command line parameters in 1.2.0 and
> consolidated the individual arguments into a single ContainerLaunchInfo
> proto buf message. Update ThermosExecutor to use the new `--launch_info`
> parameter to be compatible with MesosContainerizer also check the
> containerizer binary interface to determine to be backward-compatible.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/thermos/common/process_util.py 
> 54e716b726fc02f3901f1b9143d3fa253511e29b 
>   src/test/python/apache/aurora/executor/common/test_health_checker.py 
> a3b5a22a805753d42e97f8f1e4e3dab7d3c9c507 
>   src/test/python/apache/thermos/core/test_process.py 
> 520390217f691b9136cb4d36262be3d372a16509 
> 
> Diff: https://reviews.apache.org/r/55951/diff/
> 
> 
> Testing
> ---
> 
> build-support/jenkins/build.sh
> src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 55951: Use --launch_info when invoking MesosContainerizer.

2017-02-14 Thread Santhosh Kumar Shanmugham



> On Feb. 14, 2017, 9:52 p.m., Aurora ReviewBot wrote:
> > Master (9ea8979) is red with this patch.
> >   ./build-support/jenkins/build.sh
> > 
> >  
> >   FAILURES 
> >   test_simple_process_filesystem_isolator_launch_info 
> >  
> >  args = (), kw = {}
> >  
> >  [1m@wraps(f)[0m
> >  [1mdef _inner(*args, **kw):[0m
> >  [1mself._patch_dict()[0m
> >  [1mtry:[0m
> >  [1m>   return f(*args, **kw)[0m
> >  
> >  
> > .pants.d/python-setup/chroots/22bbf744265cd5201eb0521a411354ed4e0941bf/.deps/mock-1.0.1-py2-none-any.whl/mock.py:1618:
> >  
> >  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> >  
> > src/test/python/apache/thermos/core/test_process.py:190: in 
> > test_simple_process_filesystem_isolator_launch_info
> >  [1m'"working_directory": "%s", "user": "None", 
> > "rootfs": "/some/path/taskfs"}\n' % (sandbox))[0m
> >  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> >  
> >  taskpath =  > at 0x7f6b822fcd90>
> >  log_name = 'stdout'
> >  expected_content = 'launch --unshare_namespace_mnt 
> > --launch_info={"command": {"shell": false, "arguments": ["/bin/bash", "-c", 
> > "echo hell..."value": "/bin/bash"}, "working_directory": 
> > "/tmp/tmpc0WFNT/sandbox", "user": "None", "rootfs": "/some/path/taskfs"}\n'
> >  
> >  [1mdef assert_log_content(taskpath, log_name, 
> > expected_content):[0m
> >  [1m  log = 
> > taskpath.with_filename(log_name).getpath('process_logdir')[0m
> >  [1m  assert os.path.exists(log)[0m
> >  [1m  with open(log, 'r') as fp:[0m
> >  [1m>   assert fp.read() == expected_content[0m
> >  [1m[31mE   assert 'launch --uns.../bin/bash"}\n' 
> > == 'launch --unsh...th/taskfs"}\n'[0m
> >  [1m[31mE - launch --unshare_namespace_mnt 
> > --working_directory=/tmp/tmpc0WFNT/sandbox --rootfs=/some/path/taskfs 
> > --user=None --command={"shell": false, "arguments": ["/bin/bash", "-c", 
> > "echo hello world"], "value": "/bin/bash"}[0m
> >  [1m[31mE + launch --unshare_namespace_mnt 
> > --launch_info={"command": {"shell": false, "arguments": ["/bin/bash", "-c", 
> > "echo hello world"], "value": "/bin/bash"}, "working_directory": 
> > "/tmp/tmpc0WFNT/sandbox", "user": "None", "rootfs": "/some/path/taskfs"}[0m
> >  
> >  
> > src/test/python/apache/thermos/core/test_process.py:364: AssertionError
> >   generated xml file: 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/aaf4d108c31293299a0839bdc404a91802f80937.xml
> >  
> >  [1m[31m 1 failed, 780 passed, 6 skipped, 1 warnings 
> > in 288.49 seconds [0m
> >  
> > FAILURE
> > 
> > 
> > 05:52:42 05:26   [complete][31m
> >FAILURE[0m
> > 
> > 
> > I will refresh this build result if you post a review containing 
> > "@ReviewBot retry"

@ReviewBot retry


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55951/#review165649
---


On Feb. 14, 2017, 9:40 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55951/
> ---
> 
> (Updated Feb. 14, 2017, 9:40 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Stephan Erb, and Karthik Anantha 
> Padmanabhan.
> 
> 
> Bugs: AURORA-1882
> https://issues.apache.org/jira/browse/AURORA-1882
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MesosContainerizer has updated the command line parameters in 1.2.0 and
> consolidated the individual arguments into a single ContainerLaunchInfo
> proto buf message. Update ThermosExecutor to use the new `--launch_info`
> parameter to be compatible with MesosContainerizer also check the
> containerizer binary interface to determine to be backward-compatible.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/thermos/common/process_util.py 
> 54e716b726fc02f3901f1b9143d3fa253511e29b 
>   src/test/python/apache/aurora/executor/common/test_health_checker.py 
> a3b5a22a805753d42e97f8f1e4e3dab7d3c9c507 
>   src/test/pyth

Re: Review Request 56575: AURORA-1837 Improve task history pruning

2017-02-14 Thread Santhosh Kumar Shanmugham



> On Feb. 14, 2017, 4:02 p.m., Mehrdad Nurolahzade wrote:
> > Looking at it as is, I'm not sure if there is much value to be gained from 
> > pushing this down to `TaskStore`.
> > Do you see any value in pursuing this idea any further? Or shall I restore 
> > it to previous state?
> 
> Zameer Manji wrote:
> I think there is value in moving operations like this down to storage.
> First, it mirrors the update store for pruning old updates.
> Second, I think we can benefit from more precise storage operations. It 
> allows us to leverage the power of DbTaskStore (ie we don't need to hydrate a 
> full object for deletion) and ensures that callers don't do excessive work.
> 
> Mehrdad Nurolahzade wrote:
> Right, that was the motivation. 
> 
> However the way task deletion is handled now, it needs to go through 
> `StateManager` to (a) verify transition to `DELETED` is allowrd and (b) 
> publish `TaskDeleted` events to other subscribers like `TaskGroups`, 
> `TaskVars`, and so on. For this to work at storage level, like it's done for 
> job updates, we need to address its rather inefficient dependency on 
> `TaskManager`.

Looking how there might be more work needed here, why don't we use this RB to 
fix the O(N^2) issue on startup and handle the other improvements in a separate 
RB. The way the current `TaskHistoryPruner` works on startup is going need a 
fix.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56575/#review165599
---


On Feb. 14, 2017, 3:38 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56575/
> ---
> 
> (Updated Feb. 14, 2017, 3:38 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1837
> https://issues.apache.org/jira/browse/AURORA-1837
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch addressed efficiency issues in the current implementation of 
> `TaskHistoryPruner`. The new design is similar to that of 
> `JobUpdateHistoryPruner`: (a) Instead of registering a `DelayExecutor` run 
> upon terminal task state transitions, it runs on preconfigured intervals, 
> finds all terminal state tasks that meet pruning criteria and deletes them. 
> (b) Makes the initial task history pruning delay configurable so that it does 
> not hamper scheduler upon start.
> 
> The new design addressed the following two efficiecy problems:
> 
> 1. Upon scheduler restart/failure, the in-memory state of task history 
> pruning scheduled with `DelayExecutor` is lost. `TaskHistoryPruner` learns 
> about these dead tasks upon restart when log is replayed. These expired tasks 
> are picked up by the second call to `executor.execute()` that performs job 
> level pruning immediately (i.e., without delay). Hence, most task history 
> pruning happens after scheduler restarts and can severely hamper scheduler 
> performance (or cause consecutive fail-overs on test clusters when we put 
> load test on scheduler).
> 
> 2. Expired tasks can be picked up for pruning multiple times. The 
> asynchronous nature of `BatchWorker` which used to process task deletions 
> introduces some delay between delete enqueue and delete execution. As a 
> result, tasks already queued for deletion in a previous evaluation round 
> might get picked up, evaluated and enqueued for deletion again. This is 
> evident in `tasks_pruned` metric which reflects numbers much higher than the 
> actual number of expired tasks deleted.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/pruning/PruningModule.java 
> 735199ac1ab343c24471890aa330d6635c26 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> f77849498ff23616f1d56d133eb218f837ac3413 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> 1094a122fe836e53d0481ee5c097447f1e91fa0a 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> a649a6e3d2f2d0aeaf6d7ac704ed24911c310a1e 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> d89e715b1b08faf95f8b5788c9c28cbbb33af093 
>   
> src/test/java/org/apache/aurora/scheduler/pruning/TaskHistoryPrunerTest.java 
> 14e4040e0b94e96f77068b41454311fa3bf53573 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 02719c312294b58525c1fddd3ed096a9b1cef601 
> 
> Diff: https://reviews.apache.org/r/56575/diff/
> 
> 
> Testing
> ---
> 
> Manual testing under Vagrant
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 55951: Use --launch_info when invoking MesosContainerizer.

2017-02-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55951/#review165649
---



Master (9ea8979) is red with this patch.
  ./build-support/jenkins/build.sh

 
  FAILURES 
  test_simple_process_filesystem_isolator_launch_info 
 
 args = (), kw = {}
 
 [1m@wraps(f)[0m
 [1mdef _inner(*args, **kw):[0m
 [1mself._patch_dict()[0m
 [1mtry:[0m
 [1m>   return f(*args, **kw)[0m
 
 
.pants.d/python-setup/chroots/22bbf744265cd5201eb0521a411354ed4e0941bf/.deps/mock-1.0.1-py2-none-any.whl/mock.py:1618:
 
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 src/test/python/apache/thermos/core/test_process.py:190: 
in test_simple_process_filesystem_isolator_launch_info
 [1m'"working_directory": "%s", "user": "None", 
"rootfs": "/some/path/taskfs"}\n' % (sandbox))[0m
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 
 taskpath = 
 log_name = 'stdout'
 expected_content = 'launch --unshare_namespace_mnt 
--launch_info={"command": {"shell": false, "arguments": ["/bin/bash", "-c", 
"echo hell..."value": "/bin/bash"}, "working_directory": 
"/tmp/tmpc0WFNT/sandbox", "user": "None", "rootfs": "/some/path/taskfs"}\n'
 
 [1mdef assert_log_content(taskpath, log_name, 
expected_content):[0m
 [1m  log = 
taskpath.with_filename(log_name).getpath('process_logdir')[0m
 [1m  assert os.path.exists(log)[0m
 [1m  with open(log, 'r') as fp:[0m
 [1m>   assert fp.read() == expected_content[0m
 [1m[31mE   assert 'launch --uns.../bin/bash"}\n' == 
'launch --unsh...th/taskfs"}\n'[0m
 [1m[31mE - launch --unshare_namespace_mnt 
--working_directory=/tmp/tmpc0WFNT/sandbox --rootfs=/some/path/taskfs 
--user=None --command={"shell": false, "arguments": ["/bin/bash", "-c", "echo 
hello world"], "value": "/bin/bash"}[0m
 [1m[31mE + launch --unshare_namespace_mnt 
--launch_info={"command": {"shell": false, "arguments": ["/bin/bash", "-c", 
"echo hello world"], "value": "/bin/bash"}, "working_directory": 
"/tmp/tmpc0WFNT/sandbox", "user": "None", "rootfs": "/some/path/taskfs"}[0m
 
 src/test/python/apache/thermos/core/test_process.py:364: 
AssertionError
  generated xml file: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/aaf4d108c31293299a0839bdc404a91802f80937.xml
 
 [1m[31m 1 failed, 780 passed, 6 skipped, 1 warnings in 
288.49 seconds [0m
 
FAILURE


05:52:42 05:26   [complete][31m
   FAILURE[0m


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 15, 2017, 5:40 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55951/
> ---
> 
> (Updated Feb. 15, 2017, 5:40 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Stephan Erb, and Karthik Anantha 
> Padmanabhan.
> 
> 
> Bugs: AURORA-1882
> https://issues.apache.org/jira/browse/AURORA-1882
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MesosContainerizer has updated the command line parameters in 1.2.0 and
> consolidated the individual arguments into a single ContainerLaunchInfo
> proto buf message. Update ThermosExecutor to use the new `--launch_info`
> parameter to be compatible with MesosContainerizer also check the
> containerizer binary interface to determine to be backward-compatible.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/thermos/common/process_util.py 
> 54e716b726fc02f3901f1b9143d3fa253511e29b 
>   src/test/python/apache/aurora/executor/common/test_health_checker.py 
> a3b5a22a805753d42e97f8f1e4e3dab7d3c9c507 
>   src/test/python/apache/thermos/core/test_process.py 
> 520390217f691b9136cb4d36262be3d372a16509 
> 
> Diff: https://reviews.apache.org/r/55951/diff/
> 
> 
> Testing
> ---
> 
> build-support/jenkins/build.sh
> src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 56575: AURORA-1837 Improve task history pruning

2017-02-14 Thread Mehrdad Nurolahzade



> On Feb. 14, 2017, 4:02 p.m., Mehrdad Nurolahzade wrote:
> > Looking at it as is, I'm not sure if there is much value to be gained from 
> > pushing this down to `TaskStore`.
> > Do you see any value in pursuing this idea any further? Or shall I restore 
> > it to previous state?
> 
> Zameer Manji wrote:
> I think there is value in moving operations like this down to storage.
> First, it mirrors the update store for pruning old updates.
> Second, I think we can benefit from more precise storage operations. It 
> allows us to leverage the power of DbTaskStore (ie we don't need to hydrate a 
> full object for deletion) and ensures that callers don't do excessive work.

Right, that was the motivation. 

However the way task deletion is handled now, it needs to go through 
`StateManager` to (a) verify transition to `DELETED` is allowrd and (b) publish 
`TaskDeleted` events to other subscribers like `TaskGroups`, `TaskVars`, and so 
on. For this to work at storage level, like it's done for job updates, we need 
to address its rather inefficient dependency on `TaskManager`.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56575/#review165599
---


On Feb. 14, 2017, 3:38 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56575/
> ---
> 
> (Updated Feb. 14, 2017, 3:38 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1837
> https://issues.apache.org/jira/browse/AURORA-1837
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch addressed efficiency issues in the current implementation of 
> `TaskHistoryPruner`. The new design is similar to that of 
> `JobUpdateHistoryPruner`: (a) Instead of registering a `DelayExecutor` run 
> upon terminal task state transitions, it runs on preconfigured intervals, 
> finds all terminal state tasks that meet pruning criteria and deletes them. 
> (b) Makes the initial task history pruning delay configurable so that it does 
> not hamper scheduler upon start.
> 
> The new design addressed the following two efficiecy problems:
> 
> 1. Upon scheduler restart/failure, the in-memory state of task history 
> pruning scheduled with `DelayExecutor` is lost. `TaskHistoryPruner` learns 
> about these dead tasks upon restart when log is replayed. These expired tasks 
> are picked up by the second call to `executor.execute()` that performs job 
> level pruning immediately (i.e., without delay). Hence, most task history 
> pruning happens after scheduler restarts and can severely hamper scheduler 
> performance (or cause consecutive fail-overs on test clusters when we put 
> load test on scheduler).
> 
> 2. Expired tasks can be picked up for pruning multiple times. The 
> asynchronous nature of `BatchWorker` which used to process task deletions 
> introduces some delay between delete enqueue and delete execution. As a 
> result, tasks already queued for deletion in a previous evaluation round 
> might get picked up, evaluated and enqueued for deletion again. This is 
> evident in `tasks_pruned` metric which reflects numbers much higher than the 
> actual number of expired tasks deleted.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/pruning/PruningModule.java 
> 735199ac1ab343c24471890aa330d6635c26 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> f77849498ff23616f1d56d133eb218f837ac3413 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> 1094a122fe836e53d0481ee5c097447f1e91fa0a 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> a649a6e3d2f2d0aeaf6d7ac704ed24911c310a1e 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> d89e715b1b08faf95f8b5788c9c28cbbb33af093 
>   
> src/test/java/org/apache/aurora/scheduler/pruning/TaskHistoryPrunerTest.java 
> 14e4040e0b94e96f77068b41454311fa3bf53573 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 02719c312294b58525c1fddd3ed096a9b1cef601 
> 
> Diff: https://reviews.apache.org/r/56575/diff/
> 
> 
> Testing
> ---
> 
> Manual testing under Vagrant
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 55951: Use --launch_info when invoking MesosContainerizer.

2017-02-14 Thread Santhosh Kumar Shanmugham


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55951/
---

(Updated Feb. 14, 2017, 9:40 p.m.)


Review request for Aurora, Joshua Cohen, Stephan Erb, and Karthik Anantha 
Padmanabhan.


Changes
---

Adding support for backward-compatibility check.


Bugs: AURORA-1882
https://issues.apache.org/jira/browse/AURORA-1882


Repository: aurora


Description (updated)
---

MesosContainerizer has updated the command line parameters in 1.2.0 and
consolidated the individual arguments into a single ContainerLaunchInfo
proto buf message. Update ThermosExecutor to use the new `--launch_info`
parameter to be compatible with MesosContainerizer also check the
containerizer binary interface to determine to be backward-compatible.


Diffs (updated)
-

  src/main/python/apache/thermos/common/process_util.py 
54e716b726fc02f3901f1b9143d3fa253511e29b 
  src/test/python/apache/aurora/executor/common/test_health_checker.py 
a3b5a22a805753d42e97f8f1e4e3dab7d3c9c507 
  src/test/python/apache/thermos/core/test_process.py 
520390217f691b9136cb4d36262be3d372a16509 

Diff: https://reviews.apache.org/r/55951/diff/


Testing (updated)
---

build-support/jenkins/build.sh
src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh


Thanks,

Santhosh Kumar Shanmugham

Re: Review Request 55951: Use --launch_info when invoking MesosContainerizer.

2017-02-14 Thread Santhosh Kumar Shanmugham



> On Jan. 25, 2017, 11:21 a.m., Joshua Cohen wrote:
> > build-support/packer/build.sh, line 20
> > 
> >
> > This should probably be reverted?
> 
> Santhosh Kumar Shanmugham wrote:
> The current end-to-end test breaks since the Mesos-1.2.0 is not available 
> in the deb repo. So eventually we will have to upgrade to 1.2.0.
> 
> But making the version check I mentioned above should get us over this 
> issue. I will revert this along with that change.

Reverted this.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55951/#review163003
---


On Feb. 14, 2017, 9:40 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55951/
> ---
> 
> (Updated Feb. 14, 2017, 9:40 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Stephan Erb, and Karthik Anantha 
> Padmanabhan.
> 
> 
> Bugs: AURORA-1882
> https://issues.apache.org/jira/browse/AURORA-1882
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MesosContainerizer has updated the command line parameters in 1.2.0 and
> consolidated the individual arguments into a single ContainerLaunchInfo
> proto buf message. Update ThermosExecutor to use the new `--launch_info`
> parameter to be compatible with MesosContainerizer also check the
> containerizer binary interface to determine to be backward-compatible.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/thermos/common/process_util.py 
> 54e716b726fc02f3901f1b9143d3fa253511e29b 
>   src/test/python/apache/aurora/executor/common/test_health_checker.py 
> a3b5a22a805753d42e97f8f1e4e3dab7d3c9c507 
>   src/test/python/apache/thermos/core/test_process.py 
> 520390217f691b9136cb4d36262be3d372a16509 
> 
> Diff: https://reviews.apache.org/r/55951/diff/
> 
> 
> Testing
> ---
> 
> build-support/jenkins/build.sh
> src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 55951: Use --launch_info when invoking MesosContainerizer.

2017-02-14 Thread Santhosh Kumar Shanmugham



> On Jan. 25, 2017, 11:21 a.m., Joshua Cohen wrote:
> > src/main/python/apache/thermos/common/process_util.py, lines 25-37
> > 
> >
> > Rather than continuing to manually construct a json representation of a 
> > protobuf, we might want to look at just creating the actual `LaunchInfo` 
> > protobuf and converting it directly to json?
> > 
> > I looked into this at one point, and it was problematic due to the 
> > version of the protobuf library we use (tied to the version in use by 
> > Mesos). I remember thinking the problem wasn't intractable though. Would 
> > you mind taking a swing at that now that the json object is more complex?
> 
> Santhosh Kumar Shanmugham wrote:
> I could not find any examples, so I will address, but will a while.

This change will need protobuf-3.0 or later which has support for json. But 
mesos.interface=1.1.0 is not compatible with this version. Using 3rdparty 
libraries to achieve the same conversion, only increases our dependency. We can 
update this when Mesos upgrades protobuf, which is probably downline the line, 
considering that they have need to utilize the increased protobuf size limit. 
(https://issues.apache.org/jira/browse/MESOS-7123)


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55951/#review163003
---


On Jan. 25, 2017, 11:02 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55951/
> ---
> 
> (Updated Jan. 25, 2017, 11:02 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Stephan Erb, and Karthik Anantha 
> Padmanabhan.
> 
> 
> Bugs: AURORA-1882
> https://issues.apache.org/jira/browse/AURORA-1882
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MesosContainerizer has updated the command line parameters and
> consolidated the individual arguments into a single ContainerLaunchInfo
> proto buf message. Update ThermosExecutor to use the new `--launch_info`
> parameter to be compatible with MesosContainerizer.
> 
> 
> Diffs
> -
> 
>   build-support/packer/build.sh 548cf37e097c6ed56fc6cc718a642b105afb9331 
>   src/main/python/apache/thermos/common/process_util.py 
> 54e716b726fc02f3901f1b9143d3fa253511e29b 
>   src/test/python/apache/thermos/core/test_process.py 
> 520390217f691b9136cb4d36262be3d372a16509 
> 
> Diff: https://reviews.apache.org/r/55951/diff/
> 
> 
> Testing
> ---
> 
> build-support/jenkins/build.sh
> 
> TBD: End-to-end test needs Mesos 1.2.0 which has not been released?
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 55951: Use --launch_info when invoking MesosContainerizer.

2017-02-14 Thread Santhosh Kumar Shanmugham



> On Jan. 25, 2017, 1:07 p.m., Stephan Erb wrote:
> > The code change itself is fine. However, it's implications are a little bit 
> > horrible :)
> > 
> > * We can only apply it once we have released 0.17 (build against Mesos 1.1)
> > * Mesos broke +-1 version compatibility here, so users have to update to 
> > Thermos 0.18 (i.e. this patch) and Mesos 1.2 at the exact same time. 
> > 
> > I feel like we should escalate this and ask the Mesos people if they are 
> > willing to re-establish compatibility for one version.
> 
> Santhosh Kumar Shanmugham wrote:
> I am in agreement. I will ping the Mesos folks to find out if they can 
> rollout the change via a deprecation cycle.
> 
> Santhosh Kumar Shanmugham wrote:
> Capturing the outcome from the conversation with the Mesos folks.
> 
> MesosContainerizer is considered internal to Mesos and hence will not 
> follow a deprecation cycle. We need an alternative approach here.

Okay. The binary does not support a `version` switch, so we will perform a 
`launch` command without arguments to determine the interface to the binary and 
choose our path accordingly. We will need to have this ugly check until we can 
migrate +1 version of Mesos, after which we can drop this or move to Pod API.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55951/#review163018
---


On Jan. 25, 2017, 11:02 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55951/
> ---
> 
> (Updated Jan. 25, 2017, 11:02 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Stephan Erb, and Karthik Anantha 
> Padmanabhan.
> 
> 
> Bugs: AURORA-1882
> https://issues.apache.org/jira/browse/AURORA-1882
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MesosContainerizer has updated the command line parameters and
> consolidated the individual arguments into a single ContainerLaunchInfo
> proto buf message. Update ThermosExecutor to use the new `--launch_info`
> parameter to be compatible with MesosContainerizer.
> 
> 
> Diffs
> -
> 
>   build-support/packer/build.sh 548cf37e097c6ed56fc6cc718a642b105afb9331 
>   src/main/python/apache/thermos/common/process_util.py 
> 54e716b726fc02f3901f1b9143d3fa253511e29b 
>   src/test/python/apache/thermos/core/test_process.py 
> 520390217f691b9136cb4d36262be3d372a16509 
> 
> Diff: https://reviews.apache.org/r/55951/diff/
> 
> 
> Testing
> ---
> 
> build-support/jenkins/build.sh
> 
> TBD: End-to-end test needs Mesos 1.2.0 which has not been released?
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 56690: [Patch 1/2] RFC for first patch implementing scheduling changes for Dynamic Reservations

2017-02-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56690/#review165623
---



Master (9ea8979) is red with this patch.
  ./build-support/jenkins/build.sh

Note: Recompile with -Xlint:unchecked for details.

:commons:generateThriftResources
:commons:processResources
:commons:classes
:commons:jar
:compileJava/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/storage/log/WriteAheadStorage.java:74:
 Note: Wrote forwarder 
org.apache.aurora.scheduler.storage.log.WriteAheadStorageForwarder
@Forward({
^
Note: Writing 
file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/org/apache/aurora/common/args/apt/cmdline.arg.info.txt.2
Note: Writing 
file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/META-INF/compiler/resource-mappings/org.apache.aurora.common.args.apt.CmdLineProcessor

:generateBuildProperties
:processResources
:classes
:jar
:startScripts
:distTar
:distZip
:assemble
:compileJmhJavaNote: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java
 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:processJmhResources UP-TO-DATE
:jmhClasses
:checkstyleJmh[ant:checkstyle] [ERROR] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java:38:
 Line is longer than 100 characters (found 112). [LineLength]
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':checkstyleJmh'.
> Checkstyle rule violations were found. See the report at: 
> file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/jmh.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 1 mins 9.388 secs


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 15, 2017, 1:44 a.m., Dmitriy Shirchenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56690/
> ---
> 
> (Updated Feb. 15, 2017, 1:44 a.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.
> 
> 
> Bugs: AURORA-1819
> https://issues.apache.org/jira/browse/AURORA-1819
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This is an RFC (without tests) for dynamic reservations proposal. If there is 
> consensus on the approach, I will add tests. This patch was also tested 
> locally and works as expected.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> f2296a9d7a88be7e43124370edecfe64415df00f 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 
> 6f2ca35c5d83dde29c24865b4826d4932e96da80 
>   src/main/java/org/apache/aurora/scheduler/TaskVars.java 
> 676dfd9f9d7ee0633c05424f788fd0ab116976bb 
>   src/main/java/org/apache/aurora/scheduler/TierInfo.java 
> c45b949ae7946fc92d7e62f94696ddc4f0790cfa 
>   src/main/java/org/apache/aurora/scheduler/TierManager.java 
> c6ad2b1c48673ca2c14ddd308684d81ce536beca 
>   src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java 
> b12ac83168401c15fb1d30179ea8e4816f09cd3d 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 0136afb8f6049a6d88cd42b5e3f17d61fcd629d5 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> f0b148cd158d61cd89cc51dca9f3fa4c6feb1b49 
>   
> src/main/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilter.java
>  f6c759f03c4152ae93317692fc9db202fe251122 
>   src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 
> bb1a960a4c77f48b0ceaa213bd27546551f384f9 
>   src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 
> 60097d91d836e2686d6e90571f13a2fbfd88ae14 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 0d639f66db456858278b0485c91c40975c3b45ac 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
> 8c000cb0626bd34f6f30e23fe2b3a045f2b44e35 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferSettings.java 
> e16e36ed360ef9ca371df9084365ea88cfb6e7ce 
>   src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 
> 202cae96ffc5b49e638b973a273f7983137b5baf 
>   src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 
> 9aa263a9cfae03a9a0c5bc7fe3a1405397d3009c 
>   src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java 
> 203f62bacc47470545d095e4d25f7e0f25990ed9 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> da378e84ee65a658ff2382489d3ab6d5f6

Re: Review Request 56691: [Patch 2/2] RFC for 2nd patch implements OfferReconciler for dynamic reservations proposal

2017-02-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56691/#review165621
---



Master (9ea8979) is red with this patch.
  ./build-support/jenkins/build.sh

Note: Recompile with -Xlint:unchecked for details.

:commons:generateThriftResources
:commons:processResources
:commons:classes
:commons:jar
:compileJava/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/storage/log/WriteAheadStorage.java:74:
 Note: Wrote forwarder 
org.apache.aurora.scheduler.storage.log.WriteAheadStorageForwarder
@Forward({
^
Note: Writing 
file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/org/apache/aurora/common/args/apt/cmdline.arg.info.txt.2
Note: Writing 
file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/META-INF/compiler/resource-mappings/org.apache.aurora.common.args.apt.CmdLineProcessor
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/HostOffer.java:67:
 error: incompatible types: List cannot be 
converted to List
List resourceList = offer.getResourcesList();
^
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/offers/OfferReconciler.java:65:
 error: incompatible types: List cannot be 
converted to List
  List resourceList = offer.getOffer().getResourcesList();
 ^
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/offers/OfferReconciler.java:87:
 error: incompatible types: inference variable T has incompatible bounds
  offerManager.unReserveOffer(offer.getOffer().getId(), 
Collections.singletonList(resource));

 ^
equality constraints: org.apache.mesos.v1.Protos.Resource
lower bounds: org.apache.mesos.Protos.Resource
  where T is a type-variable:
T extends Object declared in method singletonList(T)
Note: Some messages have been simplified; recompile with -Xdiags:verbose to get 
full output
3 errors
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 1 mins 3.245 secs


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 15, 2017, 1:48 a.m., Dmitriy Shirchenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56691/
> ---
> 
> (Updated Feb. 15, 2017, 1:48 a.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.
> 
> 
> Bugs: AURORA-1816
> https://issues.apache.org/jira/browse/AURORA-1816
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Second patch that implements mechanism for unreserving previously reserved 
> resources for dynamic reservations proposal.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 
> 6f2ca35c5d83dde29c24865b4826d4932e96da80 
>   src/main/java/org/apache/aurora/scheduler/HostOffer.java 
> 23f0600d64e1e15f4856f397e839e3d1c87f3b96 
>   src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java 
> b12ac83168401c15fb1d30179ea8e4816f09cd3d 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 0136afb8f6049a6d88cd42b5e3f17d61fcd629d5 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> c76b365f43eb6a3b9b0b63a879b43eb04dcd8fac 
>   src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 
> 0637eb7f85125cf70b588d56fa7dc88130947837 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
> 8c000cb0626bd34f6f30e23fe2b3a045f2b44e35 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferReconciler.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 
> 202cae96ffc5b49e638b973a273f7983137b5baf 
>   src/test/java/org/apache/aurora/scheduler/http/OffersTest.java 
> 30699596a1c95199df7504f62c5c18cab1be1c6c 
>   src/test/java/org/apache/aurora/scheduler/offers/HostOffers.java 
> PRE-CREATION 
>   src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
> 49d4e82cc03144b80292fe43066a6cc4d7aed88f 
>   src/test/java/org/apache/aurora/scheduler/offers/OfferReconcilerTest.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/56691/diff/
> 
> 
> Testing
> ---
> 
> Ran locally on vagrant.
> 
> 
> Thanks,
> 
> D

Re: Review Request 56691: [Patch 2/2] RFC for 2nd patch implements OfferReconciler for dynamic reservations proposal

2017-02-14 Thread Dmitriy Shirchenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56691/
---

(Updated Feb. 15, 2017, 1:48 a.m.)


Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.


Bugs: AURORA-1816
https://issues.apache.org/jira/browse/AURORA-1816


Repository: aurora


Description
---

Second patch that implements mechanism for unreserving previously reserved 
resources for dynamic reservations proposal.


Diffs (updated)
-

  src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 
6f2ca35c5d83dde29c24865b4826d4932e96da80 
  src/main/java/org/apache/aurora/scheduler/HostOffer.java 
23f0600d64e1e15f4856f397e839e3d1c87f3b96 
  src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java 
b12ac83168401c15fb1d30179ea8e4816f09cd3d 
  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
0136afb8f6049a6d88cd42b5e3f17d61fcd629d5 
  src/main/java/org/apache/aurora/scheduler/base/Query.java 
c76b365f43eb6a3b9b0b63a879b43eb04dcd8fac 
  src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 
0637eb7f85125cf70b588d56fa7dc88130947837 
  src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
8c000cb0626bd34f6f30e23fe2b3a045f2b44e35 
  src/main/java/org/apache/aurora/scheduler/offers/OfferReconciler.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 
202cae96ffc5b49e638b973a273f7983137b5baf 
  src/test/java/org/apache/aurora/scheduler/http/OffersTest.java 
30699596a1c95199df7504f62c5c18cab1be1c6c 
  src/test/java/org/apache/aurora/scheduler/offers/HostOffers.java PRE-CREATION 
  src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
49d4e82cc03144b80292fe43066a6cc4d7aed88f 
  src/test/java/org/apache/aurora/scheduler/offers/OfferReconcilerTest.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/56691/diff/


Testing
---

Ran locally on vagrant.


Thanks,

Dmitriy Shirchenko

Re: Review Request 56690: [Patch 1/2] RFC for first patch implementing scheduling changes for Dynamic Reservations

2017-02-14 Thread Dmitriy Shirchenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56690/
---

(Updated Feb. 15, 2017, 1:44 a.m.)


Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.


Bugs: AURORA-1819
https://issues.apache.org/jira/browse/AURORA-1819


Repository: aurora


Description
---

This is an RFC (without tests) for dynamic reservations proposal. If there is 
consensus on the approach, I will add tests. This patch was also tested locally 
and works as expected.


Diffs (updated)
-

  src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
f2296a9d7a88be7e43124370edecfe64415df00f 
  src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 
6f2ca35c5d83dde29c24865b4826d4932e96da80 
  src/main/java/org/apache/aurora/scheduler/TaskVars.java 
676dfd9f9d7ee0633c05424f788fd0ab116976bb 
  src/main/java/org/apache/aurora/scheduler/TierInfo.java 
c45b949ae7946fc92d7e62f94696ddc4f0790cfa 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
c6ad2b1c48673ca2c14ddd308684d81ce536beca 
  src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java 
b12ac83168401c15fb1d30179ea8e4816f09cd3d 
  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
0136afb8f6049a6d88cd42b5e3f17d61fcd629d5 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
f0b148cd158d61cd89cc51dca9f3fa4c6feb1b49 
  
src/main/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilter.java 
f6c759f03c4152ae93317692fc9db202fe251122 
  src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 
bb1a960a4c77f48b0ceaa213bd27546551f384f9 
  src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 
60097d91d836e2686d6e90571f13a2fbfd88ae14 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
0d639f66db456858278b0485c91c40975c3b45ac 
  src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
8c000cb0626bd34f6f30e23fe2b3a045f2b44e35 
  src/main/java/org/apache/aurora/scheduler/offers/OfferSettings.java 
e16e36ed360ef9ca371df9084365ea88cfb6e7ce 
  src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 
202cae96ffc5b49e638b973a273f7983137b5baf 
  src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 
9aa263a9cfae03a9a0c5bc7fe3a1405397d3009c 
  src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java 
203f62bacc47470545d095e4d25f7e0f25990ed9 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
da378e84ee65a658ff2382489d3ab6d5f6451b5f 
  src/main/resources/org/apache/aurora/scheduler/tiers.json 
34ddb1dc769a73115c209c9b2ee158cd364392d8 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
82e40d509d84c37a19b6a9ef942283d908833840 
  src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
49d4e82cc03144b80292fe43066a6cc4d7aed88f 
  src/test/java/org/apache/aurora/scheduler/resources/AcceptedOfferTest.java 
dded9c34749cf599d197ed312ffb6bf63b6033f1 
  src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java 
b8b8edb1a21ba89b8b60f8f8451c8c776fc23ae8 
  src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java 
e04f6113c43eca4555ee0719f8208d7c4ebb8d61 
  src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
cf2d25ec2e407df7159e0021ddb44adf937e1777 

Diff: https://reviews.apache.org/r/56690/diff/


Testing
---


Thanks,

Dmitriy Shirchenko

Re: Review Request 53333: Add DSL and E2E changes for per task volume mounts.

2017-02-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/#review165616
---


Ship it!




Master (0e9c086) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 15, 2017, 12:45 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5/
> ---
> 
> (Updated Feb. 15, 2017, 12:45 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1107
> https://issues.apache.org/jira/browse/AURORA-1107
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Enables the client DSL to set per task volume mounts. This also adds a E2E 
> test that tests per task volume mounting.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md ff382ffb62f2720c5f860009157bb89c87b48198 
>   docs/reference/configuration.md 6c7114256ea6f57d6e09e07896e9f4fde19139ca 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 91f0f0c18062cb8ca36c104a270b1d16ff13cd52 
>   src/main/python/apache/aurora/config/schema/base.py 
> b15b93942f6d92d8f4c84c82d61950414f4f3d34 
>   src/main/python/apache/aurora/config/thrift.py 
> 3539469d243638c0acd08bf0859d0ce858d8977c 
>   src/test/python/apache/aurora/config/test_thrift.py 
> e213184739167e01f3614c20a809af39b3a6b3d6 
>   src/test/sh/org/apache/aurora/e2e/Dockerfile.netcat 
> c8b2f46a36d2ddecf030bc7f005afc1f7fe5e99d 
>   src/test/sh/org/apache/aurora/e2e/check-fs.sh PRE-CREATION 
>   src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
> b2b977b0c248e7156061653aacec43947cc49cd5 
>   src/test/sh/org/apache/aurora/e2e/run-server.sh 
> 1fe09090bfa3eeb0f6897e8c895782fb1ff949a1 
> 
> Diff: https://reviews.apache.org/r/5/diff/
> 
> 
> Testing
> ---
> 
> sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 53333: Add DSL and E2E changes for per task volume mounts.

2017-02-14 Thread Zameer Manji


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/
---

(Updated Feb. 14, 2017, 4:45 p.m.)


Review request for Aurora, Joshua Cohen, Santhosh Kumar Shanmugham, and Stephan 
Erb.


Bugs: AURORA-1107
https://issues.apache.org/jira/browse/AURORA-1107


Repository: aurora


Description
---

Enables the client DSL to set per task volume mounts. This also adds a E2E test 
that tests per task volume mounting.


Diffs (updated)
-

  RELEASE-NOTES.md ff382ffb62f2720c5f860009157bb89c87b48198 
  docs/reference/configuration.md 6c7114256ea6f57d6e09e07896e9f4fde19139ca 
  examples/vagrant/upstart/aurora-scheduler.conf 
91f0f0c18062cb8ca36c104a270b1d16ff13cd52 
  src/main/python/apache/aurora/config/schema/base.py 
b15b93942f6d92d8f4c84c82d61950414f4f3d34 
  src/main/python/apache/aurora/config/thrift.py 
3539469d243638c0acd08bf0859d0ce858d8977c 
  src/test/python/apache/aurora/config/test_thrift.py 
e213184739167e01f3614c20a809af39b3a6b3d6 
  src/test/sh/org/apache/aurora/e2e/Dockerfile.netcat 
c8b2f46a36d2ddecf030bc7f005afc1f7fe5e99d 
  src/test/sh/org/apache/aurora/e2e/check-fs.sh PRE-CREATION 
  src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
b2b977b0c248e7156061653aacec43947cc49cd5 
  src/test/sh/org/apache/aurora/e2e/run-server.sh 
1fe09090bfa3eeb0f6897e8c895782fb1ff949a1 

Diff: https://reviews.apache.org/r/5/diff/


Testing
---

sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh


Thanks,

Zameer Manji

Re: Review Request 53333: Add DSL and E2E changes for per task volume mounts.

2017-02-14 Thread Zameer Manji



> On Feb. 13, 2017, 9:28 p.m., Santhosh Kumar Shanmugham wrote:
> > src/test/sh/org/apache/aurora/e2e/check-fs.sh, line 4
> > 
> >
> > Verify `RO` mode?

That feels more like testing what mesos does than what we do.

We already have unit tests to ensure that the 'RO' is passed all the way down 
to mesos.


> On Feb. 13, 2017, 9:28 p.m., Santhosh Kumar Shanmugham wrote:
> > src/test/sh/org/apache/aurora/e2e/http/http_example.aurora, line 130
> > 
> >
> > Mount another volume with `RW` mode?

It seems exessive to put it in our e2e tests.


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/#review165452
---


On Feb. 13, 2017, 6:26 p.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5/
> ---
> 
> (Updated Feb. 13, 2017, 6:26 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1107
> https://issues.apache.org/jira/browse/AURORA-1107
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Enables the client DSL to set per task volume mounts. This also adds a E2E 
> test that tests per task volume mounting.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   docs/reference/configuration.md 6c7114256ea6f57d6e09e07896e9f4fde19139ca 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 91f0f0c18062cb8ca36c104a270b1d16ff13cd52 
>   src/main/python/apache/aurora/config/schema/base.py 
> b15b93942f6d92d8f4c84c82d61950414f4f3d34 
>   src/main/python/apache/aurora/config/thrift.py 
> 3539469d243638c0acd08bf0859d0ce858d8977c 
>   src/test/python/apache/aurora/config/test_thrift.py 
> e213184739167e01f3614c20a809af39b3a6b3d6 
>   src/test/sh/org/apache/aurora/e2e/Dockerfile.netcat 
> c8b2f46a36d2ddecf030bc7f005afc1f7fe5e99d 
>   src/test/sh/org/apache/aurora/e2e/check-fs.sh PRE-CREATION 
>   src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
> b2b977b0c248e7156061653aacec43947cc49cd5 
>   src/test/sh/org/apache/aurora/e2e/run-server.sh 
> 1fe09090bfa3eeb0f6897e8c895782fb1ff949a1 
> 
> Diff: https://reviews.apache.org/r/5/diff/
> 
> 
> Testing
> ---
> 
> sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 53333: Add DSL and E2E changes for per task volume mounts.

2017-02-14 Thread Zameer Manji



> On Feb. 14, 2017, 3:56 p.m., Jason Lai wrote:
> > src/main/python/apache/aurora/config/schema/base.py, line 144
> > 
> >
> > Actually according to 
> > [mesos.proto](https://github.com/apache/mesos/blob/1.1.0/include/mesos/mesos.proto#L2124),
> >  `volumes` is not part of the `MesosInfo` schema, but instead part of 
> > `ContainerInfo` that wraps both `DockerInfo` and `MesosInfo`.
> > 
> > Do you intend to add `volumes` to the `Mesos` schema in the Aurora 
> > land? Or have you thought about maintaining consistency between the Aurora 
> > schema and its Mesos schema counterpart?

I omitted support for this in the DSL because the API definition and the 
implementation don't plumb volume mounts to Docker Containers. I am under the 
impression that DockerContainerizer doesn't support it and I am open to 
revising the DSL and API if there is demand for it.


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/#review165597
---


On Feb. 13, 2017, 6:26 p.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5/
> ---
> 
> (Updated Feb. 13, 2017, 6:26 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1107
> https://issues.apache.org/jira/browse/AURORA-1107
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Enables the client DSL to set per task volume mounts. This also adds a E2E 
> test that tests per task volume mounting.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   docs/reference/configuration.md 6c7114256ea6f57d6e09e07896e9f4fde19139ca 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 91f0f0c18062cb8ca36c104a270b1d16ff13cd52 
>   src/main/python/apache/aurora/config/schema/base.py 
> b15b93942f6d92d8f4c84c82d61950414f4f3d34 
>   src/main/python/apache/aurora/config/thrift.py 
> 3539469d243638c0acd08bf0859d0ce858d8977c 
>   src/test/python/apache/aurora/config/test_thrift.py 
> e213184739167e01f3614c20a809af39b3a6b3d6 
>   src/test/sh/org/apache/aurora/e2e/Dockerfile.netcat 
> c8b2f46a36d2ddecf030bc7f005afc1f7fe5e99d 
>   src/test/sh/org/apache/aurora/e2e/check-fs.sh PRE-CREATION 
>   src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
> b2b977b0c248e7156061653aacec43947cc49cd5 
>   src/test/sh/org/apache/aurora/e2e/run-server.sh 
> 1fe09090bfa3eeb0f6897e8c895782fb1ff949a1 
> 
> Diff: https://reviews.apache.org/r/5/diff/
> 
> 
> Testing
> ---
> 
> sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 53333: Add DSL and E2E changes for per task volume mounts.

2017-02-14 Thread Zameer Manji



> On Feb. 14, 2017, 7:19 a.m., Joshua Cohen wrote:
> > RELEASE-NOTES.md, line 7
> > 
> >
> > s/*/-
> > 
> > Also I think, "Add support for per-task volume mounts for Mesos 
> > containers to the Aurora config DSL." reads a little bit better?

Done.


> On Feb. 14, 2017, 7:19 a.m., Joshua Cohen wrote:
> > docs/reference/configuration.md, lines 475-476
> > 
> >
> > Can you fix spacing so the columns line up? ;)

Done.


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/#review165512
---


On Feb. 13, 2017, 6:26 p.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5/
> ---
> 
> (Updated Feb. 13, 2017, 6:26 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1107
> https://issues.apache.org/jira/browse/AURORA-1107
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Enables the client DSL to set per task volume mounts. This also adds a E2E 
> test that tests per task volume mounting.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   docs/reference/configuration.md 6c7114256ea6f57d6e09e07896e9f4fde19139ca 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 91f0f0c18062cb8ca36c104a270b1d16ff13cd52 
>   src/main/python/apache/aurora/config/schema/base.py 
> b15b93942f6d92d8f4c84c82d61950414f4f3d34 
>   src/main/python/apache/aurora/config/thrift.py 
> 3539469d243638c0acd08bf0859d0ce858d8977c 
>   src/test/python/apache/aurora/config/test_thrift.py 
> e213184739167e01f3614c20a809af39b3a6b3d6 
>   src/test/sh/org/apache/aurora/e2e/Dockerfile.netcat 
> c8b2f46a36d2ddecf030bc7f005afc1f7fe5e99d 
>   src/test/sh/org/apache/aurora/e2e/check-fs.sh PRE-CREATION 
>   src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
> b2b977b0c248e7156061653aacec43947cc49cd5 
>   src/test/sh/org/apache/aurora/e2e/run-server.sh 
> 1fe09090bfa3eeb0f6897e8c895782fb1ff949a1 
> 
> Diff: https://reviews.apache.org/r/5/diff/
> 
> 
> Testing
> ---
> 
> sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 56690: [Patch 1/2] RFC for first patch implementing scheduling changes for Dynamic Reservations

2017-02-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56690/#review165608
---



This patch does not apply cleanly against master (0e9c086), do you need to 
rebase?

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 14, 2017, 11:56 p.m., Dmitriy Shirchenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56690/
> ---
> 
> (Updated Feb. 14, 2017, 11:56 p.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.
> 
> 
> Bugs: AURORA-1819
> https://issues.apache.org/jira/browse/AURORA-1819
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This is an RFC (without tests) for dynamic reservations proposal. If there is 
> consensus on the approach, I will add tests. This patch was also tested 
> locally and works as expected.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> f2296a9d7a88be7e43124370edecfe64415df00f 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 
> fbd24ea4a58e28c14a343170de137c0e0ae437a2 
>   src/main/java/org/apache/aurora/scheduler/TaskVars.java 
> 676dfd9f9d7ee0633c05424f788fd0ab116976bb 
>   src/main/java/org/apache/aurora/scheduler/TierInfo.java 
> c45b949ae7946fc92d7e62f94696ddc4f0790cfa 
>   src/main/java/org/apache/aurora/scheduler/TierManager.java 
> c6ad2b1c48673ca2c14ddd308684d81ce536beca 
>   src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java 
> b12ac83168401c15fb1d30179ea8e4816f09cd3d 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 0136afb8f6049a6d88cd42b5e3f17d61fcd629d5 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> aba73005019c13ac943be9c53ac58c8ce5bfba94 
>   
> src/main/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilter.java
>  f6c759f03c4152ae93317692fc9db202fe251122 
>   src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 
> bb1a960a4c77f48b0ceaa213bd27546551f384f9 
>   src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 
> 60097d91d836e2686d6e90571f13a2fbfd88ae14 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 3d5c3bd139206e970811aa95bd74b78987bb9cfe 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
> 2b12696d0218a48d71e89df4d3a38c6bf84c74c7 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferSettings.java 
> e16e36ed360ef9ca371df9084365ea88cfb6e7ce 
>   src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 
> 202cae96ffc5b49e638b973a273f7983137b5baf 
>   src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 
> d751d017522c682a1373932e286ee7b4c447e2aa 
>   src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java 
> 203f62bacc47470545d095e4d25f7e0f25990ed9 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> a030cebea13d92766584643ddda33c50dec7bdbf 
>   src/main/resources/org/apache/aurora/scheduler/tiers.json 
> 34ddb1dc769a73115c209c9b2ee158cd364392d8 
>   src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
> 82e40d509d84c37a19b6a9ef942283d908833840 
>   src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
> fb8bd85be93280a9b8366b1398a1fa0a5817bb55 
>   src/test/java/org/apache/aurora/scheduler/resources/AcceptedOfferTest.java 
> 2777d7276d55cc0fe75e7470d0dc8182796d67f4 
>   
> src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java 
> 133cd23427485ea329ef10489e5d4a0029cb54cd 
>   src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java 
> 7dc8c179736fab93ca03e174b16e104ba0118127 
>   src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
> f8bb25b612ae42748205901dd4b8cfcdedf5b136 
> 
> Diff: https://reviews.apache.org/r/56690/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Dmitriy Shirchenko
> 
>

Re: Review Request 56691: [Patch 2/2] RFC for 2nd patch implements OfferReconciler for dynamic reservations proposal

2017-02-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56691/#review165607
---



This patch does not apply cleanly against master (0e9c086), do you need to 
rebase?

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 14, 2017, 11:59 p.m., Dmitriy Shirchenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56691/
> ---
> 
> (Updated Feb. 14, 2017, 11:59 p.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.
> 
> 
> Bugs: AURORA-1816
> https://issues.apache.org/jira/browse/AURORA-1816
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Second patch that implements mechanism for unreserving previously reserved 
> resources for dynamic reservations proposal.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 
> fbd24ea4a58e28c14a343170de137c0e0ae437a2 
>   src/main/java/org/apache/aurora/scheduler/HostOffer.java 
> ad30bf978ae5aa278fa9b5e01294c43892b08762 
>   src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java 
> b12ac83168401c15fb1d30179ea8e4816f09cd3d 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 0136afb8f6049a6d88cd42b5e3f17d61fcd629d5 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> c76b365f43eb6a3b9b0b63a879b43eb04dcd8fac 
>   src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 
> 70b5470b9dad1af838b5222cae5ac86487e2f2e4 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
> 6c2b6d20658a7fe75725487c9a983e884d9ddfe5 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferReconciler.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 
> 202cae96ffc5b49e638b973a273f7983137b5baf 
>   src/test/java/org/apache/aurora/scheduler/http/OffersTest.java 
> add0eb8f861a36551ed87ba13a75b82c5cd5bdfa 
>   src/test/java/org/apache/aurora/scheduler/offers/HostOffers.java 
> PRE-CREATION 
>   src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
> 5e570b6341c55be8ef27469077932d1ea8378b55 
>   src/test/java/org/apache/aurora/scheduler/offers/OfferReconcilerTest.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/56691/diff/
> 
> 
> Testing
> ---
> 
> Ran locally on vagrant.
> 
> 
> Thanks,
> 
> Dmitriy Shirchenko
> 
>

Re: Review Request 56575: AURORA-1837 Improve task history pruning

2017-02-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56575/#review165606
---


Ship it!




Master (0e9c086) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 14, 2017, 11:38 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56575/
> ---
> 
> (Updated Feb. 14, 2017, 11:38 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1837
> https://issues.apache.org/jira/browse/AURORA-1837
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch addressed efficiency issues in the current implementation of 
> `TaskHistoryPruner`. The new design is similar to that of 
> `JobUpdateHistoryPruner`: (a) Instead of registering a `DelayExecutor` run 
> upon terminal task state transitions, it runs on preconfigured intervals, 
> finds all terminal state tasks that meet pruning criteria and deletes them. 
> (b) Makes the initial task history pruning delay configurable so that it does 
> not hamper scheduler upon start.
> 
> The new design addressed the following two efficiecy problems:
> 
> 1. Upon scheduler restart/failure, the in-memory state of task history 
> pruning scheduled with `DelayExecutor` is lost. `TaskHistoryPruner` learns 
> about these dead tasks upon restart when log is replayed. These expired tasks 
> are picked up by the second call to `executor.execute()` that performs job 
> level pruning immediately (i.e., without delay). Hence, most task history 
> pruning happens after scheduler restarts and can severely hamper scheduler 
> performance (or cause consecutive fail-overs on test clusters when we put 
> load test on scheduler).
> 
> 2. Expired tasks can be picked up for pruning multiple times. The 
> asynchronous nature of `BatchWorker` which used to process task deletions 
> introduces some delay between delete enqueue and delete execution. As a 
> result, tasks already queued for deletion in a previous evaluation round 
> might get picked up, evaluated and enqueued for deletion again. This is 
> evident in `tasks_pruned` metric which reflects numbers much higher than the 
> actual number of expired tasks deleted.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/pruning/PruningModule.java 
> 735199ac1ab343c24471890aa330d6635c26 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> f77849498ff23616f1d56d133eb218f837ac3413 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> 1094a122fe836e53d0481ee5c097447f1e91fa0a 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> a649a6e3d2f2d0aeaf6d7ac704ed24911c310a1e 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> d89e715b1b08faf95f8b5788c9c28cbbb33af093 
>   
> src/test/java/org/apache/aurora/scheduler/pruning/TaskHistoryPrunerTest.java 
> 14e4040e0b94e96f77068b41454311fa3bf53573 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 02719c312294b58525c1fddd3ed096a9b1cef601 
> 
> Diff: https://reviews.apache.org/r/56575/diff/
> 
> 
> Testing
> ---
> 
> Manual testing under Vagrant
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 56575: AURORA-1837 Improve task history pruning

2017-02-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56575/#review165605
---



Master (0e9c086) is red with this patch.
  ./build-support/jenkins/build.sh

Execution failed for task ':test'.
> Process 'Gradle Test Executor 6' finished with non-zero exit value 137

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

2: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':analyzeReport'.
> Test coverage missing for org/apache/aurora/scheduler/storage/db/views/DbImage
  Test coverage missing for org/apache/aurora/scheduler/http/Mname
  Test coverage missing for org/apache/aurora/scheduler/http/Services
  Test coverage missing for org/apache/aurora/scheduler/http/QuitCallback
  Test coverage missing for org/apache/aurora/scheduler/http/Cron
  Test coverage missing for org/apache/aurora/scheduler/app/VolumeParser
  Test coverage missing for 
org/apache/aurora/scheduler/stats/AsyncStatsModule$OfferAdapter
  Test coverage missing for 
org/apache/aurora/scheduler/http/api/security/IniShiroRealmModule
  Test coverage missing for 
org/apache/aurora/scheduler/http/api/security/ShiroUtils
  Test coverage missing for 
org/apache/aurora/scheduler/http/api/security/HttpSecurityModule$3
  Test coverage missing for 
org/apache/aurora/scheduler/http/api/security/HttpSecurityModule$2
  Test coverage missing for 
org/apache/aurora/scheduler/http/api/security/KerberosPrincipalParser
  Test coverage missing for 
org/apache/aurora/scheduler/http/api/security/Kerberos5ShiroRealmModule
  Test coverage missing for 
org/apache/aurora/scheduler/http/api/security/Kerberos5ShiroRealmModule$1
  Test coverage missing for 
org/apache/aurora/scheduler/discovery/CommonsServiceDiscoveryModule
  Test coverage missing for 
org/apache/aurora/scheduler/reconciliation/KillRetry$KillAttempt
  Test coverage missing for 
org/apache/aurora/scheduler/cron/quartz/CronPredictorImpl
  Test coverage missing for 
org/apache/aurora/scheduler/preemptor/Preemptor$PreemptorImpl
  Test coverage missing for 
org/apache/aurora/scheduler/events/PubsubEvent$DriverDisconnected
  Test coverage missing for 
org/apache/aurora/scheduler/events/PubsubEvent$DriverRegistered
  Test coverage missing for 
org/apache/aurora/scheduler/storage/db/typehandlers/VolumeModeTypeHandler

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

BUILD FAILED

Total time: 29 mins 57.137 secs


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 14, 2017, 11:38 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56575/
> ---
> 
> (Updated Feb. 14, 2017, 11:38 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1837
> https://issues.apache.org/jira/browse/AURORA-1837
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch addressed efficiency issues in the current implementation of 
> `TaskHistoryPruner`. The new design is similar to that of 
> `JobUpdateHistoryPruner`: (a) Instead of registering a `DelayExecutor` run 
> upon terminal task state transitions, it runs on preconfigured intervals, 
> finds all terminal state tasks that meet pruning criteria and deletes them. 
> (b) Makes the initial task history pruning delay configurable so that it does 
> not hamper scheduler upon start.
> 
> The new design addressed the following two efficiecy problems:
> 
> 1. Upon scheduler restart/failure, the in-memory state of task history 
> pruning scheduled with `DelayExecutor` is lost. `TaskHistoryPruner` learns 
> about these dead tasks upon restart when log is replayed. These expired tasks 
> are picked up by the second call to `executor.execute()` that performs job 
> level pruning immediately (i.e., without delay). Hence, most task history 
> pruning happens after scheduler restarts and can severely hamper scheduler 
> performance (or cause consecutive fail-overs on test clusters when we put 
> load test on scheduler).
> 
> 2. Expired tasks can be picked up for pruning multiple times. The 
> asynchronous nature of `BatchWorker` which used to process task deletions 
> introduces some delay between delete enqueue and delete execution. As a 
> result, tasks already queued for deletion in a previous evaluation round 
> might get picked up, evaluated and enqueued for deletion agai

Re: Review Request 56575: AURORA-1837 Improve task history pruning

2017-02-14 Thread Zameer Manji



> On Feb. 14, 2017, 4:02 p.m., Mehrdad Nurolahzade wrote:
> > Looking at it as is, I'm not sure if there is much value to be gained from 
> > pushing this down to `TaskStore`.
> > Do you see any value in pursuing this idea any further? Or shall I restore 
> > it to previous state?

I think there is value in moving operations like this down to storage.
First, it mirrors the update store for pruning old updates.
Second, I think we can benefit from more precise storage operations. It allows 
us to leverage the power of DbTaskStore (ie we don't need to hydrate a full 
object for deletion) and ensures that callers don't do excessive work.


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56575/#review165599
---


On Feb. 14, 2017, 3:38 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56575/
> ---
> 
> (Updated Feb. 14, 2017, 3:38 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1837
> https://issues.apache.org/jira/browse/AURORA-1837
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch addressed efficiency issues in the current implementation of 
> `TaskHistoryPruner`. The new design is similar to that of 
> `JobUpdateHistoryPruner`: (a) Instead of registering a `DelayExecutor` run 
> upon terminal task state transitions, it runs on preconfigured intervals, 
> finds all terminal state tasks that meet pruning criteria and deletes them. 
> (b) Makes the initial task history pruning delay configurable so that it does 
> not hamper scheduler upon start.
> 
> The new design addressed the following two efficiecy problems:
> 
> 1. Upon scheduler restart/failure, the in-memory state of task history 
> pruning scheduled with `DelayExecutor` is lost. `TaskHistoryPruner` learns 
> about these dead tasks upon restart when log is replayed. These expired tasks 
> are picked up by the second call to `executor.execute()` that performs job 
> level pruning immediately (i.e., without delay). Hence, most task history 
> pruning happens after scheduler restarts and can severely hamper scheduler 
> performance (or cause consecutive fail-overs on test clusters when we put 
> load test on scheduler).
> 
> 2. Expired tasks can be picked up for pruning multiple times. The 
> asynchronous nature of `BatchWorker` which used to process task deletions 
> introduces some delay between delete enqueue and delete execution. As a 
> result, tasks already queued for deletion in a previous evaluation round 
> might get picked up, evaluated and enqueued for deletion again. This is 
> evident in `tasks_pruned` metric which reflects numbers much higher than the 
> actual number of expired tasks deleted.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/pruning/PruningModule.java 
> 735199ac1ab343c24471890aa330d6635c26 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> f77849498ff23616f1d56d133eb218f837ac3413 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> 1094a122fe836e53d0481ee5c097447f1e91fa0a 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> a649a6e3d2f2d0aeaf6d7ac704ed24911c310a1e 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> d89e715b1b08faf95f8b5788c9c28cbbb33af093 
>   
> src/test/java/org/apache/aurora/scheduler/pruning/TaskHistoryPrunerTest.java 
> 14e4040e0b94e96f77068b41454311fa3bf53573 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 02719c312294b58525c1fddd3ed096a9b1cef601 
> 
> Diff: https://reviews.apache.org/r/56575/diff/
> 
> 
> Testing
> ---
> 
> Manual testing under Vagrant
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 56575: AURORA-1837 Improve task history pruning

2017-02-14 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56575/#review165599
---



Looking at it as is, I'm not sure if there is much value to be gained from 
pushing this down to `TaskStore`.
Do you see any value in pursuing this idea any further? Or shall I restore it 
to previous state?

- Mehrdad Nurolahzade


On Feb. 14, 2017, 3:38 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56575/
> ---
> 
> (Updated Feb. 14, 2017, 3:38 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1837
> https://issues.apache.org/jira/browse/AURORA-1837
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch addressed efficiency issues in the current implementation of 
> `TaskHistoryPruner`. The new design is similar to that of 
> `JobUpdateHistoryPruner`: (a) Instead of registering a `DelayExecutor` run 
> upon terminal task state transitions, it runs on preconfigured intervals, 
> finds all terminal state tasks that meet pruning criteria and deletes them. 
> (b) Makes the initial task history pruning delay configurable so that it does 
> not hamper scheduler upon start.
> 
> The new design addressed the following two efficiecy problems:
> 
> 1. Upon scheduler restart/failure, the in-memory state of task history 
> pruning scheduled with `DelayExecutor` is lost. `TaskHistoryPruner` learns 
> about these dead tasks upon restart when log is replayed. These expired tasks 
> are picked up by the second call to `executor.execute()` that performs job 
> level pruning immediately (i.e., without delay). Hence, most task history 
> pruning happens after scheduler restarts and can severely hamper scheduler 
> performance (or cause consecutive fail-overs on test clusters when we put 
> load test on scheduler).
> 
> 2. Expired tasks can be picked up for pruning multiple times. The 
> asynchronous nature of `BatchWorker` which used to process task deletions 
> introduces some delay between delete enqueue and delete execution. As a 
> result, tasks already queued for deletion in a previous evaluation round 
> might get picked up, evaluated and enqueued for deletion again. This is 
> evident in `tasks_pruned` metric which reflects numbers much higher than the 
> actual number of expired tasks deleted.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/pruning/PruningModule.java 
> 735199ac1ab343c24471890aa330d6635c26 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> f77849498ff23616f1d56d133eb218f837ac3413 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> 1094a122fe836e53d0481ee5c097447f1e91fa0a 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> a649a6e3d2f2d0aeaf6d7ac704ed24911c310a1e 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> d89e715b1b08faf95f8b5788c9c28cbbb33af093 
>   
> src/test/java/org/apache/aurora/scheduler/pruning/TaskHistoryPrunerTest.java 
> 14e4040e0b94e96f77068b41454311fa3bf53573 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 02719c312294b58525c1fddd3ed096a9b1cef601 
> 
> Diff: https://reviews.apache.org/r/56575/diff/
> 
> 
> Testing
> ---
> 
> Manual testing under Vagrant
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Review Request 56691: [Patch 2/2] RFC for 2nd patch implements OfferReconciler for dynamic reservations proposal

2017-02-14 Thread Dmitriy Shirchenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56691/
---

Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.


Bugs: AURORA-1816
https://issues.apache.org/jira/browse/AURORA-1816


Repository: aurora


Description
---

Second patch that implements mechanism for unreserving previously reserved 
resources for dynamic reservations proposal.


Diffs
-

  src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 
fbd24ea4a58e28c14a343170de137c0e0ae437a2 
  src/main/java/org/apache/aurora/scheduler/HostOffer.java 
ad30bf978ae5aa278fa9b5e01294c43892b08762 
  src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java 
b12ac83168401c15fb1d30179ea8e4816f09cd3d 
  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
0136afb8f6049a6d88cd42b5e3f17d61fcd629d5 
  src/main/java/org/apache/aurora/scheduler/base/Query.java 
c76b365f43eb6a3b9b0b63a879b43eb04dcd8fac 
  src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 
70b5470b9dad1af838b5222cae5ac86487e2f2e4 
  src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
6c2b6d20658a7fe75725487c9a983e884d9ddfe5 
  src/main/java/org/apache/aurora/scheduler/offers/OfferReconciler.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 
202cae96ffc5b49e638b973a273f7983137b5baf 
  src/test/java/org/apache/aurora/scheduler/http/OffersTest.java 
add0eb8f861a36551ed87ba13a75b82c5cd5bdfa 
  src/test/java/org/apache/aurora/scheduler/offers/HostOffers.java PRE-CREATION 
  src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
5e570b6341c55be8ef27469077932d1ea8378b55 
  src/test/java/org/apache/aurora/scheduler/offers/OfferReconcilerTest.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/56691/diff/


Testing
---

Ran locally on vagrant.


Thanks,

Dmitriy Shirchenko

Re: Review Request 53333: Add DSL and E2E changes for per task volume mounts.

2017-02-14 Thread Jason Lai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/#review165597
---




src/main/python/apache/aurora/config/schema/base.py (line 144)


Actually according to 
[mesos.proto](https://github.com/apache/mesos/blob/1.1.0/include/mesos/mesos.proto#L2124),
 `volumes` is not part of the `MesosInfo` schema, but instead part of 
`ContainerInfo` that wraps both `DockerInfo` and `MesosInfo`.

Do you intend to add `volumes` to the `Mesos` schema in the Aurora land? Or 
have you thought about maintaining consistency between the Aurora schema and 
its Mesos schema counterpart?


- Jason Lai


On Feb. 14, 2017, 2:26 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5/
> ---
> 
> (Updated Feb. 14, 2017, 2:26 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1107
> https://issues.apache.org/jira/browse/AURORA-1107
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Enables the client DSL to set per task volume mounts. This also adds a E2E 
> test that tests per task volume mounting.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   docs/reference/configuration.md 6c7114256ea6f57d6e09e07896e9f4fde19139ca 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 91f0f0c18062cb8ca36c104a270b1d16ff13cd52 
>   src/main/python/apache/aurora/config/schema/base.py 
> b15b93942f6d92d8f4c84c82d61950414f4f3d34 
>   src/main/python/apache/aurora/config/thrift.py 
> 3539469d243638c0acd08bf0859d0ce858d8977c 
>   src/test/python/apache/aurora/config/test_thrift.py 
> e213184739167e01f3614c20a809af39b3a6b3d6 
>   src/test/sh/org/apache/aurora/e2e/Dockerfile.netcat 
> c8b2f46a36d2ddecf030bc7f005afc1f7fe5e99d 
>   src/test/sh/org/apache/aurora/e2e/check-fs.sh PRE-CREATION 
>   src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
> b2b977b0c248e7156061653aacec43947cc49cd5 
>   src/test/sh/org/apache/aurora/e2e/run-server.sh 
> 1fe09090bfa3eeb0f6897e8c895782fb1ff949a1 
> 
> Diff: https://reviews.apache.org/r/5/diff/
> 
> 
> Testing
> ---
> 
> sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Review Request 56690: [Patch 1/2] RFC for first patch implementing scheduling changes for Dynamic Reservations

2017-02-14 Thread Dmitriy Shirchenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56690/
---

Review request for Aurora, Mehrdad Nurolahzade, Stephan Erb, and Zameer Manji.


Bugs: AURORA-1819
https://issues.apache.org/jira/browse/AURORA-1819


Repository: aurora


Description
---

This is an RFC (without tests) for dynamic reservations proposal. If there is 
consensus on the approach, I will add tests. This patch was also tested locally 
and works as expected.


Diffs
-

  src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
f2296a9d7a88be7e43124370edecfe64415df00f 
  src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java 
fbd24ea4a58e28c14a343170de137c0e0ae437a2 
  src/main/java/org/apache/aurora/scheduler/TaskVars.java 
676dfd9f9d7ee0633c05424f788fd0ab116976bb 
  src/main/java/org/apache/aurora/scheduler/TierInfo.java 
c45b949ae7946fc92d7e62f94696ddc4f0790cfa 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
c6ad2b1c48673ca2c14ddd308684d81ce536beca 
  src/main/java/org/apache/aurora/scheduler/base/InstanceKeys.java 
b12ac83168401c15fb1d30179ea8e4816f09cd3d 
  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
0136afb8f6049a6d88cd42b5e3f17d61fcd629d5 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
aba73005019c13ac943be9c53ac58c8ce5bfba94 
  
src/main/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilter.java 
f6c759f03c4152ae93317692fc9db202fe251122 
  src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 
bb1a960a4c77f48b0ceaa213bd27546551f384f9 
  src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 
60097d91d836e2686d6e90571f13a2fbfd88ae14 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
3d5c3bd139206e970811aa95bd74b78987bb9cfe 
  src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
2b12696d0218a48d71e89df4d3a38c6bf84c74c7 
  src/main/java/org/apache/aurora/scheduler/offers/OfferSettings.java 
e16e36ed360ef9ca371df9084365ea88cfb6e7ce 
  src/main/java/org/apache/aurora/scheduler/offers/OffersModule.java 
202cae96ffc5b49e638b973a273f7983137b5baf 
  src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 
d751d017522c682a1373932e286ee7b4c447e2aa 
  src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java 
203f62bacc47470545d095e4d25f7e0f25990ed9 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
a030cebea13d92766584643ddda33c50dec7bdbf 
  src/main/resources/org/apache/aurora/scheduler/tiers.json 
34ddb1dc769a73115c209c9b2ee158cd364392d8 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
82e40d509d84c37a19b6a9ef942283d908833840 
  src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
fb8bd85be93280a9b8366b1398a1fa0a5817bb55 
  src/test/java/org/apache/aurora/scheduler/resources/AcceptedOfferTest.java 
2777d7276d55cc0fe75e7470d0dc8182796d67f4 
  src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java 
133cd23427485ea329ef10489e5d4a0029cb54cd 
  src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java 
7dc8c179736fab93ca03e174b16e104ba0118127 
  src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
f8bb25b612ae42748205901dd4b8cfcdedf5b136 

Diff: https://reviews.apache.org/r/56690/diff/


Testing
---


Thanks,

Dmitriy Shirchenko

Re: Review Request 56575: AURORA-1837 Improve task history pruning

2017-02-14 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56575/
---

(Updated Feb. 14, 2017, 3:38 p.m.)


Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, and 
Stephan Erb.


Changes
---

WIP: pushing prunable history selection logic down to `TaskStore`.
- Done: `MemTaskStore`
- ToDo: `DbTaskStore`


Bugs: AURORA-1837
https://issues.apache.org/jira/browse/AURORA-1837


Repository: aurora


Description
---

This patch addressed efficiency issues in the current implementation of 
`TaskHistoryPruner`. The new design is similar to that of 
`JobUpdateHistoryPruner`: (a) Instead of registering a `DelayExecutor` run upon 
terminal task state transitions, it runs on preconfigured intervals, finds all 
terminal state tasks that meet pruning criteria and deletes them. (b) Makes the 
initial task history pruning delay configurable so that it does not hamper 
scheduler upon start.

The new design addressed the following two efficiecy problems:

1. Upon scheduler restart/failure, the in-memory state of task history pruning 
scheduled with `DelayExecutor` is lost. `TaskHistoryPruner` learns about these 
dead tasks upon restart when log is replayed. These expired tasks are picked up 
by the second call to `executor.execute()` that performs job level pruning 
immediately (i.e., without delay). Hence, most task history pruning happens 
after scheduler restarts and can severely hamper scheduler performance (or 
cause consecutive fail-overs on test clusters when we put load test on 
scheduler).

2. Expired tasks can be picked up for pruning multiple times. The asynchronous 
nature of `BatchWorker` which used to process task deletions introduces some 
delay between delete enqueue and delete execution. As a result, tasks already 
queued for deletion in a previous evaluation round might get picked up, 
evaluated and enqueued for deletion again. This is evident in `tasks_pruned` 
metric which reflects numbers much higher than the actual number of expired 
tasks deleted.


Diffs (updated)
-

  src/main/java/org/apache/aurora/scheduler/pruning/PruningModule.java 
735199ac1ab343c24471890aa330d6635c26 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
f77849498ff23616f1d56d133eb218f837ac3413 
  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
1094a122fe836e53d0481ee5c097447f1e91fa0a 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
a649a6e3d2f2d0aeaf6d7ac704ed24911c310a1e 
  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
d89e715b1b08faf95f8b5788c9c28cbbb33af093 
  src/test/java/org/apache/aurora/scheduler/pruning/TaskHistoryPrunerTest.java 
14e4040e0b94e96f77068b41454311fa3bf53573 
  src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
02719c312294b58525c1fddd3ed096a9b1cef601 

Diff: https://reviews.apache.org/r/56575/diff/


Testing
---

Manual testing under Vagrant


Thanks,

Mehrdad Nurolahzade

Re: Review Request 56577: Add basic test scripts for RPM and DEB packages

2017-02-14 Thread Stephan Erb


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56577/
---

(Updated Feb. 14, 2017, 11:33 p.m.)


Review request for Aurora, John Sirois and Renan DelValle.


Changes
---

Review changes.


Repository: aurora-packaging


Description
---

Add basic test scripts for RPM and DEB packages

This reduces some of the manual effort for verifying the binary packages.


Diffs (updated)
-

  .gitignore 2d30051c482a2416026421808876fe456337290f 
  build-artifact.sh e4efe5f1ac36c9e19082ef9e8be10507b901076f 
  test/deb/debian-jessie/README.md 2e9bcb70cb0052c48ad325d9cd29b4a1c0a7eaf6 
  test/deb/debian-jessie/Vagrantfile f79c3379c113190784c40211318a8f19bc8e3fa4 
  test/deb/debian-jessie/install-aurora.sh PRE-CREATION 
  test/deb/debian-jessie/provision.sh 175f1ba81c2b1104b13d5c7ea6c463544c85a11c 
  test/deb/ubuntu-trusty/README.md 116ae8eb95afbafb5f350be5f41fd2f9efc7d9a8 
  test/deb/ubuntu-trusty/Vagrantfile ed5ec8e6b42d2960df9823bfafa13bf182ea2040 
  test/deb/ubuntu-trusty/install-aurora.sh PRE-CREATION 
  test/deb/ubuntu-trusty/provision.sh 123590e63a1440b787836eee937634bd63648517 
  test/deb/ubuntu-xenial/README.md a5284ecaf6803209037a2915b24469a7e154dc8c 
  test/deb/ubuntu-xenial/Vagrantfile 3ce6209ba616a4d156c5c4c6f24d8f148a0f7522 
  test/deb/ubuntu-xenial/install-aurora.sh PRE-CREATION 
  test/deb/ubuntu-xenial/provision.sh eda79f69395769e02771e290135b0fef05215b91 
  test/rpm/centos-7/README.md 7fcd5a0da87cd62b72a84cc70be96ab93a1cd3c5 
  test/rpm/centos-7/Vagrantfile 5059b8410dd271dbae6cd23f5909fd41fb88244a 
  test/rpm/centos-7/install-aurora.sh PRE-CREATION 
  test/test-artifact.sh PRE-CREATION 
  test/test-aurora.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/56577/diff/


Testing
---

See the distribution specific README.md.


Thanks,

Stephan Erb

Re: Review Request 53333: Add DSL and E2E changes for per task volume mounts.

2017-02-14 Thread Stephan Erb


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/#review165579
---


Ship it!




Ship It!

- Stephan Erb


On Feb. 14, 2017, 3:26 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5/
> ---
> 
> (Updated Feb. 14, 2017, 3:26 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1107
> https://issues.apache.org/jira/browse/AURORA-1107
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Enables the client DSL to set per task volume mounts. This also adds a E2E 
> test that tests per task volume mounting.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   docs/reference/configuration.md 6c7114256ea6f57d6e09e07896e9f4fde19139ca 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 91f0f0c18062cb8ca36c104a270b1d16ff13cd52 
>   src/main/python/apache/aurora/config/schema/base.py 
> b15b93942f6d92d8f4c84c82d61950414f4f3d34 
>   src/main/python/apache/aurora/config/thrift.py 
> 3539469d243638c0acd08bf0859d0ce858d8977c 
>   src/test/python/apache/aurora/config/test_thrift.py 
> e213184739167e01f3614c20a809af39b3a6b3d6 
>   src/test/sh/org/apache/aurora/e2e/Dockerfile.netcat 
> c8b2f46a36d2ddecf030bc7f005afc1f7fe5e99d 
>   src/test/sh/org/apache/aurora/e2e/check-fs.sh PRE-CREATION 
>   src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
> b2b977b0c248e7156061653aacec43947cc49cd5 
>   src/test/sh/org/apache/aurora/e2e/run-server.sh 
> 1fe09090bfa3eeb0f6897e8c895782fb1ff949a1 
> 
> Diff: https://reviews.apache.org/r/5/diff/
> 
> 
> Testing
> ---
> 
> sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

2017-02-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165577
---



Master (40d91fe) is red with this patch.
  ./build-support/jenkins/build.sh

  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/EntrySerializer$EntrySerializerImpl$1
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$8
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$7
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$4
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$3
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$6
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$5
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$2
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$1
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/LogStorage$Settings
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/LogStorage$ScheduledExecutorSchedulingService
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/LogStorageModule
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/TemporaryStorage$TemporaryStorageFactory$1
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/BackupModule
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/Recovery$RecoveryImpl
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/TemporaryStorage$TemporaryStorageFactory
  Test coverage missing for 
org/apache/aurora/scheduler/storage/backup/Recovery$RecoveryImpl$PendingRecovery
  Test coverage missing for org/apache/aurora/scheduler/TaskVars
  Test coverage missing for 
org/apache/aurora/scheduler/SchedulerLifecycle$DefaultDelayedActions
  Test coverage missing for 
org/apache/aurora/scheduler/TierManager$TierManagerImpl$TierConfig
  Test coverage missing for org/apache/aurora/scheduler/TaskVars$Counter
  Test coverage missing for org/apache/aurora/scheduler/TaskVars$1
  Test coverage missing for 
org/apache/aurora/scheduler/SchedulerModule$TaskEventBatchWorker
  Test coverage missing for org/apache/aurora/scheduler/HostOffer$1
  Test coverage missing for org/apache/aurora/scheduler/SchedulerModule
  Test coverage missing for 
org/apache/aurora/scheduler/TaskIdGenerator$TaskIdGeneratorImpl
  Test coverage missing for org/apache/aurora/scheduler/SchedulerModule$1
  Test coverage missing for org/apache/aurora/scheduler/TaskStatusHandlerImpl
  Test coverage missing for org/apache/aurora/scheduler/TaskStatusHandlerImpl$1
  Test coverage missing for org/apache/aurora/scheduler/TierModule
  Test coverage missing for 
org/apache/aurora/scheduler/storage/db/typehandlers/VolumeModeTypeHandler

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

BUILD FAILED

Total time: 6 mins 30.931 secs


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 14, 2017, 9:13 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 9:13 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Bugs: AURORA-1893
> https://issues.apache.org/jira/browse/AURORA-1893
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d397

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

2017-02-14 Thread David McLaughlin



> On Feb. 14, 2017, 7:34 p.m., Zameer Manji wrote:
> > Linking this to a ticket would be nice for posterity.
> > 
> > This seems like a useful admin tool. Thanks for adding it. I would probably 
> > use this in cases where some large jobs go flapping and populate the DB 
> > with thousands of terminal tasks.

Filed AURORA-1893.


> On Feb. 14, 2017, 7:34 p.m., Zameer Manji wrote:
> > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java,
> >  line 1130
> > 
> >
> > Could you refer to AURORA-1892?

Done.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165562
---


On Feb. 14, 2017, 9:13 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 9:13 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Bugs: AURORA-1893
> https://issues.apache.org/jira/browse/AURORA-1893
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

2017-02-14 Thread David McLaughlin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/
---

(Updated Feb. 14, 2017, 9:13 p.m.)


Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
Manji.


Changes
---

Add ticket to comment.


Bugs: AURORA-1893
https://issues.apache.org/jira/browse/AURORA-1893


Repository: aurora


Description
---

Expose task pruning endpoint in aurora_admin. Useful for scale testing in order 
to 'clean up' after a test run, but also useful in production if you have a bad 
actor inflating the size of your task index.


Diffs (updated)
-

  RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
a211483caf59755b1f9b286b58f183e00db6eee6 
  src/main/python/apache/aurora/admin/admin.py 
070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py 
1250ccd16f906eec08c159df3300d1b94b306d8e 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0cdd9829417bb3ef0215278a9458a3dd78a49a20 
  src/test/python/apache/aurora/admin/test_admin.py 
66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py 
fab97986dcecf7761e161f4074d5108a8c362e6e 

Diff: https://reviews.apache.org/r/56629/diff/


Testing
---

Ran various combinations of task pruning via aurora_admin in Vagrant. Works as 
expected. 

Also:

./pants test src/test/python/apache/aurora/admin::


Thanks,

David McLaughlin

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

2017-02-14 Thread David McLaughlin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/
---

(Updated Feb. 14, 2017, 9:11 p.m.)


Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
Manji.


Changes
---

Filed JIRA ticket to track motivation.


Bugs: AURORA-1893
https://issues.apache.org/jira/browse/AURORA-1893


Repository: aurora


Description
---

Expose task pruning endpoint in aurora_admin. Useful for scale testing in order 
to 'clean up' after a test run, but also useful in production if you have a bad 
actor inflating the size of your task index.


Diffs
-

  RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
a211483caf59755b1f9b286b58f183e00db6eee6 
  src/main/python/apache/aurora/admin/admin.py 
070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py 
1250ccd16f906eec08c159df3300d1b94b306d8e 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0cdd9829417bb3ef0215278a9458a3dd78a49a20 
  src/test/python/apache/aurora/admin/test_admin.py 
66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py 
fab97986dcecf7761e161f4074d5108a8c362e6e 

Diff: https://reviews.apache.org/r/56629/diff/


Testing
---

Ran various combinations of task pruning via aurora_admin in Vagrant. Works as 
expected. 

Also:

./pants test src/test/python/apache/aurora/admin::


Thanks,

David McLaughlin

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

2017-02-14 Thread Reza Motamedi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165565
---


Ship it!




Ship It!

- Reza Motamedi


On Feb. 14, 2017, 5:41 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 5:41 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

2017-02-14 Thread Zameer Manji


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165562
---


Ship it!




Linking this to a ticket would be nice for posterity.

This seems like a useful admin tool. Thanks for adding it. I would probably use 
this in cases where some large jobs go flapping and populate the DB with 
thousands of terminal tasks.


src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
(line 1130)


Could you refer to AURORA-1892?


- Zameer Manji


On Feb. 13, 2017, 9:41 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 13, 2017, 9:41 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 53333: Add DSL and E2E changes for per task volume mounts.

2017-02-14 Thread Joshua Cohen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/#review165512
---


Ship it!




lgtm modulo some truly nitpicky nits below ;).


RELEASE-NOTES.md (line 7)


s/*/-

Also I think, "Add support for per-task volume mounts for Mesos containers 
to the Aurora config DSL." reads a little bit better?



docs/reference/configuration.md (lines 475 - 476)


Can you fix spacing so the columns line up? ;)


- Joshua Cohen


On Feb. 14, 2017, 2:26 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5/
> ---
> 
> (Updated Feb. 14, 2017, 2:26 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Santhosh Kumar Shanmugham, and 
> Stephan Erb.
> 
> 
> Bugs: AURORA-1107
> https://issues.apache.org/jira/browse/AURORA-1107
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Enables the client DSL to set per task volume mounts. This also adds a E2E 
> test that tests per task volume mounting.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   docs/reference/configuration.md 6c7114256ea6f57d6e09e07896e9f4fde19139ca 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 91f0f0c18062cb8ca36c104a270b1d16ff13cd52 
>   src/main/python/apache/aurora/config/schema/base.py 
> b15b93942f6d92d8f4c84c82d61950414f4f3d34 
>   src/main/python/apache/aurora/config/thrift.py 
> 3539469d243638c0acd08bf0859d0ce858d8977c 
>   src/test/python/apache/aurora/config/test_thrift.py 
> e213184739167e01f3614c20a809af39b3a6b3d6 
>   src/test/sh/org/apache/aurora/e2e/Dockerfile.netcat 
> c8b2f46a36d2ddecf030bc7f005afc1f7fe5e99d 
>   src/test/sh/org/apache/aurora/e2e/check-fs.sh PRE-CREATION 
>   src/test/sh/org/apache/aurora/e2e/http/http_example.aurora 
> b2b977b0c248e7156061653aacec43947cc49cd5 
>   src/test/sh/org/apache/aurora/e2e/run-server.sh 
> 1fe09090bfa3eeb0f6897e8c895782fb1ff949a1 
> 
> Diff: https://reviews.apache.org/r/5/diff/
> 
> 
> Testing
> ---
> 
> sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

37 matches

Mail list logo