Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

> On Feb. 14, 2017, 7:34 p.m., Zameer Manji wrote:
> > Linking this to a ticket would be nice for posterity.
> > 
> > This seems like a useful admin tool. Thanks for adding it. I would probably 
> > use this in cases where some large jobs go flapping and populate the DB 
> > with thousands of terminal tasks.

Filed AURORA-1893.


> On Feb. 14, 2017, 7:34 p.m., Zameer Manji wrote:
> > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java,
> >  line 1130
> > 
> >
> > Could you refer to AURORA-1892?

Done.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165562
---


On Feb. 14, 2017, 9:13 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 9:13 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Bugs: AURORA-1893
> https://issues.apache.org/jira/browse/AURORA-1893
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/
---

(Updated Feb. 14, 2017, 9:11 p.m.)


Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
Manji.


Changes
---

Filed JIRA ticket to track motivation.


Bugs: AURORA-1893
https://issues.apache.org/jira/browse/AURORA-1893


Repository: aurora


Description
---

Expose task pruning endpoint in aurora_admin. Useful for scale testing in order 
to 'clean up' after a test run, but also useful in production if you have a bad 
actor inflating the size of your task index.


Diffs
-

  RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
a211483caf59755b1f9b286b58f183e00db6eee6 
  src/main/python/apache/aurora/admin/admin.py 
070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py 
1250ccd16f906eec08c159df3300d1b94b306d8e 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0cdd9829417bb3ef0215278a9458a3dd78a49a20 
  src/test/python/apache/aurora/admin/test_admin.py 
66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py 
fab97986dcecf7761e161f4074d5108a8c362e6e 

Diff: https://reviews.apache.org/r/56629/diff/


Testing
---

Ran various combinations of task pruning via aurora_admin in Vagrant. Works as 
expected. 

Also:

./pants test src/test/python/apache/aurora/admin::


Thanks,

David McLaughlin

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Zameer Manji]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165562
---


Ship it!




Linking this to a ticket would be nice for posterity.

This seems like a useful admin tool. Thanks for adding it. I would probably use 
this in cases where some large jobs go flapping and populate the DB with 
thousands of terminal tasks.


src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
(line 1130)


Could you refer to AURORA-1892?


- Zameer Manji


On Feb. 13, 2017, 9:41 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 13, 2017, 9:41 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

> On Feb. 14, 2017, 5:44 a.m., Santhosh Kumar Shanmugham wrote:
> > LGTM modulo my pervious comment,
> > 
> > "Since this command has the potential to hurt the Scheduler, should we make 
> > sure that atleast one of the filters is provided? Or force a minimum 
> > default limit to avoid operators from accidentally bringing down the 
> > scheduler?"

Not a huge fan of client-side limits like that. We should address how expensive 
task deletion is on the server side.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165456
---


On Feb. 14, 2017, 5:41 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 5:41 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165460
---


Ship it!




Master (40d91fe) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 14, 2017, 5:41 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 5:41 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Santhosh Kumar Shanmugham]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165456
---


Ship it!




LGTM modulo my pervious comment,

"Since this command has the potential to hurt the Scheduler, should we make 
sure that atleast one of the filters is provided? Or force a minimum default 
limit to avoid operators from accidentally bringing down the scheduler?"

- Santhosh Kumar Shanmugham


On Feb. 13, 2017, 9:41 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 13, 2017, 9:41 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 3e9880249aa40215a08424b58f85a634337cc67e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165450
---


Ship it!




Master (40d91fe) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 14, 2017, 5:06 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 5:06 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Santhosh Kumar Shanmugham]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165448
---



Add entry in RELEASE-NOTES.

- Santhosh Kumar Shanmugham


On Feb. 13, 2017, 9:06 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 13, 2017, 9:06 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Santhosh Kumar Shanmugham]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165447
---




src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
(lines 1130 - 1134)


Limit needs to be applied at the Store, both `MemTaskStore` and 
`DBTaskStore` do not appear to use the `limit` and `offset` filters. We can fix 
this in another RB.


- Santhosh Kumar Shanmugham


On Feb. 13, 2017, 9:06 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 13, 2017, 9:06 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Santhosh Kumar Shanmugham]

> On Feb. 13, 2017, 5:54 p.m., Reza Motamedi wrote:
> > LGTM!
> > 
> > But, should we test this in the test cluster? At least to check the effect 
> > of _limit_?
> 
> David McLaughlin wrote:
> I did test this in Vagrant. All the options work correctly.
> 
> David McLaughlin wrote:
> Weird, verifying the StateManager change in Vagrant and now limit does 
> not appear to work.

Weird - `TaskQuery.getLimit()` is only used in 
`ReadOnlySchedulerImpl.getTasks`. (I wonder how your previous test passed?) 

Even this implementation is of limit in `getTasks` is inefficient, since we 
fetch all the tasks from the store and then apply the limit. Limit needs to be 
applied as part of `TaskStore.queryFilter`.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165425
---


On Feb. 13, 2017, 8:28 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 13, 2017, 8:28 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/
---

(Updated Feb. 14, 2017, 5:06 a.m.)


Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
Manji.


Changes
---

Support limit option of query.


Repository: aurora


Description
---

Expose task pruning endpoint in aurora_admin. Useful for scale testing in order 
to 'clean up' after a test run, but also useful in production if you have a bad 
actor inflating the size of your task index.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
a211483caf59755b1f9b286b58f183e00db6eee6 
  src/main/python/apache/aurora/admin/admin.py 
070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py 
1250ccd16f906eec08c159df3300d1b94b306d8e 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0cdd9829417bb3ef0215278a9458a3dd78a49a20 
  src/test/python/apache/aurora/admin/test_admin.py 
66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py 
fab97986dcecf7761e161f4074d5108a8c362e6e 

Diff: https://reviews.apache.org/r/56629/diff/


Testing
---

Ran various combinations of task pruning via aurora_admin in Vagrant. Works as 
expected. 

Also:

./pants test src/test/python/apache/aurora/admin::


Thanks,

David McLaughlin

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Santhosh Kumar Shanmugham]

> On Feb. 13, 2017, 8:48 p.m., Santhosh Kumar Shanmugham wrote:
> > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java,
> >  lines 1128-1129
> > 
> >
> > Depending on th query this can be a full-scan on the store, which can 
> > be inefficient. As mentioned in https://reviews.apache.org/r/56575/ this 
> > approach can cause GC pressure. We should use the `TaskStore.pruneHistory` 
> > implementation which might be more efficient.
> 
> David McLaughlin wrote:
> pruneHistory would search for candidates that are eligible for pruning 
> according to the pruning settings. This is a way to circumvent the grace 
> periods and other limits, and prune all terminal tasks that match the 
> provided query. Since this would be called explicitly and can be supplied 
> with a limit, the operator is able to manage the GC pressure themselves.

Since this command has the potential to hurt the Scheduler, should we make sure 
that atleast one of the filters is provided? Or force a minimum default `limit` 
to avoid operators from accidentally bringing down the scheduler?


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165440
---


On Feb. 13, 2017, 8:28 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 13, 2017, 8:28 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Santhosh Kumar Shanmugham]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165440
---




src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
(lines 1128 - 1129)


Depending on th query this can be a full-scan on the store, which can be 
inefficient. As mentioned in https://reviews.apache.org/r/56575/ this approach 
can cause GC pressure. We should use the `TaskStore.pruneHistory` 
implementation which might be more efficient.


- Santhosh Kumar Shanmugham


On Feb. 13, 2017, 8:28 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 13, 2017, 8:28 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

> On Feb. 14, 2017, 1:54 a.m., Reza Motamedi wrote:
> > LGTM!
> > 
> > But, should we test this in the test cluster? At least to check the effect 
> > of _limit_?
> 
> David McLaughlin wrote:
> I did test this in Vagrant. All the options work correctly.

Weird, verifying the StateManager change in Vagrant and now limit does not 
appear to work.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165425
---


On Feb. 14, 2017, 4:28 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 4:28 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/
---

(Updated Feb. 14, 2017, 4:28 a.m.)


Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
Manji.


Repository: aurora


Description
---

Expose task pruning endpoint in aurora_admin. Useful for scale testing in order 
to 'clean up' after a test run, but also useful in production if you have a bad 
actor inflating the size of your task index.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
a211483caf59755b1f9b286b58f183e00db6eee6 
  src/main/python/apache/aurora/admin/admin.py 
070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py 
1250ccd16f906eec08c159df3300d1b94b306d8e 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0cdd9829417bb3ef0215278a9458a3dd78a49a20 
  src/test/python/apache/aurora/admin/test_admin.py 
66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py 
fab97986dcecf7761e161f4074d5108a8c362e6e 

Diff: https://reviews.apache.org/r/56629/diff/


Testing
---

Ran various combinations of task pruning via aurora_admin in Vagrant. Works as 
expected. 

Also:

./pants test src/test/python/apache/aurora/admin::


Thanks,

David McLaughlin

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/
---

(Updated Feb. 14, 2017, 4:27 a.m.)


Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
Manji.


Changes
---

Use StateManager to delete tasks.


Repository: aurora


Description
---

Expose task pruning endpoint in aurora_admin. Useful for scale testing in order 
to 'clean up' after a test run, but also useful in production if you have a bad 
actor inflating the size of your task index.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
a211483caf59755b1f9b286b58f183e00db6eee6 
  src/main/python/apache/aurora/admin/admin.py 
070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py 
1250ccd16f906eec08c159df3300d1b94b306d8e 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0cdd9829417bb3ef0215278a9458a3dd78a49a20 
  src/test/python/apache/aurora/admin/test_admin.py 
66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py 
fab97986dcecf7761e161f4074d5108a8c362e6e 

Diff: https://reviews.apache.org/r/56629/diff/


Testing
---

Ran various combinations of task pruning via aurora_admin in Vagrant. Works as 
expected. 

Also:

./pants test src/test/python/apache/aurora/admin::


Thanks,

David McLaughlin

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

> On Feb. 14, 2017, 3:48 a.m., Mehrdad Nurolahzade wrote:
> > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java,
> >  line 1135
> > 
> >
> > We are skipping task delete through `StateManager` which publishes 
> > `PubsubEvent.TasksDeleted` to subscribers like `TaskGroups` and 
> > `JobUpdateController`.
> > 
> > In the case of `JobUpdateController` the dependency does not seem to 
> > matter. Did you verify ignoring the `TaskGroups` dependency is safe?

Looks like I need to go through StateManager for TaskVars. The other 
subscribers only deal with PENDING tasks. 

It's a shame because the pubsub path is really ineffecient.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165434
---


On Feb. 14, 2017, 4:12 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 4:12 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
>   src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> fab97986dcecf7761e161f4074d5108a8c362e6e 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/
---

(Updated Feb. 14, 2017, 4:12 a.m.)


Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
Manji.


Changes
---

Fix Python tests.


Repository: aurora


Description
---

Expose task pruning endpoint in aurora_admin. Useful for scale testing in order 
to 'clean up' after a test run, but also useful in production if you have a bad 
actor inflating the size of your task index.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
a211483caf59755b1f9b286b58f183e00db6eee6 
  src/main/python/apache/aurora/admin/admin.py 
070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py 
1250ccd16f906eec08c159df3300d1b94b306d8e 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0cdd9829417bb3ef0215278a9458a3dd78a49a20 
  src/test/python/apache/aurora/admin/test_admin.py 
66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
  src/test/python/apache/aurora/client/api/test_scheduler_client.py 
fab97986dcecf7761e161f4074d5108a8c362e6e 

Diff: https://reviews.apache.org/r/56629/diff/


Testing
---

Ran various combinations of task pruning via aurora_admin in Vagrant. Works as 
expected. 

Also:

./pants test src/test/python/apache/aurora/admin::


Thanks,

David McLaughlin

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Mehrdad Nurolahzade]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165434
---




src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
(line 1135)


We are skipping task delete through `StateManager` which publishes 
`PubsubEvent.TasksDeleted` to subscribers like `TaskGroups` and 
`JobUpdateController`.

In the case of `JobUpdateController` the dependency does not seem to 
matter. Did you verify ignoring the `TaskGroups` dependency is safe?


- Mehrdad Nurolahzade


On Feb. 13, 2017, 6:01 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 13, 2017, 6:01 p.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165428
---



Master (40d91fe) is red with this patch.
  ./build-support/jenkins/build.sh

 
src/test/python/apache/thermos/common/test_task_planner.py::test_task_process_cannot_depend_upon_daemon
 PASSED
 
src/test/python/apache/thermos/common/test_task_planner.py::test_task_non_ephemeral_process_cannot_depend_on_ephemeral_process
 PASSED
 
src/test/python/apache/thermos/common/test_task_planner.py::test_task_failed_predecessor_does_not_make_process_runnable
 PASSED
 
src/test/python/apache/thermos/common/test_task_planner.py::test_task_daemon_duration
 PASSED
 
src/test/python/apache/thermos/common/test_task_planner.py::test_task_waits 
PASSED
 
src/test/python/apache/thermos/common/test_task_planner.py::test_task_fails 
PASSED
 
src/test/python/apache/thermos/common/test_task_planner.py::test_task_lost 
PASSED
 
src/test/python/apache/thermos/common/test_task_planner.py::test_task_filters 
PASSED
 
src/test/python/apache/thermos/common/test_task_planner.py::test_task_max_runs 
PASSED
 
src/test/python/apache/thermos/common/test_pathspec.py::test_legacy_task_roots 
PASSED
 
src/test/python/apache/thermos/common/test_pathspec.py::test_legacy_log_dirs 
PASSED
 
src/test/python/apache/thermos/common/test_pathspec.py::test_exception_on_none_keys
 PASSED
 
src/test/python/apache/thermos/common/test_planner.py::test_planner_empty 
PASSED
 
src/test/python/apache/thermos/common/test_planner.py::test_planner_unordered 
PASSED
 
src/test/python/apache/thermos/common/test_planner.py::test_planner_ordered 
PASSED
 
src/test/python/apache/thermos/common/test_planner.py::test_planner_mixed 
PASSED
 
src/test/python/apache/thermos/common/test_planner.py::test_planner_unsatisfiables
 PASSED
 
  FAILURES 
 _ test_coverage __
 
 def test_coverage():
   """Make sure a new thrift RPC doesn't get added 
without minimal test coverage."""
   for name, klass in 
inspect.getmembers(AuroraAdmin) + 
inspect.getmembers(AuroraSchedulerManager):
 if name.endswith('_args'):
   rpc_name = name[:-len('_args')]
 > assert 
hasattr(TestSchedulerProxyAdminInjection, 'test_%s' % rpc_name), (
   'No test defined for RPC %s' % 
rpc_name)
 E AssertionError: No test defined for RPC 
pruneTasks
 E assert 
hasattr(TestSchedulerProxyAdminInjection, ('test_%s' % 'pruneTasks'))
 
 
src/test/python/apache/aurora/client/api/test_scheduler_client.py:65: 
AssertionError
  generated xml file: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/aaf4d108c31293299a0839bdc404a91802f80937.xml
 
  1 failed, 777 passed, 6 skipped, 1 warnings in 
326.98 seconds 
 
FAILURE


02:23:08 06:04   [complete]
   FAILURE


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 14, 2017, 2:01 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 2:01 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

> On Feb. 14, 2017, 1:54 a.m., Reza Motamedi wrote:
> > LGTM!
> > 
> > But, should we test this in the test cluster? At least to check the effect 
> > of _limit_?

I did test this in Vagrant. All the options work correctly.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165425
---


On Feb. 14, 2017, 2:01 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 2:01 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/
---

(Updated Feb. 14, 2017, 2:01 a.m.)


Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
Manji.


Changes
---

Checkstyle and pmd fixes.


Repository: aurora


Description
---

Expose task pruning endpoint in aurora_admin. Useful for scale testing in order 
to 'clean up' after a test run, but also useful in production if you have a bad 
actor inflating the size of your task index.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
a211483caf59755b1f9b286b58f183e00db6eee6 
  src/main/python/apache/aurora/admin/admin.py 
070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py 
1250ccd16f906eec08c159df3300d1b94b306d8e 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0cdd9829417bb3ef0215278a9458a3dd78a49a20 
  src/test/python/apache/aurora/admin/test_admin.py 
66abade378d3974dbf031a7bbed9cd6fc4f22d5f 

Diff: https://reviews.apache.org/r/56629/diff/


Testing
---

Ran various combinations of task pruning via aurora_admin in Vagrant. Works as 
expected. 

Also:

./pants test src/test/python/apache/aurora/admin::


Thanks,

David McLaughlin

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Reza Motamedi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165425
---



LGTM!

But, should we test this in the test cluster? At least to check the effect of 
_limit_?

- Reza Motamedi


On Feb. 14, 2017, 1:36 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 1:36 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/aop/AnnotatedAuroraAdmin.java
>  d63b11d36748de796e96685622a3290bb94e7106 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Re: Review Request 56629: Expose task pruning endpoint in aurora_admin

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/#review165421
---



Master (40d91fe) is red with this patch.
  ./build-support/jenkins/build.sh

:commons:generateThriftResources
:commons:processResources
:commons:classes
:commons:jar
:compileJava/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/storage/log/WriteAheadStorage.java:74:
 Note: Wrote forwarder 
org.apache.aurora.scheduler.storage.log.WriteAheadStorageForwarder
@Forward({
^
Note: Writing 
file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/org/apache/aurora/common/args/apt/cmdline.arg.info.txt.2
Note: Writing 
file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/META-INF/compiler/resource-mappings/org.apache.aurora.common.args.apt.CmdLineProcessor

:generateBuildProperties
:processResources
:classes
:jar
:startScripts
:distTar
:distZip
:assemble
:compileJmhJavaNote: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java
 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:processJmhResources UP-TO-DATE
:jmhClasses
:checkstyleJmh
:jsHint
:checkstyleMain[ant:checkstyle] [ERROR] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/thrift/aop/AnnotatedAuroraAdmin.java:27:8:
 Unused import - org.apache.aurora.gen.TaskQuery. [UnusedImports]
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':checkstyleMain'.
> Checkstyle rule violations were found. See the report at: 
> file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/main.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 1 mins 22.991 secs


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 14, 2017, 1:36 a.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56629/
> ---
> 
> (Updated Feb. 14, 2017, 1:36 a.m.)
> 
> 
> Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
> Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose task pruning endpoint in aurora_admin. Useful for scale testing in 
> order to 'clean up' after a test run, but also useful in production if you 
> have a bad actor inflating the size of your task index.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  a211483caf59755b1f9b286b58f183e00db6eee6 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/aop/AnnotatedAuroraAdmin.java
>  d63b11d36748de796e96685622a3290bb94e7106 
>   src/main/python/apache/aurora/admin/admin.py 
> 070c348d2ca5db1edecf832efd9aa5481bddaa4b 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 1250ccd16f906eec08c159df3300d1b94b306d8e 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0cdd9829417bb3ef0215278a9458a3dd78a49a20 
>   src/test/python/apache/aurora/admin/test_admin.py 
> 66abade378d3974dbf031a7bbed9cd6fc4f22d5f 
> 
> Diff: https://reviews.apache.org/r/56629/diff/
> 
> 
> Testing
> ---
> 
> Ran various combinations of task pruning via aurora_admin in Vagrant. Works 
> as expected. 
> 
> Also:
> 
> ./pants test src/test/python/apache/aurora/admin::
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

Review Request 56629: Expose task pruning endpoint in aurora_admin

[David McLaughlin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56629/
---

Review request for Aurora, Santhosh Kumar Shanmugham, Stephan Erb, and Zameer 
Manji.


Repository: aurora


Description
---

Expose task pruning endpoint in aurora_admin. Useful for scale testing in order 
to 'clean up' after a test run, but also useful in production if you have a bad 
actor inflating the size of your task index.


Diffs
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
6205c2ed1e2d5898a322de5bc2bcba6b2c0cd4b1 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
a211483caf59755b1f9b286b58f183e00db6eee6 
  
src/main/java/org/apache/aurora/scheduler/thrift/aop/AnnotatedAuroraAdmin.java 
d63b11d36748de796e96685622a3290bb94e7106 
  src/main/python/apache/aurora/admin/admin.py 
070c348d2ca5db1edecf832efd9aa5481bddaa4b 
  src/main/python/apache/aurora/client/api/__init__.py 
1250ccd16f906eec08c159df3300d1b94b306d8e 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0cdd9829417bb3ef0215278a9458a3dd78a49a20 
  src/test/python/apache/aurora/admin/test_admin.py 
66abade378d3974dbf031a7bbed9cd6fc4f22d5f 

Diff: https://reviews.apache.org/r/56629/diff/


Testing
---

Ran various combinations of task pruning via aurora_admin in Vagrant. Works as 
expected. 

Also:

./pants test src/test/python/apache/aurora/admin::


Thanks,

David McLaughlin