subject:"Review Request 67696\: Enable SLA\-aware updates"

Re: Review Request 67696: Enable SLA-aware updates

2018-07-18 Thread Stephan Erb


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206212
---


Ship it!




Ship It!

- Stephan Erb


On July 18, 2018, 12:15 a.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 18, 2018, 12:15 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  6a28bc274acdd6d3ac239166771ef2d45648d60f 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 
> 3992aa77fc305adc390a4aaeb1d3939d6241ddbd 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> abee0951ca998894b29ee32c5362ef30da6421c7 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  aa1cb2b287642e87d787e160e04a17ad0e4690d9 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/U

Re: Review Request 67696: Enable SLA-aware updates

2018-07-18 Thread Stephan Erb



> On July 17, 2018, 10:52 p.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
> > Lines 413-417 (patched)
> > 
> >
> > I am a bit confused about this. Can you please elaborate a bit more 
> > when/how you observed this problem?
> > 
> > I am mostly concerned about race conditions here. 
> > 
> > For example, say I trigger a job update from config A to B. Shortly 
> > before I update an instance the instance fails or transitions to LOST. 
> > Concurrently we might end up hopping into the `instanceChanged` method 
> > here, but bail due to the new guard without ever handling the switch from 
> > config A to B.
> > 
> > Could be that I am missing something entirely here. Would be great if 
> > you could elaborate.
> 
> Jordan Ly wrote:
> Good question.
> 
> When we determine what action we should perform on an instance, we use 
> the config/state we get from these task events. If we are using stale 
> information, we may be producing inaccurate actions to perform. This is a 
> rare scenario but it relies on tons of async events happening at the same 
> time so there is a large delay in event processing. The scenario that I 
> encountered that lead me to introduce this guard was the following:
> 
> 1. Task "ABC-old-1" is being scheduled and goes from ASSIGNED -> RUNNING 
> (RUNNING is now on queue to be evaluated as an update)
> 2. A different previous task finishes updating so instance 1 is added to 
> the working group to be updated. The initial evaluation causes "ABC-old-1" to 
> be moved to KILLING.
> 3. "ABC-old-1" is KILLED and "ABC-new-1" is created and running.
> 4. RUNNING from the async queue is finally evaluated so we see that this 
> instance number needs to be updated (since it is "ABC-old-1") so we kill the 
> task at 1 (which is now "ABC-new-1").
> 5. Before this patch, this would mean the task needs to be killed and 
> added again. Now, since there is an async aspect to killing, this could mean 
> an instance # is permanently killed post-update.
> 
> I am ensuring this condition is met within the evaluation of state 
> changes. From `StateEvaluator`:
> 
> ```
>* It is the responsibility of the caller to ensure that the {@code 
> actualState} is the latest
>* value.  Note: the caller should avoid calling this when a terminal 
> task is moving to another
>* terminal state.  It should also suppress deletion events for tasks 
> that have been replaced by
>* an active task.
> ```
> 
> In the scenario you provide, the condition will work as intended and the 
> update should still move to completion successfully. A write lock governs all 
> actions dealing with updates so we do not have race conditions.
> 
> 1. Instance A needs to be updated.
> 2. Instance A fails before being evaluated for an update.
> 3. Since A failed, it will be rescheduled into A'.
>   - If A is evaluated for the update before being rescheduled (e.g. we 
> evaluate the state change into PENDING or FAILED before rescheduling), then 
> we kill that task and update the configuration
>   - If A is rescheduled before being evaluated for the update, then we 
> will kill A' and update the configuration
> 
> Lots of moving parts for updates so hopefully what I said makes sense :/

Thanks for the detailed writeup. I can follow your argument and it makes sense 
to me. However, I also have to admit that I still haven't wrapped my head 
around all the details of the update procedure 100%. I trust your judgment and 
analysis though :)


- Stephan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206149
---


On July 18, 2018, 12:15 a.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 18, 2018, 12:15 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two op

Re: Review Request 67696: Enable SLA-aware updates

2018-07-17 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206188
---


Ship it!




Master (efe8656) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 17, 2018, 10:15 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 17, 2018, 10:15 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  6a28bc274acdd6d3ac239166771ef2d45648d60f 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 
> 3992aa77fc305adc390a4aaeb1d3939d6241ddbd 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> abee0951ca998894b29ee32c5362ef30da6421c7 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  aa1cb2b287642e87d787e160e04a17ad0e4690d9 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a679

Re: Review Request 67696: Enable SLA-aware updates

2018-07-17 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated July 17, 2018, 10:15 p.m.)


Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
Santhosh Kumar Shanmugham, and Stephan Erb.


Changes
---

Address styling


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
  docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
  docs/reference/scheduler-configuration.md 
805e516689be019101f7c220c89fd9c391bb93b3 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
6a28bc274acdd6d3ac239166771ef2d45648d60f 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 
3992aa77fc305adc390a4aaeb1d3939d6241ddbd 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
abee0951ca998894b29ee32c5362ef30da6421c7 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 aa1cb2b287642e87d787e160e04a17ad0e4690d9 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
2baba2aa55865ec298a4c9e5af3952b56cb9a910 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/11/

Changes: https://reviews.apache.org/r/67696/diff/10-11/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

Re: Review Request 67696: Enable SLA-aware updates

2018-07-17 Thread Jordan Ly

> On July 17, 2018, 8:52 p.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
> > Lines 413-417 (patched)
> > 
> >
> > I am a bit confused about this. Can you please elaborate a bit more 
> > when/how you observed this problem?
> > 
> > I am mostly concerned about race conditions here. 
> > 
> > For example, say I trigger a job update from config A to B. Shortly 
> > before I update an instance the instance fails or transitions to LOST. 
> > Concurrently we might end up hopping into the `instanceChanged` method 
> > here, but bail due to the new guard without ever handling the switch from 
> > config A to B.
> > 
> > Could be that I am missing something entirely here. Would be great if 
> > you could elaborate.

Good question.

When we determine what action we should perform on an instance, we use the 
config/state we get from these task events. If we are using stale information, 
we may be producing inaccurate actions to perform. This is a rare scenario but 
it relies on tons of async events happening at the same time so there is a 
large delay in event processing. The scenario that I encountered that lead me 
to introduce this guard was the following:

1. Task "ABC-old-1" is being scheduled and goes from ASSIGNED -> RUNNING 
(RUNNING is now on queue to be evaluated as an update)
2. A different previous task finishes updating so instance 1 is added to the 
working group to be updated. The initial evaluation causes "ABC-old-1" to be 
moved to KILLING.
3. "ABC-old-1" is KILLED and "ABC-new-1" is created and running.
4. RUNNING from the async queue is finally evaluated so we see that this 
instance number needs to be updated (since it is "ABC-old-1") so we kill the 
task at 1 (which is now "ABC-new-1").
5. Before this patch, this would mean the task needs to be killed and added 
again. Now, since there is an async aspect to killing, this could mean an 
instance # is permanently killed post-update.

I am ensuring this condition is met within the evaluation of state changes. 
From `StateEvaluator`:

```
   * It is the responsibility of the caller to ensure that the {@code 
actualState} is the latest
   * value.  Note: the caller should avoid calling this when a terminal task is 
moving to another
   * terminal state.  It should also suppress deletion events for tasks that 
have been replaced by
   * an active task.
```

In the scenario you provide, the condition will work as intended and the update 
should still move to completion successfully. A write lock governs all actions 
dealing with updates so we do not have race conditions.

1. Instance A needs to be updated.
2. Instance A fails before being evaluated for an update.
3. Since A failed, it will be rescheduled into A'.
  - If A is evaluated for the update before being rescheduled (e.g. we evaluate 
the state change into PENDING or FAILED before rescheduling), then we kill that 
task and update the configuration
  - If A is rescheduled before being evaluated for the update, then we will 
kill A' and update the configuration

Lots of moving parts for updates so hopefully what I said makes sense :/

- Jordan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206149
---

On July 17, 2018, 6:21 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 17, 2018, 6:21 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6

Re: Review Request 67696: Enable SLA-aware updates

2018-07-17 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206173
---


Ship it!




Master (efe8656) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 17, 2018, 6:21 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 17, 2018, 6:21 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  6a28bc274acdd6d3ac239166771ef2d45648d60f 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 
> 3992aa77fc305adc390a4aaeb1d3939d6241ddbd 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> abee0951ca998894b29ee32c5362ef30da6421c7 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  aa1cb2b287642e87d787e160e04a17ad0e4690d9 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989

Re: Review Request 67696: Enable SLA-aware updates

2018-07-17 Thread Stephan Erb


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206149
---



Looks good, thanks!

I have just one question that is currently confusing me and a few smaller 
nitpicks. (I would even be okay if you punt on the nitpicks since the patch has 
already seen quite a few iterations)


src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java
Line 145 (original), 145 (patched)


Nitpick: Please switch to the logger-builtin formatting. This reduces a bit 
of overhead for messages that never end up being logged.



src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java
Lines 178-184 (patched)


Nitpick: We normally give the parameters a consistent indentation 
indepentent of the length of the method/function name. This leads to code that 
is a bit less jagged. For an example, see `getReevaluationDelay` in this file. 

The same remark also applies to a few other places, e.g. 
`SlaKillController`.

Reference 
https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/styleguide.md#indent-style
 and 
https://github.com/twitter/commons/blob/master/src/python/twitter/common/styleguide.md#best-practices



src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
Lines 413-417 (patched)


I am a bit confused about this. Can you please elaborate a bit more 
when/how you observed this problem?

I am mostly concerned about race conditions here. 

For example, say I trigger a job update from config A to B. Shortly before 
I update an instance the instance fails or transitions to LOST. Concurrently we 
might end up hopping into the `instanceChanged` method here, but bail due to 
the new guard without ever handling the switch from config A to B.

Could be that I am missing something entirely here. Would be great if you 
could elaborate.


- Stephan Erb


On July 17, 2018, 8:21 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 17, 2018, 8:21 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  6a28bc274acdd6d3ac239166771ef2d45648d60f 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 
> 3992aa77fc305adc390a4aaeb1d3939d6241ddbd 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src

Re: Review Request 67696: Enable SLA-aware updates

2018-07-17 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated July 17, 2018, 6:21 p.m.)


Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
Santhosh Kumar Shanmugham, and Stephan Erb.


Changes
---

Throw exception on bad state, fix typo


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
  docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
  docs/reference/scheduler-configuration.md 
805e516689be019101f7c220c89fd9c391bb93b3 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
6a28bc274acdd6d3ac239166771ef2d45648d60f 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 
3992aa77fc305adc390a4aaeb1d3939d6241ddbd 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
abee0951ca998894b29ee32c5362ef30da6421c7 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 aa1cb2b287642e87d787e160e04a17ad0e4690d9 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
2baba2aa55865ec298a4c9e5af3952b56cb9a910 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/10/

Changes: https://reviews.apache.org/r/67696/diff/9-10/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

Re: Review Request 67696: Enable SLA-aware updates

2018-07-17 Thread Santhosh Kumar Shanmugham


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206137
---


Fix it, then Ship it!




LGTM. Minor comments.


src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java
Lines 212-216 (patched)


It will be better if we hard-fail here, since this is an unexpected 
behavior. Allowing it to be silently ignored might hide issues.



src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
Lines 1080 (patched)


s/policy/sla-policy/


- Santhosh Kumar Shanmugham


On July 16, 2018, 4 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 16, 2018, 4 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  6a28bc274acdd6d3ac239166771ef2d45648d60f 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 
> 3992aa77fc305adc390a4aaeb1d3939d6241ddbd 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> abee0951ca998894b29ee32c5362ef30da6421c7 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  aa1cb2b287642e87d787e160e04a17ad0e4690d9 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/clie

Re: Review Request 67696: Enable SLA-aware updates

2018-07-16 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206143
---


Ship it!




Master (a6d9288) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 16, 2018, 11 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 16, 2018, 11 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  6a28bc274acdd6d3ac239166771ef2d45648d60f 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 
> 3992aa77fc305adc390a4aaeb1d3939d6241ddbd 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> abee0951ca998894b29ee32c5362ef30da6421c7 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  aa1cb2b287642e87d787e160e04a17ad0e4690d9 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a

Re: Review Request 67696: Enable SLA-aware updates

2018-07-16 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated July 16, 2018, 11 p.m.)


Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
Santhosh Kumar Shanmugham, and Stephan Erb.


Changes
---

Fixed bug where starting an `slaAware` update on a task with no `slaPolicy` 
would cause the scheduler to die.

Tested on vagrant + added unit tests.


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
  docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
  docs/reference/scheduler-configuration.md 
805e516689be019101f7c220c89fd9c391bb93b3 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
6a28bc274acdd6d3ac239166771ef2d45648d60f 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java 
3992aa77fc305adc390a4aaeb1d3939d6241ddbd 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
abee0951ca998894b29ee32c5362ef30da6421c7 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 aa1cb2b287642e87d787e160e04a17ad0e4690d9 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
2baba2aa55865ec298a4c9e5af3952b56cb9a910 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/9/

Changes: https://reviews.apache.org/r/67696/diff/8-9/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affe

Re: Review Request 67696: Enable SLA-aware updates

2018-07-12 Thread Daniel Knightly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206034
---


Ship it!




Ship It!

- Daniel Knightly


On July 12, 2018, 9:10 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 12, 2018, 9:10 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/8/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update events occuring 
> concurrently. Scheduler stability did not seem to be affected.
> 
> 
> Thanks,
> 
> Jordan Ly
> 
>

Re: Review Request 67696: Enable SLA-aware updates

2018-07-12 Thread Daniel Knightly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206028
---




src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java
Lines 182 (patched)


Nit:
slf4j lets you do LOG.info("Killing {} while {}", instance, status);

Though I see the style is string concat


- Daniel Knightly


On July 12, 2018, 9:10 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 12, 2018, 9:10 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/8/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware

Re: Review Request 67696: Enable SLA-aware updates

2018-07-12 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206026
---


Ship it!




Master (a6d9288) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 12, 2018, 9:10 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 12, 2018, 9:10 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/8/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update events occuring 
> concurrently. Scheduler stability did not seem to be affected.
> 
> 
>

Re: Review Request 67696: Enable SLA-aware updates

2018-07-12 Thread Santhosh Kumar Shanmugham


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review206024
---


Ship it!




Ship It!

- Santhosh Kumar Shanmugham


On July 12, 2018, 2:10 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 12, 2018, 2:10 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/8/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update events occuring 
> concurrently. Scheduler stability did not seem to be affected.
> 
> 
> Thanks,
> 
> Jordan Ly
> 
>

Re: Review Request 67696: Enable SLA-aware updates

2018-07-12 Thread Jordan Ly



> On July 9, 2018, 9:04 p.m., Santhosh Kumar Shanmugham wrote:
> > src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java
> > Lines 180 (patched)
> > 
> >
> > Drop `_` here and everywhere. `SLA_CHECKING` sounds like a valid event 
> > that is definied in code.
> 
> Jordan Ly wrote:
> Can you elaborate on this? Did you mean drop `_MESSAGE`? I added 
> `_MESSAGE` because it is not a true "event" in the sense that it piggybacks 
> on `INSTANCE_UPDATING` or `INSTANCE_ROLLING_BACK` with a message showing 
> progress.
> 
> Santhosh Kumar Shanmugham wrote:
> Can use `SLA_CHECKING_MESSAGE` or simply `sla checking event is added`. I 
> started searching for an event type of `SLA_CHECKING` after reading this 
> comment, which is a little confusing.

Done.


- Jordan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205866
---


On July 10, 2018, 2:22 a.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 10, 2018, 2:22 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobIn

Re: Review Request 67696: Enable SLA-aware updates

2018-07-12 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated July 12, 2018, 9:10 p.m.)


Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
Santhosh Kumar Shanmugham, and Stephan Erb.


Changes
---

Clarify comments in test


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
  docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
  docs/reference/scheduler-configuration.md 
805e516689be019101f7c220c89fd9c391bb93b3 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
2baba2aa55865ec298a4c9e5af3952b56cb9a910 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/8/

Changes: https://reviews.apache.org/r/67696/diff/7-8/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

Re: Review Request 67696: Enable SLA-aware updates

2018-07-12 Thread Santhosh Kumar Shanmugham



> On July 9, 2018, 2:04 p.m., Santhosh Kumar Shanmugham wrote:
> > src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java
> > Lines 140 (patched)
> > 
> >
> > Repeating my comment from internal review - 
> > 
> > Technically this is can be any kind of work. Should we call this class 
> > `SlaAwareInstanceUpdater`? Since this has more to do with actually update 
> > logic than kill itself. And perhaps the method can become 
> > `startSlaAwareUpdate`?
> > 
> > The job of this class is around checking safety and scheduling the 
> > supplied work to be evaulated. Naming the class based on it usage rather 
> > than its function seems weird to me.
> 
> Jordan Ly wrote:
> This would be ideal but I think there are some small parts of the logic 
> that changing the name would be misleading towards.
> 
> For example:
> 
> ```
> storeProvider
> .getTaskStore()
> .fetchTask(taskId)
> .filter(task -> isKillable(task.getStatus()))
> .ifPresent(task -> {
>   incrementJobStatCounter(killAttemptsByJob, 
> SLA_KILL_ATTEMPT, instance.getJobKey());
>   slaManager.checkSlaThenAct(
>   task,
>   instructions.getDesiredState().getTask().getSlaPolicy(),
>   slaStoreProvider -> performKill(
>   slaStoreProvider,
>   instance,
>   key,
>   status,
>   killCommand),
>   ImmutableMap.of(),
>   // If the task is not assigned, force the update since 
> it does not affect the
>   // SLA. For example, if a task is THROTTLED or PENDING, 
> we probably don't care
>   // if the update replaces it with a new instance.
>   !SLAVE_ASSIGNED_STATES.contains(task.getStatus()));
> ```
> 
> Within the SLA aware code, we have kill-specific checks in order to 
> determine if we should continue with the action (or if it should be a NOOP), 
> or if we should force the kill regardless (the task would not affect SLA). 
> Changing the name would require a refactoring for a more complex interface 
> but I am not sure if we would ever use it for anything else.
> 
> I am leaning towards keeping the current name and functionality right now 
> until another sla-aware update use case comes in and we have a better idea on 
> how we can refactor the current abstraction (from kill to generic, which I 
> don't think should be too difficult but I'm sure I'll regret those words...)

These checks can be moved into the `killCommand`. I think the abstraction is 
not perfect as it is today.


> On July 9, 2018, 2:04 p.m., Santhosh Kumar Shanmugham wrote:
> > src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java
> > Lines 180 (patched)
> > 
> >
> > Drop `_` here and everywhere. `SLA_CHECKING` sounds like a valid event 
> > that is definied in code.
> 
> Jordan Ly wrote:
> Can you elaborate on this? Did you mean drop `_MESSAGE`? I added 
> `_MESSAGE` because it is not a true "event" in the sense that it piggybacks 
> on `INSTANCE_UPDATING` or `INSTANCE_ROLLING_BACK` with a message showing 
> progress.

Can use `SLA_CHECKING_MESSAGE` or simply `sla checking event is added`. I 
started searching for an event type of `SLA_CHECKING` after reading this 
comment, which is a little confusing.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205866
---


On July 9, 2018, 7:22 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 9, 2018, 7:22 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `

Re: Review Request 67696: Enable SLA-aware updates

2018-07-11 Thread David McLaughlin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205973
---


Ship it!




Ship It!

- David McLaughlin


On July 10, 2018, 2:22 a.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 10, 2018, 2:22 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/7/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update events occuring 
> concurrently. Scheduler stability did not seem to be affected.
> 
> 
> Thanks,
> 
> Jordan Ly
> 
>

Re: Review Request 67696: Enable SLA-aware updates

2018-07-10 Thread Reza Motamedi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205843
---




docs/reference/configuration.md
Lines 379 (patched)


`Default: False`?



docs/reference/configuration.md
Lines 386 (patched)


s/the update/the updater


- Reza Motamedi


On July 10, 2018, 2:22 a.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 10, 2018, 2:22 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/7/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update eve

Re: Review Request 67696: Enable SLA-aware updates

2018-07-09 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205886
---


Ship it!




Master (a6d9288) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 10, 2018, 2:22 a.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 10, 2018, 2:22 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/7/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update events occuring 
> concurrently. Scheduler stability did not seem to be affected.
> 
> 
>

Re: Review Request 67696: Enable SLA-aware updates

2018-07-09 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated July 10, 2018, 2:22 a.m.)


Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
Santhosh Kumar Shanmugham, and Stephan Erb.


Changes
---

Fix broken test


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
  docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
  docs/reference/scheduler-configuration.md 
805e516689be019101f7c220c89fd9c391bb93b3 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
2baba2aa55865ec298a4c9e5af3952b56cb9a910 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/7/

Changes: https://reviews.apache.org/r/67696/diff/6-7/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

Re: Review Request 67696: Enable SLA-aware updates

2018-07-09 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205880
---



Master (a6d9288) is red with this patch.
  ./build-support/jenkins/build.sh

Pass 2: Analyzing classes (329 / 348) - 94% complete
Pass 2: Analyzing classes (330 / 348) - 94% complete
Pass 2: Analyzing classes (331 / 348) - 95% complete
Pass 2: Analyzing classes (332 / 348) - 95% complete
Pass 2: Analyzing classes (333 / 348) - 95% complete
Pass 2: Analyzing classes (334 / 348) - 95% complete
Pass 2: Analyzing classes (335 / 348) - 96% complete
Pass 2: Analyzing classes (336 / 348) - 96% complete
Pass 2: Analyzing classes (337 / 348) - 96% complete
Pass 2: Analyzing classes (338 / 348) - 97% complete
Pass 2: Analyzing classes (339 / 348) - 97% complete
Pass 2: Analyzing classes (340 / 348) - 97% complete
Pass 2: Analyzing classes (341 / 348) - 97% complete
Pass 2: Analyzing classes (342 / 348) - 98% complete
Pass 2: Analyzing classes (343 / 348) - 98% complete
Pass 2: Analyzing classes (344 / 348) - 98% complete
Pass 2: Analyzing classes (345 / 348) - 99% complete
Pass 2: Analyzing classes (346 / 348) - 99% complete
Pass 2: Analyzing classes (347 / 348) - 99% complete
Pass 2: Analyzing classes (348 / 348) - 100% complete
Done with analysis
:testI0709 23:38:37.507 [ShutdownHook, SchedulerMain] Stopping scheduler 
services. 
 FAILED
:jacocoTestReport
Coverage report generated: 
file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html
:jacocoTestCoverageVerification

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> Process 'Gradle Test Executor 20' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 7m 28s
45 actionable tasks: 36 executed, 9 up-to-date


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 9, 2018, 4:12 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 9, 2018, 4:12 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc18

Re: Review Request 67696: Enable SLA-aware updates

2018-07-09 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205879
---



@ReviewBot retry

- Jordan Ly


On July 9, 2018, 11:12 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 9, 2018, 11:12 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/6/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update events occuring 
> concurrently. Scheduler stability did not seem to be affected.
> 
> 
> Thanks,
> 
> Jordan Ly
> 
>

Re: Review Request 67696: Enable SLA-aware updates

2018-07-09 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205878
---



Master (a6d9288) is red with this patch.
  ./build-support/jenkins/build.sh

 50% .  23930 KB
 52% .. 24890 KB
 54% ...25850 KB
 56%    26800 KB
 58% .  27760 KB
 60% .. 28720 KB
 62% ...29670 KB
 64%    30630 KB
 66% .  31590 KB
 68% .. 32550 KB
 70% ...33500 KB
 72%    34460 KB
 74% .  35420 KB
 76% .. 36370 KB
 78% ...37330 KB
 80%    38290 KB
 82% .  39240 KB
 84% .. 40200 KB
 86% ...41160 KB
 88%    42120 KB
 90% .  43070 KB
 92% .. 44030 KB
 94% ...44990 KB
 96%    45940 KB
 98% .  46900 KB
100% .. 47853 KB 5.210s




:api:classesThrift
:api:checkPython
:api:generateThriftEntitiesJava
:api:classesThriftEntities
:api:compileJava NO-SOURCE
:api:generateThriftResources
:api:processResources NO-SOURCE
:api:classes
:api:jar
:commons:compileJavaNote: /

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 9, 2018, 11:12 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 9, 2018, 11:12 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>

Re: Review Request 67696: Enable SLA-aware updates

2018-07-09 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated July 9, 2018, 11:12 p.m.)


Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
Santhosh Kumar Shanmugham, and Stephan Erb.


Changes
---

Respond to some of Santhosh's comments


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
  docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
  docs/reference/scheduler-configuration.md 
805e516689be019101f7c220c89fd9c391bb93b3 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
2baba2aa55865ec298a4c9e5af3952b56cb9a910 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/6/

Changes: https://reviews.apache.org/r/67696/diff/5-6/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

Re: Review Request 67696: Enable SLA-aware updates

2018-07-09 Thread Jordan Ly

> On July 9, 2018, 9:04 p.m., Santhosh Kumar Shanmugham wrote:
> > end-to-end test for this feature?

I think that the integration test should be sufficient for this feature 
(similar to coordinated updates which has fairly comprehensive integration 
tests but no e2e tests).

> On July 9, 2018, 9:04 p.m., Santhosh Kumar Shanmugham wrote:
> > docs/reference/scheduler-configuration.md
> > Lines 162 (patched)
> > 
> >
> > s/The number of/The maximum number of/
> > 
> > This is only used for SLA-aware updates, should we name it so that they 
> > are grouped together with the other settings?

Good point, done.

> On July 9, 2018, 9:04 p.m., Santhosh Kumar Shanmugham wrote:
> > docs/reference/scheduler-configuration.md
> > Lines 227-230 (patched)
> > 
> >
> > `sla_aware_kill_retry_min_delay` will be more readable. Same for the 
> > max delay.

Done.

> On July 9, 2018, 9:04 p.m., Santhosh Kumar Shanmugham wrote:
> > src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java
> > Lines 56 (patched)
> > 
> >
> > s/SLA passes/SLA passes or update is cancelled/

Added "or the update changes state." which is clarified in the next sentence.

> On July 9, 2018, 9:04 p.m., Santhosh Kumar Shanmugham wrote:
> > src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java
> > Lines 140 (patched)
> > 
> >
> > Repeating my comment from internal review - 
> > 
> > Technically this is can be any kind of work. Should we call this class 
> > `SlaAwareInstanceUpdater`? Since this has more to do with actually update 
> > logic than kill itself. And perhaps the method can become 
> > `startSlaAwareUpdate`?
> > 
> > The job of this class is around checking safety and scheduling the 
> > supplied work to be evaulated. Naming the class based on it usage rather 
> > than its function seems weird to me.

This would be ideal but I think there are some small parts of the logic that 
changing the name would be misleading towards.

For example:

```
storeProvider
.getTaskStore()
.fetchTask(taskId)
.filter(task -> isKillable(task.getStatus()))
.ifPresent(task -> {
  incrementJobStatCounter(killAttemptsByJob, SLA_KILL_ATTEMPT, 
instance.getJobKey());
  slaManager.checkSlaThenAct(
  task,
  instructions.getDesiredState().getTask().getSlaPolicy(),
  slaStoreProvider -> performKill(
  slaStoreProvider,
  instance,
  key,
  status,
  killCommand),
  ImmutableMap.of(),
  // If the task is not assigned, force the update since it 
does not affect the
  // SLA. For example, if a task is THROTTLED or PENDING, we 
probably don't care
  // if the update replaces it with a new instance.
  !SLAVE_ASSIGNED_STATES.contains(task.getStatus()));
```

Within the SLA aware code, we have kill-specific checks in order to determine 
if we should continue with the action (or if it should be a NOOP), or if we 
should force the kill regardless (the task would not affect SLA). Changing the 
name would require a refactoring for a more complex interface but I am not sure 
if we would ever use it for anything else.

I am leaning towards keeping the current name and functionality right now until 
another sla-aware update use case comes in and we have a better idea on how we 
can refactor the current abstraction (from kill to generic, which I don't think 
should be too difficult but I'm sure I'll regret those words...)

> On July 9, 2018, 9:04 p.m., Santhosh Kumar Shanmugham wrote:
> > src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java
> > Lines 78 (patched)
> > 
> >
> > test case when sla kill is a noop (task is already killed for a 
> > different reason) - is this already covered?

This is covered at the end of `testSlaKillRetry`.

> On July 9, 2018, 9:04 p.m., Santhosh Kumar Shanmugham wrote:
> > src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java
> > Lines 180 (patched)
> > 
> >
> > Drop `_` here and everywhere. `SLA_CHECKING` sounds like a valid event 
> > that is definied in code.

Can you elaborate on this? Did you mean drop `_MESSAGE`? I added `_MESSAGE` 
because it is not a true "event" in the sense that it piggybacks on 
`INSTANCE_UPDATING` or `INSTANCE_ROLLING_BACK` with a message showi

Re: Review Request 67696: Enable SLA-aware updates

2018-07-09 Thread Santhosh Kumar Shanmugham


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205866
---



end-to-end test for this feature?


docs/reference/scheduler-configuration.md
Lines 162 (patched)


s/The number of/The maximum number of/

This is only used for SLA-aware updates, should we name it so that they are 
grouped together with the other settings?



docs/reference/scheduler-configuration.md
Lines 227-230 (patched)


`sla_aware_kill_retry_min_delay` will be more readable. Same for the max 
delay.



src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java
Lines 56 (patched)


s/SLA passes/SLA passes or update is cancelled/



src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java
Lines 140 (patched)


Repeating my comment from internal review - 

Technically this is can be any kind of work. Should we call this class 
`SlaAwareInstanceUpdater`? Since this has more to do with actually update logic 
than kill itself. And perhaps the method can become `startSlaAwareUpdate`?

The job of this class is around checking safety and scheduling the supplied 
work to be evaulated. Naming the class based on it usage rather than its 
function seems weird to me.



src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java
Lines 78 (patched)


test case when sla kill is a noop (task is already killed for a different 
reason) - is this already covered?



src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java
Lines 180 (patched)


Drop `_` here and everywhere. `SLA_CHECKING` sounds like a valid event that 
is definied in code.



src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java
Lines 194 (patched)


Drop `_` here and everywhere. `SLA_PASSED` sounds like a real event.


- Santhosh Kumar Shanmugham


On July 9, 2018, 11:20 a.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 9, 2018, 11:20 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Daniel Knightly, Renan DelValle, 
> Santhosh Kumar Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da

Re: Review Request 67696: Enable SLA-aware updates

2018-07-08 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205838
---


Ship it!




Master (a6d9288) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 8, 2018, 6:48 a.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 8, 2018, 6:48 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
> Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/5/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update events occuring 
> concurrently. Scheduler stability did not seem to be affected.
> 
> 
> Thanks,
> 
> Jordan

Re: Review Request 67696: Enable SLA-aware updates

2018-07-07 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated July 8, 2018, 6:48 a.m.)


Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
Shanmugham, and Stephan Erb.


Changes
---

Add more user documentation around using the feature


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
  docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
  docs/reference/scheduler-configuration.md 
805e516689be019101f7c220c89fd9c391bb93b3 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
2baba2aa55865ec298a4c9e5af3952b56cb9a910 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/5/

Changes: https://reviews.apache.org/r/67696/diff/4-5/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

Re: Review Request 67696: Enable SLA-aware updates

2018-07-05 Thread David McLaughlin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205783
---



This looks great! Very easy to read. One thing I noticed is that the comments 
in the code have better insight into how this is supposed to work than the 
documentation. Would be nice to get more details in the user-facing docs. 
Particularly the overlap with maintenance coordinators and this. Example 
questions I can imagine users having include: how would I run an update which 
adheres to the maintenance SLA but also never updates more than N total 
instances? They may be tempted to use the batch size for this and here it would 
drastically limit their throughput.

- David McLaughlin


On July 3, 2018, 11:29 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 3, 2018, 11:29 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
> Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91

Re: Review Request 67696: Enable SLA-aware updates

2018-07-03 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205702
---


Ship it!




Master (a6d9288) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 3, 2018, 11:29 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 3, 2018, 11:29 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
> Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
>   docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
>   docs/reference/scheduler-configuration.md 
> 805e516689be019101f7c220c89fd9c391bb93b3 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/4/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update events occuring 
> concurrently. Scheduler stability did not seem to be affected.
> 
> 
> Thanks,
> 
> Jord

Re: Review Request 67696: Enable SLA-aware updates

2018-07-03 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated July 3, 2018, 11:29 p.m.)


Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
Shanmugham, and Stephan Erb.


Changes
---

Update documentation


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  RELEASE-NOTES.md edc081f502370190597ad028f3275cdfd572f5ca 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  docs/features/job-updates.md b52eb35a1de9da40f8a00ef0b905df30069029d3 
  docs/reference/configuration.md acab4c58d9ab3c04d156fed3636e77aed6d1faf4 
  docs/reference/scheduler-configuration.md 
805e516689be019101f7c220c89fd9c391bb93b3 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
2baba2aa55865ec298a4c9e5af3952b56cb9a910 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/4/

Changes: https://reviews.apache.org/r/67696/diff/3-4/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

Re: Review Request 67696: Enable SLA-aware updates

2018-07-03 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205697
---


Ship it!




Master (a6d9288) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On July 3, 2018, 6:30 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated July 3, 2018, 6:30 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
> Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5667a1b59681a6de87149d7161d760bff5da3818 
>   src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
> 2c27ec7136ff22a3570a4ec278c73f7ee310f628 
>   
> src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
> PRE-CREATION 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> 2baba2aa55865ec298a4c9e5af3952b56cb9a910 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
>  48902d39b9d2cbeae1a52180669aba8349e4dd65 
>   
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
>  08dfa5b9a67989083a0d405ce8100698a4d096ae 
>   ui/src/main/js/components/UpdateInstanceEvents.js 
> 8351f2c4c256e27625d94b70842be0e91065a551 
>   ui/src/main/js/components/UpdateSettings.js 
> d756f5916fd8c39dcbb5578ee5eda198f807f458 
> 
> 
> Diff: https://reviews.apache.org/r/67696/diff/3/
> 
> 
> Testing
> ---
> 
> Added unit tests, `./gradlew test`.
> 
> Tested at scale with over 10,000 SLA-aware instance update events occuring 
> concurrently. Scheduler stability did not seem to be affected.
> 
> 
> Thanks,
> 
> Jordan Ly
> 
>

Re: Review Request 67696: Enable SLA-aware updates

2018-07-03 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated July 3, 2018, 6:30 p.m.)


Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
Shanmugham, and Stephan Erb.


Changes
---

Fix build add additional IT tests.


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
2baba2aa55865ec298a4c9e5af3952b56cb9a910 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/3/

Changes: https://reviews.apache.org/r/67696/diff/2-3/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

Re: Review Request 67696: Enable SLA-aware updates

2018-06-21 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205226
---



Master (8508aec) is red with this patch.
  ./build-support/jenkins/build.sh

 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[create_job]
 <- 
pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 53%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[kill_job]
 <- 
pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 60%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[restart]
 <- 
pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 66%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[start_cronjob]
 <- 
pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 73%]
 
src/test/python/apache/aurora/client/hooks/test_hooked_api.py::test_api_methods_params[start_job_update]
 <- 
pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/client/hooks/test_hooked_api.py
 PASSED [ 80%]
 
src/test/python/apache/aurora/client/hooks/test_non_hooked_api.py::TestNonHookedAuroraClientAPI::test_kill_job_discards_config
 <- 
pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/client/hooks/test_non_hooked_api.py
 PASSED [ 86%]
 
src/test/python/apache/aurora/client/hooks/test_non_hooked_api.py::TestNonHookedAuroraClientAPI::test_restart_discards_config
 <- 
pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/client/hooks/test_non_hooked_api.py
 PASSED [ 93%]
 
src/test/python/apache/aurora/client/hooks/test_non_hooked_api.py::TestNonHookedAuroraClientAPI::test_start_cronjob_discards_config
 <- 
pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/client/hooks/test_non_hooked_api.py
 PASSED [100%]
 
  generated xml file: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/.pants.d/test/pytest/src.test.python.apache.aurora.client.hooks.hooks/junitxml/TEST-src.test.python.apache.aurora.client.hooks.hooks.xml
 
 === 15 passed in 0.33 seconds 
 
   src/test/python/apache/aurora/admin  
   .   SUCCESS
   src/test/python/apache/aurora/client 
   .   SUCCESS
   src/test/python/apache/aurora/client/api 
   .   FAILURE
   src/test/python/apache/aurora/client/cli 
   .   FAILURE
   src/test/python/apache/aurora/client/docker  
   .   SUCCESS
   src/test/python/apache/aurora/client/hooks   
   .   SUCCESS
   src/test/python/apache/aurora/common 
   .   SUCCESS
   src/test/python/apache/aurora/common/health_check
   .   SUCCESS
   src/test/python/apache/aurora/config 
   .   SUCCESS
   src/test/python/apache/aurora/executor   
   .   SUCCESS
   src/test/python/apache/aurora/executor/bin   
   .   SUCCESS
   src/test/python/apache/aurora/executor/common
   .   SUCCESS
   src/test/python/apache/aurora/tools  
   .   SUCCESS
   src/test/python/apache/thermos/cli   
   .   SUCCESS
   src/test/python/apache/thermos/cli/commands  
   .   SUCCESS
   src/test/python/apache/thermos/common
   .   SUCCESS
   src/test/python/apache/thermos/config
   .   SUCCESS
   src/test/python/apache/thermos/core  
   .   SUCCESS
   src/test/python/apache/thermos/monitoring
   .   SUCCESS
   src/test/python/apache/thermos/observer

Re: Review Request 67696: Enable SLA-aware updates

2018-06-21 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

(Updated June 22, 2018, 12:50 a.m.)


Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
Shanmugham, and Stephan Erb.


Changes
---

Fixing build, small style stuff.


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/2/

Changes: https://reviews.apache.org/r/67696/diff/1-2/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

Re: Review Request 67696: Enable SLA-aware updates

2018-06-21 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/#review205222
---



Master (8508aec) is red with this patch.
  ./build-support/jenkins/build.sh

 [221] ./src/main/js/index.js 3.43 kB {0} [built]
+ 264 hidden modules
:processResources
:classes
:jar
:moreStartScripts
:startScripts
:distTar
:distZip
:assemble
:compileTestJavaNote: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

:processTestResources
:testClasses
:compileJmhJavaNote: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java
 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:processJmhResources NO-SOURCE
:jmhClasses
:checkstyleJmh
:checkstyleMain
:checkstyleTest[ant:checkstyle] [ERROR] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java:48:15:
 Unused import - org.easymock.EasyMock.newCapture. [UnusedImports]
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':checkstyleTest'.
> Checkstyle rule violations were found. See the report at: 
> file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/test.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 5m 0s
33 actionable tasks: 27 executed, 6 up-to-date


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On June 21, 2018, 10:53 p.m., Jordan Ly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67696/
> ---
> 
> (Updated June 21, 2018, 10:53 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
> Shanmugham, and Stephan Erb.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch enables SLA-aware updates.
> 
> Following https://reviews.apache.org/r/66716/, tasks may now specify custom 
> SLA policies that will be respected by the scheduler during maintenance. This 
> patch integrates into the same system to allow users to specify if they want 
> their updates to also respect SLA. Please see 
> https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
>  for a more detailed description.
> 
> This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
> and `message` to `JobInstanceUpdateEvent`. These should be forward and 
> backwards compatible.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 7265b11103aa12743c42355163ae64e98e965d7f 
>   src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
> 2e13aacf576e648d9fffe989e4fc05c8954e72d8 
>   src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
> 9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
>   
> src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
> 9fa68b2dd55b4e4f5436356c1b94af1393967679 
>   src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
> a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  ec577cccb86914ebd679ca235103f79dd7e7b79d 
>   src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
> f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
>   src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
> 74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
>   src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
> f949fd54f524780672167e12fcadf268da08e679 
>   src/main/python/apache/aurora/client/api/updater_util.py 
> 4e3986220aaa4c9b138394b962120b176185af12 
>   src/main/python/apache/aurora/config/schema/base.py 
> 7baded79acdf863670afc183d740dcad602490c2 
>   src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
> dcf58896f1e866c0369ba1b78060236e98d9d46b 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
>  3a06a451da4ef3acccb33b5495b9fae141557148 
>   
> src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
> 0aea369d8a8f75291de9691b6d61f3d48895507c 
>   src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
> 43f857d893a54e19e71b36f2f06fef3a3ef6e874 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 5

Review Request 67696: Enable SLA-aware updates

2018-06-21 Thread Jordan Ly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67696/
---

Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar 
Shanmugham, and Stephan Erb.


Repository: aurora


Description
---

This patch enables SLA-aware updates.

Following https://reviews.apache.org/r/66716/, tasks may now specify custom SLA 
policies that will be respected by the scheduler during maintenance. This patch 
integrates into the same system to allow users to specify if they want their 
updates to also respect SLA. Please see 
https://docs.google.com/document/d/1lCoDyoX26qrGrptrgO7vJHqYR_L2CBRGFIywsAd8uQo/edit?usp=sharing
 for a more detailed description.

This patch adds two optional Thrift fields, `slaAware` to `JobUpdateSettings` 
and `message` to `JobInstanceUpdateEvent`. These should be forward and 
backwards compatible.


Diffs
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
7265b11103aa12743c42355163ae64e98e965d7f 
  src/main/java/org/apache/aurora/scheduler/base/Tasks.java 
2e13aacf576e648d9fffe989e4fc05c8954e72d8 
  src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java 
9aa51c3637df74cca088bd65c5539e1ebb8e5f0d 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceActionHandler.java 
9fa68b2dd55b4e4f5436356c1b94af1393967679 
  src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 
a002d955c3bc7b7c39da5e130e8c10c536bdcebd 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
ec577cccb86914ebd679ca235103f79dd7e7b79d 
  src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 
f2d33fb9ab6bd2c3ff199ab03dc75b1d6d618f3a 
  src/main/java/org/apache/aurora/scheduler/updater/SlaKillController.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/updater/UpdaterModule.java 
74ee1745e1fd7c308e2bdfa46aeb18a7ecfe14d2 
  src/main/java/org/apache/aurora/scheduler/updater/Updates.java 
f949fd54f524780672167e12fcadf268da08e679 
  src/main/python/apache/aurora/client/api/updater_util.py 
4e3986220aaa4c9b138394b962120b176185af12 
  src/main/python/apache/aurora/config/schema/base.py 
7baded79acdf863670afc183d740dcad602490c2 
  src/test/java/org/apache/aurora/scheduler/config/CommandLineTest.java 
dcf58896f1e866c0369ba1b78060236e98d9d46b 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java
 3a06a451da4ef3acccb33b5495b9fae141557148 
  src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 
0aea369d8a8f75291de9691b6d61f3d48895507c 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
43f857d893a54e19e71b36f2f06fef3a3ef6e874 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
5667a1b59681a6de87149d7161d760bff5da3818 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
2c27ec7136ff22a3570a4ec278c73f7ee310f628 
  src/test/java/org/apache/aurora/scheduler/updater/SlaKillControllerTest.java 
PRE-CREATION 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent
 48902d39b9d2cbeae1a52180669aba8349e4dd65 
  
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate
 08dfa5b9a67989083a0d405ce8100698a4d096ae 
  ui/src/main/js/components/UpdateInstanceEvents.js 
8351f2c4c256e27625d94b70842be0e91065a551 
  ui/src/main/js/components/UpdateSettings.js 
d756f5916fd8c39dcbb5578ee5eda198f807f458 


Diff: https://reviews.apache.org/r/67696/diff/1/


Testing
---

Added unit tests, `./gradlew test`.

Tested at scale with over 10,000 SLA-aware instance update events occuring 
concurrently. Scheduler stability did not seem to be affected.


Thanks,

Jordan Ly

39 matches

Mail list logo