Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Stephan Erb]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170932
---


Ship it!




Ship It!

- Stephan Erb


On March 31, 2017, 2:45 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 31, 2017, 2:45 a.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[David McLaughlin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170720
---


Ship it!




Ship It!

- David McLaughlin


On March 31, 2017, 12:45 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 31, 2017, 12:45 a.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170663
---


Ship it!




Master (076d917) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On March 31, 2017, 12:45 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 31, 2017, 12:45 a.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Zameer Manji]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/
---

(Updated March 30, 2017, 5:45 p.m.)


Review request for Aurora and Stephan Erb.


Changes
---

feedback.


Bugs: AURORA-1911
https://issues.apache.org/jira/browse/AURORA-1911


Repository: aurora


Description
---

As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
if they fail. This means that after a leader subscribes and disconnects, it is 
possible for it to never re subscribe again if the Mesos Master is unhealthy.

To fix this, I have moved the subscription into the dedicated 
`SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
binary backoff. It only stops if we are disconnected or if we sucessfully 
connect.


Diffs (updated)
-

  src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
10d4f1b515b91d85b283cb7c655275c22fb133f9 
  
src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
 67d356ab66c926a3b56860b906a453d57d6b694d 
  
src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
 756d0d9e30a447f9fba75c1c60f2f2f3c610399b 


Diff: https://reviews.apache.org/r/58053/diff/4/

Changes: https://reviews.apache.org/r/58053/diff/3-4/


Testing
---


Thanks,

Zameer Manji

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170662
---



Master (076d917) is red with this patch.
  ./build-support/jenkins/build.sh

:generateBuildProperties
:processResources
:classes
:jar
:startScripts
:distTar
:distZip
:assemble
:compileJmhJavaNote: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java
 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:processJmhResources UP-TO-DATE
:jmhClasses
:checkstyleJmh
:jsHint
:checkstyleMain
:compileTestJava/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/thrift/aop/MockDecoratedThrift.java:38:
 Note: Wrote forwarder 
org.apache.aurora.scheduler.thrift.aop.MockDecoratedThriftForwarder
@Forward(AnnotatedAuroraAdmin.class)
^
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:processTestResources
:testClasses
:checkstyleTest[ant:checkstyle] [ERROR] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java:379:
 Line is longer than 100 characters (found 101). [LineLength]
[ant:checkstyle] [ERROR] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java:405:
 Line is longer than 100 characters (found 106). [LineLength]
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':checkstyleTest'.
> Checkstyle rule violations were found. See the report at: 
> file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/test.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 1 mins 45.207 secs


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On March 31, 2017, 12:22 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 31, 2017, 12:22 a.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Zameer Manji]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/
---

(Updated March 30, 2017, 5:22 p.m.)


Review request for Aurora and Stephan Erb.


Changes
---

Feedback.


Bugs: AURORA-1911
https://issues.apache.org/jira/browse/AURORA-1911


Repository: aurora


Description
---

As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
if they fail. This means that after a leader subscribes and disconnects, it is 
possible for it to never re subscribe again if the Mesos Master is unhealthy.

To fix this, I have moved the subscription into the dedicated 
`SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
binary backoff. It only stops if we are disconnected or if we sucessfully 
connect.


Diffs (updated)
-

  src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
10d4f1b515b91d85b283cb7c655275c22fb133f9 
  
src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
 67d356ab66c926a3b56860b906a453d57d6b694d 
  
src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
 756d0d9e30a447f9fba75c1c60f2f2f3c610399b 


Diff: https://reviews.apache.org/r/58053/diff/3/

Changes: https://reviews.apache.org/r/58053/diff/2-3/


Testing
---


Thanks,

Zameer Manji

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Zameer Manji]

> On March 30, 2017, 4:08 p.m., David McLaughlin wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
> > Lines 168 (patched)
> > 
> >
> > Since it has to be true to execute this code block, isn't this 
> > redundant?

Yes, this is an error.

I have fixed it and added a full test case for this scenario.


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170655
---


On March 30, 2017, 11:37 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 30, 2017, 11:37 a.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[David McLaughlin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170655
---




src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
Lines 168 (patched)


Since it has to be true to execute this code block, isn't this redundant?


- David McLaughlin


On March 30, 2017, 6:37 p.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 30, 2017, 6:37 p.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Zameer Manji]

> On March 30, 2017, 8:13 a.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
> > Lines 125-129 (patched)
> > 
> >
> > Does the Mesos docs say anything about simultanous `SUBSCRIBE` calls?
> > 
> > If the backoff time is still pretty low we might end up sending another 
> > subscribe before we have received an answer for the previous one.
> 
> Zameer Manji wrote:
> From what I understand, multiple subscription per framework is not 
> allowed and subsequent subscribe attempts will fail if a connection was 
> already established. The underlying driver ignores those failures so we 
> should be fine.

The `V1Mesos` driver drops them silently so we are fine if we send extra calls. 
https://github.com/apache/mesos/blob/1a1fa95d0de179d7efab002a99a0e6261ce307f9/src/scheduler/scheduler.cpp#L230


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170580
---


On March 30, 2017, 11:37 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 30, 2017, 11:37 a.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170627
---


Ship it!




Master (076d917) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On March 30, 2017, 6:37 p.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 30, 2017, 6:37 p.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Zameer Manji]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/
---

(Updated March 30, 2017, 11:37 a.m.)


Review request for Aurora and Stephan Erb.


Changes
---

Feedback.


Bugs: AURORA-1911
https://issues.apache.org/jira/browse/AURORA-1911


Repository: aurora


Description
---

As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
if they fail. This means that after a leader subscribes and disconnects, it is 
possible for it to never re subscribe again if the Mesos Master is unhealthy.

To fix this, I have moved the subscription into the dedicated 
`SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
binary backoff. It only stops if we are disconnected or if we sucessfully 
connect.


Diffs (updated)
-

  src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
10d4f1b515b91d85b283cb7c655275c22fb133f9 
  
src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
 67d356ab66c926a3b56860b906a453d57d6b694d 
  
src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
 756d0d9e30a447f9fba75c1c60f2f2f3c610399b 


Diff: https://reviews.apache.org/r/58053/diff/2/

Changes: https://reviews.apache.org/r/58053/diff/1-2/


Testing
---


Thanks,

Zameer Manji

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Zameer Manji]

> On March 30, 2017, 8:13 a.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
> > Lines 120 (patched)
> > 
> >
> > Do we have to give up eventually? (I suppose not...)

I don't think so. If we give up, I assume the scheduler is going to shut down. 
Suppose if Mesos is down, on scheduler shutdown means we will elect a new 
leader. A new leader (by default) has a one minute timeout to register to 
Mesos. If we give up, we will just be flapping between leaders until the system 
heals. I think that's pretty undesirable.


> On March 30, 2017, 8:13 a.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
> > Lines 125-129 (patched)
> > 
> >
> > Does the Mesos docs say anything about simultanous `SUBSCRIBE` calls?
> > 
> > If the backoff time is still pretty low we might end up sending another 
> > subscribe before we have received an answer for the previous one.

>From what I understand, multiple subscription per framework is not allowed and 
>subsequent subscribe attempts will fail if a connection was already 
>established. The underlying driver ignores those failures so we should be fine.


> On March 30, 2017, 8:13 a.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
> > Lines 128-130 (original), 165-167 (patched)
> > 
> >
> > You are unsetting `isSubscribed` in the `disconnected` handler. Doesn't 
> > this imply we will never run the reregistration code here?

Good catch, fixed.


> On March 30, 2017, 8:13 a.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
> > Lines 137-138 (original), 174-175 (patched)
> > 
> >
> > I am wondering why we need this here for `OFFERS` but not for 
> > `RESCIND`, `INVERSE_OFFERS`, etc.

I put it in here for the same kind of errors are the unversioned driver. 
Technically we could put it everywhere. I'm not opposed if you think we should 
do it.


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170580
---


On March 29, 2017, 4:52 p.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 29, 2017, 4:52 p.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Stephan Erb]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170580
---



Not sure if I got everything :) A couple of questions below.


src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
Lines 120 (patched)


Do we have to give up eventually? (I suppose not...)



src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
Lines 125-129 (patched)


Does the Mesos docs say anything about simultanous `SUBSCRIBE` calls?

If the backoff time is still pretty low we might end up sending another 
subscribe before we have received an answer for the previous one.



src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
Lines 128-130 (original), 165-167 (patched)


You are unsetting `isSubscribed` in the `disconnected` handler. Doesn't 
this imply we will never run the reregistration code here?



src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
Lines 137-138 (original), 174-175 (patched)


I am wondering why we need this here for `OFFERS` but not for `RESCIND`, 
`INVERSE_OFFERS`, etc.


- Stephan Erb


On March 30, 2017, 1:52 a.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 30, 2017, 1:52 a.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 58053: Reliably subscribe to Mesos in the HTTP Driver.

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58053/#review170504
---


Ship it!




Master (3a9aabd) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On March 29, 2017, 11:52 p.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58053/
> ---
> 
> (Updated March 29, 2017, 11:52 p.m.)
> 
> 
> Review request for Aurora and Stephan Erb.
> 
> 
> Bugs: AURORA-1911
> https://issues.apache.org/jira/browse/AURORA-1911
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> As noted in AURORA-1911 the `V1Mesos` driver doesn't re try `SUBSCRIBE` calls 
> if they fail. This means that after a leader subscribes and disconnects, it 
> is possible for it to never re subscribe again if the Mesos Master is 
> unhealthy.
> 
> To fix this, I have moved the subscription into the dedicated 
> `SchedulerExecutor` and it coninutes to attempt to subscribe using truncated 
> binary backoff. It only stops if we are disconnected or if we sucessfully 
> connect.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 206b11458da2b0f938f0fcab5e5d3259a88ac9ee 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosCallbackHandler.java 
> 5bf1e4e8c46044cb69b266cd203b5ec2f8b9ab61 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> 10d4f1b515b91d85b283cb7c655275c22fb133f9 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImpl.java
>  67d356ab66c926a3b56860b906a453d57d6b694d 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/VersionedMesosSchedulerImplTest.java
>  756d0d9e30a447f9fba75c1c60f2f2f3c610399b 
> 
> 
> Diff: https://reviews.apache.org/r/58053/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>