-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52766/
-----------------------------------------------------------
Review request for Aurora, Joshua Cohen and Zameer Manji.
Bugs: AURORA-1791
https://issues.apache.org/jira/browse/AURORA-1791
Repository: aurora
Description
-------
Fix a bug in commit ca683cb. The commit is related to this review
https://reviews.apache.org/r/51876/. Please see it for more details and
backgrounds.
Currently, health checks are performed during a grace period called
initial_interval_secs. It is likely that HealthChecker fails to see sufficient
number of successes before the intitial_interval_secs expires. For example, for
a task with HealthCheckConfig(initital_interval_secs=15, interval_secs=10,
min_consecutive_successes=1). If the task sleeps during the first 12 seconds
and becomes healthy afterwards, the health checker will report the task status
as "TASK_FAILED" and miss the "healthy" status between second 12-15. This is
because only one health check is performed at second 10 before the
initial_interval_secs expires. This is an implementation flaw that breaks
backward-compatability.
To address this problem, I rewrite the function that is responsible for
updating the failure counts and the healthy status. The expected behavior is
that for the task described above, the health checker will performs a health
check after the initial_interval_secs expires and sets the health check status
to be healthy. Please see this review for more details.
Will add some more tests since the current e2e tests does not include the above
test case.
Diffs
-----
src/main/python/apache/aurora/executor/common/health_checker.py
1e0be108b49480d57c5ab94b1d2903bb57bae20a
src/test/python/apache/aurora/executor/common/test_health_checker.py
28769dca68a6353fc1283a8bb279fae05173aaac
Diff: https://reviews.apache.org/r/52766/diff/
Testing
-------
./build-support/jenkins/build.sh
./pants test.pytest src/test/python/apache/aurora/executor::
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
Modified the test in http_example.py. Let the http server sleep for the first
10 seconds.
Launch a job that contains the task with Default
HealthCheckConfig(initial_interval_secs=15, interval_secs=10,
min_consecutive_successes=1) in vagrant aurora cluster. The task transitions to
TASK_RUNNING state after ~20 seconds.
File Attachments
----------------
Task with default Health Check Config
https://reviews.apache.org/media/uploaded/files/2016/10/12/64cf6610-9294-46cb-b159-6e5721da5fff__Screen_Shot_2016-10-11_at_6.17.00_PM.png
Thanks,
Kai Huang
