[jira] [Commented] (AURORA-1791) Commit ca683 is not backwards compatible.
[
https://issues.apache.org/jira/browse/AURORA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566749#comment-15566749
]
Stephan Erb commented on AURORA-1791:
-
Another alternative would be to only ever increment the
{{current_consecutive_failures}} counter when the {{initial_interval_sec}} has
expired.
> Commit ca683 is not backwards compatible.
> -
>
> Key: AURORA-1791
> URL: https://issues.apache.org/jira/browse/AURORA-1791
> Project: Aurora
> Issue Type: Bug
>Reporter: Zameer Manji
>Assignee: Kai Huang
>Priority: Blocker
>
> The commit [ca683cb9e27bae76424a687bc6c3af5a73c501b9 |
> https://github.com/apache/aurora/commit/ca683cb9e27bae76424a687bc6c3af5a73c501b9]
> is not backwards compatible. The last section of the commit
> {quote}
> 4. Modified the Health Checker and redefined the meaning
> initial_interval_secs.
> {quote}
> has serious, unintended consequences.
> Consider the following health check config:
> {noformat}
> initial_interval_secs: 10
> interval_secs: 5
> max_consecutive_failures: 1
> {noformat}
> On the 0.16.0 executor, no health checking will occur for the first 10
> seconds. Here the earliest a task can cause failure is at the 10th second.
> On master, health checking starts right away which means the task can fail at
> the first second since {{max_consecutive_failures}} is set to 1.
> This is not backwards compatible and needs to be fixed.
> I think a good solution would be to revert the meaning change to
> initial_interval_secs and have the task transition into RUNNING when
> {{max_consecutive_successes}} is met.
> An investigation shows {{initial_interval_secs}} was set to 5 but the task
> failed health checks right away:
> {noformat}
> D1011 19:52:13.295877 6 health_checker.py:107] Health checks enabled.
> Performing health check.
> D1011 19:52:13.306816 6 health_checker.py:126] Reset consecutive failures
> counter.
> D1011 19:52:13.307032 6 health_checker.py:132] Initial interval expired.
> W1011 19:52:13.307130 6 health_checker.py:135] Failed to reach minimum
> consecutive successes.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1791) Commit ca683 is not backwards compatible.
[
https://issues.apache.org/jira/browse/AURORA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566721#comment-15566721
]
Kai Huang commented on AURORA-1791:
---
In your case, it's likely your task becomes health at the 6th second. However,
within the 10 secs initial_interval_secs, we only performed one health check at
the 5th second. So by definition, we "failed" to detect at least one health
check within the given period, so we moved the task state to FAILED.
We can enforce a health check at the end of the initial_interval_secs.
> Commit ca683 is not backwards compatible.
> -
>
> Key: AURORA-1791
> URL: https://issues.apache.org/jira/browse/AURORA-1791
> Project: Aurora
> Issue Type: Bug
>Reporter: Zameer Manji
>Assignee: Kai Huang
>Priority: Blocker
>
> The commit [ca683cb9e27bae76424a687bc6c3af5a73c501b9 |
> https://github.com/apache/aurora/commit/ca683cb9e27bae76424a687bc6c3af5a73c501b9]
> is not backwards compatible. The last section of the commit
> {quote}
> 4. Modified the Health Checker and redefined the meaning
> initial_interval_secs.
> {quote}
> has serious, unintended consequences.
> Consider the following health check config:
> {noformat}
> initial_interval_secs: 10
> interval_secs: 5
> max_consecutive_failures: 1
> {noformat}
> On the 0.16.0 executor, no health checking will occur for the first 10
> seconds. Here the earliest a task can cause failure is at the 10th second.
> On master, health checking starts right away which means the task can fail at
> the first second since {{max_consecutive_failures}} is set to 1.
> This is not backwards compatible and needs to be fixed.
> I think a good solution would be to revert the meaning change to
> initial_interval_secs and have the task transition into RUNNING when
> {{max_consecutive_successes}} is met.
> An investigation shows {{initial_interval_secs}} was set to 5 but the task
> failed health checks right away:
> {noformat}
> D1011 19:52:13.295877 6 health_checker.py:107] Health checks enabled.
> Performing health check.
> D1011 19:52:13.306816 6 health_checker.py:126] Reset consecutive failures
> counter.
> D1011 19:52:13.307032 6 health_checker.py:132] Initial interval expired.
> W1011 19:52:13.307130 6 health_checker.py:135] Failed to reach minimum
> consecutive successes.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1791) Commit ca683 is not backwards compatible.
[
https://issues.apache.org/jira/browse/AURORA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566711#comment-15566711
]
Kai Huang commented on AURORA-1791:
---
Thanks for pointing it out. The current implementation is: if we failed to
see(even if it exist) enough successful health checks within the
initial_interval_secs, we will move the task state to failed. This behavior is
not back-ward compatible and not safe. I'll revised the design.
> Commit ca683 is not backwards compatible.
> -
>
> Key: AURORA-1791
> URL: https://issues.apache.org/jira/browse/AURORA-1791
> Project: Aurora
> Issue Type: Bug
>Reporter: Zameer Manji
>Assignee: Kai Huang
>Priority: Blocker
>
> The commit [ca683cb9e27bae76424a687bc6c3af5a73c501b9 |
> https://github.com/apache/aurora/commit/ca683cb9e27bae76424a687bc6c3af5a73c501b9]
> is not backwards compatible. The last section of the commit
> {quote}
> 4. Modified the Health Checker and redefined the meaning
> initial_interval_secs.
> {quote}
> has serious, unintended consequences.
> Consider the following health check config:
> {noformat}
> initial_interval_secs: 10
> interval_secs: 5
> max_consecutive_failures: 1
> {noformat}
> On the 0.16.0 executor, no health checking will occur for the first 10
> seconds. Here the earliest a task can cause failure is at the 10th second.
> On master, health checking starts right away which means the task can fail at
> the first second since {{max_consecutive_failures}} is set to 1.
> This is not backwards compatible and needs to be fixed.
> I think a good solution would be to revert the meaning change to
> initial_interval_secs and have the task transition into RUNNING when
> {{max_consecutive_successes}} is met.
> An investigation shows {{initial_interval_secs}} was set to 5 but the task
> failed health checks right away:
> {noformat}
> D1011 19:52:13.295877 6 health_checker.py:107] Health checks enabled.
> Performing health check.
> D1011 19:52:13.306816 6 health_checker.py:126] Reset consecutive failures
> counter.
> D1011 19:52:13.307032 6 health_checker.py:132] Initial interval expired.
> W1011 19:52:13.307130 6 health_checker.py:135] Failed to reach minimum
> consecutive successes.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AURORA-1792) Executor does not log full task information.
Zameer Manji created AURORA-1792:
Summary: Executor does not log full task information.
Key: AURORA-1792
URL: https://issues.apache.org/jira/browse/AURORA-1792
Project: Aurora
Issue Type: Bug
Reporter: Zameer Manji
I launched a task that has an {{initial_interval_secs}} in the health check
config. However the log contains no information about this field:
{noformat}
$ grep "initial_interval_secs" __main__.log
{noformat}
We should log the entire ExecutorInfo blob.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AURORA-1791) Commit ca683 is not backwards compatible.
[
https://issues.apache.org/jira/browse/AURORA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zameer Manji updated AURORA-1791:
-
Description:
The commit [ca683cb9e27bae76424a687bc6c3af5a73c501b9 |
https://github.com/apache/aurora/commit/ca683cb9e27bae76424a687bc6c3af5a73c501b9]
is not backwards compatible. The last section of the commit
{quote}
4. Modified the Health Checker and redefined the meaning initial_interval_secs.
{quote}
has serious, unintended consequences.
Consider the following health check config:
{noformat}
initial_interval_secs: 10
interval_secs: 5
max_consecutive_failures: 1
{noformat}
On the 0.16.0 executor, no health checking will occur for the first 10 seconds.
Here the earliest a task can cause failure is at the 10th second.
On master, health checking starts right away which means the task can fail at
the first second since {{max_consecutive_failures}} is set to 1.
This is not backwards compatible and needs to be fixed.
I think a good solution would be to revert the meaning change to
initial_interval_secs and have the task transition into RUNNING when
{{max_consecutive_successes}} is met.
An investigation shows {{initial_interval_secs}} was set to 5 but the task
failed health checks right away:
{noformat}
D1011 19:52:13.295877 6 health_checker.py:107] Health checks enabled.
Performing health check.
D1011 19:52:13.306816 6 health_checker.py:126] Reset consecutive failures
counter.
D1011 19:52:13.307032 6 health_checker.py:132] Initial interval expired.
W1011 19:52:13.307130 6 health_checker.py:135] Failed to reach minimum
consecutive successes.
{noformat}
was:
The commit [ca683cb9e27bae76424a687bc6c3af5a73c501b9 |
https://github.com/apache/aurora/commit/ca683cb9e27bae76424a687bc6c3af5a73c501b9]
is not backwards compatible. The last section of the commit
{quote}
4. Modified the Health Checker and redefined the meaning initial_interval_secs.
{quote}
has serious, unintended consequences.
Consider the following health check config:
{noformat}
initial_interval_secs: 10
interval_secs: 5
max_consecutive_failures: 1
{noformat}
On the 0.16.0 executor, no health checking will occur for the first 10 seconds.
Here the earliest a task can cause failure is at the 10th second.
On master, health checking starts right away which means the task can fail at
the first second since {{max_consecutive_failures}} is set to 1.
This is not backwards compatible and needs to be fixed.
I think a good solution would be to revert the meaning change to
initial_interval_secs and have the task transition into RUNNING when
{{max_consecutive_successes}} is met.
> Commit ca683 is not backwards compatible.
> -
>
> Key: AURORA-1791
> URL: https://issues.apache.org/jira/browse/AURORA-1791
> Project: Aurora
> Issue Type: Bug
>Reporter: Zameer Manji
>Assignee: Kai Huang
>Priority: Blocker
>
> The commit [ca683cb9e27bae76424a687bc6c3af5a73c501b9 |
> https://github.com/apache/aurora/commit/ca683cb9e27bae76424a687bc6c3af5a73c501b9]
> is not backwards compatible. The last section of the commit
> {quote}
> 4. Modified the Health Checker and redefined the meaning
> initial_interval_secs.
> {quote}
> has serious, unintended consequences.
> Consider the following health check config:
> {noformat}
> initial_interval_secs: 10
> interval_secs: 5
> max_consecutive_failures: 1
> {noformat}
> On the 0.16.0 executor, no health checking will occur for the first 10
> seconds. Here the earliest a task can cause failure is at the 10th second.
> On master, health checking starts right away which means the task can fail at
> the first second since {{max_consecutive_failures}} is set to 1.
> This is not backwards compatible and needs to be fixed.
> I think a good solution would be to revert the meaning change to
> initial_interval_secs and have the task transition into RUNNING when
> {{max_consecutive_successes}} is met.
> An investigation shows {{initial_interval_secs}} was set to 5 but the task
> failed health checks right away:
> {noformat}
> D1011 19:52:13.295877 6 health_checker.py:107] Health checks enabled.
> Performing health check.
> D1011 19:52:13.306816 6 health_checker.py:126] Reset consecutive failures
> counter.
> D1011 19:52:13.307032 6 health_checker.py:132] Initial interval expired.
> W1011 19:52:13.307130 6 health_checker.py:135] Failed to reach minimum
> consecutive successes.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1791) Commit ca683 is not backwards compatible.
[
https://issues.apache.org/jira/browse/AURORA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1557#comment-1557
]
Zameer Manji commented on AURORA-1791:
--
Note, I could be wrong here but this was deployed to a cluster and tasks that
were healthy before started to fail.
> Commit ca683 is not backwards compatible.
> -
>
> Key: AURORA-1791
> URL: https://issues.apache.org/jira/browse/AURORA-1791
> Project: Aurora
> Issue Type: Bug
>Reporter: Zameer Manji
>Assignee: Kai Huang
>Priority: Blocker
>
> The commit [ca683cb9e27bae76424a687bc6c3af5a73c501b9 |
> https://github.com/apache/aurora/commit/ca683cb9e27bae76424a687bc6c3af5a73c501b9]
> is not backwards compatible. The last section of the commit
> {quote}
> 4. Modified the Health Checker and redefined the meaning
> initial_interval_secs.
> {quote}
> has serious, unintended consequences.
> Consider the following health check config:
> {noformat}
> initial_interval_secs: 10
> interval_secs: 5
> max_consecutive_failures: 1
> {noformat}
> On the 0.16.0 executor, no health checking will occur for the first 10
> seconds. Here the earliest a task can cause failure is at the 10th second.
> On master, health checking starts right away which means the task can fail at
> the first second since {{max_consecutive_failures}} is set to 1.
> This is not backwards compatible and needs to be fixed.
> I think a good solution would be to revert the meaning change to
> initial_interval_secs and have the task transition into RUNNING when
> {{max_consecutive_successes}} is met.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AURORA-1791) Commit ca683 is not backwards compatible.
Zameer Manji created AURORA-1791:
Summary: Commit ca683 is not backwards compatible.
Key: AURORA-1791
URL: https://issues.apache.org/jira/browse/AURORA-1791
Project: Aurora
Issue Type: Bug
Reporter: Zameer Manji
Assignee: Kai Huang
Priority: Blocker
The commit [ca683cb9e27bae76424a687bc6c3af5a73c501b9 |
https://github.com/apache/aurora/commit/ca683cb9e27bae76424a687bc6c3af5a73c501b9]
is not backwards compatible. The last section of the commit
{quote}
4. Modified the Health Checker and redefined the meaning initial_interval_secs.
{quote}
has serious, unintended consequences.
Consider the following health check config:
{noformat}
initial_interval_secs: 10
interval_secs: 5
max_consecutive_failures: 1
{noformat}
On the 0.16.0 executor, no health checking will occur for the first 10 seconds.
Here the earliest a task can cause failure is at the 10th second.
On master, health checking starts right away which means the task can fail at
the first second since {{max_consecutive_failures}} is set to 1.
This is not backwards compatible and needs to be fixed.
I think a good solution would be to revert the meaning change to
initial_interval_secs and have the task transition into RUNNING when
{{max_consecutive_successes}} is met.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AURORA-1790) Aurora CNI Support
[
https://issues.apache.org/jira/browse/AURORA-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jake Farrell updated AURORA-1790:
-
Assignee: (was: Jake Farrell)
> Aurora CNI Support
> --
>
> Key: AURORA-1790
> URL: https://issues.apache.org/jira/browse/AURORA-1790
> Project: Aurora
> Issue Type: Epic
>Reporter: Stephan Erb
>
> The [Container Network Interface
> (CNI)|https://github.com/containernetworking/cni/blob/master/SPEC.md] is a
> plug-in based networking solution for containers. CNI is [supported by the
> Mesos Unified
> Containerizer|https://github.com/apache/mesos/blob/master/docs/cni.md].
> CNI support in Aurora would enable cluster operators to isolate tasks on the
> network level. This includes features such as IP-per container, or security
> policies ensuring that only designated subsets of containers can communicate
> with each other. Both are important feature for multi-tenant environments.
> h2. Mesos Protobufs
> In order to launch a task using CNI, Mesos requires frameworks to populate
> the
> [NetworkInfo|https://github.com/apache/mesos/blob/0f97117bac3e1382744e9a847ce11b7589fc45bd/include/mesos/mesos.proto#L1916-L1999]
> protobuf. The following shows relevant subset of fields:
> {code}
> /**
> * Describes a container configuration and allows extensible
> * configurations for different container implementations.
> *
> * NOTE: In the Aurora case, this is set as part of ExecutorInfo
> */
> message ContainerInfo {
> ...
> // A list of network requests. A framework can request multiple IP addresses
> // for the container.
> repeated NetworkInfo network_infos = 7;
> ...
> }
> /**
> * Describes a network request from a framework as well as network resolution
> * provided by Mesos.
> */
> message NetworkInfo {
> ...
> // For the CNI case, empty during task/executor launch and only used
> // in TaskStatus messages to inform the framework scheduler about
> // the IP addresses bound to a container
> repeated IPAddress ip_addresses = 5;
> // Name of the network which will be used by network isolator to determine
> // the network that the container joins. It's up to the network isolator
> // to decide how to interpret this field.
> optional string name = 6;
> // To tag certain metadata to be used by Isolator/IPAM, e.g., rack, etc.
> // Opaque to Mesos but interpreted by the CNI plugin
> optional Labels labels = 4;
> ...
> }
> /**
> * Container related information that is resolved during container
> * setup. The information is sent back to the framework as part of the
> * TaskStatus message.
> */
> message ContainerStatus {
> // This field can be reliably used to identify the container IP address.
> repeated NetworkInfo network_infos = 1;
> ...
> }
> {code}
> h2. Challenges
> * In contrast to ports or other resources, this is the first time an
> important detail is only discovered asynchronously after a task has been
> launched, i.e. the scheduler will only learn about the IP addresses of the
> launched task after having received its first {{TaskStatus}}.
> * A task can now live in multiple networks and can have multiple IP addresses.
> h2. Necessary Changes
> In order to implement CNI support in Aurora, several changes across the
> entire code base are needed.
> h3. Mesos
> * As of today, it seems like there is no reliable way to discover
> CNI-assigned IPs from within an executor (see MESOS-6281). This is crucial
> for us, as Thermos is responsible to announce itself into Zookeeper
> serversets.
> h3. Thermos
> * The Observer UI needs to be updated to handle multiple IP addresses.
> * The ZK serverset announcement needs to be adjusted to publish all
> IP-addresses.
> * A replacement/addition for pystachio mesos.hostname is required so
> that usercode can discover its current IP addresses. This relates to
> MESOS-6281.
> h3. Aurora Scheduler
> * Feature toggle allowing operators to enabe/disable CNI support.
> * Plumbing of NetworkInfo name and labels touching Thrift API, storage, and
> task launch mechanism.
> * Extension of {{TaskStatusHandlerImpl}} and
> [{{StateManager}}|https://github.com/apache/aurora/blob/783baaefb9a814ca01fad78181fe3df3de5b34af/src/main/java/org/apache/aurora/scheduler/state/StateManager.java]
> storage layer to persist received IP addresses.
> h3. Aurora Client
> * Extension of the Pystachio configuration so that user-defined jobs can join
> operator enabled networks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (AURORA-1790) Aurora CNI Support
[
https://issues.apache.org/jira/browse/AURORA-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jake Farrell reassigned AURORA-1790:
Assignee: Jake Farrell
> Aurora CNI Support
> --
>
> Key: AURORA-1790
> URL: https://issues.apache.org/jira/browse/AURORA-1790
> Project: Aurora
> Issue Type: Epic
>Reporter: Stephan Erb
>Assignee: Jake Farrell
>
> The [Container Network Interface
> (CNI)|https://github.com/containernetworking/cni/blob/master/SPEC.md] is a
> plug-in based networking solution for containers. CNI is [supported by the
> Mesos Unified
> Containerizer|https://github.com/apache/mesos/blob/master/docs/cni.md].
> CNI support in Aurora would enable cluster operators to isolate tasks on the
> network level. This includes features such as IP-per container, or security
> policies ensuring that only designated subsets of containers can communicate
> with each other. Both are important feature for multi-tenant environments.
> h2. Mesos Protobufs
> In order to launch a task using CNI, Mesos requires frameworks to populate
> the
> [NetworkInfo|https://github.com/apache/mesos/blob/0f97117bac3e1382744e9a847ce11b7589fc45bd/include/mesos/mesos.proto#L1916-L1999]
> protobuf. The following shows relevant subset of fields:
> {code}
> /**
> * Describes a container configuration and allows extensible
> * configurations for different container implementations.
> *
> * NOTE: In the Aurora case, this is set as part of ExecutorInfo
> */
> message ContainerInfo {
> ...
> // A list of network requests. A framework can request multiple IP addresses
> // for the container.
> repeated NetworkInfo network_infos = 7;
> ...
> }
> /**
> * Describes a network request from a framework as well as network resolution
> * provided by Mesos.
> */
> message NetworkInfo {
> ...
> // For the CNI case, empty during task/executor launch and only used
> // in TaskStatus messages to inform the framework scheduler about
> // the IP addresses bound to a container
> repeated IPAddress ip_addresses = 5;
> // Name of the network which will be used by network isolator to determine
> // the network that the container joins. It's up to the network isolator
> // to decide how to interpret this field.
> optional string name = 6;
> // To tag certain metadata to be used by Isolator/IPAM, e.g., rack, etc.
> // Opaque to Mesos but interpreted by the CNI plugin
> optional Labels labels = 4;
> ...
> }
> /**
> * Container related information that is resolved during container
> * setup. The information is sent back to the framework as part of the
> * TaskStatus message.
> */
> message ContainerStatus {
> // This field can be reliably used to identify the container IP address.
> repeated NetworkInfo network_infos = 1;
> ...
> }
> {code}
> h2. Challenges
> * In contrast to ports or other resources, this is the first time an
> important detail is only discovered asynchronously after a task has been
> launched, i.e. the scheduler will only learn about the IP addresses of the
> launched task after having received its first {{TaskStatus}}.
> * A task can now live in multiple networks and can have multiple IP addresses.
> h2. Necessary Changes
> In order to implement CNI support in Aurora, several changes across the
> entire code base are needed.
> h3. Mesos
> * As of today, it seems like there is no reliable way to discover
> CNI-assigned IPs from within an executor (see MESOS-6281). This is crucial
> for us, as Thermos is responsible to announce itself into Zookeeper
> serversets.
> h3. Thermos
> * The Observer UI needs to be updated to handle multiple IP addresses.
> * The ZK serverset announcement needs to be adjusted to publish all
> IP-addresses.
> * A replacement/addition for pystachio mesos.hostname is required so
> that usercode can discover its current IP addresses. This relates to
> MESOS-6281.
> h3. Aurora Scheduler
> * Feature toggle allowing operators to enabe/disable CNI support.
> * Plumbing of NetworkInfo name and labels touching Thrift API, storage, and
> task launch mechanism.
> * Extension of {{TaskStatusHandlerImpl}} and
> [{{StateManager}}|https://github.com/apache/aurora/blob/783baaefb9a814ca01fad78181fe3df3de5b34af/src/main/java/org/apache/aurora/scheduler/state/StateManager.java]
> storage layer to persist received IP addresses.
> h3. Aurora Client
> * Extension of the Pystachio configuration so that user-defined jobs can join
> operator enabled networks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
