[jira] [Commented] (MESOS-4671) Status updates from executor can be forwarded out of order by the Agent.
[ https://issues.apache.org/jira/browse/MESOS-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145712#comment-15145712 ] Anand Mazumdar commented on MESOS-4671: --- cc: [~avin...@mesosphere.io], [~jieyu] > Status updates from executor can be forwarded out of order by the Agent. > > > Key: MESOS-4671 > URL: https://issues.apache.org/jira/browse/MESOS-4671 > Project: Mesos > Issue Type: Bug > Components: containerization, HTTP API >Affects Versions: 0.28.0 >Reporter: Anand Mazumdar > Labels: mesosphere > > Previously, all status update messages from the executor were forwarded by > the agent to the master in the order that they had been received. > However, that seems to be no longer valid due to a recently introduced change > in the agent: > {code} > // Before sending update, we need to retrieve the container status. > containerizer->status(executor->containerId) > .onAny(defer(self(), > &Slave::_statusUpdate, > update, > pid, > executor->id, > lambda::_1)); > {code} > This can sometimes lead to status updates being sent out of order depending > on the order the {{Future}} is fulfilled from the call to {{status(...)}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4671) Status updates from executor can be forwarded out of order by the Agent.
[ https://issues.apache.org/jira/browse/MESOS-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146809#comment-15146809 ] Avinash Sridharan commented on MESOS-4671: -- >From the logs attached to MESOS-4661 it does look like the status updates for >TASK_RUNNING and TASK_FINISHED got inverted. Here are the relevant logs: I0212 00:23:11.639288 731 slave.cpp:3002] Handling status update TASK_RUNNING (UUID: b0810058-a1c0-4918-b699-61c16eb0f46a) for task 1 of framework 6508f198-e145-4d76-844f-0460dc5d7d39- I0212 00:23:11.639622 731 slave.cpp:3002] Handling status update TASK_FINISHED (UUID: 4674114a-c022-4945-ac98-52c0fae325e5) for task 1 of framework 6508f198-e145-4d76-844f-0460dc5d7d39- I0212 00:23:11.641832 729 slave.cpp:5661] Terminating task 1 I0212 00:23:11.643185 721 status_update_manager.cpp:320] Received status update TASK_FINISHED (UUID: 4674114a-c022-4945-ac98-52c0fae325e5) for task 1 of framework 6508f198-e145-4d76-844f-0460dc5d7d39- I0212 00:23:11.643405 7480 executor.cpp:588] Enqueuing event SUBSCRIBED received from http://172.17.0.2:57200/slave/api/v1/executor I0212 00:23:11.643427 721 status_update_manager.cpp:497] Creating StatusUpdate stream for task 1 of framework 6508f198-e145-4d76-844f-0460dc5d7d39- I0212 00:23:11.644057 721 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FINISHED (UUID: 4674114a-c022-4945-ac98-52c0fae325e5) for task 1 of framework 6508f198-e145-4d76-844f-0460dc5d7d39- Received a SUBSCRIBED event Since `dispatch` maintains the causality of events in a `process`, as long as the messages go through the same set of `processes` we should never see a re-ordering of events. However, the implementation of the status method in `MesosContainerizer` uses `await` to collect all the `ContainerStatus` `Futures` from the isolators, before completing the `Promise` given to the agent. The `await` method internally launches a `process` to wait for these futures. Thus, due to the use of `await` multiple `StatusUpdate` messages might end up being processed by a different set of `libprocess` thread causing a race. > Status updates from executor can be forwarded out of order by the Agent. > > > Key: MESOS-4671 > URL: https://issues.apache.org/jira/browse/MESOS-4671 > Project: Mesos > Issue Type: Bug > Components: containerization, HTTP API >Affects Versions: 0.28.0 >Reporter: Anand Mazumdar >Assignee: Avinash Sridharan > Labels: mesosphere > > Previously, all status update messages from the executor were forwarded by > the agent to the master in the order that they had been received. > However, that seems to be no longer valid due to a recently introduced change > in the agent: > {code} > // Before sending update, we need to retrieve the container status. > containerizer->status(executor->containerId) > .onAny(defer(self(), > &Slave::_statusUpdate, > update, > pid, > executor->id, > lambda::_1)); > {code} > This can sometimes lead to status updates being sent out of order depending > on the order the {{Future}} is fulfilled from the call to {{status(...)}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4671) Status updates from executor can be forwarded out of order by the Agent.
[ https://issues.apache.org/jira/browse/MESOS-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146874#comment-15146874 ] Klaus Ma commented on MESOS-4671: - In {{MesosContainerizerProcess}}, it'll also handle {{status}} in order; is it possible to return {{FINISHED}} before {{RUNNING}} for {{StatusUpdate}}? > Status updates from executor can be forwarded out of order by the Agent. > > > Key: MESOS-4671 > URL: https://issues.apache.org/jira/browse/MESOS-4671 > Project: Mesos > Issue Type: Bug > Components: containerization, HTTP API >Affects Versions: 0.28.0 >Reporter: Anand Mazumdar >Assignee: Avinash Sridharan > Labels: mesosphere > > Previously, all status update messages from the executor were forwarded by > the agent to the master in the order that they had been received. > However, that seems to be no longer valid due to a recently introduced change > in the agent: > {code} > // Before sending update, we need to retrieve the container status. > containerizer->status(executor->containerId) > .onAny(defer(self(), > &Slave::_statusUpdate, > update, > pid, > executor->id, > lambda::_1)); > {code} > This can sometimes lead to status updates being sent out of order depending > on the order the {{Future}} is fulfilled from the call to {{status(...)}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4671) Status updates from executor can be forwarded out of order by the Agent.
[ https://issues.apache.org/jira/browse/MESOS-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146882#comment-15146882 ] Avinash Sridharan commented on MESOS-4671: -- Hi [~klaus1982], Sorry didn't understand your comment? > Status updates from executor can be forwarded out of order by the Agent. > > > Key: MESOS-4671 > URL: https://issues.apache.org/jira/browse/MESOS-4671 > Project: Mesos > Issue Type: Bug > Components: containerization, HTTP API >Affects Versions: 0.28.0 >Reporter: Anand Mazumdar >Assignee: Avinash Sridharan > Labels: mesosphere > > Previously, all status update messages from the executor were forwarded by > the agent to the master in the order that they had been received. > However, that seems to be no longer valid due to a recently introduced change > in the agent: > {code} > // Before sending update, we need to retrieve the container status. > containerizer->status(executor->containerId) > .onAny(defer(self(), > &Slave::_statusUpdate, > update, > pid, > executor->id, > lambda::_1)); > {code} > This can sometimes lead to status updates being sent out of order depending > on the order the {{Future}} is fulfilled from the call to {{status(...)}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4671) Status updates from executor can be forwarded out of order by the Agent.
[ https://issues.apache.org/jira/browse/MESOS-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148125#comment-15148125 ] Klaus Ma commented on MESOS-4671: - When we call {{containerizer->status(...)}}, it'll dispatch event to {{MesosContainerizerProcess}}; it's also handled in order although it used {{await}}. Does there any case that make {{FINISHED}} status return firstly? One possibility is that: two {{containerizer->status()}} between the two {{await}}. > Status updates from executor can be forwarded out of order by the Agent. > > > Key: MESOS-4671 > URL: https://issues.apache.org/jira/browse/MESOS-4671 > Project: Mesos > Issue Type: Bug > Components: containerization, HTTP API >Affects Versions: 0.28.0 >Reporter: Anand Mazumdar >Assignee: Avinash Sridharan > Labels: mesosphere > > Previously, all status update messages from the executor were forwarded by > the agent to the master in the order that they had been received. > However, that seems to be no longer valid due to a recently introduced change > in the agent: > {code} > // Before sending update, we need to retrieve the container status. > containerizer->status(executor->containerId) > .onAny(defer(self(), > &Slave::_statusUpdate, > update, > pid, > executor->id, > lambda::_1)); > {code} > This can sometimes lead to status updates being sent out of order depending > on the order the {{Future}} is fulfilled from the call to {{status(...)}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)