[jira] [Commented] (MESOS-1410) Keep terminal unacknowledged tasks in the master's state.
[ https://issues.apache.org/jira/browse/MESOS-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131014#comment-14131014 ] Benjamin Mahler commented on MESOS-1410: https://reviews.apache.org/r/25565/ https://reviews.apache.org/r/25566/ https://reviews.apache.org/r/25567/ https://reviews.apache.org/r/25568/ > Keep terminal unacknowledged tasks in the master's state. > - > > Key: MESOS-1410 > URL: https://issues.apache.org/jira/browse/MESOS-1410 > Project: Mesos > Issue Type: Task >Affects Versions: 0.19.0 >Reporter: Benjamin Mahler >Assignee: Benjamin Mahler > Fix For: 0.21.0 > > > Once we are sending acknowledgments through the master as per MESOS-1409, we > need to keep terminal tasks that are *unacknowledged* in the Master's memory. > This will allow us to identify these tasks to frameworks when we haven't yet > forwarded them an update. Without this, we're susceptible to MESOS-1389. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1696) Improve reconciliation between master and slave.
[
https://issues.apache.org/jira/browse/MESOS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Mahler updated MESOS-1696:
---
Description:
As we update the Master to keep tasks in memory until they are both terminal
and acknowledged (MESOS-1410), the lifetime of tasks in Mesos will look as
follows:
{code}
Master Slave
{} {}
{Tn} {} // Master receives Task T, non-terminal. Forwards to
slave.
{Tn} {Tn} // Slave receives Task T, non-terminal.
{Tn} {Tt} // Task becomes terminal on slave. Update forwarded.
{Tt} {Tt} // Master receives update, forwards to framework.
{} {Tt} // Master receives ack, forwards to slave.
{} {} // Slave receives ack.
{code}
In the current form of reconciliation, the slave sends to the master all tasks
that are not both terminal and acknowledged. At any point in the above
lifecycle, the slave's re-registration message can reach the master.
Note the following properties:
*(1)* The master may have a non-terminal task, not present in the slave's
re-registration message.
*(2)* The master may have a non-terminal task, present in the slave's
re-registration message but in a different state.
*(3)* The slave's re-registration message may contain a terminal unacknowledged
task unknown to the master.
In the current master / slave
[reconciliation|https://github.com/apache/mesos/blob/0.19.1/src/master/master.cpp#L3146]
code, the master assumes that case (1) is because a launch task message was
dropped, and it sends TASK_LOST. We've seen above that (1) can happen even when
the task reaches the slave correctly, so this can lead to inconsistency!
After chatting with [~vinodkone], we're considering updating the reconciliation
to occur as follows:
→ Slave sends all tasks that are not both terminal and acknowledged, during
re-registration. This is the same as before.
→ If the master sees tasks that are missing in the slave, the master sends the
tasks that need to be reconciled to the slave for the tasks. This can be
piggy-backed on the re-registration message.
→ The slave will send TASK_LOST if the task is not known to it. Preferably in a
retried manner, unless we update socket closure on the slave to force a
re-registration.
was:
As we update the Master to keep tasks in memory until they are both terminal
and acknowledged (MESOS-1410), the lifetime of tasks in Mesos will look as
follows:
{code}
Master Slave
{} {}
{Tn} {} // Master receives Task T, non-terminal. Forwards to
slave.
{Tn} {Tn} // Slave receives Task T, non-terminal.
{Tn} {Tt} // Task becomes terminal on slave. Update forwarded.
{Tt} {Tt} // Master receives update, forwards to framework.
{} {Tt} // Master receives ack, forwards to slave.
{} {} // Slave receives ack.
{code}
In the current form of reconciliation, the slave sends to the master all tasks
that are not both terminal and acknowledged. At any point in the above
lifecycle, the slave's re-registration message can reach the master.
Note the following properties:
*(1)* The master may have a non-terminal task, not present in the slave's
re-registration message.
*(2)* The master may have a non-terminal task, present in the slave's
re-registration message but in a different state.
*(3)* The slave's re-registration message may contain a terminal unacknowledged
task unknown to the master.
In the current master / slave
[reconciliation|https://github.com/apache/mesos/blob/0.19.1/src/master/master.cpp#L3146]
code, the master assumes that case (1) is because a launch task message was
dropped, and it sends TASK_LOST. We've seen above that (1) can happen even when
the task reaches the slave correctly, so this can lead to inconsistency!
After chatting with [~vinodkone], we're considering updating the reconciliation
to occur as follows:
→ Slave sends all tasks that are not both terminal and acknowledged, during
re-registration. This is the same as before.
→ If the master sees tasks that are missing in the slave, the master sends a
reconcile message to the slave for the tasks.
→ The slave will reply to reconcile messages with the latest state, or
TASK_LOST if the task is not known to it. Preferably in a retried manner,
unless we update socket closure on the slave to force a re-registration.
> Improve reconciliation between master and slave.
>
>
> Key: MESOS-1696
> URL: https://issues.apache.org/jira/browse/MESOS-1696
> Project: Mesos
> Issue Type: Bug
> Components: master, slave
>Reporter: Benjamin Mahler
>Assignee: Vinod Kone
>
> As we update the Master to keep tasks in memory until they are both terminal
> and acknowledge
[jira] [Updated] (MESOS-1668) Handle a temporary one-way master --> slave socket closure.
[ https://issues.apache.org/jira/browse/MESOS-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kone updated MESOS-1668: -- Sprint: Mesos Q3 Sprint 5 Assignee: Vinod Kone Story Points: 2 The plan is to handle this by piggybacking the current slave state (e.g., bool registered) on the ping/pong messages. When the slave receives a ping message which says that the master thinks the slave is disconnected but slave doesn't know it yet (socket only broke on the master side), slave will attempt a re-registration. > Handle a temporary one-way master --> slave socket closure. > --- > > Key: MESOS-1668 > URL: https://issues.apache.org/jira/browse/MESOS-1668 > Project: Mesos > Issue Type: Bug > Components: master, slave >Reporter: Benjamin Mahler >Assignee: Vinod Kone >Priority: Minor > Labels: reliability > > In MESOS-1529, we realized that it's possible for a slave to remain > disconnected in the master if the following occurs: > → Master and Slave connected operating normally. > → Temporary one-way network failure, master→slave link breaks. > → Master marks slave as disconnected. > → Network restored and health checking continues normally, slave is not > removed as a result. Slave does not attempt to re-register since it is > receiving pings once again. > → Slave remains disconnected according to the master, and the slave does not > try to re-register. Bad! > We were originally thinking of using a failover timeout in the master to > remove these slaves that don't re-register. However, it can be dangerous when > ZooKeeper issues are preventing the slave from re-registering with the > master; we do not want to remove a ton of slaves in this situation. > Rather, when the slave is health checking correctly but does not re-register > within a timeout, we could send a registration request from the master to the > slave, telling the slave that it must re-register. This message could also be > used when receiving status updates (or other messages) from slaves that are > disconnected in the master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1783) MasterTest.LaunchDuplicateOfferTest is flaky
[
https://issues.apache.org/jira/browse/MESOS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130940#comment-14130940
]
Niklas Quarfot Nielsen commented on MESOS-1783:
---
Right - I see the problem. The launch task request fails and the resources are
reoffered and it worked before because we raced out of the test.
We can either discard any subsequent offers or wait for the recovered
resources. Both work, but the first is faster.
I will get a patch ready.
> MasterTest.LaunchDuplicateOfferTest is flaky
>
>
> Key: MESOS-1783
> URL: https://issues.apache.org/jira/browse/MESOS-1783
> Project: Mesos
> Issue Type: Bug
> Components: test
>Affects Versions: 0.20.0
> Environment: ubuntu-14.04-gcc Jenkins VM
>Reporter: Yan Xu
>
> {noformat:title=}
> [ RUN ] MasterTest.LaunchDuplicateOfferTest
> Using temporary directory '/tmp/MasterTest_LaunchDuplicateOfferTest_3ifzmg'
> I0909 22:46:59.212977 21883 leveldb.cpp:176] Opened db in 20.307533ms
> I0909 22:46:59.219717 21883 leveldb.cpp:183] Compacted db in 6.470397ms
> I0909 22:46:59.219925 21883 leveldb.cpp:198] Created db iterator in 5571ns
> I0909 22:46:59.220100 21883 leveldb.cpp:204] Seeked to beginning of db in
> 1365ns
> I0909 22:46:59.220268 21883 leveldb.cpp:273] Iterated through 0 keys in the
> db in 658ns
> I0909 22:46:59.220448 21883 replica.cpp:741] Replica recovered with log
> positions 0 -> 0 with 1 holes and 0 unlearned
> I0909 22:46:59.220855 21903 recover.cpp:425] Starting replica recovery
> I0909 22:46:59.221103 21903 recover.cpp:451] Replica is in EMPTY status
> I0909 22:46:59.221626 21903 replica.cpp:638] Replica in EMPTY status received
> a broadcasted recover request
> I0909 22:46:59.221914 21903 recover.cpp:188] Received a recover response from
> a replica in EMPTY status
> I0909 22:46:59.04 21903 recover.cpp:542] Updating replica status to
> STARTING
> I0909 22:46:59.232590 21900 master.cpp:286] Master
> 20140909-224659-16842879-44263-21883 (trusty) started on 127.0.1.1:44263
> I0909 22:46:59.233278 21900 master.cpp:332] Master only allowing
> authenticated frameworks to register
> I0909 22:46:59.233543 21900 master.cpp:337] Master only allowing
> authenticated slaves to register
> I0909 22:46:59.233934 21900 credentials.hpp:36] Loading credentials for
> authentication from
> '/tmp/MasterTest_LaunchDuplicateOfferTest_3ifzmg/credentials'
> I0909 22:46:59.236431 21900 master.cpp:366] Authorization enabled
> I0909 22:46:59.237522 21898 hierarchical_allocator_process.hpp:299]
> Initializing hierarchical allocator process with master :
> master@127.0.1.1:44263
> I0909 22:46:59.237877 21904 master.cpp:120] No whitelist given. Advertising
> offers for all slaves
> I0909 22:46:59.238723 21903 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 16.245391ms
> I0909 22:46:59.238916 21903 replica.cpp:320] Persisted replica status to
> STARTING
> I0909 22:46:59.239203 21903 recover.cpp:451] Replica is in STARTING status
> I0909 22:46:59.239724 21903 replica.cpp:638] Replica in STARTING status
> received a broadcasted recover request
> I0909 22:46:59.239967 21903 recover.cpp:188] Received a recover response from
> a replica in STARTING status
> I0909 22:46:59.240304 21903 recover.cpp:542] Updating replica status to VOTING
> I0909 22:46:59.240684 21900 master.cpp:1212] The newly elected leader is
> master@127.0.1.1:44263 with id 20140909-224659-16842879-44263-21883
> I0909 22:46:59.240846 21900 master.cpp:1225] Elected as the leading master!
> I0909 22:46:59.241149 21900 master.cpp:1043] Recovering from registrar
> I0909 22:46:59.241509 21898 registrar.cpp:313] Recovering registrar
> I0909 22:46:59.248440 21903 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 7.864221ms
> I0909 22:46:59.248644 21903 replica.cpp:320] Persisted replica status to
> VOTING
> I0909 22:46:59.248846 21903 recover.cpp:556] Successfully joined the Paxos
> group
> I0909 22:46:59.249330 21897 log.cpp:656] Attempting to start the writer
> I0909 22:46:59.249809 21897 replica.cpp:474] Replica received implicit
> promise request with proposal 1
> I0909 22:46:59.250075 21903 recover.cpp:440] Recover process terminated
> I0909 22:46:59.258286 21897 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 8.292514ms
> I0909 22:46:59.258489 21897 replica.cpp:342] Persisted promised to 1
> I0909 22:46:59.258848 21897 coordinator.cpp:230] Coordinator attemping to
> fill missing position
> I0909 22:46:59.259454 21897 replica.cpp:375] Replica received explicit
> promise request for position 0 with proposal 2
> I0909 22:46:59.267755 21897 leveldb.cpp:343] Persisting action (8 bytes) to
> leveldb took 8.109338ms
> I0909 22:46:59.267916 21897 replica.cpp:676] Persisted action at 0
> I0909 22:46:59.2701
[jira] [Assigned] (MESOS-1696) Improve reconciliation between master and slave.
[
https://issues.apache.org/jira/browse/MESOS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kone reassigned MESOS-1696:
-
Assignee: Vinod Kone
> Improve reconciliation between master and slave.
>
>
> Key: MESOS-1696
> URL: https://issues.apache.org/jira/browse/MESOS-1696
> Project: Mesos
> Issue Type: Bug
> Components: master, slave
>Reporter: Benjamin Mahler
>Assignee: Vinod Kone
>
> As we update the Master to keep tasks in memory until they are both terminal
> and acknowledged (MESOS-1410), the lifetime of tasks in Mesos will look as
> follows:
> {code}
> Master Slave
> {} {}
> {Tn} {} // Master receives Task T, non-terminal. Forwards to
> slave.
> {Tn} {Tn} // Slave receives Task T, non-terminal.
> {Tn} {Tt} // Task becomes terminal on slave. Update forwarded.
> {Tt} {Tt} // Master receives update, forwards to framework.
> {} {Tt} // Master receives ack, forwards to slave.
> {} {} // Slave receives ack.
> {code}
> In the current form of reconciliation, the slave sends to the master all
> tasks that are not both terminal and acknowledged. At any point in the above
> lifecycle, the slave's re-registration message can reach the master.
> Note the following properties:
> *(1)* The master may have a non-terminal task, not present in the slave's
> re-registration message.
> *(2)* The master may have a non-terminal task, present in the slave's
> re-registration message but in a different state.
> *(3)* The slave's re-registration message may contain a terminal
> unacknowledged task unknown to the master.
> In the current master / slave
> [reconciliation|https://github.com/apache/mesos/blob/0.19.1/src/master/master.cpp#L3146]
> code, the master assumes that case (1) is because a launch task message was
> dropped, and it sends TASK_LOST. We've seen above that (1) can happen even
> when the task reaches the slave correctly, so this can lead to inconsistency!
> After chatting with [~vinodkone], we're considering updating the
> reconciliation to occur as follows:
> → Slave sends all tasks that are not both terminal and acknowledged, during
> re-registration. This is the same as before.
> → If the master sees tasks that are missing in the slave, the master sends a
> reconcile message to the slave for the tasks.
> → The slave will reply to reconcile messages with the latest state, or
> TASK_LOST if the task is not known to it. Preferably in a retried manner,
> unless we update socket closure on the slave to force a re-registration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-1696) Improve reconciliation between master and slave.
[
https://issues.apache.org/jira/browse/MESOS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kone updated MESOS-1696:
--
Sprint: Mesos Q3 Sprint 5
Story Points: 3
> Improve reconciliation between master and slave.
>
>
> Key: MESOS-1696
> URL: https://issues.apache.org/jira/browse/MESOS-1696
> Project: Mesos
> Issue Type: Bug
> Components: master, slave
>Reporter: Benjamin Mahler
>Assignee: Vinod Kone
>
> As we update the Master to keep tasks in memory until they are both terminal
> and acknowledged (MESOS-1410), the lifetime of tasks in Mesos will look as
> follows:
> {code}
> Master Slave
> {} {}
> {Tn} {} // Master receives Task T, non-terminal. Forwards to
> slave.
> {Tn} {Tn} // Slave receives Task T, non-terminal.
> {Tn} {Tt} // Task becomes terminal on slave. Update forwarded.
> {Tt} {Tt} // Master receives update, forwards to framework.
> {} {Tt} // Master receives ack, forwards to slave.
> {} {} // Slave receives ack.
> {code}
> In the current form of reconciliation, the slave sends to the master all
> tasks that are not both terminal and acknowledged. At any point in the above
> lifecycle, the slave's re-registration message can reach the master.
> Note the following properties:
> *(1)* The master may have a non-terminal task, not present in the slave's
> re-registration message.
> *(2)* The master may have a non-terminal task, present in the slave's
> re-registration message but in a different state.
> *(3)* The slave's re-registration message may contain a terminal
> unacknowledged task unknown to the master.
> In the current master / slave
> [reconciliation|https://github.com/apache/mesos/blob/0.19.1/src/master/master.cpp#L3146]
> code, the master assumes that case (1) is because a launch task message was
> dropped, and it sends TASK_LOST. We've seen above that (1) can happen even
> when the task reaches the slave correctly, so this can lead to inconsistency!
> After chatting with [~vinodkone], we're considering updating the
> reconciliation to occur as follows:
> → Slave sends all tasks that are not both terminal and acknowledged, during
> re-registration. This is the same as before.
> → If the master sees tasks that are missing in the slave, the master sends a
> reconcile message to the slave for the tasks.
> → The slave will reply to reconcile messages with the latest state, or
> TASK_LOST if the task is not known to it. Preferably in a retried manner,
> unless we update socket closure on the slave to force a re-registration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-1787) Expose historical executor statistics / time series data over slave endpoint
Niklas Quarfot Nielsen created MESOS-1787: - Summary: Expose historical executor statistics / time series data over slave endpoint Key: MESOS-1787 URL: https://issues.apache.org/jira/browse/MESOS-1787 Project: Mesos Issue Type: Improvement Components: slave Reporter: Niklas Quarfot Nielsen Per https://github.com/apache/mesos/blob/master/src/slave/monitor.hpp#L118 , we already collect historical statistics but do not expose them over the monitor http endpoints yet. As [~bmahler] suggested, this would involve adding archive.json, presenting them (https://github.com/apache/mesos/blob/master/src/slave/monitor.cpp#L223) and preferably ways to query the available samples (along the lines of https://github.com/apache/mesos/blob/master/src/master/http.cpp#L776) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1621) Docker run networking should be configurable and support bridge network
[ https://issues.apache.org/jira/browse/MESOS-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130719#comment-14130719 ] Thomas Hoppe commented on MESOS-1621: - Thx! > Docker run networking should be configurable and support bridge network > --- > > Key: MESOS-1621 > URL: https://issues.apache.org/jira/browse/MESOS-1621 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: Timothy Chen >Assignee: Timothy Chen > Labels: Docker > > Currently to easily support running executors in Docker image, we hardcode > --net=host into Docker run so slave and executor and reuse the same mechanism > to communicate, which is to pass the slave IP/PORT for the framework to > respond with it's own hostname and port information back to setup the tunnel. > We want to see how to abstract this or even get rid of host networking > altogether if we have a good way to not rely on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1675) Decouple version of the mesos library from the package release version
[ https://issues.apache.org/jira/browse/MESOS-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130653#comment-14130653 ] Timothy St. Clair commented on MESOS-1675: -- This is pretty much what http://reviews.apache.org/r/25551/ starts. However the release wrangler will need to add a step to their punch-list prior to adoption. > Decouple version of the mesos library from the package release version > -- > > Key: MESOS-1675 > URL: https://issues.apache.org/jira/browse/MESOS-1675 > Project: Mesos > Issue Type: Bug >Reporter: Vinod Kone > > This discussion should be rolled into the larger discussion around how to > version Mesos (APIs, packages, libraries etc). > Some notes from libtool docs. > http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html > http://www.gnu.org/software/libtool/manual/html_node/Release-numbers.html#Release-numbers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1621) Docker run networking should be configurable and support bridge network
[ https://issues.apache.org/jira/browse/MESOS-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130627#comment-14130627 ] Connor Doyle commented on MESOS-1621: - [~thomas] Here is a link to the associated issue in Marathon if you'd like to follow along: https://github.com/mesosphere/marathon/issues/587 > Docker run networking should be configurable and support bridge network > --- > > Key: MESOS-1621 > URL: https://issues.apache.org/jira/browse/MESOS-1621 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: Timothy Chen >Assignee: Timothy Chen > Labels: Docker > > Currently to easily support running executors in Docker image, we hardcode > --net=host into Docker run so slave and executor and reuse the same mechanism > to communicate, which is to pass the slave IP/PORT for the framework to > respond with it's own hostname and port information back to setup the tunnel. > We want to see how to abstract this or even get rid of host networking > altogether if we have a good way to not rely on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1764) Build Fixes from 0.20 release
[ https://issues.apache.org/jira/browse/MESOS-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130604#comment-14130604 ] Timothy St. Clair commented on MESOS-1764: -- add initial -version-info for shared library http://reviews.apache.org/r/25551/ > Build Fixes from 0.20 release > - > > Key: MESOS-1764 > URL: https://issues.apache.org/jira/browse/MESOS-1764 > Project: Mesos > Issue Type: Bug > Components: build >Affects Versions: 0.20.0 >Reporter: Timothy St. Clair >Assignee: Timothy St. Clair > > This ticket is a catch all for minor issues caught during a rebase and > testing. > + Add package configuration file to deployment > + Updates deploy_dir from localstatedir to sysconfdir -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-1765) Use PID namespace to avoid freezing cgroup
[ https://issues.apache.org/jira/browse/MESOS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Downes reassigned MESOS-1765: - Assignee: Ian Downes (was: Vinod Kone) > Use PID namespace to avoid freezing cgroup > -- > > Key: MESOS-1765 > URL: https://issues.apache.org/jira/browse/MESOS-1765 > Project: Mesos > Issue Type: Story > Components: containerization >Reporter: Cong Wang >Assignee: Ian Downes > > There is some known kernel issue when we freeze the whole cgroup upon OOM. > Mesos probably can just use PID namespace so that we will only need to kill > the "init" of the pid namespace, instead of freezing all the processes and > killing them one by one. But I am not quite sure if this would break the > existing code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1621) Docker run networking should be configurable and support bridge network
[ https://issues.apache.org/jira/browse/MESOS-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130546#comment-14130546 ] Timothy Chen commented on MESOS-1621: - [~megh] once the task is finished with an offer all the resources that is associated with the task is automatically reaccounted back to the master, so yes it will be released. > Docker run networking should be configurable and support bridge network > --- > > Key: MESOS-1621 > URL: https://issues.apache.org/jira/browse/MESOS-1621 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: Timothy Chen >Assignee: Timothy Chen > Labels: Docker > > Currently to easily support running executors in Docker image, we hardcode > --net=host into Docker run so slave and executor and reuse the same mechanism > to communicate, which is to pass the slave IP/PORT for the framework to > respond with it's own hostname and port information back to setup the tunnel. > We want to see how to abstract this or even get rid of host networking > altogether if we have a good way to not rely on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MESOS-1740) Bad error message when docker containerizer isn't enabled
[
https://issues.apache.org/jira/browse/MESOS-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Chen resolved MESOS-1740.
-
Resolution: Fixed
> Bad error message when docker containerizer isn't enabled
> -
>
> Key: MESOS-1740
> URL: https://issues.apache.org/jira/browse/MESOS-1740
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Jay Buffington
>Assignee: Timothy Chen
>Priority: Minor
> Labels: docker
>
> If I set container in TaskInfo's executor (aka DockerInfo) but I do not start
> the slave with {{--containerizer=docker,...}} then I get this error message
> in the log:
> {noformat}
> E0827 17:53:16.422735 20090 slave.cpp:2491] Container 'xxx' for executor
> 'yyy' of framework 'zzz' failed to start: TaskInfo/ExecutorInfo not supported
> {noformat}
> A better error message would have been:
> {noformat}
> No enabled containerizers could create a container for the provided
> TaskInfo/ExecutorInfo message. Enabled containerizers are: mesos.
> {noformat}
> An even better error message would have been:
> {noformat}
> DockerInfo was sent, but docker containerizer is not enabled. Try adding
> --containerizer=docker,... to command line args
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-1761) docker containerizer should override entrypoint when starting executor.
[
https://issues.apache.org/jira/browse/MESOS-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130509#comment-14130509
]
Timothy Chen commented on MESOS-1761:
-
Currently I'm changing the docker containerizer to override entrypoint when
shell is enabled from the CommandInfo, and I believe that should be suffice.
Override --entrypoint on ExecutorInfo seems more limiting to me, as it should
be valid for a image to just to use the default entrypoint.
> docker containerizer should override entrypoint when starting executor.
> ---
>
> Key: MESOS-1761
> URL: https://issues.apache.org/jira/browse/MESOS-1761
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Jay Buffington
>Assignee: Timothy Chen
> Labels: docker
>
> When I specify an ExecutorInfo with a ContainerInfo set, the docker
> containerizer will do {{docker run ... /bin/sh -c ./my-executor}}.
> If the docker image the user specifies has an entrypoint configured the
> executor command will be appended to the entrypoint rather than overwriting
> it. The docker docs says:
> {quote}
> Unlike the behavior of the CMD instruction, The ENTRYPOINT instruction adds
> an entry command that will not be overwritten when arguments are passed to
> docker run. This allows arguments to be passed to the entry point, i.e.
> docker run -d will pass the -d argument to the entry point.
> {quote}
> This results in docker running the entrypoint inside the container with the
> executorinfo's command as an argument. If you specified an executor this is
> never what you want.
> I suggest docker run use {{--entrypoint}} when starting an executor inside
> the container.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-1761) docker containerizer should override entrypoint when starting executor.
[
https://issues.apache.org/jira/browse/MESOS-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130510#comment-14130510
]
Timothy Chen commented on MESOS-1761:
-
Btw MESOS-1770 is the ticket for changing the entrypoint.
> docker containerizer should override entrypoint when starting executor.
> ---
>
> Key: MESOS-1761
> URL: https://issues.apache.org/jira/browse/MESOS-1761
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Jay Buffington
>Assignee: Timothy Chen
> Labels: docker
>
> When I specify an ExecutorInfo with a ContainerInfo set, the docker
> containerizer will do {{docker run ... /bin/sh -c ./my-executor}}.
> If the docker image the user specifies has an entrypoint configured the
> executor command will be appended to the entrypoint rather than overwriting
> it. The docker docs says:
> {quote}
> Unlike the behavior of the CMD instruction, The ENTRYPOINT instruction adds
> an entry command that will not be overwritten when arguments are passed to
> docker run. This allows arguments to be passed to the entry point, i.e.
> docker run -d will pass the -d argument to the entry point.
> {quote}
> This results in docker running the entrypoint inside the container with the
> executorinfo's command as an argument. If you specified an executor this is
> never what you want.
> I suggest docker run use {{--entrypoint}} when starting an executor inside
> the container.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-1586) Isolate system directories, e.g., per-container /tmp
[ https://issues.apache.org/jira/browse/MESOS-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130484#comment-14130484 ] Ian Downes commented on MESOS-1586: --- Updated with new review: https://reviews.apache.org/r/25549/ > Isolate system directories, e.g., per-container /tmp > > > Key: MESOS-1586 > URL: https://issues.apache.org/jira/browse/MESOS-1586 > Project: Mesos > Issue Type: Improvement > Components: isolation >Affects Versions: 0.20.0 >Reporter: Ian Downes >Assignee: Ian Downes > > Ideally, tasks should not write outside their sandbox (executor work > directory) but pragmatically they may need to write to /tmp, /var/tmp, or > some other directory. > 1) We should include any such files in disk usage and quota. > 2) We should make these "shared" directories private, i.e., each container > has their own. > 3) We should make the lifetime of any such files the same as the executor > work directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1688) No offers if no memory is allocatable
[
https://issues.apache.org/jira/browse/MESOS-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dominic Hamon updated MESOS-1688:
-
Shepherd: Vinod Kone
> No offers if no memory is allocatable
> -
>
> Key: MESOS-1688
> URL: https://issues.apache.org/jira/browse/MESOS-1688
> Project: Mesos
> Issue Type: Bug
> Components: master
>Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1
>Reporter: Martin Weindel
>Priority: Critical
>
> The [Spark
> scheduler|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala]
> allocates memory only for the executor and cpu only for its tasks.
> So it can happen that all memory is nearly completely allocated by Spark
> executors, but all cpu resources are idle.
> In this case Mesos does not offer resources anymore, as less than MIN_MEM
> (=32MB) memory is allocatable.
> This effectively causes a dead lock in the Spark job, as it is not offered
> cpu resources needed for launching new tasks.
> see {{HierarchicalAllocatorProcess::allocatable(const Resources&)}} called in
> {{HierarchicalAllocatorProcess::allocate(const hashset&)}}
> {code}
> template
> bool
> HierarchicalAllocatorProcess::allocatable(
> const Resources& resources)
> {
> ...
> Option cpus = resources.cpus();
> Option mem = resources.mem();
> if (cpus.isSome() && mem.isSome()) {
> return cpus.get() >= MIN_CPUS && mem.get() > MIN_MEM;
> }
> return false;
> }
> {code}
> A possible solution may to completely drop the condition on allocatable
> memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-1688) No offers if no memory is allocatable
[
https://issues.apache.org/jira/browse/MESOS-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130279#comment-14130279
]
Dominic Hamon commented on MESOS-1688:
--
https://reviews.apache.org/r/25035
> No offers if no memory is allocatable
> -
>
> Key: MESOS-1688
> URL: https://issues.apache.org/jira/browse/MESOS-1688
> Project: Mesos
> Issue Type: Bug
> Components: master
>Affects Versions: 0.18.1, 0.18.2, 0.19.0, 0.19.1
>Reporter: Martin Weindel
>Priority: Critical
>
> The [Spark
> scheduler|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala]
> allocates memory only for the executor and cpu only for its tasks.
> So it can happen that all memory is nearly completely allocated by Spark
> executors, but all cpu resources are idle.
> In this case Mesos does not offer resources anymore, as less than MIN_MEM
> (=32MB) memory is allocatable.
> This effectively causes a dead lock in the Spark job, as it is not offered
> cpu resources needed for launching new tasks.
> see {{HierarchicalAllocatorProcess::allocatable(const Resources&)}} called in
> {{HierarchicalAllocatorProcess::allocate(const hashset&)}}
> {code}
> template
> bool
> HierarchicalAllocatorProcess::allocatable(
> const Resources& resources)
> {
> ...
> Option cpus = resources.cpus();
> Option mem = resources.mem();
> if (cpus.isSome() && mem.isSome()) {
> return cpus.get() >= MIN_CPUS && mem.get() > MIN_MEM;
> }
> return false;
> }
> {code}
> A possible solution may to completely drop the condition on allocatable
> memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
