[jira] [Created] (MESOS-9891) Add AGENT_UPDATED event with drain state.
Greg Mann created MESOS-9891: Summary: Add AGENT_UPDATED event with drain state. Key: MESOS-9891 URL: https://issues.apache.org/jira/browse/MESOS-9891 Project: Mesos Issue Type: Task Reporter: Greg Mann The master's event stream for schedulers and operators should get a new event whenever an agent's drain or deactivation state changes: {code} message Event { . . . enum Type { . . . AGENT_UPDATED = 10; } message AgentUpdated { optional bool deactivated = 1; optional DrainInfo drain_info = 2; } . . . optional AgentUpdated agent_updated = 10; } {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (MESOS-9816) Add draining state information to master event stream and state endpoints
[ https://issues.apache.org/jira/browse/MESOS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Mann reassigned MESOS-9816: Assignee: Greg Mann (was: Joseph Wu) > Add draining state information to master event stream and state endpoints > - > > Key: MESOS-9816 > URL: https://issues.apache.org/jira/browse/MESOS-9816 > Project: Mesos > Issue Type: Task > Components: master >Reporter: Joseph Wu >Assignee: Greg Mann >Priority: Major > Labels: foundations, mesosphere > > The response for {{GET_STATE}} and {{GET_AGENTS}} should include the new > fields indicating deactivation or draining states: > {code} > message Response { > . . . > message GetAgents { > message Agent { > . . . > optional bool deactivated = 12; > optional DrainInfo drain_info = 13; > . . . > } > } > . . . > } > {code} > Additionally, the master's event stream should get a new event whenever these > states change: > {code} > message Event { > . . . > enum Type { > . . . > AGENT_UPDATED = 10; > } > message AgentUpdated { > optional bool deactivated = 1; > optional DrainInfo drain_info = 2; > } > . . . > optional AgentUpdated agent_updated = 10; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (MESOS-6566) The Docker executor should not leak task env variables in the Docker command cmd line.
[ https://issues.apache.org/jira/browse/MESOS-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kone reassigned MESOS-6566: - Assignee: (was: Till Toenshoff) > The Docker executor should not leak task env variables in the Docker command > cmd line. > -- > > Key: MESOS-6566 > URL: https://issues.apache.org/jira/browse/MESOS-6566 > Project: Mesos > Issue Type: Bug > Components: docker, security >Reporter: Gastón Kleiman >Priority: Major > Labels: containerization > > Task environment variables are sensitive, as they might contain secrets. > The Docker executor starts tasks by executing a {{docker run}} command, and > it includes the env variables in the cmd line of the docker command, exposing > them to all the users in the machine: > {code} > $ ./src/mesos-execute --command="sleep 200" --containerizer=docker > --docker_image=alpine --env='{"foo": "bar"}' --master=10.0.2.15:5050 > --name=test > $ ps aux | grep bar > [...] docker -H unix:///var/run/docker.sock run [...] -e foo=bar [...] alpine > -c sleep 200 > $ > {code} > The Docker executor could pass Docker the {{--env-file}} flag, pointing it to > a file with the environment variables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (MESOS-9890) /roles and GET_ROLES does not always expose parent roles.
Benjamin Mahler created MESOS-9890: -- Summary: /roles and GET_ROLES does not always expose parent roles. Key: MESOS-9890 URL: https://issues.apache.org/jira/browse/MESOS-9890 Project: Mesos Issue Type: Bug Reporter: Benjamin Mahler If some descendant roles are present in frameworks, then the parent roles will not be exposed in the /roles and GET_ROLES endpoints. This is because the tracking is currently based on frameworks being subscribed to the role. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (MESOS-9889) Master CPU high due to unexpected foreachkey behaviour in Master::__reregisterSlave
haosdent created MESOS-9889: --- Summary: Master CPU high due to unexpected foreachkey behaviour in Master::__reregisterSlave Key: MESOS-9889 URL: https://issues.apache.org/jira/browse/MESOS-9889 Project: Mesos Issue Type: Bug Reporter: haosdent At https://github.com/apache/mesos/blob/9932550e9632e7fbb9a45b217793c7f508f57001/src/master/master.cpp#L7707-L7708 {code} void Master::__reregisterSlave( ... foreachkey (FrameworkID frameworkId, slaves.unreachableTasks.at(slaveInfo.id())) { ... foreach (TaskID taskId, slaves.unreachableTasks.at(slaveInfo.id()).get(frameworkId)) { {code} Our case is when network flapping, 3~4 agents reregister, then master would CPU full and could not process any requests during that period. After change {code} -foreachkey (FrameworkID frameworkId, - slaves.unreachableTasks.at(slaveInfo.id())) { +foreach (FrameworkID frameworkId, + slaves.unreachableTasks.at(slaveInfo.id()).keys()) { {code} The problem gone. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[ https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884055#comment-16884055 ] haosdent commented on MESOS-5037: - [~bmahler] Create at https://issues.apache.org/jira/browse/MESOS-9889 > foreachkey behaviour is not expected in multimap > > > Key: MESOS-5037 > URL: https://issues.apache.org/jira/browse/MESOS-5037 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: haosdent >Priority: Major > Labels: foundations, stout > > Currently the {{foreachkey}} implementation is > {code} > #define foreachkey(VAR, COL)\ > foreachpair (VAR, __foreach__::ignore, COL) > {code} > This works in most structures. But in multimap, one key may map to multi > values. This means there are multi pairs which have same key. So when call > {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve > this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method > exists in {{COL}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[ https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884051#comment-16884051 ] haosdent commented on MESOS-5037: - [~bmahler] No problem. > foreachkey behaviour is not expected in multimap > > > Key: MESOS-5037 > URL: https://issues.apache.org/jira/browse/MESOS-5037 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: haosdent >Priority: Major > Labels: foundations, stout > > Currently the {{foreachkey}} implementation is > {code} > #define foreachkey(VAR, COL)\ > foreachpair (VAR, __foreach__::ignore, COL) > {code} > This works in most structures. But in multimap, one key may map to multi > values. This means there are multi pairs which have same key. So when call > {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve > this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method > exists in {{COL}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[ https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884048#comment-16884048 ] Benjamin Mahler commented on MESOS-5037: [~haosd...@gmail.com] Can you file a separate ticket for the performance problem? And we can keep this ticket as a foreachkey issue? > foreachkey behaviour is not expected in multimap > > > Key: MESOS-5037 > URL: https://issues.apache.org/jira/browse/MESOS-5037 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: haosdent >Priority: Major > Labels: foundations, stout > > Currently the {{foreachkey}} implementation is > {code} > #define foreachkey(VAR, COL)\ > foreachpair (VAR, __foreach__::ignore, COL) > {code} > This works in most structures. But in multimap, one key may map to multi > values. This means there are multi pairs which have same key. So when call > {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve > this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method > exists in {{COL}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (MESOS-5037) foreachkey behaviour is not expected in multimap
[ https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884044#comment-16884044 ] Benjamin Mahler edited comment on MESOS-5037 at 7/12/19 5:59 PM: - [~bmahler] Sure, it is https://github.com/apache/mesos/blob/9932550e9632e7fbb9a45b217793c7f508f57001/src/master/master.cpp#L7707-L7708 {code} void Master::__reregisterSlave( ... foreachkey (FrameworkID frameworkId, slaves.unreachableTasks.at(slaveInfo.id())) { ... foreach (TaskID taskId, slaves.unreachableTasks.at(slaveInfo.id()).get(frameworkId)) { {code} Our case is when network flapping, 3~4 agents reregister, then master would CPU full and could not process any requests during that period. was (Author: haosd...@gmail.com): [~bmahler] Sure, it is https://github.com/apache/mesos/blob/master/src/master/master.cpp#L7707-L7708 {code} void Master::__reregisterSlave( ... foreachkey (FrameworkID frameworkId, slaves.unreachableTasks.at(slaveInfo.id())) { ... foreach (TaskID taskId, slaves.unreachableTasks.at(slaveInfo.id()).get(frameworkId)) { {code} Our case is when network flapping, 3~4 agents reregister, then master would CPU full and could not process any requests during that period. > foreachkey behaviour is not expected in multimap > > > Key: MESOS-5037 > URL: https://issues.apache.org/jira/browse/MESOS-5037 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: haosdent >Priority: Major > Labels: foundations, stout > > Currently the {{foreachkey}} implementation is > {code} > #define foreachkey(VAR, COL)\ > foreachpair (VAR, __foreach__::ignore, COL) > {code} > This works in most structures. But in multimap, one key may map to multi > values. This means there are multi pairs which have same key. So when call > {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve > this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method > exists in {{COL}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[ https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884045#comment-16884045 ] haosdent commented on MESOS-5037: - After change {code} -foreachkey (FrameworkID frameworkId, - slaves.unreachableTasks.at(slaveInfo.id())) { +foreach (FrameworkID frameworkId, + slaves.unreachableTasks.at(slaveInfo.id()).keys()) { {code} The problem gone. > foreachkey behaviour is not expected in multimap > > > Key: MESOS-5037 > URL: https://issues.apache.org/jira/browse/MESOS-5037 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: haosdent >Priority: Major > Labels: foundations, stout > > Currently the {{foreachkey}} implementation is > {code} > #define foreachkey(VAR, COL)\ > foreachpair (VAR, __foreach__::ignore, COL) > {code} > This works in most structures. But in multimap, one key may map to multi > values. This means there are multi pairs which have same key. So when call > {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve > this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method > exists in {{COL}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[ https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884044#comment-16884044 ] haosdent commented on MESOS-5037: - [~bmahler] Sure, it is https://github.com/apache/mesos/blob/master/src/master/master.cpp#L7707-L7708 {code} void Master::__reregisterSlave( ... foreachkey (FrameworkID frameworkId, slaves.unreachableTasks.at(slaveInfo.id())) { ... foreach (TaskID taskId, slaves.unreachableTasks.at(slaveInfo.id()).get(frameworkId)) { {code} Our case is when network flapping, 3~4 agents reregister, then master would CPU full and could not process any requests during that period. > foreachkey behaviour is not expected in multimap > > > Key: MESOS-5037 > URL: https://issues.apache.org/jira/browse/MESOS-5037 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: haosdent >Priority: Major > Labels: foundations, stout > > Currently the {{foreachkey}} implementation is > {code} > #define foreachkey(VAR, COL)\ > foreachpair (VAR, __foreach__::ignore, COL) > {code} > This works in most structures. But in multimap, one key may map to multi > values. This means there are multi pairs which have same key. So when call > {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve > this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method > exists in {{COL}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[ https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884011#comment-16884011 ] Benjamin Mahler commented on MESOS-5037: [~haosd...@gmail.com] can you post a link to the code in question? > foreachkey behaviour is not expected in multimap > > > Key: MESOS-5037 > URL: https://issues.apache.org/jira/browse/MESOS-5037 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: haosdent >Priority: Major > Labels: foundations, stout > > Currently the {{foreachkey}} implementation is > {code} > #define foreachkey(VAR, COL)\ > foreachpair (VAR, __foreach__::ignore, COL) > {code} > This works in most structures. But in multimap, one key may map to multi > values. This means there are multi pairs which have same key. So when call > {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve > this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method > exists in {{COL}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-6566) The Docker executor should not leak task env variables in the Docker command cmd line.
[ https://issues.apache.org/jira/browse/MESOS-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883994#comment-16883994 ] Vinod Kone commented on MESOS-6566: --- See the description in MESOS-6951 for a potential solution using `--env` argument. > The Docker executor should not leak task env variables in the Docker command > cmd line. > -- > > Key: MESOS-6566 > URL: https://issues.apache.org/jira/browse/MESOS-6566 > Project: Mesos > Issue Type: Bug > Components: docker, security >Reporter: Gastón Kleiman >Assignee: Till Toenshoff >Priority: Major > > Task environment variables are sensitive, as they might contain secrets. > The Docker executor starts tasks by executing a {{docker run}} command, and > it includes the env variables in the cmd line of the docker command, exposing > them to all the users in the machine: > {code} > $ ./src/mesos-execute --command="sleep 200" --containerizer=docker > --docker_image=alpine --env='{"foo": "bar"}' --master=10.0.2.15:5050 > --name=test > $ ps aux | grep bar > [...] docker -H unix:///var/run/docker.sock run [...] -e foo=bar [...] alpine > -c sleep 200 > $ > {code} > The Docker executor could pass Docker the {{--env-file}} flag, pointing it to > a file with the environment variables. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[ https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883939#comment-16883939 ] haosdent commented on MESOS-5037: - [~bmahler] Sry for the delay. The context is we have an agent which has 1k tasks, and when the agent reregister, it would trigger 1000,000 rounds on this loop. I have a flamegraph last week, but it is cleared by incident. > foreachkey behaviour is not expected in multimap > > > Key: MESOS-5037 > URL: https://issues.apache.org/jira/browse/MESOS-5037 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: haosdent >Priority: Major > Labels: foundations, stout > > Currently the {{foreachkey}} implementation is > {code} > #define foreachkey(VAR, COL)\ > foreachpair (VAR, __foreach__::ignore, COL) > {code} > This works in most structures. But in multimap, one key may map to multi > values. This means there are multi pairs which have same key. So when call > {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve > this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method > exists in {{COL}}. -- This message was sent by Atlassian JIRA (v7.6.14#76016)