[jira] [Assigned] (MESOS-7416) Filter results of `/master/slaves` and the v1 call GET_AGENTS
[ https://issues.apache.org/jira/browse/MESOS-7416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rojas reassigned MESOS-7416: -- Assignee: Alexander Rojas > Filter results of `/master/slaves` and the v1 call GET_AGENTS > - > > Key: MESOS-7416 > URL: https://issues.apache.org/jira/browse/MESOS-7416 > Project: Mesos > Issue Type: Task > Components: HTTP API, master >Reporter: Alexander Rojas >Assignee: Alexander Rojas > Labels: mesosphere, security > > The results returned by both the endpoint {{/master/slaves}} and the API v1 > {{GET_AGENTS}} return full information about the agent state which probably > need to be filtered for certain uses, particularly in a multi-tenancy > scenario. > The kind of leaked data includes specific role names and their specific > allocations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (MESOS-7415) Add authorization to master's operator maintenance API in v0 and v1
[ https://issues.apache.org/jira/browse/MESOS-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rojas reassigned MESOS-7415: -- Assignee: Alexander Rojas > Add authorization to master's operator maintenance API in v0 and v1 > --- > > Key: MESOS-7415 > URL: https://issues.apache.org/jira/browse/MESOS-7415 > Project: Mesos > Issue Type: Task > Components: c++ api, HTTP API, master >Reporter: Alexander Rojas >Assignee: Alexander Rojas > Labels: authorization, mesosphere, security > > None of the maintenance primitives in either API v0 or API v1 have any kind > of authorization, which allows any user with valid credentials to do things > such as shutting down a machine, schedule time off on an agent, modify > maintenance schedule, etc. > The authorization support needs to be added to the v0 endpoints: > * {{/master/machine/up}} > * {{/master/machine/down}} > * {{/master/maintenance/schedule}} > * {{/master/maintenance/status}} > as well as to the v1 calls: > * {{GET_MAINTENANCE_STATUS}} > * {{GET_MAINTENANCE_SCHEDULE}} > * {{UPDATE_MAINTENANCE_SCHEDULE}} > * {{START_MAINTENANCE}} > * {{STOP_MAINTENANCE}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (MESOS-7414) Enable authorization for master's logging API calls: GET_LOGGING_LEVEL and SET_LOGGING_LEVEL
[ https://issues.apache.org/jira/browse/MESOS-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rojas reassigned MESOS-7414: -- Assignee: Alexander Rojas > Enable authorization for master's logging API calls: GET_LOGGING_LEVEL and > SET_LOGGING_LEVEL > - > > Key: MESOS-7414 > URL: https://issues.apache.org/jira/browse/MESOS-7414 > Project: Mesos > Issue Type: Task > Components: HTTP API, master >Reporter: Alexander Rojas >Assignee: Alexander Rojas > Labels: mesosphere, operator, security > > The Operator API calls {{GET_LOGGING_LEVEL}} and {{SET_LOGGING_LEVEL}}, as > well as the v0 endpoint {{/logging/toggle}} lack authorization so any > recognized user will be able to change the logging level of a given master. > Note that there are already actions defined for authorization of these > actions as they were already implemented in the agent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7416) Filter results of `/master/slaves` and the v1 call GET_AGENTS
[ https://issues.apache.org/jira/browse/MESOS-7416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-7416: -- Labels: mesosphere security (was: mesosphere) > Filter results of `/master/slaves` and the v1 call GET_AGENTS > - > > Key: MESOS-7416 > URL: https://issues.apache.org/jira/browse/MESOS-7416 > Project: Mesos > Issue Type: Task > Components: HTTP API, master >Reporter: Alexander Rojas > Labels: mesosphere, security > > The results returned by both the endpoint {{/master/slaves}} and the API v1 > {{GET_AGENTS}} return full information about the agent state which probably > need to be filtered for certain uses, particularly in a multi-tenancy > scenario. > The kind of leaked data includes specific role names and their specific > allocations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7416) Filter results of `/master/slaves` and the v1 call GET_AGENTS
[ https://issues.apache.org/jira/browse/MESOS-7416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-7416: -- Labels: mesosphere (was: ) > Filter results of `/master/slaves` and the v1 call GET_AGENTS > - > > Key: MESOS-7416 > URL: https://issues.apache.org/jira/browse/MESOS-7416 > Project: Mesos > Issue Type: Task > Components: HTTP API, master >Reporter: Alexander Rojas > Labels: mesosphere > > The results returned by both the endpoint {{/master/slaves}} and the API v1 > {{GET_AGENTS}} return full information about the agent state which probably > need to be filtered for certain uses, particularly in a multi-tenancy > scenario. > The kind of leaked data includes specific role names and their specific > allocations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7414) Enable authorization for master's logging API calls: GET_LOGGING_LEVEL and SET_LOGGING_LEVEL
[ https://issues.apache.org/jira/browse/MESOS-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-7414: -- Labels: mesosphere operator security (was: mesosphere operator) > Enable authorization for master's logging API calls: GET_LOGGING_LEVEL and > SET_LOGGING_LEVEL > - > > Key: MESOS-7414 > URL: https://issues.apache.org/jira/browse/MESOS-7414 > Project: Mesos > Issue Type: Task > Components: HTTP API, master >Reporter: Alexander Rojas > Labels: mesosphere, operator, security > > The Operator API calls {{GET_LOGGING_LEVEL}} and {{SET_LOGGING_LEVEL}}, as > well as the v0 endpoint {{/logging/toggle}} lack authorization so any > recognized user will be able to change the logging level of a given master. > Note that there are already actions defined for authorization of these > actions as they were already implemented in the agent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (MESOS-7423) Information on Mesos CI
[ https://issues.apache.org/jira/browse/MESOS-7423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988138#comment-15988138 ] Nayana Thorat edited comment on MESOS-7423 at 4/28/17 3:49 AM: --- I have subscribed for the d...@mesos.apache.org. Also, sent an email to d...@mesos.apache.org for CI support. was (Author: nayana): I have subscribed for the d...@mesos.apache.org. Also, sent an emial to d...@mesos.apache.org for CI support. > Information on Mesos CI > --- > > Key: MESOS-7423 > URL: https://issues.apache.org/jira/browse/MESOS-7423 > Project: Mesos > Issue Type: Task >Reporter: Nayana Thorat > > Hi Vinod, > We had raised an issue to add s390x support for mesos which was fixed and > resolved. > https://issues.apache.org/jira/browse/MESOS-6742 > We also want to know about Mesos CI. > We need following details about current Mesos CI: > 1. How is the current Mesos CI infrastructure? Travis/Jenkins? > 2. Can Mesos CI extended to support s390x systems? > We are not sure if this is right channel to discuss this topic. > Please let us know if you want to start this discussion on some other channel. > Thanks, -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7423) Information on Mesos CI
[ https://issues.apache.org/jira/browse/MESOS-7423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988138#comment-15988138 ] Nayana Thorat commented on MESOS-7423: -- I have subscribed for the d...@mesos.apache.org. Also, sent an emial to d...@mesos.apache.org for CI support. > Information on Mesos CI > --- > > Key: MESOS-7423 > URL: https://issues.apache.org/jira/browse/MESOS-7423 > Project: Mesos > Issue Type: Task >Reporter: Nayana Thorat > > Hi Vinod, > We had raised an issue to add s390x support for mesos which was fixed and > resolved. > https://issues.apache.org/jira/browse/MESOS-6742 > We also want to know about Mesos CI. > We need following details about current Mesos CI: > 1. How is the current Mesos CI infrastructure? Travis/Jenkins? > 2. Can Mesos CI extended to support s390x systems? > We are not sure if this is right channel to discuss this topic. > Please let us know if you want to start this discussion on some other channel. > Thanks, -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7228) Upgrade Mesos to build with proto3.
[ https://issues.apache.org/jira/browse/MESOS-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988106#comment-15988106 ] Jay Guo commented on MESOS-7228: [~zhitao] Thank you for clarifying it! I think I misinterpreted it as upgrading both bundle and syntax. Sounds good! > Upgrade Mesos to build with proto3. > --- > > Key: MESOS-7228 > URL: https://issues.apache.org/jira/browse/MESOS-7228 > Project: Mesos > Issue Type: Improvement >Reporter: Anand Mazumdar >Assignee: Zhitao Li >Priority: Critical > > We currently build Mesos with protobuf 2.6.1 and bundle it as a dependency. > We should upgrade it to use v3.2.0 instead. This would help us use arenas to > improve performance (MESOS-6971) and also help resolve some bugs around the > Mesos master not able to handle large protobufs (>64mbs in size, MESOS-4210) > etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MESOS-7434) SlaveTest.RestartSlaveRequireExecutorAuthentication is flaky
Greg Mann created MESOS-7434: Summary: SlaveTest.RestartSlaveRequireExecutorAuthentication is flaky Key: MESOS-7434 URL: https://issues.apache.org/jira/browse/MESOS-7434 Project: Mesos Issue Type: Bug Reporter: Greg Mann This test failure has been observed on an internal CI system. It occurs on a variety of Linux distributions. It seems that using {{cat}} as the task command may be problematic; see attached log file {{SlaveTest.RestartSlaveRequireExecutorAuthentication.txt}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7434) SlaveTest.RestartSlaveRequireExecutorAuthentication is flaky
[ https://issues.apache.org/jira/browse/MESOS-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987981#comment-15987981 ] Greg Mann commented on MESOS-7434: -- Potential fix here: https://reviews.apache.org/r/58754/ But I would like to have a better understanding of why this is occurring > SlaveTest.RestartSlaveRequireExecutorAuthentication is flaky > > > Key: MESOS-7434 > URL: https://issues.apache.org/jira/browse/MESOS-7434 > Project: Mesos > Issue Type: Bug >Reporter: Greg Mann >Assignee: Greg Mann > Labels: flaky, flaky-test, mesosphere > > This test failure has been observed on an internal CI system. It occurs on a > variety of Linux distributions. It seems that using {{cat}} as the task > command may be problematic; see attached log file > {{SlaveTest.RestartSlaveRequireExecutorAuthentication.txt}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (MESOS-7434) SlaveTest.RestartSlaveRequireExecutorAuthentication is flaky
[ https://issues.apache.org/jira/browse/MESOS-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Mann reassigned MESOS-7434: Assignee: Greg Mann > SlaveTest.RestartSlaveRequireExecutorAuthentication is flaky > > > Key: MESOS-7434 > URL: https://issues.apache.org/jira/browse/MESOS-7434 > Project: Mesos > Issue Type: Bug >Reporter: Greg Mann >Assignee: Greg Mann > Labels: flaky, flaky-test, mesosphere > > This test failure has been observed on an internal CI system. It occurs on a > variety of Linux distributions. It seems that using {{cat}} as the task > command may be problematic; see attached log file > {{SlaveTest.RestartSlaveRequireExecutorAuthentication.txt}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7340) Log HTTP accesses to the /files endpoint
[ https://issues.apache.org/jira/browse/MESOS-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987974#comment-15987974 ] Zhitao Li commented on MESOS-7340: -- Just spotted this story: totally +1 for making this configurable, maybe also include processing time of the entire HTTP request? > Log HTTP accesses to the /files endpoint > > > Key: MESOS-7340 > URL: https://issues.apache.org/jira/browse/MESOS-7340 > Project: Mesos > Issue Type: Bug > Components: HTTP API >Reporter: James Peach >Assignee: James Peach >Priority: Minor > > The Mesos master and agent log HTTP accesses, but the {{Files}} process does > not. We should log accessed to the {{/files}} endpoint in the same way. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7429) Allow isolators to inject task-specific environment variables.
[ https://issues.apache.org/jira/browse/MESOS-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987836#comment-15987836 ] James Peach commented on MESOS-7429: {quote} What we do for them already is passing the ContainerLaunchInfo returned environment on to the executor command environment execvpe. {quote} Do you mean that environment variables an isolator sets in {{ContainerLaunchInfo.task_environment}} will be placed into the environment of the custom executor? That sounds like the problem you are solving here. Though I can see it is not easy to solve for custom executors, I think this is a very useful capability to have for them. > Allow isolators to inject task-specific environment variables. > -- > > Key: MESOS-7429 > URL: https://issues.apache.org/jira/browse/MESOS-7429 > Project: Mesos > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Till Toenshoff >Assignee: Till Toenshoff > Labels: containerizer, isolator, mesosphere > > For command tasks, we currently have no way for isolators to inject task > specific environments into data path. > Isolators in their {{prepare}} implementation can return a > {{ContainerLaunchInfo}} which currently supports setting the member > {{environment}} - but that is effective for the executor already and will get > inherited by the task environment. > Given that the command-executor is active in the host-fs context but the task > may be running in a container-fs context ({{has_rootfs=true}}), using the > above will possibly cause failures, depending on the nature of the variables > set -- consider e.g. {{LD_LIBRARY_PATH}} as a fatal one. > The command-executor does support the flag {{task_environment}} to receive > environment variables which are meant exclusively for the task itself but not > the executor. > Runtime isolators (docker/runtime, appc/runtime) already make use of this > flag. However, they do it in a way that is a bit unfortunate as it does not > allow for other isolators to make use of it - there is no merging taking > place in the containerizer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (MESOS-7430) Per-role Suppress call implementation is broken.
[ https://issues.apache.org/jira/browse/MESOS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Mahler reassigned MESOS-7430: -- Assignee: Benjamin Mahler > Per-role Suppress call implementation is broken. > > > Key: MESOS-7430 > URL: https://issues.apache.org/jira/browse/MESOS-7430 > Project: Mesos > Issue Type: Bug > Components: allocation, master >Reporter: Benjamin Mahler >Assignee: Benjamin Mahler >Priority: Blocker > > The per-role Suppress call implementation is broken currently in the > allocator, since it still uses a global 'suppress' bit for the framework. > Before fixing, we should discuss whether we want keep role within Suppress > (it hasn't been released yet), or add calls to move towards consistent > naming, e.g. {{Call::ActivateRole}} / {{Call::DeactivateRole}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (MESOS-6782) Inherit Environment from parent container when launching DEBUG container.
[ https://issues.apache.org/jira/browse/MESOS-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961054#comment-15961054 ] Alexander Rukletsov edited comment on MESOS-6782 at 4/27/17 10:18 PM: -- https://reviews.apache.org/r/58262/ https://reviews.apache.org/r/58718/ was (Author: alexr): https://reviews.apache.org/r/58262/ > Inherit Environment from parent container when launching DEBUG container. > - > > Key: MESOS-6782 > URL: https://issues.apache.org/jira/browse/MESOS-6782 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Alexander Rukletsov > Labels: check, debugging, health-check, mesosphere > > Right now whenever we enter a DEBUG container we have a fresh environment. > For a better user experience, we should have the DEBUG container inherit the > environment set up in its parent container image spec (if there is one). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-6782) Inherit Environment from parent container when launching DEBUG container.
[ https://issues.apache.org/jira/browse/MESOS-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rukletsov updated MESOS-6782: --- Story Points: 3 (was: 1) > Inherit Environment from parent container when launching DEBUG container. > - > > Key: MESOS-6782 > URL: https://issues.apache.org/jira/browse/MESOS-6782 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Alexander Rukletsov > Labels: check, debugging, health-check, mesosphere > > Right now whenever we enter a DEBUG container we have a fresh environment. > For a better user experience, we should have the DEBUG container inherit the > environment set up in its parent container image spec (if there is one). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-6782) Inherit Environment from parent container when launching DEBUG container.
[ https://issues.apache.org/jira/browse/MESOS-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rukletsov updated MESOS-6782: --- Summary: Inherit Environment from parent container when launching DEBUG container. (was: Inherit Environment from Parent containers image spec when launching DEBUG container) > Inherit Environment from parent container when launching DEBUG container. > - > > Key: MESOS-6782 > URL: https://issues.apache.org/jira/browse/MESOS-6782 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Alexander Rukletsov > Labels: check, debugging, health-check, mesosphere > > Right now whenever we enter a DEBUG container we have a fresh environment. > For a better user experience, we should have the DEBUG container inherit the > environment set up in its parent container image spec (if there is one). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7433) Set working directory in DEBUG containers.
[ https://issues.apache.org/jira/browse/MESOS-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rukletsov updated MESOS-7433: --- Story Points: 3 Labels: check debugging health-check mesosphere (was: ) Description: Currently working directory is not set for DEBUG containers. The most reasonable value seems to be parent's working directory. > Set working directory in DEBUG containers. > -- > > Key: MESOS-7433 > URL: https://issues.apache.org/jira/browse/MESOS-7433 > Project: Mesos > Issue Type: Task > Components: agent, containerization >Reporter: Alexander Rukletsov >Assignee: Alexander Rukletsov > Labels: check, debugging, health-check, mesosphere > > Currently working directory is not set for DEBUG containers. The most > reasonable value seems to be parent's working directory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MESOS-7433) Set working directory in DEBUG containers.
Alexander Rukletsov created MESOS-7433: -- Summary: Set working directory in DEBUG containers. Key: MESOS-7433 URL: https://issues.apache.org/jira/browse/MESOS-7433 Project: Mesos Issue Type: Task Components: agent, containerization Reporter: Alexander Rukletsov Assignee: Alexander Rukletsov -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-6782) Inherit Environment from Parent containers image spec when launching DEBUG container
[ https://issues.apache.org/jira/browse/MESOS-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rukletsov updated MESOS-6782: --- Labels: check debugging health-check mesosphere (was: debugging mesosphere) > Inherit Environment from Parent containers image spec when launching DEBUG > container > > > Key: MESOS-6782 > URL: https://issues.apache.org/jira/browse/MESOS-6782 > Project: Mesos > Issue Type: Improvement >Reporter: Kevin Klues >Assignee: Alexander Rukletsov > Labels: check, debugging, health-check, mesosphere > > Right now whenever we enter a DEBUG container we have a fresh environment. > For a better user experience, we should have the DEBUG container inherit the > environment set up in its parent container image spec (if there is one). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7429) Allow isolators to inject task-specific environment variables.
[ https://issues.apache.org/jira/browse/MESOS-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987758#comment-15987758 ] Till Toenshoff commented on MESOS-7429: --- [~mcypark] that is perfectly fine. It is not a blocker and back porting, if really needed, will not be a problem. > Allow isolators to inject task-specific environment variables. > -- > > Key: MESOS-7429 > URL: https://issues.apache.org/jira/browse/MESOS-7429 > Project: Mesos > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Till Toenshoff >Assignee: Till Toenshoff > Labels: containerizer, isolator, mesosphere > > For command tasks, we currently have no way for isolators to inject task > specific environments into data path. > Isolators in their {{prepare}} implementation can return a > {{ContainerLaunchInfo}} which currently supports setting the member > {{environment}} - but that is effective for the executor already and will get > inherited by the task environment. > Given that the command-executor is active in the host-fs context but the task > may be running in a container-fs context ({{has_rootfs=true}}), using the > above will possibly cause failures, depending on the nature of the variables > set -- consider e.g. {{LD_LIBRARY_PATH}} as a fatal one. > The command-executor does support the flag {{task_environment}} to receive > environment variables which are meant exclusively for the task itself but not > the executor. > Runtime isolators (docker/runtime, appc/runtime) already make use of this > flag. However, they do it in a way that is a bit unfortunate as it does not > allow for other isolators to make use of it - there is no merging taking > place in the containerizer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (MESOS-7429) Allow isolators to inject task-specific environment variables.
[ https://issues.apache.org/jira/browse/MESOS-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987753#comment-15987753 ] Till Toenshoff edited comment on MESOS-7429 at 4/27/17 9:43 PM: [~jpe...@apache.org] Custom executors currently have no way of receiving task specific environments injected by isolators - isolators do no mutate the {{TaskInfo}}. What we do for them already is passing the {{ContainerLaunchInfo}} returned {{environment}} on to the executor command environment {{execvpe}}. The custom executor then has to control the passing on onto the task itself. was (Author: tillt): [~jpe...@apache.org] Custom executors currently have no way of receiving task specific environments injected by isolators - isolators do no mutate the {{TaskInfo}}. What we do for them already is passing the {{ContainerLaunchInfo}} returned {{environment}} on to the executor command environment {{execvpe}}. > Allow isolators to inject task-specific environment variables. > -- > > Key: MESOS-7429 > URL: https://issues.apache.org/jira/browse/MESOS-7429 > Project: Mesos > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Till Toenshoff >Assignee: Till Toenshoff > Labels: containerizer, isolator, mesosphere > > For command tasks, we currently have no way for isolators to inject task > specific environments into data path. > Isolators in their {{prepare}} implementation can return a > {{ContainerLaunchInfo}} which currently supports setting the member > {{environment}} - but that is effective for the executor already and will get > inherited by the task environment. > Given that the command-executor is active in the host-fs context but the task > may be running in a container-fs context ({{has_rootfs=true}}), using the > above will possibly cause failures, depending on the nature of the variables > set -- consider e.g. {{LD_LIBRARY_PATH}} as a fatal one. > The command-executor does support the flag {{task_environment}} to receive > environment variables which are meant exclusively for the task itself but not > the executor. > Runtime isolators (docker/runtime, appc/runtime) already make use of this > flag. However, they do it in a way that is a bit unfortunate as it does not > allow for other isolators to make use of it - there is no merging taking > place in the containerizer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7429) Allow isolators to inject task-specific environment variables.
[ https://issues.apache.org/jira/browse/MESOS-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987753#comment-15987753 ] Till Toenshoff commented on MESOS-7429: --- [~jpe...@apache.org] Custom executors currently have no way of receiving task specific environments injected by isolators - isolators do no mutate the {{TaskInfo}}. What we do for them already is passing the {{ContainerLaunchInfo}} returned {{environment}} on to the executor command environment {{execvpe}}. > Allow isolators to inject task-specific environment variables. > -- > > Key: MESOS-7429 > URL: https://issues.apache.org/jira/browse/MESOS-7429 > Project: Mesos > Issue Type: Improvement >Affects Versions: 1.2.0 >Reporter: Till Toenshoff >Assignee: Till Toenshoff > Labels: containerizer, isolator, mesosphere > > For command tasks, we currently have no way for isolators to inject task > specific environments into data path. > Isolators in their {{prepare}} implementation can return a > {{ContainerLaunchInfo}} which currently supports setting the member > {{environment}} - but that is effective for the executor already and will get > inherited by the task environment. > Given that the command-executor is active in the host-fs context but the task > may be running in a container-fs context ({{has_rootfs=true}}), using the > above will possibly cause failures, depending on the nature of the variables > set -- consider e.g. {{LD_LIBRARY_PATH}} as a fatal one. > The command-executor does support the flag {{task_environment}} to receive > environment variables which are meant exclusively for the task itself but not > the executor. > Runtime isolators (docker/runtime, appc/runtime) already make use of this > flag. However, they do it in a way that is a bit unfortunate as it does not > allow for other isolators to make use of it - there is no merging taking > place in the containerizer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-5967) Add support for 'docker image inspect' in our docker abstraction.
[ https://issues.apache.org/jira/browse/MESOS-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Mahler updated MESOS-5967: --- Target Version/s: (was: 1.4.0) > Add support for 'docker image inspect' in our docker abstraction. > - > > Key: MESOS-5967 > URL: https://issues.apache.org/jira/browse/MESOS-5967 > Project: Mesos > Issue Type: Improvement > Components: containerization, docker >Reporter: Kevin Klues >Assignee: Guangya Liu > Labels: gpu > > Docker's command line tool for {{docker inspect}} can take either a > {{container}}, an {{image}}, or a {{task}} as its argument, and return a JSON > array containing low-level information about that container, image or task. > However, the current {{docker inspect}} support in our docker abstraction > only supports inspecting containers (not images or tasks). We should expand > this to (at least) support images. > In particular, this additional functionality is motivated by the upcoming GPU > support, which needs to inspect the labels in a docker image to decide if it > should inject the required Nvidia volumes into a container. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-5918) Replace jsonp with a more secure alternative
[ https://issues.apache.org/jira/browse/MESOS-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987624#comment-15987624 ] Anand Mazumdar commented on MESOS-5918: --- We intend to move the Web UI eventually to use the v1 Operator API. When that happens, we won't be able to use jsonp at all owing to not being able to use {{POST}} requests. Based on previous discussions with UI folks, we did not want to use CORS due to security implications. Instead, the plan was to expose an endpoint on the master that would proxy requests to the agent (e.g., {{/forward}}). The endpoint would still be guarded by AuthN. See MESOS-5735 for more context. > Replace jsonp with a more secure alternative > > > Key: MESOS-5918 > URL: https://issues.apache.org/jira/browse/MESOS-5918 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Yan Xu > > We currently use the {{jsonp}} technique to bypass CORS check. This practice > has many security concerns (see discussions on MESOS-5911) so we should > replace it with a better alternative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (MESOS-5918) Replace jsonp with a more secure alternative
[ https://issues.apache.org/jira/browse/MESOS-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983568#comment-15983568 ] Jacob Janco edited comment on MESOS-5918 at 4/27/17 8:35 PM: - [~greggomann] [~anandmazumdar] [~mlunoe] [~xujyan] [~haosdent] Reopening a bit of discussion on replacing the jsonp workaround with CORS handling server side. An initial idea is to have a configurable regex for domains available for cross origin requests which will match against sent Origin headers. At this point I don't think we'll have to support preflighting requests to add this functionality. Another consideration, should this be a libprocess level configuration or perhaps a flag set on masters and agents? was (Author: jjanco): [~greggomann] [~anandmazumdar] [~mlunoe] [~xujyan] Reopening a bit of discussion on replacing the jsonp workaround with CORS handling server side. An initial idea is to have a configurable regex for domains available for cross origin requests which will match against sent Origin headers. At this point I don't think we'll have to support preflighting requests to add this functionality. Another consideration, should this be a libprocess level configuration or perhaps a flag set on masters and agents? > Replace jsonp with a more secure alternative > > > Key: MESOS-5918 > URL: https://issues.apache.org/jira/browse/MESOS-5918 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Yan Xu > > We currently use the {{jsonp}} technique to bypass CORS check. This practice > has many security concerns (see discussions on MESOS-5911) so we should > replace it with a better alternative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-5946) Consider supporting compression for the event stream.
[ https://issues.apache.org/jira/browse/MESOS-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987506#comment-15987506 ] Zhitao Li commented on MESOS-5946: -- Proposal: Similar to how [grpc handles compression | https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md#requests ], the {{api/v1}} endpoint will add support to a new optional header {{Message-Accept-Encoding}}. If this header is present with a supported value (initially only {{identity}} and {{gzip}}), each line in recordio response will be processed accordingly (no-op for {{identity}} which is the default case, and gzip-compressed for {{gzip}}). > Consider supporting compression for the event stream. > - > > Key: MESOS-5946 > URL: https://issues.apache.org/jira/browse/MESOS-5946 > Project: Mesos > Issue Type: Improvement >Reporter: Anand Mazumdar >Assignee: Zhitao Li > Labels: mesosphere > > Currently, we support GZIP compression for HTTP based responses in > Libprocess. However, for events streamed on the persistent connection > (RecordIO encoded) for schedulers/executors/subscribers, we don't yet have > support for compression. > We would need an implementation of GZIP/other compression technique that > would work efficiently for streaming data and is not fixed length based. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7106) Test ContentTypeAndSSLConfig/SchedulerSSLTest.RunTaskAndTeardown/1 segfaults
[ https://issues.apache.org/jira/browse/MESOS-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kone updated MESOS-7106: -- Sprint: Mesosphere Sprint 52, Mesosphere Sprint 53 (was: Mesosphere Sprint 52, Mesosphere Sprint 53, Mesosphere Sprint 55) > Test ContentTypeAndSSLConfig/SchedulerSSLTest.RunTaskAndTeardown/1 segfaults > > > Key: MESOS-7106 > URL: https://issues.apache.org/jira/browse/MESOS-7106 > Project: Mesos > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: centos7, SSL build >Reporter: Benjamin Bannier >Assignee: Joseph Wu > Labels: flaky-test, mesosphere, test > > {{ContentTypeAndSSLConfig/SchedulerSSLTest.RunTaskAndTeardown/1}} segfaulted > in our internal CI: > {noformat} > [ RUN ] ContentTypeAndSSLConfig/SchedulerSSLTest.RunTaskAndTeardown/1 > W0210 03:08:05.018744 1020 process.cpp:3029] Attempted to spawn a process > (__http_connection__(1079)@10.168.212.35:42363) after finalizing libprocess! > *** Aborted at 1486696085 (unix time) try "date -d @1486696085" if you are > using GNU date *** > I0210 03:08:05.023609 6019 process.cpp:1246] libprocess is initialized on > 10.168.212.35:44850 with 8 worker threads > I0210 03:08:05.024163 6019 cluster.cpp:160] Creating default 'local' > authorizer > I0210 03:08:05.025065 1025 master.cpp:383] Master > 7adcbe15-38a9-4512-aa9c-8d5f7538e4ee (ip-10-168-212-35.ec2.internal) started > on 10.168.212.35:44850 > I0210 03:08:05.025089 1025 master.cpp:385] Flags at startup: --acls="" > --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" > --allocation_interval="1secs" --allocator="HierarchicalDRF" > --authenticate_agents="true" --authenticate_frameworks="true" > --authenticate_http_frameworks="true" --authenticate_http_readonly="true" > --authenticate_http_readwrite="true" --authenticators="crammd5" > --authorizers="local" --credentials="/tmp/5DRa8u/credentials" > --framework_sorter="drf" --help="false" --hostname_lookup="true" > --http_authenticators="basic" --http_framework_authenticators="basic" > --initialize_driver_logging="true" --log_auto_initialize="true" > --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" > --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" > --max_unreachable_tasks_per_framework="1000" --quiet="false" > --recovery_agent_removal_limit="100%" --registry="in_memory" > --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" > --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" > --registry_store_timeout="100secs" --registry_strict="false" > --root_submissions="true" --user_sorter="drf" --version="false" > --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/5DRa8u/master" > --zk_session_timeout="10secs" > I0210 03:08:05.025264 1025 master.cpp:435] Master only allowing > authenticated frameworks to register > I0210 03:08:05.025276 1025 master.cpp:449] Master only allowing > authenticated agents to register > I0210 03:08:05.025285 1025 master.cpp:462] Master only allowing > authenticated HTTP frameworks to register > I0210 03:08:05.025293 1025 credentials.hpp:37] Loading credentials for > authentication from '/tmp/5DRa8u/credentials' > I0210 03:08:05.025387 1025 master.cpp:507] Using default 'crammd5' > authenticator > I0210 03:08:05.025441 1025 http.cpp:919] Using default 'basic' HTTP > authenticator for realm 'mesos-master-readonly' > I0210 03:08:05.025512 1025 http.cpp:919] Using default 'basic' HTTP > authenticator for realm 'mesos-master-readwrite' > I0210 03:08:05.025560 1025 http.cpp:919] Using default 'basic' HTTP > authenticator for realm 'mesos-master-scheduler' > I0210 03:08:05.025619 1025 master.cpp:587] Authorization enabled > I0210 03:08:05.025728 1023 hierarchical.cpp:161] Initialized hierarchical > allocator process > I0210 03:08:05.025754 1027 whitelist_watcher.cpp:77] No whitelist given > PC: @ 0x7f69d2296012 process::ProcessManager::spawn() > *** SIGSEGV (@0x0) received by PID 6019 (TID 0x7f69c46d5700) from PID 0; > stack trace: *** > @ 0x7f69c2408725 (unknown) > I0210 03:08:05.026340 1023 master.cpp:2124] Elected as the leading master! > I0210 03:08:05.026357 1023 master.cpp:1646] Recovering from registrar > I0210 03:08:05.026406 1025 registrar.cpp:329] Recovering registrar > @ 0x7f69c240d2f1 (unknown) > @ 0x7f69c24011e8 (unknown) > I0210 03:08:05.027294 1024 registrar.cpp:362] Successfully fetched the > registry (0B) in 865024ns > I0210 03:08:05.027330 1024 registrar.cpp:461] Applied 1 operations in > 2848ns; attempting to update the registry > @ 0x7f69d027b370 (unknown) > I0210 03:08:05.028261 1028 registrar.cpp:506] Successfully updated the > registry in 916992ns > I0210 03
[jira] [Updated] (MESOS-7355) Set MESOS_SANDBOX in debug containers.
[ https://issues.apache.org/jira/browse/MESOS-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Rukletsov updated MESOS-7355: --- Description: Currently {{MESOS_SANDBOX}} is not set for debug containers, see [https://github.com/apache/mesos/blob/7f04cf886fc2ed59414bf0056a2f351959a2d1f8/src/slave/containerizer/mesos/containerizer.cpp#L1392-L1407]. The most reasonable value seems to be task's sandbox. (was: Currently {{MESOS_SANDBOX}} is not set for debug containers, see [https://github.com/apache/mesos/blob/7f04cf886fc2ed59414bf0056a2f351959a2d1f8/src/slave/containerizer/mesos/containerizer.cpp#L1392-L1407]. The most reasonable value seem to be task's sandbox.) > Set MESOS_SANDBOX in debug containers. > -- > > Key: MESOS-7355 > URL: https://issues.apache.org/jira/browse/MESOS-7355 > Project: Mesos > Issue Type: Improvement > Components: agent, containerization >Reporter: Alexander Rukletsov >Assignee: Alexander Rukletsov > Labels: check, health-check, mesosphere > > Currently {{MESOS_SANDBOX}} is not set for debug containers, see > [https://github.com/apache/mesos/blob/7f04cf886fc2ed59414bf0056a2f351959a2d1f8/src/slave/containerizer/mesos/containerizer.cpp#L1392-L1407]. > The most reasonable value seems to be task's sandbox. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (MESOS-7355) Set MESOS_SANDBOX in debug containers.
[ https://issues.apache.org/jira/browse/MESOS-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kone reassigned MESOS-7355: - Assignee: Alexander Rukletsov > Set MESOS_SANDBOX in debug containers. > -- > > Key: MESOS-7355 > URL: https://issues.apache.org/jira/browse/MESOS-7355 > Project: Mesos > Issue Type: Improvement > Components: agent, containerization >Reporter: Alexander Rukletsov >Assignee: Alexander Rukletsov > Labels: check, health-check, mesosphere > > Currently {{MESOS_SANDBOX}} is not set for debug containers, see > [https://github.com/apache/mesos/blob/7f04cf886fc2ed59414bf0056a2f351959a2d1f8/src/slave/containerizer/mesos/containerizer.cpp#L1392-L1407]. > The most reasonable value seem to be task's sandbox. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MESOS-7432) Agent state can become corrupted
Dmitry Zhuk created MESOS-7432: -- Summary: Agent state can become corrupted Key: MESOS-7432 URL: https://issues.apache.org/jira/browse/MESOS-7432 Project: Mesos Issue Type: Bug Components: agent, master Affects Versions: 1.1.0 Reporter: Dmitry Zhuk Under some circumstances master can crash with the following (Mesos 1.1): {noformat} I0415 02:51:35.302822 21132 master.cpp:8272] Adding task task1-d74842f3-52e7-49cf-9f96-8176b5e9aa1a with resources … on agent b8f47842-5f08-446a-9b1f-532fe3bd47ed-S3371 (agent_host) I0415 02:51:35.311982 21132 master.cpp:5426] Re-registered agent b8f47842-5f08-446a-9b1f-532fe3bd47ed-S3371 at slave(1)@agent_ip:5051 (agent_host) with cpus(*):24; mem(*):61261; ports(*):[31000-32000]; disk(*):66123; ephemeral_ports(*):[32768-57344] I0415 02:51:35.312072 21132 master.cpp:5440] Shutting down framework 201205082337-03- at reregistered agent b8f47842-5f08-446a-9b1f-532fe3bd47ed-S3371 at slave(1)@agent_ip:5051 (agent_host) because the framework is not partition-aware I0415 02:51:35.320315 21119 hierarchical.cpp:485] Added agent b8f47842-5f08-446a-9b1f-532fe3bd47ed-S3371 (agent_host) with cpus(*):24; mem(*):61261; ports(*):[31000-32000]; disk(*):66123; ephemeral_ports(*):[32768-57344] (allocated: {}) F0415 02:51:35.525313 21132 master.cpp:7729] Check failed: 'slave' Must be non NULL *** Check failure stack trace: *** @ 0x7f1270cb92bd (unknown) @ 0x7f1270cbb104 (unknown) @ 0x7f1270cb8eac (unknown) @ 0x7f1270cbb9f9 (unknown) @ 0x7f127024aded (unknown) @ 0x7f127021ad2f (unknown) @ 0x7f1270240724 (unknown) @ 0x7f1270242be3 (unknown) @ 0x7f1270c56371 (unknown) @ 0x7f1270c56677 (unknown) @ 0x7f1270d8f760 (unknown) @ 0x7f126f23c83d start_thread @ 0x7f126ea2efdd clone I0415 02:51:38.058533 2 mesos-master.sh:101] Master Exit Status: 134 {noformat} Decoded stack trace: {noformat} Master::removeTask master/master.cpp:7729 Master::_reregisterSlave master/master.cpp:5450 {noformat} This part of the code has been changed in 1.2 and tasks for removed agents are no longer added on re-registration, however issues that lead to this crash still exist. This crash seems confusing at first, because it fails to lookup agent, that was just added. However agent is registered using {{SlaveInfo}}, while lookup is performed based on {{slave_id}} in {{TaskInfo}}, indicating corrupted agent registration data. Below is the sequence on events that lead to inconsistent agent state, which crashed master when it was sent by agent upon re-registration. 1. Agent b8f47842-5f08-446a-9b1f-532fe3bd47ed-S12 registered with the master, and task task1-d74842f3-52e7-49cf-9f96-8176b5e9aa1a was assigned to it: {noformat} I0411 23:52:28.516815 1748 slave.cpp:1115] Registered with master master@master_ip:5050; given agent ID b8f47842-5f08-446a-9b1f-532fe3bd47ed-S12 ... I0415 00:37:34.436111 1735 slave.cpp:1539] Got assigned task 'task1-d74842f3-52e7-49cf-9f96-8176b5e9aa1a' for framework 201205082337-03- {noformat} 2. Agent host was rebooted and agent started recovery when restarted. Agent correctly detected reboot, so it didn’t recover any data and started registration as a new agent. However agent was killed before it received registration confirmation from master. {noformat} I0415 01:08:13.300375 1772 state.cpp:57] Recovering state from '/var/lib/mesos/meta' I0415 01:08:13.300979 1772 state.cpp:698] No committed checkpointed resources found at '/var/lib/mesos/meta/resources/resources.info' I0415 01:08:13.301540 1772 state.cpp:88] Agent host rebooted I0415 01:08:13.301772 1783 status_update_manager.cpp:203] Recovering status update manager I0415 01:08:13.302006 1767 containerizer.cpp:555] Recovering containerizer I0415 01:08:13.304095 1766 port_mapping.cpp:2302] Network isolator recovery complete I0415 01:08:13.306538 1781 provisioner.cpp:253] Provisioner recovery complete I0415 01:08:13.306686 1785 slave.cpp:5281] Finished recovery I0415 01:08:13.308131 1785 slave.cpp:5314] Garbage collecting old agent b8f47842-5f08-446a-9b1f-532fe3bd47ed-S12 I0415 01:08:13.308218 1768 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/b8f47842-5f08-446a-9b1f-532fe3bd47ed-S12' for gc 6.9643315852days in the future I0415 01:08:13.308629 1781 gc.cpp:55] Scheduling '/var/lib/mesos/meta/slaves/b8f47842-5f08-446a-9b1f-532fe3bd47ed-S12' for gc 6.9642825185days in the future 2017-04-15 01:08:16,587:1512(0x7fb7a28c5700):ZOO_INFO@auth_completion_func@1300: Authentication scheme digest succeeded I0415 01:08:16.587770 1771 group.cpp:418] Trying to create path '/home/mesos/prod/master' in ZooKeeper I0415 01:08:16.589260 1777 detector.cpp:152] Detected a new leader: (id='316') I0415 01:08:16.589419 1788 group.cpp:697] Trying to get
[jira] [Commented] (MESOS-7026) Update authorization / authorization-filtering to handle hierarchical roles.
[ https://issues.apache.org/jira/browse/MESOS-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986184#comment-15986184 ] Adam B commented on MESOS-7026: --- commit d55148afbe4c86dc97907abffcbf01dec7e8467a Author: Alexander Rojas Date: Thu Apr 27 00:04:57 2017 -0700 Added test for authorization of hierarchical roles. Adds tests for each of the actions which support hierarchical roles. Review: https://reviews.apache.org/r/57474/ commit d46d33079805379515de0c300d7e6f4286adeb77 Author: Alexander Rojas Date: Thu Apr 27 00:02:06 2017 -0700 Added support for authorization of Hierachical roles. Adds mechanisms to support authorization of hierarchical roles, that is, it allows operators to write ACLs of the form role/% which will enforce the rule for any nested role, e.g. role/a, role/b and such. Review: https://reviews.apache.org/r/57473/ > Update authorization / authorization-filtering to handle hierarchical roles. > > > Key: MESOS-7026 > URL: https://issues.apache.org/jira/browse/MESOS-7026 > Project: Mesos > Issue Type: Task > Components: agent, HTTP API, master >Reporter: Benjamin Mahler >Assignee: Alexander Rojas >Priority: Blocker > Fix For: 1.3.0 > > > Authorization and endpoint filtering will need to be updated in order to > allow the authorization to be performed in a hierarchical manner (e.g. a user > can see all beneath /eng/* vs. a user can see all beneath /eng/frontend/*). -- This message was sent by Atlassian JIRA (v6.3.15#6346)