[jira] [Commented] (MESOS-6162) Add support for cgroups blkio subsystem
[ https://issues.apache.org/jira/browse/MESOS-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044040#comment-16044040 ] Jason Lai commented on MESOS-6162: -- [~haoyixin] Hi! We didn't get to prioritize this as the diff was pending for review. But we'll resurrect this task, for the sake of incoming demands on this. Will keep you updated as we progress > Add support for cgroups blkio subsystem > --- > > Key: MESOS-6162 > URL: https://issues.apache.org/jira/browse/MESOS-6162 > Project: Mesos > Issue Type: Task >Reporter: haosdent >Assignee: Jason Lai > > Noted that cgroups blkio subsystem may have performance issue, refer to > https://github.com/opencontainers/runc/issues/861 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6162) Add support for cgroups blkio subsystem
[ https://issues.apache.org/jira/browse/MESOS-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044032#comment-16044032 ] Gilbert Song commented on MESOS-6162: - [~haoyixin], sorry for the delay. I chatted with Jason. He already has a local implementation. Considering the fact that a couple companies are interested in this feature. We will try to ship it by the end of next week. I will shepherd. > Add support for cgroups blkio subsystem > --- > > Key: MESOS-6162 > URL: https://issues.apache.org/jira/browse/MESOS-6162 > Project: Mesos > Issue Type: Task >Reporter: haosdent >Assignee: Jason Lai > > Noted that cgroups blkio subsystem may have performance issue, refer to > https://github.com/opencontainers/runc/issues/861 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-6162) Add support for cgroups blkio subsystem
[ https://issues.apache.org/jira/browse/MESOS-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gilbert Song updated MESOS-6162: Shepherd: Gilbert Song > Add support for cgroups blkio subsystem > --- > > Key: MESOS-6162 > URL: https://issues.apache.org/jira/browse/MESOS-6162 > Project: Mesos > Issue Type: Task >Reporter: haosdent >Assignee: Jason Lai > > Noted that cgroups blkio subsystem may have performance issue, refer to > https://github.com/opencontainers/runc/issues/861 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6162) Add support for cgroups blkio subsystem
[ https://issues.apache.org/jira/browse/MESOS-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043980#comment-16043980 ] Hao Yixin commented on MESOS-6162: -- [~gilbert] Hi, Gilbert, we've met at qihoo 360 and talked about this. And, how does this going? I found it dead in the water for months. > Add support for cgroups blkio subsystem > --- > > Key: MESOS-6162 > URL: https://issues.apache.org/jira/browse/MESOS-6162 > Project: Mesos > Issue Type: Task >Reporter: haosdent >Assignee: Jason Lai > > Noted that cgroups blkio subsystem may have performance issue, refer to > https://github.com/opencontainers/runc/issues/861 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (MESOS-6162) Add support for cgroups blkio subsystem
[ https://issues.apache.org/jira/browse/MESOS-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043980#comment-16043980 ] Hao Yixin edited comment on MESOS-6162 at 6/9/17 5:58 AM: -- Hi, [~gilbert], we've met at qihoo 360 and talked about this. And, how does this going? I found it dead in the water for months. was (Author: haoyixin): [~gilbert] Hi, Gilbert, we've met at qihoo 360 and talked about this. And, how does this going? I found it dead in the water for months. > Add support for cgroups blkio subsystem > --- > > Key: MESOS-6162 > URL: https://issues.apache.org/jira/browse/MESOS-6162 > Project: Mesos > Issue Type: Task >Reporter: haosdent >Assignee: Jason Lai > > Noted that cgroups blkio subsystem may have performance issue, refer to > https://github.com/opencontainers/runc/issues/861 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7159) AgentAPIStreamingTest.AttachInputToNestedContainerSession is flakey
[ https://issues.apache.org/jira/browse/MESOS-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-7159: -- Labels: flaky flaky-test mesosphere (was: mesosphere) > AgentAPIStreamingTest.AttachInputToNestedContainerSession is flakey > --- > > Key: MESOS-7159 > URL: https://issues.apache.org/jira/browse/MESOS-7159 > Project: Mesos > Issue Type: Bug >Reporter: Neil Conway >Assignee: Anand Mazumdar > Labels: flaky, flaky-test, mesosphere > Attachments: test_failure.txt > > > This test fails intermittently for me (10% of the time but easy to repro) on > a VM running Ubuntu 16.04. Test log is attached. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7159) AgentAPIStreamingTest.AttachInputToNestedContainerSession is flaky
[ https://issues.apache.org/jira/browse/MESOS-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam B updated MESOS-7159: -- Summary: AgentAPIStreamingTest.AttachInputToNestedContainerSession is flaky (was: AgentAPIStreamingTest.AttachInputToNestedContainerSession is flakey) > AgentAPIStreamingTest.AttachInputToNestedContainerSession is flaky > -- > > Key: MESOS-7159 > URL: https://issues.apache.org/jira/browse/MESOS-7159 > Project: Mesos > Issue Type: Bug >Reporter: Neil Conway >Assignee: Anand Mazumdar > Labels: flaky, flaky-test, mesosphere > Attachments: test_failure.txt > > > This test fails intermittently for me (10% of the time but easy to repro) on > a VM running Ubuntu 16.04. Test log is attached. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7159) AgentAPIStreamingTest.AttachInputToNestedContainerSession is flakey
[ https://issues.apache.org/jira/browse/MESOS-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043727#comment-16043727 ] Adam B commented on MESOS-7159: --- Seen again on ASF CI for Mesos 1.2.1-rc1, logs captured here: https://gist.github.com/bmahler/5ae340b4de3341f3c1f072250006dc64 > AgentAPIStreamingTest.AttachInputToNestedContainerSession is flakey > --- > > Key: MESOS-7159 > URL: https://issues.apache.org/jira/browse/MESOS-7159 > Project: Mesos > Issue Type: Bug >Reporter: Neil Conway >Assignee: Anand Mazumdar > Labels: mesosphere > Attachments: test_failure.txt > > > This test fails intermittently for me (10% of the time but easy to repro) on > a VM running Ubuntu 16.04. Test log is attached. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MESOS-7647) Add a common ResourceProviderManager.
Jie Yu created MESOS-7647: - Summary: Add a common ResourceProviderManager. Key: MESOS-7647 URL: https://issues.apache.org/jira/browse/MESOS-7647 Project: Mesos Issue Type: Task Reporter: Jie Yu ResourceProviderManager will be used to manage the lifecycle of resource providers, including both local resource providers and external resource providers. It resides in both the agent and the master (future). The agent's resource provider manager will be used to manage local resource providers. It processes resource_provider::Call from the local resource providers and sends resource_provider::Event to the local resource providers. The agent will also polling the manager for updates like resource changes so that it can inform the master about that. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7469) Add resource provider driver.
[ https://issues.apache.org/jira/browse/MESOS-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-7469: -- Summary: Add resource provider driver. (was: Add local resource provider driver.) > Add resource provider driver. > - > > Key: MESOS-7469 > URL: https://issues.apache.org/jira/browse/MESOS-7469 > Project: Mesos > Issue Type: Task >Reporter: Jie Yu >Assignee: Jie Yu > > Similar to scheduler/executor driver, resource provider driver will be used > to connect the resource provider and the Resource provider manager (resides > in either agent for local resource providers, or master for external resource > providers). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7469) Add local resource provider driver.
[ https://issues.apache.org/jira/browse/MESOS-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-7469: -- Description: Similar to scheduler/executor driver, resource provider driver will be used to connect the resource provider and the Resource provider manager (resides in either agent for local resource providers, or master for external resource providers). (was: Similar to scheduler/executor driver, resource provider driver will be used to connect the resource provider and the master. For local resource providers, the driver will try to use the agent as the proxy, instead of initiating a direct connection to the master. There multiple reasons we want to do that: * Reduce the load on the master because there will less connections. * More easy to control the life cycle of a local resource provider. For instance, it'll be very straightforward to force the subscription of the local resource providers to be _after_ the agent registration.) > Add local resource provider driver. > --- > > Key: MESOS-7469 > URL: https://issues.apache.org/jira/browse/MESOS-7469 > Project: Mesos > Issue Type: Task >Reporter: Jie Yu >Assignee: Jie Yu > > Similar to scheduler/executor driver, resource provider driver will be used > to connect the resource provider and the Resource provider manager (resides > in either agent for local resource providers, or master for external resource > providers). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7570) Add a storage local resource provider.
[ https://issues.apache.org/jira/browse/MESOS-7570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-7570: -- Description: It will interact with CSI plugins to get capacity information as well as talking to the plugin for operations like CREATE/DESTROY. was: This will be a subclass of LocalResourceProvider. It will interact with CSI plugins to get capacity information as well as talking to the plugin for operations like CREATE/DESTROY. > Add a storage local resource provider. > -- > > Key: MESOS-7570 > URL: https://issues.apache.org/jira/browse/MESOS-7570 > Project: Mesos > Issue Type: Task >Reporter: Jie Yu >Assignee: Jie Yu > > It will interact with CSI plugins to get capacity information as well as > talking to the plugin for operations like CREATE/DESTROY. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7571) Add `--resource_provider_config_dir` flag to the agent.
[ https://issues.apache.org/jira/browse/MESOS-7571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-7571: -- Description: Add an agent flag `--resource_provider_config_dir` to allow operators to specify the list of local resource providers to register with the agent. (was: Add an agent flag `--resource_providers` to allow operators to specify the list of local resource providers to register with the master.) > Add `--resource_provider_config_dir` flag to the agent. > --- > > Key: MESOS-7571 > URL: https://issues.apache.org/jira/browse/MESOS-7571 > Project: Mesos > Issue Type: Task >Reporter: Jie Yu >Assignee: Jie Yu > > Add an agent flag `--resource_provider_config_dir` to allow operators to > specify the list of local resource providers to register with the agent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7571) Add `--resource_provider_config_dir` flag to the agent.
[ https://issues.apache.org/jira/browse/MESOS-7571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-7571: -- Summary: Add `--resource_provider_config_dir` flag to the agent. (was: Add `--resource_providers` flag to the agent.) > Add `--resource_provider_config_dir` flag to the agent. > --- > > Key: MESOS-7571 > URL: https://issues.apache.org/jira/browse/MESOS-7571 > Project: Mesos > Issue Type: Task >Reporter: Jie Yu >Assignee: Jie Yu > > Add an agent flag `--resource_providers` to allow operators to specify the > list of local resource providers to register with the master. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7645) Support RO mode for bind mount volumes with filesystem/linux isolator
[
https://issues.apache.org/jira/browse/MESOS-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043597#comment-16043597
]
Charles Raimbert commented on MESOS-7645:
-
https://reviews.apache.org/r/59930/
> Support RO mode for bind mount volumes with filesystem/linux isolator
> -
>
> Key: MESOS-7645
> URL: https://issues.apache.org/jira/browse/MESOS-7645
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Charles Raimbert
>Assignee: Charles Raimbert
>
> The filesystem/linux isolator currently creates all bind mount volumes as RW,
> even if a volume mode is set as RO.
> The TODO in the isolator code helps to spot the missing capability:
> https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L587
> {code}
> // TODO(jieyu): Consider the mode in the volume.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7646) Create a Backoff abstraction
Yan Xu created MESOS-7646: - Summary: Create a Backoff abstraction Key: MESOS-7646 URL: https://issues.apache.org/jira/browse/MESOS-7646 Project: Mesos Issue Type: Bug Components: stout Reporter: Yan Xu We currently have a few flavors of ad-hoc exponential backoff mechanisms implemented within the components that need to do backoff. Accordingly in code/flag/feature documentation there are ad-hoc explanations of them. We should consolidate these into one generic abstraction. We could have something that works like this: https://developers.google.com/api-client-library/java/google-http-java-client/reference/1.20.0/com/google/api/client/util/ExponentialBackOff -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (MESOS-7645) Support RO mode for bind mount volumes with filesystem/linux isolator
[
https://issues.apache.org/jira/browse/MESOS-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Yu reassigned MESOS-7645:
-
Assignee: Charles Raimbert
> Support RO mode for bind mount volumes with filesystem/linux isolator
> -
>
> Key: MESOS-7645
> URL: https://issues.apache.org/jira/browse/MESOS-7645
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Charles Raimbert
>Assignee: Charles Raimbert
>
> The filesystem/linux isolator currently creates all bind mount volumes as RW,
> even if a volume mode is set as RO.
> The TODO in the isolator code helps to spot the missing capability:
> https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L587
> {code}
> // TODO(jieyu): Consider the mode in the volume.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7645) Support RO mode for bind mount volumes with filesystem/linux isolator
[
https://issues.apache.org/jira/browse/MESOS-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Raimbert updated MESOS-7645:
Description:
The filesystem/linux isolator currently creates all bind mount volumes as RW,
even if a volume mode is set as RO.
The TODO in the isolator code helps to spot the missing capability:
https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L587
{code}
// TODO(jieyu): Consider the mode in the volume.
{code}
was:
The filesystem/linux isolator currently creates all bind mount volumes as RW,
even if a volume mode is set as RO.
The TODO in the isolator code helps to spot the missing capability:
https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L587
```
// TODO(jieyu): Consider the mode in the volume.
```
> Support RO mode for bind mount volumes with filesystem/linux isolator
> -
>
> Key: MESOS-7645
> URL: https://issues.apache.org/jira/browse/MESOS-7645
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Charles Raimbert
>
> The filesystem/linux isolator currently creates all bind mount volumes as RW,
> even if a volume mode is set as RO.
> The TODO in the isolator code helps to spot the missing capability:
> https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L587
> {code}
> // TODO(jieyu): Consider the mode in the volume.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7645) Support RO mode for bind mount volumes with filesystem/linux isolator
Charles Raimbert created MESOS-7645: --- Summary: Support RO mode for bind mount volumes with filesystem/linux isolator Key: MESOS-7645 URL: https://issues.apache.org/jira/browse/MESOS-7645 Project: Mesos Issue Type: Bug Components: containerization Reporter: Charles Raimbert The filesystem/linux isolator currently creates all bind mount volumes as RW, even if a volume mode is set as RO. The TODO in the isolator code helps to spot the missing capability: https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L587 ``` // TODO(jieyu): Consider the mode in the volume. ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7631) DefautlExecutor needs to inform tasks about IP addresses
[ https://issues.apache.org/jira/browse/MESOS-7631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avinash Sridharan updated MESOS-7631: - Sprint: Mesosphere Sprint 57 > DefautlExecutor needs to inform tasks about IP addresses > > > Key: MESOS-7631 > URL: https://issues.apache.org/jira/browse/MESOS-7631 > Project: Mesos > Issue Type: Task > Components: executor >Reporter: Avinash Sridharan >Assignee: Anand Mazumdar > > Currently, when the DefaultExecutor launches tasks it doesn’t pass any > networking information (host or CNI network IP addresses) to the tasks that > it launches. This becomes problematic for the applications which need to know > the IP address that they need to bind to and cannot rely on binding to > INADDR_ANY. The tasks launched by the DefaultExecutor can potentially look at > all the available interface IP addresses and select a non-loopback IP address > but this is very error prone when there are multiple interfaces on a host or > a container is attached to multiple networks. > Possible solution: > The DefaultExecutor sets a variable MESOS_TASK_IP into the task’s shell. The > task will always read the variable to determine the IP address it should use. > Below we describe the algorithm to set MESOS_CONTAINER_IP for various cases. > Setting MESOS_CONTAINER_IP for `host` network: > For host network the `LIBPROCESS_IP` in the DefaultExecutor is always set to > the agent IP. The DefaultExecutor can therefore set `MESOS_TASK_IP` to the > `LIBPROCESS_IP` whenever it sees that the LIBPROCESS_IP is anything other > than “0.0.0.0” or “::” (for IPv6). > Setting MESOS_TASK_IP for a single “CNI” network: > For CNI network the `LIBPROCESS_IP` is set to “0.0.0.0” and the `hostname` of > the container is set to the CNI ip address. The DefaultExecutor already > resolves the hostname for CNI network (to register with agent), so it can > just set MESOS_TASK_IP to the IP address it learnt by resolving the hostname. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7631) DefautlExecutor needs to inform tasks about IP addresses
[ https://issues.apache.org/jira/browse/MESOS-7631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avinash Sridharan updated MESOS-7631: - Shepherd: Anand Mazumdar (was: Vinod Kone) > DefautlExecutor needs to inform tasks about IP addresses > > > Key: MESOS-7631 > URL: https://issues.apache.org/jira/browse/MESOS-7631 > Project: Mesos > Issue Type: Task > Components: executor >Reporter: Avinash Sridharan >Assignee: Anand Mazumdar > > Currently, when the DefaultExecutor launches tasks it doesn’t pass any > networking information (host or CNI network IP addresses) to the tasks that > it launches. This becomes problematic for the applications which need to know > the IP address that they need to bind to and cannot rely on binding to > INADDR_ANY. The tasks launched by the DefaultExecutor can potentially look at > all the available interface IP addresses and select a non-loopback IP address > but this is very error prone when there are multiple interfaces on a host or > a container is attached to multiple networks. > Possible solution: > The DefaultExecutor sets a variable MESOS_TASK_IP into the task’s shell. The > task will always read the variable to determine the IP address it should use. > Below we describe the algorithm to set MESOS_CONTAINER_IP for various cases. > Setting MESOS_CONTAINER_IP for `host` network: > For host network the `LIBPROCESS_IP` in the DefaultExecutor is always set to > the agent IP. The DefaultExecutor can therefore set `MESOS_TASK_IP` to the > `LIBPROCESS_IP` whenever it sees that the LIBPROCESS_IP is anything other > than “0.0.0.0” or “::” (for IPv6). > Setting MESOS_TASK_IP for a single “CNI” network: > For CNI network the `LIBPROCESS_IP` is set to “0.0.0.0” and the `hostname` of > the container is set to the CNI ip address. The DefaultExecutor already > resolves the hostname for CNI network (to register with agent), so it can > just set MESOS_TASK_IP to the IP address it learnt by resolving the hostname. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6972) Improve performance of protobuf message passing by removing RepeatedPtrField to vector conversion.
[
https://issues.apache.org/jira/browse/MESOS-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043468#comment-16043468
]
Dmitry Zhuk commented on MESOS-6972:
Yeah, this will not work with arenas, but I'm not sure if arenas are a better
alternative.
I presume you're referring to using arenas for parsing incoming messages?
I just think about {{reregisterSlave}}, which basically takes all incoming data
and puts it inside master's internal data structures. This seems like a typical
flow for handling message: doing some validation, etc. and then passing the
incoming data to the same or another process with {{defer}}. So technically we
get faster message parsing with arena, but then we need to deep copy this data
for passing it further anyway. Without arenas, parsing is slower, but then we
can use {{std::move}} and {{Swap}} (I have some ideas about making something
like {{protobuf::move}} invoking {{Swap}} behind the scenes) to avoid any extra
overhead on copying. Am I missing something here?
> Improve performance of protobuf message passing by removing RepeatedPtrField
> to vector conversion.
> --
>
> Key: MESOS-6972
> URL: https://issues.apache.org/jira/browse/MESOS-6972
> Project: Mesos
> Issue Type: Improvement
>Reporter: Benjamin Mahler
> Labels: tech-debt
>
> Currently, all protobuf message handlers must take a {{vector}} for repeated
> fields, rather than a {{RepeatedPtrField}}.
> This requires that a copy be performed of the repeated field's entries (see
> [here|https://github.com/apache/mesos/blob/9228ebc239dac42825390bebc72053dbf3ae7b09/3rdparty/libprocess/include/process/protobuf.hpp#L78-L87]),
> which can be very expensive in some cases. We should avoid requiring this
> expense on the callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (MESOS-7642) Consider unbundling protobuf dependency
[ https://issues.apache.org/jira/browse/MESOS-7642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043439#comment-16043439 ] Benjamin Bannier edited comment on MESOS-7642 at 6/8/17 9:11 PM: - It looks like our patch on protobuf-2.6.1 was just to effectively silence a warning so any pre-built >=2.6.1 appears fine, https://github.com/apache/mesos/commit/a868b03c931d8d53ea18ca3b41b510d2d03efe94#diff-436a5ce2d86244f9f505b03239fbd51d. was (Author: bbannier): It looks like our patch on protobuf-2.6.1 was just to effectively silence a warning so an pre-built 2.6.1 appears fine, https://github.com/apache/mesos/commit/a868b03c931d8d53ea18ca3b41b510d2d03efe94#diff-436a5ce2d86244f9f505b03239fbd51d. > Consider unbundling protobuf dependency > --- > > Key: MESOS-7642 > URL: https://issues.apache.org/jira/browse/MESOS-7642 > Project: Mesos > Issue Type: Improvement >Affects Versions: 1.4.0 >Reporter: Benjamin Bannier > > After MESOS-7228 we do not add additional patches on top of the bundled > protobuf anymore. This should make it possible to always use an unbundled > protobuf of appropriate version, i.e., to remove the protobuf third-party > component. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7642) Consider unbundling protobuf dependency
[ https://issues.apache.org/jira/browse/MESOS-7642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043439#comment-16043439 ] Benjamin Bannier commented on MESOS-7642: - It looks like our patch on protobuf-2.6.1 was just to effectively silence a warning so an pre-built 2.6.1 appears fine, https://github.com/apache/mesos/commit/a868b03c931d8d53ea18ca3b41b510d2d03efe94#diff-436a5ce2d86244f9f505b03239fbd51d. > Consider unbundling protobuf dependency > --- > > Key: MESOS-7642 > URL: https://issues.apache.org/jira/browse/MESOS-7642 > Project: Mesos > Issue Type: Improvement >Affects Versions: 1.4.0 >Reporter: Benjamin Bannier > > After MESOS-7228 we do not add additional patches on top of the bundled > protobuf anymore. This should make it possible to always use an unbundled > protobuf of appropriate version, i.e., to remove the protobuf third-party > component. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (MESOS-6918) Prometheus exporter endpoints for metrics
[ https://issues.apache.org/jira/browse/MESOS-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Peach reassigned MESOS-6918: -- Assignee: James Peach > Prometheus exporter endpoints for metrics > - > > Key: MESOS-6918 > URL: https://issues.apache.org/jira/browse/MESOS-6918 > Project: Mesos > Issue Type: Bug > Components: statistics >Reporter: James Peach >Assignee: James Peach > > There are a couple of [Prometheus|https://prometheus.io] metrics exporters > for Mesos, of varying quality. Since the Mesos stats system actually knows > about statistics data types and semantics, and Mesos has reasonable HTTP > support we could add Prometheus metrics endpoints to directly expose > statistics in [Prometheus wire > format|https://prometheus.io/docs/instrumenting/exposition_formats/], > removing the need for operators to run separate exporter processes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7524) Basic fetcher success metrics
[ https://issues.apache.org/jira/browse/MESOS-7524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-7524: - Shepherd: Joseph Wu > Basic fetcher success metrics > - > > Key: MESOS-7524 > URL: https://issues.apache.org/jira/browse/MESOS-7524 > Project: Mesos > Issue Type: Bug > Components: fetcher >Reporter: James Peach >Assignee: James Peach > > There are no metrics for the fetcher. As minimum we should have counters for: > * successful fetcher invocations > * failed fetcher invocations > It would also be useful to know the fetch time, though that could be highly > variable depending on the cluster usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7607) Support for first-class fault domains.
[ https://issues.apache.org/jira/browse/MESOS-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043246#comment-16043246 ] Charles Allen commented on MESOS-7607: -- Can you explain a bit deeper why agent attributes are insufficient for this capability? > Support for first-class fault domains. > -- > > Key: MESOS-7607 > URL: https://issues.apache.org/jira/browse/MESOS-7607 > Project: Mesos > Issue Type: Epic >Reporter: Neil Conway >Assignee: Neil Conway > Labels: mesosphere > > Mesos should support a first-class notion of "fault domains", which > effectively provide a common vocabulary for describing the region and zone > where a node (either master or agent) is located. > Design doc: > https://drive.google.com/open?id=1gEugdkLRbBsqsiFv3urRPRNrHwUC-i1HwfFfHR_MvC8 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-6972) Improve performance of protobuf message passing by removing RepeatedPtrField to vector conversion.
[
https://issues.apache.org/jira/browse/MESOS-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043178#comment-16043178
]
Benjamin Mahler commented on MESOS-6972:
[~dzhuk] great, yeah I was thinking of this option as well but wasn't sure if
this works when we use arena allocation, since Swap performs deep copies if one
side is from an arena and the other is not:
https://developers.google.com/protocol-buffers/docs/reference/arenas#message-class-methods
> Improve performance of protobuf message passing by removing RepeatedPtrField
> to vector conversion.
> --
>
> Key: MESOS-6972
> URL: https://issues.apache.org/jira/browse/MESOS-6972
> Project: Mesos
> Issue Type: Improvement
>Reporter: Benjamin Mahler
> Labels: tech-debt
>
> Currently, all protobuf message handlers must take a {{vector}} for repeated
> fields, rather than a {{RepeatedPtrField}}.
> This requires that a copy be performed of the repeated field's entries (see
> [here|https://github.com/apache/mesos/blob/9228ebc239dac42825390bebc72053dbf3ae7b09/3rdparty/libprocess/include/process/protobuf.hpp#L78-L87]),
> which can be very expensive in some cases. We should avoid requiring this
> expense on the callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7575) Support reservation refinement
[
https://issues.apache.org/jira/browse/MESOS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Park updated MESOS-7575:
Story Points: 12
> Support reservation refinement
> --
>
> Key: MESOS-7575
> URL: https://issues.apache.org/jira/browse/MESOS-7575
> Project: Mesos
> Issue Type: Task
>Reporter: Michael Park
>Assignee: Michael Park
>
> With the introduction of hierarchical roles, Mesos provides a mechanism to
> delegate resources down a hierarchy.
> To complement this, we'll introduce a mechanism to *refine* the reservations
> down the hierarchy.
> For example, given resources allocated to role {{foo}}, it can be further
> reserved for {{foo/bar}}.
> When the resources allocated to {{foo/bar}} is unreserved, it goes back to
> where it came from. In this case, back to role {{foo}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7643) The order of isolators provided in '--isolation' flag is not preserved and instead sorted alphabetically
[ https://issues.apache.org/jira/browse/MESOS-7643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-7643: -- Affects Version/s: 1.3.0 1.1.2 1.2.0 > The order of isolators provided in '--isolation' flag is not preserved and > instead sorted alphabetically > > > Key: MESOS-7643 > URL: https://issues.apache.org/jira/browse/MESOS-7643 > Project: Mesos > Issue Type: Bug > Components: containerization >Affects Versions: 1.1.2, 1.2.0, 1.3.0 >Reporter: Michael Cherny > > According to documentation and comments in code the order of the entries in > the --isolation flag should specify the ordering of the isolators. > Specifically, the > `create` and `prepare` calls for each isolator should run serially in the > order in which they appear in the --isolation flag, while the `cleanup` call > should be serialized in reverse order (with exception of filesystem isolator > which is always first). > But in fact, the isolators provided in '--isolation' flag are sorted > alphabetically. > That happens in [this line of > code|https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/containerizer.cpp#L377]. > In this line use of 'set' is done (apparently instead of list or > vector) and set is a sorted container. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7314) Add offer operations for converting disk resources
[
https://issues.apache.org/jira/browse/MESOS-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043094#comment-16043094
]
Jie Yu commented on MESOS-7314:
---
Yes, it does preserve reservation.
> Add offer operations for converting disk resources
> --
>
> Key: MESOS-7314
> URL: https://issues.apache.org/jira/browse/MESOS-7314
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Jan Schlicht
>Assignee: Jan Schlicht
> Labels: mesosphere
>
> One should be able to convert {{RAW}} and {{BLOCK}} disk resources into a
> different types by applying operations to them. The offer operations and the
> related validation and resource handling needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-7314) Add offer operations for converting disk resources
[
https://issues.apache.org/jira/browse/MESOS-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043082#comment-16043082
]
James DeFelice commented on MESOS-7314:
---
Does this preserve reservations across conversion ops? If not, it probably
should...
> Add offer operations for converting disk resources
> --
>
> Key: MESOS-7314
> URL: https://issues.apache.org/jira/browse/MESOS-7314
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Jan Schlicht
>Assignee: Jan Schlicht
> Labels: mesosphere
>
> One should be able to convert {{RAW}} and {{BLOCK}} disk resources into a
> different types by applying operations to them. The offer operations and the
> related validation and resource handling needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7305) Adjust the recover logic of MesosContainerizer to allow standalone containers.
[ https://issues.apache.org/jira/browse/MESOS-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wu updated MESOS-7305: - Sprint: Mesosphere Sprint 57 Story Points: 5 Labels: mesosphere (was: ) > Adjust the recover logic of MesosContainerizer to allow standalone containers. > -- > > Key: MESOS-7305 > URL: https://issues.apache.org/jira/browse/MESOS-7305 > Project: Mesos > Issue Type: Task > Components: containerization >Reporter: Jie Yu >Assignee: Joseph Wu > Labels: mesosphere > > The current recovery logic in MesosContainerizer assumes that all top level > containers are tied to some Mesos executors. Adding standalone containers > will invalid this assumption. The recovery logic must be changed to adapt to > that. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7388) Update allocator interfaces to support resource providers
[ https://issues.apache.org/jira/browse/MESOS-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-7388: -- Sprint: Mesosphere Sprint 54, Mesosphere Sprint 55, Mesosphere Sprint 56 (was: Mesosphere Sprint 54, Mesosphere Sprint 55, Mesosphere Sprint 56, Mesosphere Sprint 57) > Update allocator interfaces to support resource providers > - > > Key: MESOS-7388 > URL: https://issues.apache.org/jira/browse/MESOS-7388 > Project: Mesos > Issue Type: Task >Reporter: Benjamin Bannier >Assignee: Benjamin Bannier > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7469) Add local resource provider driver.
[ https://issues.apache.org/jira/browse/MESOS-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-7469: -- Sprint: Mesosphere Sprint 56 (was: Mesosphere Sprint 56, Mesosphere Sprint 57) > Add local resource provider driver. > --- > > Key: MESOS-7469 > URL: https://issues.apache.org/jira/browse/MESOS-7469 > Project: Mesos > Issue Type: Task >Reporter: Jie Yu >Assignee: Jie Yu > > Similar to scheduler/executor driver, resource provider driver will be used > to connect the resource provider and the master. For local resource > providers, the driver will try to use the agent as the proxy, instead of > initiating a direct connection to the master. There multiple reasons we want > to do that: > * Reduce the load on the master because there will less connections. > * More easy to control the life cycle of a local resource provider. For > instance, it'll be very straightforward to force the subscription of the > local resource providers to be _after_ the agent registration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7595) Implement local resource provider registration
[
https://issues.apache.org/jira/browse/MESOS-7595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Yu updated MESOS-7595:
--
Sprint: (was: Mesosphere Sprint 57)
> Implement local resource provider registration
> --
>
> Key: MESOS-7595
> URL: https://issues.apache.org/jira/browse/MESOS-7595
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Jan Schlicht
>Assignee: Jan Schlicht
> Labels: mesosphere
>
> A {{resource_provider::Call::SUBSCRIBE}} call of a resource provider should
> add that one to the list of registered resource providers in the master.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7592) Add handling of local resource providers to the master
[ https://issues.apache.org/jira/browse/MESOS-7592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Yu updated MESOS-7592: -- Sprint: (was: Mesosphere Sprint 57) > Add handling of local resource providers to the master > -- > > Key: MESOS-7592 > URL: https://issues.apache.org/jira/browse/MESOS-7592 > Project: Mesos > Issue Type: Task > Components: master >Reporter: Jan Schlicht >Assignee: Jan Schlicht > Labels: mesosphere > > To support local resource providers the master has to keep track of the > registered ones, and their allocated resources/ outstanding offers. This is > similar to how it's already done for agents, hence this existing > functionality could be abstracted and reused for local resource providers. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7591) Update master to use resource provider IDs instead of agent ID in allocator calls.
[
https://issues.apache.org/jira/browse/MESOS-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Yu updated MESOS-7591:
--
Sprint: (was: Mesosphere Sprint 57)
> Update master to use resource provider IDs instead of agent ID in allocator
> calls.
> --
>
> Key: MESOS-7591
> URL: https://issues.apache.org/jira/browse/MESOS-7591
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Jan Schlicht
>Assignee: Jan Schlicht
> Labels: mesosphere
>
> MESOS-7388 updates the allocator interface to use {{ResourceProviderID}}
> instead of {{SlaveID}}. The usage of the allocator in the master has to be
> updated accordingly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7636) Add DomainInfo to Resource
[
https://issues.apache.org/jira/browse/MESOS-7636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neil Conway updated MESOS-7636:
---
Issue Type: Improvement (was: Bug)
> Add DomainInfo to Resource
> --
>
> Key: MESOS-7636
> URL: https://issues.apache.org/jira/browse/MESOS-7636
> Project: Mesos
> Issue Type: Improvement
>Reporter: Neil Conway
>Assignee: Neil Conway
> Labels: mesosphere
>
> Add a new {{domain}} field to {{Resource}}, which identifies the domain where
> the resources are located.
> It would be nice to add this to a higher-level message, like {{Offer}}, but
> we need to account for the possibility that different resources in a single
> offer come from different domains -- e.g., when support for external resource
> providers is introduced.
> It will now be possible for an agent's checkpointed resources to change when
> the agent is drained and restarted. Combined with the change to preserve
> AgentID across agent drains (MESOS-6223), this will mean that if an agent
> re-registers, its resources might have changed; previous copies of the
> resource in the master's in-memory state will need to be updated.
> This also means that if a framework has cached the properties of a persistent
> volume it created, subsequently attempting to destroy that volume with the
> old persistent volume JSON may not succeed, because the PV's properties have
> changed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7644) Add DomainInfo to offers
Neil Conway created MESOS-7644: -- Summary: Add DomainInfo to offers Key: MESOS-7644 URL: https://issues.apache.org/jira/browse/MESOS-7644 Project: Mesos Issue Type: Improvement Reporter: Neil Conway Assignee: Neil Conway Per discussion in MESOS-7636, this will be a convenience mechanism to allow frameworks to determine the domain of the agent associated with a resource offer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7636) Add DomainInfo to Resource
[
https://issues.apache.org/jira/browse/MESOS-7636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043042#comment-16043042
]
Neil Conway commented on MESOS-7636:
Per offline discussion with [~vinodkone] and [~jieyu], the plan is to avoid
tackling this for now. The {{domain}} in {{Offer}} will just be the agent's
domain, which we make available as a convenience mechanism for frameworks.
Depending on exactly how the ERP message flow works, we'll need some way to
describe the domain of a resource provided by an ERP, but that can be solved
later.
> Add DomainInfo to Resource
> --
>
> Key: MESOS-7636
> URL: https://issues.apache.org/jira/browse/MESOS-7636
> Project: Mesos
> Issue Type: Bug
>Reporter: Neil Conway
>Assignee: Neil Conway
> Labels: mesosphere
>
> Add a new {{domain}} field to {{Resource}}, which identifies the domain where
> the resources are located.
> It would be nice to add this to a higher-level message, like {{Offer}}, but
> we need to account for the possibility that different resources in a single
> offer come from different domains -- e.g., when support for external resource
> providers is introduced.
> It will now be possible for an agent's checkpointed resources to change when
> the agent is drained and restarted. Combined with the change to preserve
> AgentID across agent drains (MESOS-6223), this will mean that if an agent
> re-registers, its resources might have changed; previous copies of the
> resource in the master's in-memory state will need to be updated.
> This also means that if a framework has cached the properties of a persistent
> volume it created, subsequently attempting to destroy that volume with the
> old persistent volume JSON may not succeed, because the PV's properties have
> changed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7643) The order of isolators provided in '--isolation' flag is not preserved and instead sorted alphabetically
Michael Cherny created MESOS-7643: - Summary: The order of isolators provided in '--isolation' flag is not preserved and instead sorted alphabetically Key: MESOS-7643 URL: https://issues.apache.org/jira/browse/MESOS-7643 Project: Mesos Issue Type: Bug Components: containerization Reporter: Michael Cherny According to documentation and comments in code the order of the entries in the --isolation flag should specify the ordering of the isolators. Specifically, the `create` and `prepare` calls for each isolator should run serially in the order in which they appear in the --isolation flag, while the `cleanup` call should be serialized in reverse order (with exception of filesystem isolator which is always first). But in fact, the isolators provided in '--isolation' flag are sorted alphabetically. That happens in [this line of code|https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/containerizer.cpp#L377]. In this line use of 'set' is done (apparently instead of list or vector) and set is a sorted container. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7416) Filter results of `/master/slaves` and the v1 call GET_AGENTS
[
https://issues.apache.org/jira/browse/MESOS-7416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam B updated MESOS-7416:
--
Shepherd: Till Toenshoff (was: Adam B)
> Filter results of `/master/slaves` and the v1 call GET_AGENTS
> -
>
> Key: MESOS-7416
> URL: https://issues.apache.org/jira/browse/MESOS-7416
> Project: Mesos
> Issue Type: Task
> Components: HTTP API, master
>Reporter: Alexander Rojas
>Assignee: Alexander Rojas
> Labels: mesosphere, security
>
> The results returned by both the endpoint {{/master/slaves}} and the API v1
> {{GET_AGENTS}} return full information about the agent state which probably
> need to be filtered for certain uses, particularly in a multi-tenancy
> scenario.
> The kind of leaked data includes specific role names and their specific
> allocations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (MESOS-7415) Add authorization to master's operator maintenance API in v0 and v1
[
https://issues.apache.org/jira/browse/MESOS-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam B reassigned MESOS-7415:
-
Assignee: Alexander Rojas (was: Adam B)
> Add authorization to master's operator maintenance API in v0 and v1
> ---
>
> Key: MESOS-7415
> URL: https://issues.apache.org/jira/browse/MESOS-7415
> Project: Mesos
> Issue Type: Task
> Components: c++ api, HTTP API, master
>Reporter: Alexander Rojas
>Assignee: Alexander Rojas
> Labels: authorization, mesosphere, security
>
> None of the maintenance primitives in either API v0 or API v1 have any kind
> of authorization, which allows any user with valid credentials to do things
> such as shutting down a machine, schedule time off on an agent, modify
> maintenance schedule, etc.
> The authorization support needs to be added to the v0 endpoints:
> * {{/master/machine/up}}
> * {{/master/machine/down}}
> * {{/master/maintenance/schedule}}
> * {{/master/maintenance/status}}
> as well as to the v1 calls:
> * {{GET_MAINTENANCE_STATUS}}
> * {{GET_MAINTENANCE_SCHEDULE}}
> * {{UPDATE_MAINTENANCE_SCHEDULE}}
> * {{START_MAINTENANCE}}
> * {{STOP_MAINTENANCE}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (MESOS-7415) Add authorization to master's operator maintenance API in v0 and v1
[
https://issues.apache.org/jira/browse/MESOS-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam B reassigned MESOS-7415:
-
Assignee: Adam B (was: Alexander Rojas)
> Add authorization to master's operator maintenance API in v0 and v1
> ---
>
> Key: MESOS-7415
> URL: https://issues.apache.org/jira/browse/MESOS-7415
> Project: Mesos
> Issue Type: Task
> Components: c++ api, HTTP API, master
>Reporter: Alexander Rojas
>Assignee: Adam B
> Labels: authorization, mesosphere, security
>
> None of the maintenance primitives in either API v0 or API v1 have any kind
> of authorization, which allows any user with valid credentials to do things
> such as shutting down a machine, schedule time off on an agent, modify
> maintenance schedule, etc.
> The authorization support needs to be added to the v0 endpoints:
> * {{/master/machine/up}}
> * {{/master/machine/down}}
> * {{/master/maintenance/schedule}}
> * {{/master/maintenance/status}}
> as well as to the v1 calls:
> * {{GET_MAINTENANCE_STATUS}}
> * {{GET_MAINTENANCE_SCHEDULE}}
> * {{UPDATE_MAINTENANCE_SCHEDULE}}
> * {{START_MAINTENANCE}}
> * {{STOP_MAINTENANCE}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7414) Enable authorization for master's logging API calls: GET_LOGGING_LEVEL and SET_LOGGING_LEVEL
[
https://issues.apache.org/jira/browse/MESOS-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam B updated MESOS-7414:
--
Shepherd: Till Toenshoff (was: Adam B)
> Enable authorization for master's logging API calls: GET_LOGGING_LEVEL and
> SET_LOGGING_LEVEL
> -
>
> Key: MESOS-7414
> URL: https://issues.apache.org/jira/browse/MESOS-7414
> Project: Mesos
> Issue Type: Task
> Components: HTTP API, master
>Reporter: Alexander Rojas
>Assignee: Alexander Rojas
> Labels: mesosphere, operator, security
>
> The Operator API calls {{GET_LOGGING_LEVEL}} and {{SET_LOGGING_LEVEL}} lack
> authorization so any recognized user will be able to change the logging level
> of a given master.
> The v0 endpoint {{/logging/toggle}} has authorization through the
> {{GET_ENDPOINT_WITH_PATH}} action. We need to decide whether it should also
> use additional authorization.
> Note that there are already actions defined for authorization of these
> actions as they were already implemented in the agent.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7415) Add authorization to master's operator maintenance API in v0 and v1
[
https://issues.apache.org/jira/browse/MESOS-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam B updated MESOS-7415:
--
Shepherd: Till Toenshoff (was: Adam B)
> Add authorization to master's operator maintenance API in v0 and v1
> ---
>
> Key: MESOS-7415
> URL: https://issues.apache.org/jira/browse/MESOS-7415
> Project: Mesos
> Issue Type: Task
> Components: c++ api, HTTP API, master
>Reporter: Alexander Rojas
>Assignee: Adam B
> Labels: authorization, mesosphere, security
>
> None of the maintenance primitives in either API v0 or API v1 have any kind
> of authorization, which allows any user with valid credentials to do things
> such as shutting down a machine, schedule time off on an agent, modify
> maintenance schedule, etc.
> The authorization support needs to be added to the v0 endpoints:
> * {{/master/machine/up}}
> * {{/master/machine/down}}
> * {{/master/maintenance/schedule}}
> * {{/master/maintenance/status}}
> as well as to the v1 calls:
> * {{GET_MAINTENANCE_STATUS}}
> * {{GET_MAINTENANCE_SCHEDULE}}
> * {{UPDATE_MAINTENANCE_SCHEDULE}}
> * {{START_MAINTENANCE}}
> * {{STOP_MAINTENANCE}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7488) Add `--ip6` and `--ip6_discovery_command` flag to Mesos agent
[ https://issues.apache.org/jira/browse/MESOS-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kone updated MESOS-7488: -- Shepherd: Benjamin Hindman (was: Jie Yu) > Add `--ip6` and `--ip6_discovery_command` flag to Mesos agent > - > > Key: MESOS-7488 > URL: https://issues.apache.org/jira/browse/MESOS-7488 > Project: Mesos > Issue Type: Task > Components: agent >Reporter: Avinash Sridharan >Assignee: Avinash Sridharan > Labels: mesosphere > Fix For: 1.4.0 > > > As a first step to support IPv6 containers on Mesos, we need to provide > `--ip6` and `--ip6_discovery_command` flags to the agent so that the operator > can specify an IPv6 address for the `libprocess` actor on the agent. In this > ticket we will not aim to add IPv6 communication support for Mesos but will > aim to use the IPv6 address provided by the operator to fill in the v6 > address for any containers running on the host network in a dual stack > environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-6919) Libprocess reinit code leaks SSL server socket FD
[
https://issues.apache.org/jira/browse/MESOS-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kone updated MESOS-6919:
--
Sprint: Mesosphere Sprint 52, Mesosphere Sprint 53, Mesosphere Sprint 54,
Mesosphere Sprint 55 (was: Mesosphere Sprint 52, Mesosphere Sprint 53,
Mesosphere Sprint 54, Mesosphere Sprint 55, Mesosphere Sprint 57)
> Libprocess reinit code leaks SSL server socket FD
> -
>
> Key: MESOS-6919
> URL: https://issues.apache.org/jira/browse/MESOS-6919
> Project: Mesos
> Issue Type: Bug
> Components: libprocess
>Affects Versions: 1.2.0
>Reporter: Greg Mann
>Assignee: Joseph Wu
> Labels: libprocess, mesosphere, ssl
>
> After [this commit|https://github.com/apache/mesos/commit/789e9f7], it was
> discovered that tests which use {{process::reinitialize}} to switch between
> SSL and non-SSL modes will leak the file descriptor associated with the
> server socket {{\_\_s\_\_}}. This can be reproduced by running the following
> trivial test in repetition:
> {code}
> diff --git a/src/tests/scheduler_tests.cpp b/src/tests/scheduler_tests.cpp
> index 1ff423f..d5fd575 100644
> --- a/src/tests/scheduler_tests.cpp
> +++ b/src/tests/scheduler_tests.cpp
> @@ -1821,6 +1821,12 @@ INSTANTIATE_TEST_CASE_P(
> #endif // USE_SSL_SOCKET
> +TEST_P(SchedulerSSLTest, LeakTest)
> +{
> + ::sleep(1);
> +}
> +
> +
> // Tests that a scheduler can subscribe, run a task, and then tear itself
> down.
> TEST_P(SchedulerSSLTest, RunTaskAndTeardown)
> {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7312) Update Resource proto for storage resource providers.
[
https://issues.apache.org/jira/browse/MESOS-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kone updated MESOS-7312:
--
Sprint: Mesosphere Sprint 53, Mesosphere Sprint 54, Mesosphere Sprint 55,
Mesosphere Sprint 56 (was: Mesosphere Sprint 53, Mesosphere Sprint 54,
Mesosphere Sprint 55, Mesosphere Sprint 56, Mesosphere Sprint 57)
> Update Resource proto for storage resource providers.
> -
>
> Key: MESOS-7312
> URL: https://issues.apache.org/jira/browse/MESOS-7312
> Project: Mesos
> Issue Type: Bug
>Reporter: Benjamin Bannier
>Assignee: Benjamin Bannier
>
> Storage resource provider support requires a number of changes to the
> {{Resource}} proto:
> * support for {{RAW}} and {{BLOCK}} type {{Resource::DiskInfo::Source}}
> * {{ResourceProviderID}} in Resource
> * {{Resource::DiskInfo::Source::Path}} should be {{optional}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7578) Write a proposal to make the I/O Switchboards optional
[ https://issues.apache.org/jira/browse/MESOS-7578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gastón Kleiman updated MESOS-7578: -- Story Points: 5 > Write a proposal to make the I/O Switchboards optional > -- > > Key: MESOS-7578 > URL: https://issues.apache.org/jira/browse/MESOS-7578 > Project: Mesos > Issue Type: Task > Components: containerization >Reporter: Gastón Kleiman >Assignee: Gastón Kleiman > Labels: check, containerizer, health-check, mesosphere > > Right now DEBUG containers can only be started using the > LaunchNestedContainerSession API call. They will enter its parent’s > namespaces, inherit environment variables, stream its I/O, and Mesos will tie > their life-cycle to the lifetime of the HTTP connection. > Streaming the I/O of a container requires an I/O Switchboard and adds some > overhead and complexity: > - Mesos will launch an extra process, called an I/O Switchboard for each > nested container. These process aren’t free, they take some time to > create/destroy and consume resources. > - I/O Switchboards are managed by a complex isolator. > - /O Swichboards introduce new race conditions, and have been a source of > deadlocks in the past. > Some use cases require some of the features provided by DEBUG containers, but > don’t need the functionality provided by the I/O switchboard. For instance, > the Default Executor uses DEBUG containers to perform (health)checks, but it > doesn’t need to stream anything to/from the container. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-6916) Improve health checks validation
[ https://issues.apache.org/jira/browse/MESOS-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gastón Kleiman updated MESOS-6916: -- Story Points: 2 > Improve health checks validation > > > Key: MESOS-6916 > URL: https://issues.apache.org/jira/browse/MESOS-6916 > Project: Mesos > Issue Type: Bug >Reporter: Gastón Kleiman >Assignee: Gastón Kleiman > Labels: health-check, mesosphere > > The "general" fields should also be validated (i.e., `timeout_seconds`), > similar to what's done in https://reviews.apache.org/r/55458/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7327) Add a test with multiple tasks and checks for the default executor.
[ https://issues.apache.org/jira/browse/MESOS-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gastón Kleiman updated MESOS-7327: -- Story Points: 2 (was: 1) > Add a test with multiple tasks and checks for the default executor. > --- > > Key: MESOS-7327 > URL: https://issues.apache.org/jira/browse/MESOS-7327 > Project: Mesos > Issue Type: Improvement >Reporter: Alexander Rukletsov >Assignee: Gastón Kleiman > Labels: health-check, mesosphere > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7575) Support reservation refinement
[
https://issues.apache.org/jira/browse/MESOS-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042936#comment-16042936
]
Vinod Kone commented on MESOS-7575:
---
Can you move it to "In Progress" and add story points?
> Support reservation refinement
> --
>
> Key: MESOS-7575
> URL: https://issues.apache.org/jira/browse/MESOS-7575
> Project: Mesos
> Issue Type: Task
>Reporter: Michael Park
>Assignee: Michael Park
>
> With the introduction of hierarchical roles, Mesos provides a mechanism to
> delegate resources down a hierarchy.
> To complement this, we'll introduce a mechanism to *refine* the reservations
> down the hierarchy.
> For example, given resources allocated to role {{foo}}, it can be further
> reserved for {{foo/bar}}.
> When the resources allocated to {{foo/bar}} is unreserved, it goes back to
> where it came from. In this case, back to role {{foo}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7640) Docker containerizer fails to set sandbox logs ownership correctly.
[ https://issues.apache.org/jira/browse/MESOS-7640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Toenshoff updated MESOS-7640: -- Sprint: Mesosphere Sprint 57 > Docker containerizer fails to set sandbox logs ownership correctly. > --- > > Key: MESOS-7640 > URL: https://issues.apache.org/jira/browse/MESOS-7640 > Project: Mesos > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Till Toenshoff >Assignee: Till Toenshoff >Priority: Blocker > Labels: mesosphere > > When using the docker containerizer in connection with a task that has no > command user set but a framework that has "nobody" set as its default user, > the resulting sandbox logs will be owned by the agent owning user (e.g. root) > but not the framework owner (e.g. nobody). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7374) Running DOCKER images in Mesos Container Runtime without `linux/filesystem` isolation enabled renders host unusable
[
https://issues.apache.org/jira/browse/MESOS-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kone updated MESOS-7374:
--
Shepherd: Gilbert Song (was: Jie Yu)
> Running DOCKER images in Mesos Container Runtime without `linux/filesystem`
> isolation enabled renders host unusable
> ---
>
> Key: MESOS-7374
> URL: https://issues.apache.org/jira/browse/MESOS-7374
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Affects Versions: 1.2.0
>Reporter: Tim Harper
>Assignee: Chun-Hung Hsiao
>Priority: Critical
> Labels: containerizer, mesosphere
>
> If I run the pod below (using Marathon 1.4.2) against a mesos agent that has
> the flags (also below), then the overlay filesystem replaces the system root
> mount, effectively rendering the host unusable until reboot.
> flags:
> - {{--containerizers mesos,docker}}
> - {{--image_providers APPC,DOCKER}}
> - {{--isolation cgroups/cpu,cgroups/mem,docker/runtime}}
> pod definition for Marathon:
> {code:java}
> {
> "id": "/simplepod",
> "scaling": { "kind": "fixed", "instances": 1 },
> "containers": [
> {
> "name": "sleep1",
> "exec": { "command": { "shell": "sleep 1000" } },
> "resources": { "cpus": 0.1, "mem": 32 },
> "image": {
> "id": "alpine",
> "kind": "DOCKER"
> }
> }
> ],
> "networks": [ {"mode": "host"} ]
> }
> {code}
> Mesos should probably check for this and avoid replacing the system root
> mount point at startup or launch time.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-6972) Improve performance of protobuf message passing by removing RepeatedPtrField to vector conversion.
[
https://issues.apache.org/jira/browse/MESOS-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042695#comment-16042695
]
Dmitry Zhuk commented on MESOS-6972:
[~bmahler], what do you think about taking a different approach: since in many
cases {{RepeatedPtrField}} needs to be converted to {{vector}} anyway, e.g. to
pass to internal data structures, we can instead {{Swap}} values from
{{RepeatedPtrField}} to {{vector}}. This does not add too much overhead
compared to {{RepeatedPtrField}}.
Currently a handler can be installed by calling something like this:
{code}
template
void install(
void (T::*method)(const process::UPID&, P1C),
P1 (M::*param1)() const);
{code}
{{P1 (M::*param1)() const}} is an issue here, as it prevents from swapping from
returned const {{RepeatedPtrField}}.
We can either remove {{const}} from {{param1}} and update code to use
{{mutable_*}} versions to access message properties, or keep it as is, and do
{{const_cast}} as a part of conversion. The latter however, somewhat depends on
protobuf internals and could break for non-primitive defaults.
I've already tested this change ({{const_cast}} variant), and results look
promising for hotspots like {{reregisterSlave}}.
> Improve performance of protobuf message passing by removing RepeatedPtrField
> to vector conversion.
> --
>
> Key: MESOS-6972
> URL: https://issues.apache.org/jira/browse/MESOS-6972
> Project: Mesos
> Issue Type: Improvement
>Reporter: Benjamin Mahler
> Labels: tech-debt
>
> Currently, all protobuf message handlers must take a {{vector}} for repeated
> fields, rather than a {{RepeatedPtrField}}.
> This requires that a copy be performed of the repeated field's entries (see
> [here|https://github.com/apache/mesos/blob/9228ebc239dac42825390bebc72053dbf3ae7b09/3rdparty/libprocess/include/process/protobuf.hpp#L78-L87]),
> which can be very expensive in some cases. We should avoid requiring this
> expense on the callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-5962) Support multiple health checks per task.
[ https://issues.apache.org/jira/browse/MESOS-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042524#comment-16042524 ] Alexander Rukletsov commented on MESOS-5962: [~xds2000], we had some initial discussions about it and decided not to implement multiple checks back then. Here is the initial design with concerns against multiple health checks: https://docs.google.com/document/d/1rdX68Rtm4JO_uOYCnbM_hoIFEglxEyuq6Db2macYzsM/edit#heading=h.va2wsvg7t14p > Support multiple health checks per task. > > > Key: MESOS-5962 > URL: https://issues.apache.org/jira/browse/MESOS-5962 > Project: Mesos > Issue Type: Improvement >Reporter: Alexander Rukletsov > Labels: check, health-check, mesosphere > > Currently, only a single check and a single health check per task is > supported. Consider supporting multiple checks and/or health checks. There > are various approaches how to treat them: > * do health aggregation in Mesos or delegate it to a frameworks, > * have a single or multiple restart policies (one per health check), > * introduce health check ids or not. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-7634) OsTest.ChownNoAccess fails on s390x machines
[
https://issues.apache.org/jira/browse/MESOS-7634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042508#comment-16042508
]
vandita commented on MESOS-7634:
Hi Vinod,
I had started building Mesos from the link given
https://github.com/vinodkone/mesos/tree/vinod/s390x.Cloned the branch
vinod/s390x.
Followed steps given here
https://github.com/linux-on-ibm-z/docs/wiki/Building-Apache-Mesos . On make
command the build initially failed @ 3rd party module of protobuf for no s390x
support. Added that support and the build went ahead. The build latter got
stuck @ 3rd party module of leveldb for no support of atomic operations.Added
that.Now the build is stuck @ zookeeper module with error src/mt_adaptor.c:487:
Error: Unrecognized opcode: `lock'.
Please let me know if there is something I have to do to get the build
successful. Moreover please tell me the branch of mesos I need to clone from
the above mentioned repository.
> OsTest.ChownNoAccess fails on s390x machines
>
>
> Key: MESOS-7634
> URL: https://issues.apache.org/jira/browse/MESOS-7634
> Project: Mesos
> Issue Type: Bug
>Reporter: Vinod Kone
>
> Running a custom branch of Mesos (with some fixes in docker build scripts for
> s390x) on s390x based CI machines throws the following error when running
> stout tests.
> {code}
> [ RUN ] OsTest.ChownNoAccess
> ../../../../3rdparty/stout/tests/os_tests.cpp:839: Failure
> Value of: os::chown(uid.get(), gid.get(), "one", true).isError()
> Actual: false
> Expected: true
> ../../../../3rdparty/stout/tests/os_tests.cpp:840: Failure
> Value of: os::chown(uid.get(), gid.get(), "one/two", true).isError()
> Actual: false
> {code}
> One can repro this by building Mesos from my custom branch here:
> https://github.com/vinodkone/mesos/tree/vinod/s390x
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7642) Consider unbundling protobuf dependency
[ https://issues.apache.org/jira/browse/MESOS-7642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Bannier updated MESOS-7642: Description: After MESOS-7228 we do not add additional patches on top of the bundled protobuf anymore. This should make it possible to always use an unbundled protobuf of appropriate version, i.e., to remove the protobuf third-party component. (was: After MESOS-7228 we do add additional patches on top of the bundled protobuf. This should make it possible to always use an unbundled protobuf of appropriate version, i.e., to remove the protobuf third-party component.) > Consider unbundling protobuf dependency > --- > > Key: MESOS-7642 > URL: https://issues.apache.org/jira/browse/MESOS-7642 > Project: Mesos > Issue Type: Improvement >Affects Versions: 1.4.0 >Reporter: Benjamin Bannier > > After MESOS-7228 we do not add additional patches on top of the bundled > protobuf anymore. This should make it possible to always use an unbundled > protobuf of appropriate version, i.e., to remove the protobuf third-party > component. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MESOS-7642) Consider unbundling protobuf dependency
Benjamin Bannier created MESOS-7642: --- Summary: Consider unbundling protobuf dependency Key: MESOS-7642 URL: https://issues.apache.org/jira/browse/MESOS-7642 Project: Mesos Issue Type: Improvement Affects Versions: 1.4.0 Reporter: Benjamin Bannier After MESOS-7228 we do add additional patches on top of the bundled protobuf. This should make it possible to always use an unbundled protobuf of appropriate version, i.e., to remove the protobuf third-party component. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
