[jira] [Commented] (MESOS-7104) Mesos-slave aborts after failed CNI network event
[
https://issues.apache.org/jira/browse/MESOS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889046#comment-15889046
]
Dan Osborne commented on MESOS-7104:
Unfortunately I don't have any more information on this. Closing
> Mesos-slave aborts after failed CNI network event
> -
>
> Key: MESOS-7104
> URL: https://issues.apache.org/jira/browse/MESOS-7104
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Dan Osborne
>
> I'm trying to network a task with a CNI plugin. I'm not sure what I've done
> wrong, It's probably a bad CNI config or missing plugins or something. But
> the error has managed to kill the entire slave process.
> {code}
> I0209 22:57:56.841765 3949 containerizer.cpp:992] Starting container
> f6d069b8-62f2-42c4-a8ff-145e366409f0 for executor
> 'calico-cni-marathon-test.1b93ed5d-ef45-11e6-906e-024245c0fe5e' of framework
> fbf5b855-bdf9-4a12-aeee-cc341ffe0f1b-
> mesos-slave: ../../3rdparty/stout/include/stout/option.hpp:111: const T&
> Option::get() const & [with T = std::basic_string]: Assertion
> `isSome()' failed.
> *** Aborted at 1486699076 (unix time) try "date -d @1486699076" if you are
> using GNU date ***
> PC: @ 0x7effbb6121d7 __GI_raise
> *** SIGABRT (@0xf5a) received by PID 3930 (TID 0x7effb3cdf700) from PID
> 3930; stack trace: ***
> @ 0x7effbbecc370 (unknown)
> @ 0x7effbb6121d7 __GI_raise
> @ 0x7effbb6138c8 __GI_abort
> @ 0x7effbb60b146 __assert_fail_base
> slave.service: main process exited, code=killed, status=6/ABRT
> {code}
> Will update with more info when possible.
> This is using the latest nightly mesos build at time of writing.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (MESOS-7105) Start Mesos Agent even if CNI dir doesn't exist
[
https://issues.apache.org/jira/browse/MESOS-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Osborne updated MESOS-7105:
---
Description:
Mesos-agent refuses to start if a CNI config directory has been configured but
does not exist. I propose that we don't bother to check this at start.
Now that Mesos scans for CNI configs at task launch, it's feasible that even
though the directory doesn't exist at launch, it will exist and be filled with
configs by the time the a task is run.
Specifically, for my use case, I'm trying to install CNI plugins using the
Docker Containerizer in Mesos. These tasks volume mount in the CNI directories
which the docker daemon automatically creates if they don't exist. But I can't
get the slave to start unless I make the CNI conf dir, which feels unnecessary.
{code}
Failed to create a containerizer: Could not create MesosContainerizer: Failed
to create isolator 'network/cni': The CNI network configuration directory
'/etc/cni/net.d/' does not exist
{code}
was:
Mesos-agent refuses to start if a CNI config directory has been configured but
does not exist. I propose that we don't bother to check this at start.
Now that Mesos scans for CNI configs at task launch, it's feasible that even
though the directory doesn't exist at launch, it will be filled with configs by
the time the a task is run.
Specifically, for my use case, I'm trying to install CNI plugins using the
Docker Containerizer in Mesos. These tasks volume mount in the CNI directories
which the docker daemon automatically creates if they don't exist. But I can't
get the slave to start unless I make the CNI conf dir, which feels unnecessary.
{code}
Failed to create a containerizer: Could not create MesosContainerizer: Failed
to create isolator 'network/cni': The CNI network configuration directory
'/etc/cni/net.d/' does not exist
{code}
> Start Mesos Agent even if CNI dir doesn't exist
> ---
>
> Key: MESOS-7105
> URL: https://issues.apache.org/jira/browse/MESOS-7105
> Project: Mesos
> Issue Type: Improvement
>Reporter: Dan Osborne
>Assignee: Avinash Sridharan
>
> Mesos-agent refuses to start if a CNI config directory has been configured
> but does not exist. I propose that we don't bother to check this at start.
> Now that Mesos scans for CNI configs at task launch, it's feasible that even
> though the directory doesn't exist at launch, it will exist and be filled
> with configs by the time the a task is run.
> Specifically, for my use case, I'm trying to install CNI plugins using the
> Docker Containerizer in Mesos. These tasks volume mount in the CNI
> directories which the docker daemon automatically creates if they don't
> exist. But I can't get the slave to start unless I make the CNI conf dir,
> which feels unnecessary.
> {code}
> Failed to create a containerizer: Could not create MesosContainerizer: Failed
> to create isolator 'network/cni': The CNI network configuration directory
> '/etc/cni/net.d/' does not exist
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7105) Start Mesos Agent even if CNI dir doesn't exist
Dan Osborne created MESOS-7105:
--
Summary: Start Mesos Agent even if CNI dir doesn't exist
Key: MESOS-7105
URL: https://issues.apache.org/jira/browse/MESOS-7105
Project: Mesos
Issue Type: Improvement
Reporter: Dan Osborne
Assignee: Avinash Sridharan
Mesos-agent refuses to start if a CNI config directory has been configured but
does not exist. I propose that we don't bother to check this at start.
Now that Mesos scans for CNI configs at task launch, it's feasible that even
though the directory doesn't exist at launch, it will be filled with configs by
the time the a task is run.
Specifically, for my use case, I'm trying to install CNI plugins using the
Docker Containerizer in Mesos. These tasks volume mount in the CNI directories
which the docker daemon automatically creates if they don't exist. But I can't
get the slave to start unless I make the CNI conf dir, which feels unnecessary.
{code}
Failed to create a containerizer: Could not create MesosContainerizer: Failed
to create isolator 'network/cni': The CNI network configuration directory
'/etc/cni/net.d/' does not exist
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (MESOS-7104) Mesos-slave aborts after failed CNI network event
Dan Osborne created MESOS-7104:
--
Summary: Mesos-slave aborts after failed CNI network event
Key: MESOS-7104
URL: https://issues.apache.org/jira/browse/MESOS-7104
Project: Mesos
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Dan Osborne
Assignee: Jie Yu
I'm trying to network a task with a CNI plugin. I'm not sure what I've done
wrong, It's probably a bad CNI config or missing plugins or something. But the
error has managed to kill the entire slave process.
{code}
I0209 22:57:56.841765 3949 containerizer.cpp:992] Starting container
f6d069b8-62f2-42c4-a8ff-145e366409f0 for executor
'calico-cni-marathon-test.1b93ed5d-ef45-11e6-906e-024245c0fe5e' of framework
fbf5b855-bdf9-4a12-aeee-cc341ffe0f1b-
mesos-slave: ../../3rdparty/stout/include/stout/option.hpp:111: const T&
Option::get() const & [with T = std::basic_string]: Assertion
`isSome()' failed.
*** Aborted at 1486699076 (unix time) try "date -d @1486699076" if you are
using GNU date ***
PC: @ 0x7effbb6121d7 __GI_raise
*** SIGABRT (@0xf5a) received by PID 3930 (TID 0x7effb3cdf700) from PID 3930;
stack trace: ***
@ 0x7effbbecc370 (unknown)
@ 0x7effbb6121d7 __GI_raise
@ 0x7effbb6138c8 __GI_abort
@ 0x7effbb60b146 __assert_fail_base
slave.service: main process exited, code=killed, status=6/ABRT
{code}
Will update with more info when possible.
This is using the latest nightly mesos build at time of writing.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-6679) Allow `network/cni` isolator to dynamically load CNI configuration
[ https://issues.apache.org/jira/browse/MESOS-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767272#comment-15767272 ] Dan Osborne commented on MESOS-6679: Resolved? Is there a link to the changes? Can we close the dupe 6567? > Allow `network/cni` isolator to dynamically load CNI configuration > -- > > Key: MESOS-6679 > URL: https://issues.apache.org/jira/browse/MESOS-6679 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: Avinash Sridharan >Assignee: Avinash Sridharan > Labels: mesosphere > > Currently the `network/cni` isolator learns the CNI config at startup. In > case the CNI config changes after the agent has started, the agent needs to > be restarted in order to learn any modifications to the CNI config. > We would like the `network/cni` isolator to be able to load CNI config on the > fly without a restart. To achieve this we plan to introduce a new endpoint on > the `network/cni` isolator that would allow the operator to explicitly ask > the `network/cni` isolator to reload the configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6567) Actively Scan for CNI Configurations
[ https://issues.apache.org/jira/browse/MESOS-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667560#comment-15667560 ] Dan Osborne commented on MESOS-6567: Ah I didn't interpret "existing" correctly in your post. Makes sense. So since there already is a filescan happening at container runtime, it hopefully shouldn't be too difficult to expand it to networks that don't already exist too. > Actively Scan for CNI Configurations > > > Key: MESOS-6567 > URL: https://issues.apache.org/jira/browse/MESOS-6567 > Project: Mesos > Issue Type: Improvement >Reporter: Dan Osborne > > Mesos-Agent currently loads the CNI configs into memory at startup. After > this point, new configurations that are added will remain unknown to the > Mesos Agent process until it is restarted. > This ticket is to request that the Mesos Agent process can the CNI config > directory each time it is networking a task, so that modifying, adding, and > removing networks will not require a slave reboot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6567) Actively Scan for CNI Configurations
[ https://issues.apache.org/jira/browse/MESOS-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665861#comment-15665861 ] Dan Osborne commented on MESOS-6567: I've reproed this on 1.0.1 and can confirm it is not scanned and picked up at runtime, but instead requires a reboot of the slave process. I think it's because scanning of CNI networks happens in NetworkCniIsolatorProcess::create which I believe happens at boot, not during NetworkCniIsolatorProcess::Isolate, which happens at container runtime? > Actively Scan for CNI Configurations > > > Key: MESOS-6567 > URL: https://issues.apache.org/jira/browse/MESOS-6567 > Project: Mesos > Issue Type: Improvement >Reporter: Dan Osborne > > Mesos-Agent currently loads the CNI configs into memory at startup. After > this point, new configurations that are added will remain unknown to the > Mesos Agent process until it is restarted. > This ticket is to request that the Mesos Agent process can the CNI config > directory each time it is networking a task, so that modifying, adding, and > removing networks will not require a slave reboot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-6567) Actively Scan for CNI Configurations
Dan Osborne created MESOS-6567: -- Summary: Actively Scan for CNI Configurations Key: MESOS-6567 URL: https://issues.apache.org/jira/browse/MESOS-6567 Project: Mesos Issue Type: Improvement Reporter: Dan Osborne Mesos-Agent currently loads the CNI configs into memory at startup. After this point, new configurations that are added will remain unknown to the Mesos Agent process until it is restarted. This ticket is to request that the Mesos Agent process can the CNI config directory each time it is networking a task, so that modifying, adding, and removing networks will not require a slave reboot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6282) CNI isolator should print plugin's stderr
[ https://issues.apache.org/jira/browse/MESOS-6282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542751#comment-15542751 ] Dan Osborne commented on MESOS-6282: Agreed, but I think it'd be a good idea to print a low-level log message (INFO?) even when exit status is 0 as well. > CNI isolator should print plugin's stderr > - > > Key: MESOS-6282 > URL: https://issues.apache.org/jira/browse/MESOS-6282 > Project: Mesos > Issue Type: Improvement > Components: containerization, isolation, network >Reporter: Dan Osborne >Assignee: Avinash Sridharan > > It's quite difficult for both Operators and CNI plugin developers to diagnose > CNI plugin errors in production or in test when the only error information > available is the stdout error string returned by the plugin (assuming it > managed to even print its correctly formatted text to stdout). > Many CNI plugins print logging information to stderr, [as per the CNI > spec|https://github.com/containernetworking/cni/blob/master/SPEC.md#result]: > bq. In addition, stderr can be used for unstructured output such as logs. > Therefore, I propose the Mesos CNI Isolator capture the CNI plugin's stderr > output and log it to the Agent Logs, for easier diagnosis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-6282) CNI isolator should print plugin's stderr
Dan Osborne created MESOS-6282: -- Summary: CNI isolator should print plugin's stderr Key: MESOS-6282 URL: https://issues.apache.org/jira/browse/MESOS-6282 Project: Mesos Issue Type: Improvement Components: containerization, isolation, network Reporter: Dan Osborne It's quite difficult for both Operators and CNI plugin developers to diagnose CNI plugin errors in production or in test when the only error information available is the stdout error string returned by the plugin (assuming it managed to even print its correctly formatted text to stdout). Many CNI plugins print logging information to stderr, [as per the CNI spec|https://github.com/containernetworking/cni/blob/master/SPEC.md#result]: bq. In addition, stderr can be used for unstructured output such as logs. Therefore, I propose the Mesos CNI Isolator capture the CNI plugin's stderr output and log it to the Agent Logs, for easier diagnosis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-6129) Use libcurl instead of shelling out to curl
Dan Osborne created MESOS-6129:
--
Summary: Use libcurl instead of shelling out to curl
Key: MESOS-6129
URL: https://issues.apache.org/jira/browse/MESOS-6129
Project: Mesos
Issue Type: Improvement
Reporter: Dan Osborne
The unified Containerizer shells out to curl when downloading docker images.
Curl is not a listed requirement to run Mesos, so users will see a stacktrace
if they don't have curl installed.
These shells are called in the following two locations:
https://github.com/apache/mesos/blob/master/src/uri/fetchers/docker.cpp#L99-L104
https://github.com/apache/mesos/blob/master/src/uri/fetchers/curl.cpp#L99-L107
If there is no specific reason why libcurl is not used directly, then we should
switch to libcurl calls instead.
Resulting stack trace in Agent logs from not having curl installed:
{code}
E0818 00:39:07.55784311 slave.cpp:3976] Container
'af48e158-631e-4b9a-8fb9-53b481787a40' for executor
'database.2be5771a-64dc-11e6-84fd-0242ac110005' of framework
78ff8c50-738c-4aa0-8525-74b0752ea836- failed to start: Failed to perform
'curl': ABORT:
(../../../../../..//tmp/mesos-build/mesos-repo/3rdparty/libprocess/include/process/posix/subprocess.hpp:306):
Failed to os::execvpe on path 'curl': No such file or directory
*** Aborted at 1471480747 (unix time) try "date -d @1471480747" if you are
using GNU date ***
PC: @ 0x7fc2aa6c8c37 (unknown)
*** SIGABRT (@0x5f) received by PID 95 (TID 0x7fc2a1e38700) from PID 95; stack
trace: ***
@ 0x7fc2aaa67330 (unknown)
@ 0x7fc2aa6c8c37 (unknown)
@ 0x7fc2aa6cc028 (unknown)
@ 0x41336c _Abort()
@ 0x4133ac _Abort()
@ 0x7fc2ac8c764e process::internal::childMain()
@ 0x7fc2ac8c654d std::_Function_handler<>::_M_invoke()
@ 0x7fc2ac8c64f3 process::internal::defaultClone()
@ 0x7fc2ac8c8055 process::internal::cloneChild()
@ 0x7fc2ac8c59a6 process::subprocess()
@ 0x7fc2ac34b723 mesos::uri::curl()
@ 0x7fc2ac34defe mesos::uri::curl()
@ 0x7fc2ac352bf4 mesos::uri::DockerFetcherPluginProcess::fetch()
@ 0x7fc2ac357887
_ZNSt17_Function_handlerIFvPN7process11ProcessBaseEEZNS0_8dispatchI7NothingN5mesos3uri26DockerFetcherPluginProcessERKNS6_3URIERKSsS9_SsEENS0_6FutureIT_EERKNS0_3PIDIT0_EEMSI_FSG_T1_T2_ET3_T4_EUlS2_E_E9_M_invokeERKSt9_Any_dataS2_
@ 0x7fc2ac891791 process::ProcessManager::resume()
@ 0x7fc2ac891a97
_ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
@ 0x7fc2aaf3ca60 (unknown)
@ 0x7fc2aaa5f184 start_thread
@ 0x7fc2aa78c37d (unknown)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-5592) Pass NetworkInfo.Labels to CNI Plugins
[ https://issues.apache.org/jira/browse/MESOS-5592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328744#comment-15328744 ] Dan Osborne commented on MESOS-5592: The CNI maintainers have responded with an update to the CNI spec which closely matches the design doc linked in this report: https://github.com/containernetworking/cni/pull/247 I've updated the review board request to match their spec. > Pass NetworkInfo.Labels to CNI Plugins > -- > > Key: MESOS-5592 > URL: https://issues.apache.org/jira/browse/MESOS-5592 > Project: Mesos > Issue Type: Improvement >Reporter: Dan Osborne > Fix For: 1.0.0 > > > Mesos has adopted the Container Network Interface as a simple means of > networking Mesos tasks launched by the Unified Containerizer. The CNI > specification covers a minimum feature set, granting the flexibility to add > customized networking functionality in the form of agreements made between > the orchestrator and CNI plugin. > This proposal is to pass NetworkInfo.Labels to the CNI plugin by injecting it > into the CNI network configuration json during plugin invocation. > Design Doc on this change: > https://docs.google.com/document/d/1rxruCCcJqpppsQxQrzTbHFVnnW6CgQ2oTieYAmwL284/edit?usp=sharing > reviewboard: https://reviews.apache.org/r/48527/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5592) Pass NetworkInfo.Labels to CNI Plugins
Dan Osborne created MESOS-5592: -- Summary: Pass NetworkInfo.Labels to CNI Plugins Key: MESOS-5592 URL: https://issues.apache.org/jira/browse/MESOS-5592 Project: Mesos Issue Type: Improvement Reporter: Dan Osborne Fix For: 1.0.0 Mesos has adopted the Container Network Interface as a simple means of networking Mesos tasks launched by the Unified Containerizer. The CNI specification covers a minimum feature set, granting the flexibility to add customized networking functionality in the form of agreements made between the orchestrator and CNI plugin. This proposal is to pass NetworkInfo.Labels to the CNI plugin by injecting it into the CNI network configuration json during plugin invocation. Design Doc on this change: https://docs.google.com/document/d/1rxruCCcJqpppsQxQrzTbHFVnnW6CgQ2oTieYAmwL284/edit?usp=sharing reviewboard: https://reviews.apache.org/r/48527/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5325) Mesos can't determine if task IP is reachable
[ https://issues.apache.org/jira/browse/MESOS-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314793#comment-15314793 ] Dan Osborne commented on MESOS-5325: My last two comments are invalid. Since CNI was introduced, the presence of the IpAddress field no longer indicates that a task will be receiving a routable IP address. Prime use case: users can specify `NetworkInfo` to launch a bridged task using CNI. This will result in a unroutable container IP. So there is indeed no flag for Mesos to be able to tell if a task is routable via its Container IP or not. > Mesos can't determine if task IP is reachable > - > > Key: MESOS-5325 > URL: https://issues.apache.org/jira/browse/MESOS-5325 > Project: Mesos > Issue Type: Bug >Reporter: Dan Osborne > > I have uncovered a design flaw that affects ip-per-container tasks when run > in a cluster alongside non ip-per-container tasks. This affects > docker-libnetwork, netmodules, and I suspect it will also affect CNI. > After Mesos launches a docker bridge task, it fills the task's networkinfo > field with the docker bridge IP assigned to that task. Because of this > behavior, when a launched task's NetworkInfo is later utilized by Mesos > components, it is unknown if it is filled with an IP address accessible > throughout the cluster, or if it is not. > A common use case where this is a problem can be encountered when using Mesos > DNS. Mesos-DNS has a configuration setting that tells it which information to > respond to a query with: NetworkInfo, or HostIP. If it has been configured to > prefer NetworkInfo, it correctly resolves ip-per-container containers to > their unique IP. But, because the docker bridge IP is also stored in > NetworkInfo, it will incorrectly resolve docker-bridge containers to an IP > address not accessible from anywhere besides the slave they are on. This > breaks DNS resolutions in Mesos. > I believe Mesos needs a way to distinguish between tasks which are accessible > via their IP and tasks that are not. > One fix would be to prevent Mesos from filling in NetworkInfo for a task if > it is known that the task is not reachable throughout the cluster via that > address. Essentially, NetworkInfo could be interpreted as a boolean - Its > presence means this task is addressable. Its absence means the task is not. > In practice, this would mean it gets filled in for CNI tasks, netmodules > tasks, and docker tasks bound to the host networking namespace. It would not > get filled in for docker bridge tasks. > I believe this change would be fairly minimum in scope. To implement it, > Mesos would need to be changed to not store Docker Bridge IP's in NetworkInfo. > I'm also open to discussion and other suggestions on how to resolve this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5412) Support CNI_ARGS
[ https://issues.apache.org/jira/browse/MESOS-5412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304992#comment-15304992 ] Dan Osborne commented on MESOS-5412: [~hartem] Not planning on submitting a patch by then, so feel free to bump this out. I'm no longer convinced that CNI args would be the right place to inject network policy definitions for a Task. Shall we leave this issue open as a backlog item until a more pressing need / more defined use case for CNI_ARGS arises? > Support CNI_ARGS > > > Key: MESOS-5412 > URL: https://issues.apache.org/jira/browse/MESOS-5412 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: Dan Osborne > > Mesos-CNI should support the > [CNI_ARGS|https://github.com/containernetworking/cni/blob/master/SPEC.md#parameters] > field. > This would allow CNI plugins to be able to implement advanced networking > capabilities without needing modifications to Mesos. Current use case I am > facing: Allowing users to specify policy for their CNI plugin. > I'm proposing the following implementation: Pass a task's [NetworkInfo > Labels|https://github.com/apache/mesos/blob/b7e50fe8b20c96cda5546db5f2c2f47bee461edb/include/mesos/mesos.proto#L1732] > to the CNI plugin as CNI_ARGS. CNI args are simply key-value pairs split by > a '=', e.g. "FOO=BAR;ABC=123", which could be easily generated from the > NetworkInfo's key-value labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5453) CNI should not store subnet of address in NetworkInfo
[ https://issues.apache.org/jira/browse/MESOS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304948#comment-15304948 ] Dan Osborne commented on MESOS-5453: First time submitting - Is there something left for me to do to close this out? > CNI should not store subnet of address in NetworkInfo > - > > Key: MESOS-5453 > URL: https://issues.apache.org/jira/browse/MESOS-5453 > Project: Mesos > Issue Type: Bug >Reporter: Dan Osborne >Assignee: Dan Osborne > Labels: mesosphere > Fix For: 0.29.0 > > > When the CNI isolator executes the CNI plugin, that CNI plugin will return an > IP Address and Subnet (192.168.0.1/32). Mesos should strip the subnet before > storing the address in the Task.NetworkInfo.IPAddress. > Reason being - most current mesos components are not expecting a subnet in > the Task's NetworkInfo.IPAddress, and instead expect just the IP address. > This can cause errors in those components, such as Mesos-DNS failing to > return a NetworkInfo address (and instead defaulting to the next configured > IPSource), and Marathon generating invalid links to tasks (as it includes /32 > in the link) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5453) CNI should not store subnet of address in NetworkInfo
[ https://issues.apache.org/jira/browse/MESOS-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Osborne updated MESOS-5453: --- Description: When the CNI isolator executes the CNI plugin, that CNI plugin will return an IP Address and Subnet (192.168.0.1/32). Mesos should strip the subnet before storing the address in the Task.NetworkInfo.IPAddress. Reason being - most current mesos components are not expecting a subnet in the Task's NetworkInfo.IPAddress, and instead expect just the IP address. This can cause errors in those components, such as Mesos-DNS failing to return a NetworkInfo address (and instead defaulting to the next configured IPSource), and Marathon generating invalid links to tasks (as it includes /32 in the link) was:When the CNI isolator executes the CNI plugin, that CNI plugin will return an IP Address and Subnet (192.168.0.1/32). Mesos should strip the subnet before storing the address in the Task.NetworkInfo.IPAddress > CNI should not store subnet of address in NetworkInfo > - > > Key: MESOS-5453 > URL: https://issues.apache.org/jira/browse/MESOS-5453 > Project: Mesos > Issue Type: Bug >Reporter: Dan Osborne > > When the CNI isolator executes the CNI plugin, that CNI plugin will return an > IP Address and Subnet (192.168.0.1/32). Mesos should strip the subnet before > storing the address in the Task.NetworkInfo.IPAddress. > Reason being - most current mesos components are not expecting a subnet in > the Task's NetworkInfo.IPAddress, and instead expect just the IP address. > This can cause errors in those components, such as Mesos-DNS failing to > return a NetworkInfo address (and instead defaulting to the next configured > IPSource), and Marathon generating invalid links to tasks (as it includes /32 > in the link) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5453) CNI should not store subnet of address in NetworkInfo
Dan Osborne created MESOS-5453: -- Summary: CNI should not store subnet of address in NetworkInfo Key: MESOS-5453 URL: https://issues.apache.org/jira/browse/MESOS-5453 Project: Mesos Issue Type: Bug Reporter: Dan Osborne When the CNI isolator executes the CNI plugin, that CNI plugin will return an IP Address and Subnet (192.168.0.1/32). Mesos should strip the subnet before storing the address in the Task.NetworkInfo.IPAddress -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5325) Mesos can't determine if task IP is reachable
[ https://issues.apache.org/jira/browse/MESOS-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292468#comment-15292468 ] Dan Osborne commented on MESOS-5325: Basically, Mesos *can* determine if an IP Address is routable, based on whether or not a network_info was provided in the task's definition. > Mesos can't determine if task IP is reachable > - > > Key: MESOS-5325 > URL: https://issues.apache.org/jira/browse/MESOS-5325 > Project: Mesos > Issue Type: Bug >Reporter: Dan Osborne > > I have uncovered a design flaw that affects ip-per-container tasks when run > in a cluster alongside non ip-per-container tasks. This affects > docker-libnetwork, netmodules, and I suspect it will also affect CNI. > After Mesos launches a docker bridge task, it fills the task's networkinfo > field with the docker bridge IP assigned to that task. Because of this > behavior, when a launched task's NetworkInfo is later utilized by Mesos > components, it is unknown if it is filled with an IP address accessible > throughout the cluster, or if it is not. > A common use case where this is a problem can be encountered when using Mesos > DNS. Mesos-DNS has a configuration setting that tells it which information to > respond to a query with: NetworkInfo, or HostIP. If it has been configured to > prefer NetworkInfo, it correctly resolves ip-per-container containers to > their unique IP. But, because the docker bridge IP is also stored in > NetworkInfo, it will incorrectly resolve docker-bridge containers to an IP > address not accessible from anywhere besides the slave they are on. This > breaks DNS resolutions in Mesos. > I believe Mesos needs a way to distinguish between tasks which are accessible > via their IP and tasks that are not. > One fix would be to prevent Mesos from filling in NetworkInfo for a task if > it is known that the task is not reachable throughout the cluster via that > address. Essentially, NetworkInfo could be interpreted as a boolean - Its > presence means this task is addressable. Its absence means the task is not. > In practice, this would mean it gets filled in for CNI tasks, netmodules > tasks, and docker tasks bound to the host networking namespace. It would not > get filled in for docker bridge tasks. > I believe this change would be fairly minimum in scope. To implement it, > Mesos would need to be changed to not store Docker Bridge IP's in NetworkInfo. > I'm also open to discussion and other suggestions on how to resolve this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5412) Support CNI_ARGS
Dan Osborne created MESOS-5412: -- Summary: Support CNI_ARGS Key: MESOS-5412 URL: https://issues.apache.org/jira/browse/MESOS-5412 Project: Mesos Issue Type: Improvement Components: containerization Reporter: Dan Osborne Fix For: 0.29.0 Mesos-CNI should support the [CNI_ARGS](https://github.com/containernetworking/cni/blob/master/SPEC.md#parameters) field. This would allow CNI plugins to be able to implement advanced networking capabilities without needing modifications to Mesos. Current use case I am facing: Allowing users to specify policy for their CNI plugin. I'm proposing the following implementation: Pass a task's [NetworkInfo Labels](https://github.com/apache/mesos/blob/b7e50fe8b20c96cda5546db5f2c2f47bee461edb/include/mesos/mesos.proto#L1732) to the CNI plugin as CNI_ARGS. CNI args are simply key-value pairs split by a '=', e.g. "FOO=BAR;ABC=123", which could be easily generated from the NetworkInfo's key-value labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5325) Mesos can't determine if task IP is reachable
Dan Osborne created MESOS-5325: -- Summary: Mesos can't determine if task IP is reachable Key: MESOS-5325 URL: https://issues.apache.org/jira/browse/MESOS-5325 Project: Mesos Issue Type: Bug Reporter: Dan Osborne I have uncovered a design flaw that affects ip-per-container tasks when run in a cluster alongside non ip-per-container tasks. This affects docker-libnetwork, netmodules, and I suspect it will also affect CNI. After Mesos launches a docker bridge task, it fills the task's networkinfo field with the docker bridge IP assigned to that task. Because of this behavior, when a launched task's NetworkInfo is later utilized by Mesos components, it is unknown if it is filled with an IP address accessible throughout the cluster, or if it is not. A common use case where this is a problem can be encountered when using Mesos DNS. Mesos-DNS has a configuration setting that tells it which information to respond to a query with: NetworkInfo, or HostIP. If it has been configured to prefer NetworkInfo, it correctly resolves ip-per-container containers to their unique IP. But, because the docker bridge IP is also stored in NetworkInfo, it will incorrectly resolve docker-bridge containers to an IP address not accessible from anywhere besides the slave they are on. This breaks DNS resolutions in Mesos. I believe Mesos needs a way to distinguish between tasks which are accessible via their IP and tasks that are not. One fix would be to prevent Mesos from filling in NetworkInfo for a task if it is known that the task is not reachable throughout the cluster via that address. Essentially, NetworkInfo could be interpreted as a boolean - Its presence means this task is addressable. Its absence means the task is not. In practice, this would mean it gets filled in for CNI tasks, netmodules tasks, and docker tasks bound to the host networking namespace. It would not get filled in for docker bridge tasks. I believe this change would be fairly minimum in scope. To implement it, Mesos would need to be changed to not store Docker Bridge IP's in NetworkInfo. I'm also open to discussion and other suggestions on how to resolve this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5061) process.cpp:1966] Failed to shutdown socket with fd x: Transport endpoint is not connected
[
https://issues.apache.org/jira/browse/MESOS-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243063#comment-15243063
]
Dan Osborne commented on MESOS-5061:
I spent some time looking at Zogg's setup and found the following to be true.
After its initialization, the first thing the executor does is register with
the Slave. Since we're using the network isolator here, the registration
message should have a src address of the newly initialized networking
namespace. Since calico is handling the isolation, this means we'll see a
registration with an IP src from the default 192.168.0.0/16 range, like the
following example:
I0414 23:31:44.479730 205 slave.cpp:2642] Got registration for executor
'star_probe-b.0bd467c0-0299-11e6-ad3b-0242ac110005' of framework
6a1ae9aa-ad50-44c1-8809-58791c5bcbe5- from executor(1)@192.168.0.4:35454`
In Zogg's test, we're seeing a registration message that is using the Slave's
IP address. This is of course false information. When the slave then tries to
handshake with the registration request, it of course fails, since there is no
executor using that IP/port. This explains why we see tasks stuck in staging -
mesos-slave has completely lost contact with the executor.
Can someone shine light on how the Executor picks this IP, or if its just
extracted from the source IP of the registration Method?
Versioning info:
Mesos Manually built from 0.27.0
Net-modules (basically latest):
https://github.com/mesosphere/net-modules/commits/625b67992ceca535cf2c76ea980b64aa8f4b33e1
I'm going to work to get this reproducible using the net-modules docker-compose
demo. In the meantime, any thoughts?
> process.cpp:1966] Failed to shutdown socket with fd x: Transport endpoint is
> not connected
> --
>
> Key: MESOS-5061
> URL: https://issues.apache.org/jira/browse/MESOS-5061
> Project: Mesos
> Issue Type: Bug
> Components: containerization, modules
>Affects Versions: 0.27.0, 0.27.1, 0.28.0, 0.27.2
> Environment: Centos 7.1
>Reporter: Zogg
> Fix For: 0.29.0
>
>
> When launching a task through Marathon and asking the task to assign an IP
> (using Calico networking):
> {noformat}
> {
> "id":"/calico-apps",
> "apps": [
> {
> "id": "hello-world-1",
> "cmd": "ip addr && sleep 3",
> "cpus": 0.1,
> "mem": 64.0,
> "ipAddress": {
> "groups": ["calico-k8s-network"]
> }
> }
> ]
> }
> {noformat}
> Mesos slave fails to launch a task, locking in STAGING state forewer, with
> error:
> {noformat}
> [centos@rtmi-worker-001 mesos]$ tail mesos-slave.INFO
> I0325 20:35:43.420171 13495 slave.cpp:2642] Got registration for executor
> 'calico-apps_hello-world-1.23ff72e9-f2c9-11e5-bb22-be052ff413d3' of framework
> 23b404e4-700a-4348-a7c0-226239348981- from executor(1)@10.0.0.10:33443
> I0325 20:35:43.422652 13495 slave.cpp:1862] Sending queued task
> 'calico-apps_hello-world-1.23ff72e9-f2c9-11e5-bb22-be052ff413d3' to executor
> 'calico-apps_hello-world-1.23ff72e9-f2c9-11e5-bb22-be052ff413d3' of framework
> 23b404e4-700a-4348-a7c0-226239348981- at executor(1)@10.0.0.10:33443
> E0325 20:35:43.423159 13502 process.cpp:1966] Failed to shutdown socket with
> fd 22: Transport endpoint is not connected
> I0325 20:35:43.423316 13501 slave.cpp:3481] executor(1)@10.0.0.10:33443 exited
> {noformat}
> However, when deploying a task without ipAddress field, mesos slave launches
> a task successfully.
> Tested with various Mesos/Marathon/Calico versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MESOS-4823) Implement port forwarding in `network/cni` isolator
[ https://issues.apache.org/jira/browse/MESOS-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198135#comment-15198135 ] Dan Osborne edited comment on MESOS-4823 at 3/16/16 8:55 PM: - Thank you for providing the example use case. Can you explain, on a technical level, on what conditions you are planning to trigger creation of these ip-tables rules? I'm concerned that the capability you're trying to provide makes a lot of assumptions about both the mesos cluster and the CNI network's configurations, and to what degree both are accessible by the public network. I believe that if this behavior goes in, to some degree it should be opt-in or opt-out, as not all clusters nor CNI network's would want such a behavior. Some counter use cases - 1. if the CNI network _is_ assigning publicly accessible addresses, the port mapping becomes a redundant. 2. if they are using a load balancer, they would not need port forwarding as the load balancer will forward public requests onto the private CNI network. was (Author: djosborne): Thank you for providing the example use case. Can you explain, on a technical level, what condition you are planning that will trigger creation of these ip-tables rules? I'm concerned that the capability you're trying to provide makes a lot of assumptions about both the mesos cluster and the CNI network's configurations, and to what degree both are accessible by the public network. I believe that if this behavior goes in, to some degree it should be opt-in or opt-out, as not all clusters nor CNI network's would want such a behavior. Some counter use cases - 1. if the CNI network _is_ assigning publicly accessible addresses, the port mapping becomes a redundant. 2. if they are using a load balancer, they would not need port forwarding as the load balancer will forward public requests onto the private CNI network. > Implement port forwarding in `network/cni` isolator > --- > > Key: MESOS-4823 > URL: https://issues.apache.org/jira/browse/MESOS-4823 > Project: Mesos > Issue Type: Task > Components: containerization > Environment: linux >Reporter: Avinash Sridharan >Assignee: Avinash Sridharan >Priority: Critical > Labels: mesosphere > > Most docker and appc images wish ports that micro-services are listening on, > to the outside world. When containers are running on bridged (or ptp) > networking this can be achieved by installing port forwarding rules on the > agent (using iptables). This can be done in the `network/cni` isolator. > The reason we would like this functionality to be implemented in the > `network/cni` isolator, and not a CNI plugin, is that the specifications > currently do not support specifying port forwarding rules. Further, to > install these rules the isolator needs two pieces of information, the exposed > ports and the IP address associated with the container. Bother are available > to the isolator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4823) Implement port forwarding in `network/cni` isolator
[ https://issues.apache.org/jira/browse/MESOS-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198135#comment-15198135 ] Dan Osborne commented on MESOS-4823: Thank you for providing the example use case. Can you explain, on a technical level, what condition you are planning that will trigger creation of these ip-tables rules? I'm concerned that the capability you're trying to provide makes a lot of assumptions about both the mesos cluster and the CNI network's configurations, and to what degree both are accessible by the public network. I believe that if this behavior goes in, to some degree it should be opt-in or opt-out, as not all clusters nor CNI network's would want such a behavior. Some counter use cases - 1. if the CNI network _is_ assigning publicly accessible addresses, the port mapping becomes a redundant. 2. if they are using a load balancer, they would not need port forwarding as the load balancer will forward public requests onto the private CNI network. > Implement port forwarding in `network/cni` isolator > --- > > Key: MESOS-4823 > URL: https://issues.apache.org/jira/browse/MESOS-4823 > Project: Mesos > Issue Type: Task > Components: containerization > Environment: linux >Reporter: Avinash Sridharan >Assignee: Avinash Sridharan >Priority: Critical > Labels: mesosphere > > Most docker and appc images wish ports that micro-services are listening on, > to the outside world. When containers are running on bridged (or ptp) > networking this can be achieved by installing port forwarding rules on the > agent (using iptables). This can be done in the `network/cni` isolator. > The reason we would like this functionality to be implemented in the > `network/cni` isolator, and not a CNI plugin, is that the specifications > currently do not support specifying port forwarding rules. Further, to > install these rules the isolator needs two pieces of information, the exposed > ports and the IP address associated with the container. Bother are available > to the isolator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4823) Implement port forwarding in `network/cni` isolator
[ https://issues.apache.org/jira/browse/MESOS-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197757#comment-15197757 ] Dan Osborne commented on MESOS-4823: What is the use case for requiring port forwarding? I don't believe this feature request should be implemented, as I don't believe that port forwarding fits into the larger CNI story. CNI defines a container's network as "a group of entities that are uniquely addressable". In general, CNI plugins do not make use of port forwarding because addresses in their network are *uniquely* addressable. The port which a container is running services on should be accessible on the IP address the CNI network assigned to it. I believe that forwarding a port on the agent's IP to a port on the CNI network's IP is fundamentally wrong, as it suggests that the container's CNI IP is not uniquely addressable. > Implement port forwarding in `network/cni` isolator > --- > > Key: MESOS-4823 > URL: https://issues.apache.org/jira/browse/MESOS-4823 > Project: Mesos > Issue Type: Task > Components: containerization > Environment: linux >Reporter: Avinash Sridharan >Assignee: Avinash Sridharan >Priority: Critical > Labels: mesosphere > > Most docker and appc images wish ports that micro-services are listening on, > to the outside world. When containers are running on bridged (or ptp) > networking this can be achieved by installing port forwarding rules on the > agent (using iptables). This can be done in the `network/cni` isolator. > The reason we would like this functionality to be implemented in the > `network/cni` isolator, and not a CNI plugin, is that the specifications > currently do not support specifying port forwarding rules. Further, to > install these rules the isolator needs two pieces of information, the exposed > ports and the IP address associated with the container. Bother are available > to the isolator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4823) Implement port forwarding in `network/cni` isolator
[ https://issues.apache.org/jira/browse/MESOS-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200060#comment-15200060 ] Dan Osborne commented on MESOS-4823: In CNI world every container is uniquely addressable, so within a CNI network there is no requirement for port mapping. However, it is true that if the CNI network is an overlay network then a separate mechanism for getting in/out of the overlay to the rest of the world may be required. I think it is reasonable for frameworks to be network aware (i.e. specify which network they want to create a container in), but less sure it makes sense for them to be specifying how traffic gets in and out of the overlay network. That feels like it is the realm of the specific network implementation. Port mapping is one way to approach getting in/out of an overlay. But the exact mechanism required is highly dependent on the particular overlay implementation, so the proposed approach of mesos manipulating iptables on behalf of the network would only work for some networks. In general overlay network based SDNs have their own mechanisms for getting things in/out of the overlay to the rest of the world. > Implement port forwarding in `network/cni` isolator > --- > > Key: MESOS-4823 > URL: https://issues.apache.org/jira/browse/MESOS-4823 > Project: Mesos > Issue Type: Task > Components: containerization > Environment: linux >Reporter: Avinash Sridharan >Assignee: Avinash Sridharan >Priority: Critical > Labels: mesosphere > > Most docker and appc images wish to expose ports that micro-services are > listening on, to the outside world. When containers are running on bridged > (or ptp) networking this can be achieved by installing port forwarding rules > on the agent (using iptables). This can be done in the `network/cni` > isolator. > The reason we would like this functionality to be implemented in the > `network/cni` isolator, and not a CNI plugin, is that the specifications > currently do not support specifying port forwarding rules. Further, to > install these rules the isolator needs two pieces of information, the exposed > ports and the IP address associated with the container. Bother are available > to the isolator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4370) NetworkSettings.IPAddress field is deprecated in Docker
[ https://issues.apache.org/jira/browse/MESOS-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183423#comment-15183423 ] Dan Osborne commented on MESOS-4370: I don't believe this is entirely true. Regardless of whether the launched container is using a User-defined Network or a regular docker networked container, the place that Mesos expects to find the IP has been moved in the docker api. Your fix addresses that, and restores Mesos ability to get the IP. Though RA42516 also concerns networking, I don't think it should prevent this from getting merged, as this will restore DNS and service discovery in Mesos for Docker 1.10+ > NetworkSettings.IPAddress field is deprecated in Docker > --- > > Key: MESOS-4370 > URL: https://issues.apache.org/jira/browse/MESOS-4370 > Project: Mesos > Issue Type: Bug > Components: containerization, docker >Affects Versions: 0.25.0, 0.26.0, 0.27.0 > Environment: Ubuntu 14.04 > Docker 1.9.1 >Reporter: Clint Armstrong >Assignee: Travis Hegner > > The latest docker API deprecates the NetworkSettings.IPAddress field, in > favor of the NetworkSettings.Networks field. > https://docs.docker.com/engine/reference/api/docker_remote_api/#v1-21-api-changes > With this deprecation, NetworkSettings.IPAddress is not populated for > containers running with networks that use new network plugins. > As a result the mesos API has no data in > container_status.network_infos.ip_address or > container_status.network_infos.ipaddresses. > The immediate impact of this is that mesos-dns is unable to retrieve a > containers IP from the netinfo interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
