[jira] [Updated] (SPARK-11638) Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker with Bridge networking
[
https://issues.apache.org/jira/browse/SPARK-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-11638:
--
Target Version/s: (was: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0)
[~radekg] don't set Target version -- this is for committers. It doesn't make
sense to target released versions anyway
> Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker
> with Bridge networking
>
>
> Key: SPARK-11638
> URL: https://issues.apache.org/jira/browse/SPARK-11638
> Project: Spark
> Issue Type: Improvement
> Components: Mesos, Spark Core
>Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0
>Reporter: Radoslaw Gruchalski
> Attachments: 1.4.0.patch, 1.4.1.patch, 1.5.0.patch, 1.5.1.patch,
> 1.5.2.patch, 1.6.0.patch, 2.3.11.patch, 2.3.4.patch
>
>
> h4. Summary
> Provides {{spark.driver.advertisedPort}},
> {{spark.fileserver.advertisedPort}}, {{spark.broadcast.advertisedPort}} and
> {{spark.replClassServer.advertisedPort}} settings to enable running Spark in
> Mesos on Docker with Bridge networking. Provides patches for Akka Remote to
> enable Spark driver advertisement using alternative host and port.
> With these settings, it is possible to run Spark Master in a Docker container
> and have the executors running on Mesos talk back correctly to such Master.
> The problem is discussed on the Mesos mailing list here:
> https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bfotj5ljzfrh5u7ix-ghppfqknvg9mkkc...@mail.gmail.com%3E
> h4. Running Spark on Mesos - LIBPROCESS_ADVERTISE_IP opens the door
> In order for the framework to receive orders in the bridged container, Mesos
> in the container has to register for offers using the IP address of the
> Agent. Offers are sent by Mesos Master to the Docker container running on a
> different host, an Agent. Normally, prior to Mesos 0.24.0, {{libprocess}}
> would advertise itself using the IP address of the container, something like
> {{172.x.x.x}}. Obviously, Mesos Master can't reach that address, it's a
> different host, it's a different machine. Mesos 0.24.0 introduced two new
> properties for {{libprocess}} - {{LIBPROCESS_ADVERTISE_IP}} and
> {{LIBPROCESS_ADVERTISE_PORT}}. This allows the container to use the Agent's
> address to register for offers. This was provided mainly for running Mesos in
> Docker on Mesos.
> h4. Spark - how does the above relate and what is being addressed here?
> Similar to Mesos, out of the box, Spark does not allow to advertise its
> services on ports different than bind ports. Consider following scenario:
> Spark is running inside a Docker container on Mesos, it's a bridge networking
> mode. Assuming a port {{}} for the {{spark.driver.port}}, {{6677}} for
> the {{spark.fileserver.port}}, {{6688}} for the {{spark.broadcast.port}} and
> {{23456}} for the {{spark.replClassServer.port}}. If such task is posted to
> Marathon, Mesos will give 4 ports in range {{31000-32000}} mapping to the
> container ports. Starting the executors from such container results in
> executors not being able to communicate back to the Spark Master.
> This happens because of 2 things:
> Spark driver is effectively an {{akka-remote}} system with {{akka.tcp}}
> transport. {{akka-remote}} prior to version {{2.4}} can't advertise a port
> different to what it bound to. The settings discussed are here:
> https://github.com/akka/akka/blob/f8c1671903923837f22d0726a955e0893add5e9f/akka-remote/src/main/resources/reference.conf#L345-L376.
> These do not exist in Akka {{2.3.x}}. Spark driver will always advertise
> port {{}} as this is the one {{akka-remote}} is bound to.
> Any URIs the executors contact the Spark Master on, are prepared by Spark
> Master and handed over to executors. These always contain the port number
> used by the Master to find the service on. The services are:
> - {{spark.broadcast.port}}
> - {{spark.fileserver.port}}
> - {{spark.replClassServer.port}}
> all above ports are by default {{0}} (random assignment) but can be specified
> using Spark configuration ( {{-Dspark...port}} ). However, they are limited
> in the same way as the {{spark.driver.port}}; in the above example, an
> executor should not contact the file server on port {{6677}} but rather on
> the respective 31xxx assigned by Mesos.
> Spark currently does not allow any of that.
> h4. Taking on the problem, step 1: Spark Driver
> As mentioned above, Spark Driver is based on {{akka-remote}}. In order to
> take on the problem, the {{akka.remote.net.tcp.bind-hostname}} and
> {{akka.remote.net.tcp.bind-port}} settings are a must. Spark does not compile
> with Akka 2.4.x yet.
> What we want is the back port of mentioned {{akka-remote}}
[jira] [Updated] (SPARK-11638) Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker with Bridge networking
[
https://issues.apache.org/jira/browse/SPARK-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Radoslaw Gruchalski updated SPARK-11638:
Attachment: 1.6.0.patch
Updated 1.6.0 patch to include NettyRpcEnv.
> Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker
> with Bridge networking
>
>
> Key: SPARK-11638
> URL: https://issues.apache.org/jira/browse/SPARK-11638
> Project: Spark
> Issue Type: Improvement
> Components: Mesos, Spark Core
>Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0
>Reporter: Radoslaw Gruchalski
> Attachments: 1.4.0.patch, 1.4.1.patch, 1.5.0.patch, 1.5.1.patch,
> 1.5.2.patch, 1.6.0.patch, 2.3.11.patch, 2.3.4.patch
>
>
> h4. Summary
> Provides {{spark.driver.advertisedPort}},
> {{spark.fileserver.advertisedPort}}, {{spark.broadcast.advertisedPort}} and
> {{spark.replClassServer.advertisedPort}} settings to enable running Spark in
> Mesos on Docker with Bridge networking. Provides patches for Akka Remote to
> enable Spark driver advertisement using alternative host and port.
> With these settings, it is possible to run Spark Master in a Docker container
> and have the executors running on Mesos talk back correctly to such Master.
> The problem is discussed on the Mesos mailing list here:
> https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bfotj5ljzfrh5u7ix-ghppfqknvg9mkkc...@mail.gmail.com%3E
> h4. Running Spark on Mesos - LIBPROCESS_ADVERTISE_IP opens the door
> In order for the framework to receive orders in the bridged container, Mesos
> in the container has to register for offers using the IP address of the
> Agent. Offers are sent by Mesos Master to the Docker container running on a
> different host, an Agent. Normally, prior to Mesos 0.24.0, {{libprocess}}
> would advertise itself using the IP address of the container, something like
> {{172.x.x.x}}. Obviously, Mesos Master can't reach that address, it's a
> different host, it's a different machine. Mesos 0.24.0 introduced two new
> properties for {{libprocess}} - {{LIBPROCESS_ADVERTISE_IP}} and
> {{LIBPROCESS_ADVERTISE_PORT}}. This allows the container to use the Agent's
> address to register for offers. This was provided mainly for running Mesos in
> Docker on Mesos.
> h4. Spark - how does the above relate and what is being addressed here?
> Similar to Mesos, out of the box, Spark does not allow to advertise its
> services on ports different than bind ports. Consider following scenario:
> Spark is running inside a Docker container on Mesos, it's a bridge networking
> mode. Assuming a port {{}} for the {{spark.driver.port}}, {{6677}} for
> the {{spark.fileserver.port}}, {{6688}} for the {{spark.broadcast.port}} and
> {{23456}} for the {{spark.replClassServer.port}}. If such task is posted to
> Marathon, Mesos will give 4 ports in range {{31000-32000}} mapping to the
> container ports. Starting the executors from such container results in
> executors not being able to communicate back to the Spark Master.
> This happens because of 2 things:
> Spark driver is effectively an {{akka-remote}} system with {{akka.tcp}}
> transport. {{akka-remote}} prior to version {{2.4}} can't advertise a port
> different to what it bound to. The settings discussed are here:
> https://github.com/akka/akka/blob/f8c1671903923837f22d0726a955e0893add5e9f/akka-remote/src/main/resources/reference.conf#L345-L376.
> These do not exist in Akka {{2.3.x}}. Spark driver will always advertise
> port {{}} as this is the one {{akka-remote}} is bound to.
> Any URIs the executors contact the Spark Master on, are prepared by Spark
> Master and handed over to executors. These always contain the port number
> used by the Master to find the service on. The services are:
> - {{spark.broadcast.port}}
> - {{spark.fileserver.port}}
> - {{spark.replClassServer.port}}
> all above ports are by default {{0}} (random assignment) but can be specified
> using Spark configuration ( {{-Dspark...port}} ). However, they are limited
> in the same way as the {{spark.driver.port}}; in the above example, an
> executor should not contact the file server on port {{6677}} but rather on
> the respective 31xxx assigned by Mesos.
> Spark currently does not allow any of that.
> h4. Taking on the problem, step 1: Spark Driver
> As mentioned above, Spark Driver is based on {{akka-remote}}. In order to
> take on the problem, the {{akka.remote.net.tcp.bind-hostname}} and
> {{akka.remote.net.tcp.bind-port}} settings are a must. Spark does not compile
> with Akka 2.4.x yet.
> What we want is the back port of mentioned {{akka-remote}} settings to
> {{2.3.x}} versions. These patches are attached to this ticket -
> {{2.3.4.patch}} a
[jira] [Updated] (SPARK-11638) Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker with Bridge networking
[
https://issues.apache.org/jira/browse/SPARK-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Radoslaw Gruchalski updated SPARK-11638:
Attachment: (was: 1.6.0-master.patch)
> Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker
> with Bridge networking
>
>
> Key: SPARK-11638
> URL: https://issues.apache.org/jira/browse/SPARK-11638
> Project: Spark
> Issue Type: Improvement
> Components: Mesos, Spark Core
>Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0
>Reporter: Radoslaw Gruchalski
> Attachments: 1.4.0.patch, 1.4.1.patch, 1.5.0.patch, 1.5.1.patch,
> 1.5.2.patch, 2.3.11.patch, 2.3.4.patch
>
>
> h4. Summary
> Provides {{spark.driver.advertisedPort}},
> {{spark.fileserver.advertisedPort}}, {{spark.broadcast.advertisedPort}} and
> {{spark.replClassServer.advertisedPort}} settings to enable running Spark in
> Mesos on Docker with Bridge networking. Provides patches for Akka Remote to
> enable Spark driver advertisement using alternative host and port.
> With these settings, it is possible to run Spark Master in a Docker container
> and have the executors running on Mesos talk back correctly to such Master.
> The problem is discussed on the Mesos mailing list here:
> https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bfotj5ljzfrh5u7ix-ghppfqknvg9mkkc...@mail.gmail.com%3E
> h4. Running Spark on Mesos - LIBPROCESS_ADVERTISE_IP opens the door
> In order for the framework to receive orders in the bridged container, Mesos
> in the container has to register for offers using the IP address of the
> Agent. Offers are sent by Mesos Master to the Docker container running on a
> different host, an Agent. Normally, prior to Mesos 0.24.0, {{libprocess}}
> would advertise itself using the IP address of the container, something like
> {{172.x.x.x}}. Obviously, Mesos Master can't reach that address, it's a
> different host, it's a different machine. Mesos 0.24.0 introduced two new
> properties for {{libprocess}} - {{LIBPROCESS_ADVERTISE_IP}} and
> {{LIBPROCESS_ADVERTISE_PORT}}. This allows the container to use the Agent's
> address to register for offers. This was provided mainly for running Mesos in
> Docker on Mesos.
> h4. Spark - how does the above relate and what is being addressed here?
> Similar to Mesos, out of the box, Spark does not allow to advertise its
> services on ports different than bind ports. Consider following scenario:
> Spark is running inside a Docker container on Mesos, it's a bridge networking
> mode. Assuming a port {{}} for the {{spark.driver.port}}, {{6677}} for
> the {{spark.fileserver.port}}, {{6688}} for the {{spark.broadcast.port}} and
> {{23456}} for the {{spark.replClassServer.port}}. If such task is posted to
> Marathon, Mesos will give 4 ports in range {{31000-32000}} mapping to the
> container ports. Starting the executors from such container results in
> executors not being able to communicate back to the Spark Master.
> This happens because of 2 things:
> Spark driver is effectively an {{akka-remote}} system with {{akka.tcp}}
> transport. {{akka-remote}} prior to version {{2.4}} can't advertise a port
> different to what it bound to. The settings discussed are here:
> https://github.com/akka/akka/blob/f8c1671903923837f22d0726a955e0893add5e9f/akka-remote/src/main/resources/reference.conf#L345-L376.
> These do not exist in Akka {{2.3.x}}. Spark driver will always advertise
> port {{}} as this is the one {{akka-remote}} is bound to.
> Any URIs the executors contact the Spark Master on, are prepared by Spark
> Master and handed over to executors. These always contain the port number
> used by the Master to find the service on. The services are:
> - {{spark.broadcast.port}}
> - {{spark.fileserver.port}}
> - {{spark.replClassServer.port}}
> all above ports are by default {{0}} (random assignment) but can be specified
> using Spark configuration ( {{-Dspark...port}} ). However, they are limited
> in the same way as the {{spark.driver.port}}; in the above example, an
> executor should not contact the file server on port {{6677}} but rather on
> the respective 31xxx assigned by Mesos.
> Spark currently does not allow any of that.
> h4. Taking on the problem, step 1: Spark Driver
> As mentioned above, Spark Driver is based on {{akka-remote}}. In order to
> take on the problem, the {{akka.remote.net.tcp.bind-hostname}} and
> {{akka.remote.net.tcp.bind-port}} settings are a must. Spark does not compile
> with Akka 2.4.x yet.
> What we want is the back port of mentioned {{akka-remote}} settings to
> {{2.3.x}} versions. These patches are attached to this ticket -
> {{2.3.4.patch}} and {{2.3.11.patch}} files provide patche
[jira] [Updated] (SPARK-11638) Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker with Bridge networking
[
https://issues.apache.org/jira/browse/SPARK-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Radoslaw Gruchalski updated SPARK-11638:
Target Version/s: 1.5.1, 1.5.0, 1.4.1, 1.4.0, 1.5.2, 1.6.0 (was: 1.4.0,
1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0)
Description:
h4. Summary
Provides {{spark.driver.advertisedPort}}, {{spark.fileserver.advertisedPort}},
{{spark.broadcast.advertisedPort}} and {{spark.replClassServer.advertisedPort}}
settings to enable running Spark in Mesos on Docker with Bridge networking.
Provides patches for Akka Remote to enable Spark driver advertisement using
alternative host and port.
With these settings, it is possible to run Spark Master in a Docker container
and have the executors running on Mesos talk back correctly to such Master.
The problem is discussed on the Mesos mailing list here:
https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bfotj5ljzfrh5u7ix-ghppfqknvg9mkkc...@mail.gmail.com%3E
h4. Running Spark on Mesos - LIBPROCESS_ADVERTISE_IP opens the door
In order for the framework to receive orders in the bridged container, Mesos in
the container has to register for offers using the IP address of the Agent.
Offers are sent by Mesos Master to the Docker container running on a different
host, an Agent. Normally, prior to Mesos 0.24.0, {{libprocess}} would advertise
itself using the IP address of the container, something like {{172.x.x.x}}.
Obviously, Mesos Master can't reach that address, it's a different host, it's a
different machine. Mesos 0.24.0 introduced two new properties for
{{libprocess}} - {{LIBPROCESS_ADVERTISE_IP}} and {{LIBPROCESS_ADVERTISE_PORT}}.
This allows the container to use the Agent's address to register for offers.
This was provided mainly for running Mesos in Docker on Mesos.
h4. Spark - how does the above relate and what is being addressed here?
Similar to Mesos, out of the box, Spark does not allow to advertise its
services on ports different than bind ports. Consider following scenario:
Spark is running inside a Docker container on Mesos, it's a bridge networking
mode. Assuming a port {{}} for the {{spark.driver.port}}, {{6677}} for the
{{spark.fileserver.port}}, {{6688}} for the {{spark.broadcast.port}} and
{{23456}} for the {{spark.replClassServer.port}}. If such task is posted to
Marathon, Mesos will give 4 ports in range {{31000-32000}} mapping to the
container ports. Starting the executors from such container results in
executors not being able to communicate back to the Spark Master.
This happens because of 2 things:
Spark driver is effectively an {{akka-remote}} system with {{akka.tcp}}
transport. {{akka-remote}} prior to version {{2.4}} can't advertise a port
different to what it bound to. The settings discussed are here:
https://github.com/akka/akka/blob/f8c1671903923837f22d0726a955e0893add5e9f/akka-remote/src/main/resources/reference.conf#L345-L376.
These do not exist in Akka {{2.3.x}}. Spark driver will always advertise port
{{}} as this is the one {{akka-remote}} is bound to.
Any URIs the executors contact the Spark Master on, are prepared by Spark
Master and handed over to executors. These always contain the port number used
by the Master to find the service on. The services are:
- {{spark.broadcast.port}}
- {{spark.fileserver.port}}
- {{spark.replClassServer.port}}
all above ports are by default {{0}} (random assignment) but can be specified
using Spark configuration ( {{-Dspark...port}} ). However, they are limited in
the same way as the {{spark.driver.port}}; in the above example, an executor
should not contact the file server on port {{6677}} but rather on the
respective 31xxx assigned by Mesos.
Spark currently does not allow any of that.
h4. Taking on the problem, step 1: Spark Driver
As mentioned above, Spark Driver is based on {{akka-remote}}. In order to take
on the problem, the {{akka.remote.net.tcp.bind-hostname}} and
{{akka.remote.net.tcp.bind-port}} settings are a must. Spark does not compile
with Akka 2.4.x yet.
What we want is the back port of mentioned {{akka-remote}} settings to
{{2.3.x}} versions. These patches are attached to this ticket - {{2.3.4.patch}}
and {{2.3.11.patch}} files provide patches for respective akka versions. These
add mentioned settings and ensure they work as documented for Akka 2.4. In
other words, these are future compatible.
A part of that patch also exists in the patch for Spark, in the
{{org.apache.spark.util.AkkaUtils}} class. This is where Spark is creating the
driver and compiling the Akka configuration. That part of the patch tells Akka
to use {{bind-hostname}} instead of {{hostname}}, if
{{spark.driver.advertisedHost}} is given and use {{bind-port}} instead of
{{port}}, if {{spark.driver.advertisedPort}} is given. In such cases,
{{hostname}} and {{port}} are set to the advertised values, respectively.
*Worth m
[jira] [Updated] (SPARK-11638) Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker with Bridge networking
[ https://issues.apache.org/jira/browse/SPARK-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radoslaw Gruchalski updated SPARK-11638: Summary: Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker with Bridge networking (was: Apache Spark on Docker with Bridge networking / run Spark in Mesos on Docker with Bridge networking) > Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker > with Bridge networking > > > Key: SPARK-11638 > URL: https://issues.apache.org/jira/browse/SPARK-11638 > Project: Spark > Issue Type: Improvement > Components: Mesos, Spark Core >Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0 >Reporter: Radoslaw Gruchalski > Attachments: 1.4.0.patch, 1.4.1.patch, 1.5.0.patch, 1.5.1.patch, > 1.5.2.patch, 1.6.0-master.patch, 2.3.11.patch, 2.3.4.patch > > > Full description will be provided within the next few minutes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
