[jira] [Commented] (SPARK-15155) Optionally ignore default role resources
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275057#comment-15275057
]
Chris Heller commented on SPARK-15155:
--
At present, Spark will accept resources from both the default role, and the
additional role set with "spark.mesos.role", my problem is I do not want this
behavior.
I want Spark teither use the default role (in which case I will not set
"spark.mesos.role") or use the role I've selected with "spark.mesos.role".
Since I also don't want to break backwards compatibility with the current
behavior, I propose this patch to switch into an either/or mode.
> Optionally ignore default role resources
>
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new boolean property:
> {{spark.mesos.ignoreDefaultRoleResources}}. When this property is set Spark
> will only accept resources from the role passed in the {{spark.mesos.role}}
> property. If {{spark.mesos.role}} has not been set,
> {{spark.mesos.ignoreDefaultRoleResources}} has no effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15155) Optionally ignore default role resources
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274613#comment-15274613
]
Chris Heller commented on SPARK-15155:
--
My point is that each time I add another Spark cluster to the Mesos cluster
assigned to a role, it will continue to use default role resources without this
patch.
I can continue to push all frameworks into roles until I have vanished the
default role right off the cluster, are you suggesting that as a valid
alternative to needing this patch added to Spark?
> Optionally ignore default role resources
>
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new boolean property:
> {{spark.mesos.ignoreDefaultRoleResources}}. When this property is set Spark
> will only accept resources from the role passed in the {{spark.mesos.role}}
> property. If {{spark.mesos.role}} has not been set,
> {{spark.mesos.ignoreDefaultRoleResources}} has no effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15155) Optionally ignore default role resources
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274570#comment-15274570
]
Chris Heller commented on SPARK-15155:
--
Some of that ad-hoc work being other Spark jobs, they will again take resources
from the default role. Have I overlooked something else that should be
preventing Spark from taking default role resources without a patch such as
this? I'm presently stuck on mesos 0.22.2 for a while longer, has things
shifted in later versions that would obsolete this patch?
> Optionally ignore default role resources
>
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new boolean property:
> {{spark.mesos.ignoreDefaultRoleResources}}. When this property is set Spark
> will only accept resources from the role passed in the {{spark.mesos.role}}
> property. If {{spark.mesos.role}} has not been set,
> {{spark.mesos.ignoreDefaultRoleResources}} has no effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15155) Optionally ignore default role resources
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274523#comment-15274523
]
Chris Heller commented on SPARK-15155:
--
The downside for me is that I have two long running applications, but wish to
also run short lived ad-hoc work. If either of the two applications take all
the default role resources I starve out my ad-hoc work.
> Optionally ignore default role resources
>
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new boolean property:
> {{spark.mesos.ignoreDefaultRoleResources}}. When this property is set Spark
> will only accept resources from the role passed in the {{spark.mesos.role}}
> property. If {{spark.mesos.role}} has not been set,
> {{spark.mesos.ignoreDefaultRoleResources}} has no effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15155) Optionally ignore default role resources
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274493#comment-15274493
]
Chris Heller commented on SPARK-15155:
--
In my specific case I am running multiple spark clusters on mesos, and want to
restrict each to their own role.
> Optionally ignore default role resources
>
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new boolean property:
> {{spark.mesos.ignoreDefaultRoleResources}}. When this property is set Spark
> will only accept resources from the role passed in the {{spark.mesos.role}}
> property. If {{spark.mesos.role}} has not been set,
> {{spark.mesos.ignoreDefaultRoleResources}} has no effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-15155) Optionally ignore default role resources
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274461#comment-15274461
]
Chris Heller edited comment on SPARK-15155 at 5/6/16 5:58 PM:
--
Under the current implementation Spark will consider both resources from {{*}}
and from {{spark.mesos.role}} when set, this proposal will give you the option
to only consider resources from {{spark.mesos.role}} and ignore the default
role. In this case, I don't want Spark to be launching tasks using the default
role, but presently that cannot be enforced.
was (Author: chrisheller):
Under the current implementation Spark will consider both resources from {{*}}
and from {{spark.mesos.role}} when set, this proposal will give you the option
to only consider resources from {{spark.mesos.role}} and ignore {{*}}. In this
case, I don't want Spark to be launching tasks using the default role, but
presently that cannot be enforced.
> Optionally ignore default role resources
>
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new boolean property:
> {{spark.mesos.ignoreDefaultRoleResources}}. When this property is set Spark
> will only accept resources from the role passed in the {{spark.mesos.role}}
> property. If {{spark.mesos.role}} has not been set,
> {{spark.mesos.ignoreDefaultRoleResources}} has no effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15155) Optionally ignore default role resources
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274461#comment-15274461
]
Chris Heller commented on SPARK-15155:
--
Under the current implementation Spark will consider both resources from {{*}}
and from {{spark.mesos.role}} when set, this proposal will give you the option
to only consider resources from {{spark.mesos.role}} and ignore {{*}}. In this
case, I don't want Spark to be launching tasks using the default role, but
presently that cannot be enforced.
> Optionally ignore default role resources
>
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new boolean property:
> {{spark.mesos.ignoreDefaultRoleResources}}. When this property is set Spark
> will only accept resources from the role passed in the {{spark.mesos.role}}
> property. If {{spark.mesos.role}} has not been set,
> {{spark.mesos.ignoreDefaultRoleResources}} has no effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15155) Optionally ignore default role resources
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Heller updated SPARK-15155:
-
Description:
SPARK-6284 added support for Mesos roles, but the framework will still accept
resources from both the reserved role specified in {{spark.mesos.role}} and the
default role {{*}}.
I'd like to propose the addition of a new boolean property:
{{spark.mesos.ignoreDefaultRoleResources}}. When this property is set Spark
will only accept resources from the role passed in the {{spark.mesos.role}}
property. If {{spark.mesos.role}} has not been set,
{{spark.mesos.ignoreDefaultRoleResources}} has no effect.
was:
SPARK-6284 added support for Mesos roles, but the framework will still accept
resources from both the reserved role specified in {{spark.mesos.role}} and the
default role {{*}}.
I'd like to propose the addition of a new property
{{spark.mesos.acceptedResourceRoles}} which would be a comma-delimited list of
roles that the framework will accept resources from.
This is similar to {{spark.mesos.constraints}}, except that constraints look at
the attributes of an offer, and this will look at the role of a resource.
In the default case {{spark.mesos.acceptedResourceRoles}} will be set to
{{*[,spark.mesos.role]}} giving the exact same behavior to the framework if no
value is specified in the property.
> Optionally ignore default role resources
>
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new boolean property:
> {{spark.mesos.ignoreDefaultRoleResources}}. When this property is set Spark
> will only accept resources from the role passed in the {{spark.mesos.role}}
> property. If {{spark.mesos.role}} has not been set,
> {{spark.mesos.ignoreDefaultRoleResources}} has no effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15155) Optionally ignore default role resources
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Heller updated SPARK-15155:
-
Summary: Optionally ignore default role resources (was: Selectively accept
Mesos resources by role)
> Optionally ignore default role resources
>
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new property
> {{spark.mesos.acceptedResourceRoles}} which would be a comma-delimited list
> of roles that the framework will accept resources from.
> This is similar to {{spark.mesos.constraints}}, except that constraints look
> at the attributes of an offer, and this will look at the role of a resource.
> In the default case {{spark.mesos.acceptedResourceRoles}} will be set to
> {{*[,spark.mesos.role]}} giving the exact same behavior to the framework if
> no value is specified in the property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15155) Selectively accept Mesos resources by role
[
https://issues.apache.org/jira/browse/SPARK-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272227#comment-15272227
]
Chris Heller commented on SPARK-15155:
--
I have started work on an implementation targeting the master branch of spark,
and would like to eventually back port the fix, since its useful in all
versions that support {{spark.mesos.role}}.
https://github.com/hellertime/spark/tree/SPARK-15155
> Selectively accept Mesos resources by role
> --
>
> Key: SPARK-15155
> URL: https://issues.apache.org/jira/browse/SPARK-15155
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chris Heller
>
> SPARK-6284 added support for Mesos roles, but the framework will still accept
> resources from both the reserved role specified in {{spark.mesos.role}} and
> the default role {{*}}.
> I'd like to propose the addition of a new property
> {{spark.mesos.acceptedResourceRoles}} which would be a comma-delimited list
> of roles that the framework will accept resources from.
> This is similar to {{spark.mesos.constraints}}, except that constraints look
> at the attributes of an offer, and this will look at the role of a resource.
> In the default case {{spark.mesos.acceptedResourceRoles}} will be set to
> {{*[,spark.mesos.role]}} giving the exact same behavior to the framework if
> no value is specified in the property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-15155) Selectively accept Mesos resources by role
Chris Heller created SPARK-15155:
Summary: Selectively accept Mesos resources by role
Key: SPARK-15155
URL: https://issues.apache.org/jira/browse/SPARK-15155
Project: Spark
Issue Type: Improvement
Components: Mesos
Affects Versions: 1.6.0, 1.5.0
Reporter: Chris Heller
SPARK-6284 added support for Mesos roles, but the framework will still accept
resources from both the reserved role specified in {{spark.mesos.role}} and the
default role {{*}}.
I'd like to propose the addition of a new property
{{spark.mesos.acceptedResourceRoles}} which would be a comma-delimited list of
roles that the framework will accept resources from.
This is similar to {{spark.mesos.constraints}}, except that constraints look at
the attributes of an offer, and this will look at the role of a resource.
In the default case {{spark.mesos.acceptedResourceRoles}} will be set to
{{*[,spark.mesos.role]}} giving the exact same behavior to the framework if no
value is specified in the property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10854) MesosExecutorBackend: Received launchTask but executor was null
[
https://issues.apache.org/jira/browse/SPARK-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268635#comment-15268635
]
Chris Heller commented on SPARK-10854:
--
I wanted to bump this issue. I'm running into it from time to time with Spark
1.5.2, running on Mesos 0.22.2 and using Docker 1.8.3.
The stuck task will hold up an entire job from completion.
Would speculative tasks be an appropriate work around here? That feels like
overkill to always need to use them, but as a temporary fix it would be ok.
> MesosExecutorBackend: Received launchTask but executor was null
> ---
>
> Key: SPARK-10854
> URL: https://issues.apache.org/jira/browse/SPARK-10854
> Project: Spark
> Issue Type: Bug
> Components: Mesos
>Affects Versions: 1.4.0
> Environment: Spark 1.4.0
> Mesos 0.23.0
> Docker 1.8.1
>Reporter: Kevin Matzen
>Priority: Minor
>
> Sometimes my tasks get stuck in staging. Here's stdout from one such worker.
> I'm running mesos-slave inside a docker container with the host's docker
> exposed and I'm using Spark's docker support to launch the worker inside its
> own container. Both containers are running. I'm using pyspark. I can see
> mesos-slave and java running, but I do not see python running.
> {noformat}
> WARNING: Your kernel does not support swap limit capabilities, memory limited
> without swap.
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 15/09/28 15:02:09 INFO MesosExecutorBackend: Registered signal handlers for
> [TERM, HUP, INT]
> I0928 15:02:09.65854138 exec.cpp:132] Version: 0.23.0
> 15/09/28 15:02:09 ERROR MesosExecutorBackend: Received launchTask but
> executor was null
> I0928 15:02:09.70295554 exec.cpp:206] Executor registered on slave
> 20150928-044200-1140850698-5050-8-S190
> 15/09/28 15:02:09 INFO MesosExecutorBackend: Registered with Mesos as
> executor ID 20150928-044200-1140850698-5050-8-S190 with 1 cpus
> 15/09/28 15:02:09 INFO SecurityManager: Changing view acls to: root
> 15/09/28 15:02:09 INFO SecurityManager: Changing modify acls to: root
> 15/09/28 15:02:09 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root); users
> with modify permissions: Set(root)
> 15/09/28 15:02:10 INFO Slf4jLogger: Slf4jLogger started
> 15/09/28 15:02:10 INFO Remoting: Starting remoting
> 15/09/28 15:02:10 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutor@:56458]
> 15/09/28 15:02:10 INFO Utils: Successfully started service 'sparkExecutor' on
> port 56458.
> 15/09/28 15:02:10 INFO DiskBlockManager: Created local directory at
> /tmp/spark-28a21c2d-54cc-40b3-b0c2-cc3624f1a73c/blockmgr-f2336fec-e1ea-44f1-bd5c-9257049d5e7b
> 15/09/28 15:02:10 INFO MemoryStore: MemoryStore started with capacity 52.1 MB
> 15/09/28 15:02:11 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 15/09/28 15:02:11 INFO Executor: Starting executor ID
> 20150928-044200-1140850698-5050-8-S190 on host
> 15/09/28 15:02:11 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57431.
> 15/09/28 15:02:11 INFO NettyBlockTransferService: Server created on 57431
> 15/09/28 15:02:11 INFO BlockManagerMaster: Trying to register BlockManager
> 15/09/28 15:02:11 INFO BlockManagerMaster: Registered BlockManager
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11638) Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker with Bridge networking
[
https://issues.apache.org/jira/browse/SPARK-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14999554#comment-14999554
]
Chris Heller commented on SPARK-11638:
--
Can't wait to try this out. +1
> Apache Spark in Docker with Bridge networking / run Spark on Mesos, in Docker
> with Bridge networking
>
>
> Key: SPARK-11638
> URL: https://issues.apache.org/jira/browse/SPARK-11638
> Project: Spark
> Issue Type: Improvement
> Components: Mesos, Spark Core
>Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0
>Reporter: Radoslaw Gruchalski
> Attachments: 1.4.0.patch, 1.4.1.patch, 1.5.0.patch, 1.5.1.patch,
> 1.5.2.patch, 1.6.0-master.patch, 2.3.11.patch, 2.3.4.patch
>
>
> h4. Summary
> Provides {{spark.driver.advertisedPort}},
> {{spark.fileserver.advertisedPort}}, {{spark.broadcast.advertisedPort}} and
> {{spark.replClassServer.advertisedPort}} settings to enable running Spark in
> Mesos on Docker with Bridge networking. Provides patches for Akka Remote to
> enable Spark driver advertisement using alternative host and port.
> With these settings, it is possible to run Spark Master in a Docker container
> and have the executors running on Mesos talk back correctly to such Master.
> The problem is discussed on the Mesos mailing list here:
> https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bfotj5ljzfrh5u7ix-ghppfqknvg9mkkc...@mail.gmail.com%3E
> h4. Running Spark on Mesos - LIBPROCESS_ADVERTISE_IP opens the door
> In order for the framework to receive orders in the bridged container, Mesos
> in the container has to register for offers using the IP address of the
> Agent. Offers are sent by Mesos Master to the Docker container running on a
> different host, an Agent. Normally, prior to Mesos 0.24.0, {{libprocess}}
> would advertise itself using the IP address of the container, something like
> {{172.x.x.x}}. Obviously, Mesos Master can't reach that address, it's a
> different host, it's a different machine. Mesos 0.24.0 introduced two new
> properties for {{libprocess}} - {{LIBPROCESS_ADVERTISE_IP}} and
> {{LIBPROCESS_ADVERTISE_PORT}}. This allows the container to use the Agent's
> address to register for offers. This was provided mainly for running Mesos in
> Docker on Mesos.
> h4. Spark - how does the above relate and what is being addressed here?
> Similar to Mesos, out of the box, Spark does not allow to advertise its
> services on ports different than bind ports. Consider following scenario:
> Spark is running inside a Docker container on Mesos, it's a bridge networking
> mode. Assuming a port {{}} for the {{spark.driver.port}}, {{6677}} for
> the {{spark.fileserver.port}}, {{6688}} for the {{spark.broadcast.port}} and
> {{23456}} for the {{spark.replClassServer.port}}. If such task is posted to
> Marathon, Mesos will give 4 ports in range {{31000-32000}} mapping to the
> container ports. Starting the executors from such container results in
> executors not being able to communicate back to the Spark Master.
> This happens because of 2 things:
> Spark driver is effectively an {{akka-remote}} system with {{akka.tcp}}
> transport. {{akka-remote}} prior to version {{2.4}} can't advertise a port
> different to what it bound to. The settings discussed are here:
> https://github.com/akka/akka/blob/f8c1671903923837f22d0726a955e0893add5e9f/akka-remote/src/main/resources/reference.conf#L345-L376.
> These do not exist in Akka {{2.3.x}}. Spark driver will always advertise
> port {{}} as this is the one {{akka-remote}} is bound to.
> Any URIs the executors contact the Spark Master on, are prepared by Spark
> Master and handed over to executors. These always contain the port number
> used by the Master to find the service on. The services are:
> - {{spark.broadcast.port}}
> - {{spark.fileserver.port}}
> - {{spark.replClassServer.port}}
> all above ports are by default {{0}} (random assignment) but can be specified
> using Spark configuration ( {{-Dspark...port}} ). However, they are limited
> in the same way as the {{spark.driver.port}}; in the above example, an
> executor should not contact the file server on port {{6677}} but rather on
> the respective 31xxx assigned by Mesos.
> Spark currently does not allow any of that.
> h4. Taking on the problem, step 1: Spark Driver
> As mentioned above, Spark Driver is based on {{akka-remote}}. In order to
> take on the problem, the {{akka.remote.net.tcp.bind-hostname}} and
> {{akka.remote.net.tcp.bind-port}} settings are a must. Spark does not compile
> with Akka 2.4.x yet.
> What we want is the back port of mentioned {{akka-remote}} settings to
> {{2.3.x}} versions. These patches are attached to this ticket -
> {{2.3.4.
[jira] [Commented] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
[ https://issues.apache.org/jira/browse/SPARK-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935043#comment-14935043 ] Chris Heller commented on SPARK-8734: - I agree on the goal. Without a total redesign of the Spark properties format, this approach will get us most of the way there. I noticed in your patch you also unified the SparkConf/MesosProperties logic which makes maintaining these special cases simpler. Good documentation should provide a level of needed user-friendliness. +1 > Expose all Mesos DockerInfo options to Spark > > > Key: SPARK-8734 > URL: https://issues.apache.org/jira/browse/SPARK-8734 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Chris Heller >Priority: Minor > Attachments: network.diff > > > SPARK-2691 only exposed a few options from the DockerInfo message. It would > be reasonable to expose them all, especially given one can now specify > arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
[ https://issues.apache.org/jira/browse/SPARK-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933353#comment-14933353 ] Chris Heller commented on SPARK-8734: - For something like marathon, it models parameters as a list of key, value pairs -- so it nicely sidesteps this issue. Looking at the current documentation for `docker run` even with this approach of embedding part of the parameter in the property name won't work for the `--security-opt` setting: https://docs.docker.com/reference/run/#security-configuration It sounds like special casing parameters which cannot be modeled as a single property is the only approach that can support all possible parameters without resorting to data embeddings or reworking the entire Spark conf model to not use Java property files! > Expose all Mesos DockerInfo options to Spark > > > Key: SPARK-8734 > URL: https://issues.apache.org/jira/browse/SPARK-8734 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Chris Heller >Priority: Minor > Attachments: network.diff > > > SPARK-2691 only exposed a few options from the DockerInfo message. It would > be reasonable to expose them all, especially given one can now specify > arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
[ https://issues.apache.org/jira/browse/SPARK-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933193#comment-14933193 ] Chris Heller commented on SPARK-8734: - This approach eliminates the ability to pass values into the parameter which would embed a comma. For instance if I wished to pass a ZooKeeper connection string into a container by an environment variable, I would be unable to do so if the comma is reserved to allow multiple parameters since the connection string also reserves the comma for its delimiter. Just as a thought exercise, what about a method such as: spark.mesos.executor.docker.parameter.env.1: FOO=bar spark.mesos.executor.docker.parameter.env.2: BAZ=quux (where the .* suffix is just a unique value to prevent key folding by the hash table) This is ugly in that it exposes an implementation detail I'd rather not expose, but I'd rather have to suffix a ".1", than do something like: spark.mesos.executor.docker.parameter.env: ZOOKEEPER=host1:2181\,host2:2181\,host3:2181, FOO=bar Don't support comma delimited values, escape all commas, and unescape them after splitting the value in the code, or rework the property name to encode ordinal information to support multi-valued keys. Are these the only alternatives? > Expose all Mesos DockerInfo options to Spark > > > Key: SPARK-8734 > URL: https://issues.apache.org/jira/browse/SPARK-8734 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Chris Heller >Priority: Minor > Attachments: network.diff > > > SPARK-2691 only exposed a few options from the DockerInfo message. It would > be reasonable to expose them all, especially given one can now specify > arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
[ https://issues.apache.org/jira/browse/SPARK-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909541#comment-14909541 ] Chris Heller commented on SPARK-8734: - That's unfortunate the issue with the hash map. Embedding a mini-language in the value of the property isn't my favorite idea. Especially the environment variable parameter, given the value is already a free-form datum. Adding a delimiter means adding escaping that delimiter in a way that is unambiguous. > Expose all Mesos DockerInfo options to Spark > > > Key: SPARK-8734 > URL: https://issues.apache.org/jira/browse/SPARK-8734 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Chris Heller >Priority: Minor > Attachments: network.diff > > > SPARK-2691 only exposed a few options from the DockerInfo message. It would > be reasonable to expose them all, especially given one can now specify > arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
[ https://issues.apache.org/jira/browse/SPARK-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908050#comment-14908050 ] Chris Heller commented on SPARK-8734: - I pushed up to my branch some code for the parameters. Though its untested at the moment ... tried to rebase the code to master and am now getting build errors. > Expose all Mesos DockerInfo options to Spark > > > Key: SPARK-8734 > URL: https://issues.apache.org/jira/browse/SPARK-8734 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Chris Heller >Priority: Minor > Attachments: network.diff > > > SPARK-2691 only exposed a few options from the DockerInfo message. It would > be reasonable to expose them all, especially given one can now specify > arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
[ https://issues.apache.org/jira/browse/SPARK-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907952#comment-14907952 ] Chris Heller commented on SPARK-8734: - Responsibilities have sort of pulled me away from focusing on this. I did managed to get the network code in my branch. I was thinking about parameters, and considered a scheme such as: spark.mesos.executor.docker.parameter. = This follows from how you set environment variables on the executor. Would this scheme be reasonable? > Expose all Mesos DockerInfo options to Spark > > > Key: SPARK-8734 > URL: https://issues.apache.org/jira/browse/SPARK-8734 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Chris Heller >Priority: Minor > Attachments: network.diff > > > SPARK-2691 only exposed a few options from the DockerInfo message. It would > be reasonable to expose them all, especially given one can now specify > arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9442) java.lang.ArithmeticException: / by zero when reading Parquet
[
https://issues.apache.org/jira/browse/SPARK-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903260#comment-14903260
]
Chris Heller commented on SPARK-9442:
-
Curious if the issue seen here was with a parquet file created with a small
block size? I just ran into a similar case with a nested schema, but ad no
problems on many of the files, except for a larger file -- all were written
with a block size of 16MB.
> java.lang.ArithmeticException: / by zero when reading Parquet
> -
>
> Key: SPARK-9442
> URL: https://issues.apache.org/jira/browse/SPARK-9442
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.4.1
>Reporter: DB Tsai
>
> I am counting how many records in my nested parquet file with this schema,
> {code}
> scala> u1aTesting.printSchema
> root
> |-- profileId: long (nullable = true)
> |-- country: string (nullable = true)
> |-- data: array (nullable = true)
> ||-- element: struct (containsNull = true)
> |||-- videoId: long (nullable = true)
> |||-- date: long (nullable = true)
> |||-- label: double (nullable = true)
> |||-- weight: double (nullable = true)
> |||-- features: vector (nullable = true)
> {code}
> and the number of the records in the nested data array is around 10k, and
> each of the parquet file is around 600MB. The total size is around 120GB.
> I am doing a simple count
> {code}
> scala> u1aTesting.count
> parquet.io.ParquetDecodingException: Can not read value at 100 in block 0 in
> file
> hdfs://compute-1.amazonaws.com:9000/users/dbtsai/testing/u1old/20150721/part-r-00115-d70c946b-b0f0-45fe-9965-b9f062b9ec6d.gz.parquet
> at
> parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:213)
> at
> parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:204)
> at
> org.apache.spark.sql.sources.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:163)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$doExecute$1$$anonfun$6.apply(Aggregate.scala:129)
> at
> org.apache.spark.sql.execution.Aggregate$$anonfun$doExecute$1$$anonfun$6.apply(Aggregate.scala:126)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:686)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:686)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:109)
> at
> parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:193)
> ... 21 more
> {code}
> BTW, no all the tasks fail, and some of them are successful.
> Another note: By explicitly looping through the data to count, it will works.
> {code}
> sqlContext.read.load(hdfsPath + s"/testing/u1snappy/${date}/").map(x =>
> 1L).reduce((x, y) => x + y)
> {code}
> I think maybe some metadata in parquet files are corrupted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
[ https://issues.apache.org/jira/browse/SPARK-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14610289#comment-14610289 ] Chris Heller commented on SPARK-8734: - I've started work on this @ https://github.com/hellertime/spark/tree/feature/SPARK-8734 Once I have all the fields I'll submit a PR, but for those eager to try it out feel free to fetch and merge. > Expose all Mesos DockerInfo options to Spark > > > Key: SPARK-8734 > URL: https://issues.apache.org/jira/browse/SPARK-8734 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Chris Heller >Priority: Minor > > SPARK-2691 only exposed a few options from the DockerInfo message. It would > be reasonable to expose them all, especially given one can now specify > arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14608935#comment-14608935 ] Chris Heller commented on SPARK-2691: - I had already created SPARK-8734 to track this, and started work on adding in the additional properties. I should have --net support soon, just running the test suite, and then I'll open a PR. I see no problem supporting all the fields, as it is the most flexible option, but there is certainly the ability to shoot oneself in the foot when Spark, Mesos and Docker all come to the party. > Allow Spark on Mesos to be launched with Docker > --- > > Key: SPARK-2691 > URL: https://issues.apache.org/jira/browse/SPARK-2691 > Project: Spark > Issue Type: New Feature > Components: Mesos >Affects Versions: 1.0.0 >Reporter: Timothy Chen >Assignee: Chris Heller > Labels: mesos > Fix For: 1.4.0 > > Attachments: spark-docker.patch > > > Currently to launch Spark with Mesos one must upload a tarball and specifiy > the executor URI to be passed in that is to be downloaded on each slave or > even each execution depending coarse mode or not. > We want to make Spark able to support launching Executors via a Docker image > that utilizes the recent Docker and Mesos integration work. > With the recent integration Spark can simply specify a Docker image and > options that is needed and it should continue to work as-is. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
[ https://issues.apache.org/jira/browse/SPARK-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Heller updated SPARK-8734: Priority: Minor (was: Major) > Expose all Mesos DockerInfo options to Spark > > > Key: SPARK-8734 > URL: https://issues.apache.org/jira/browse/SPARK-8734 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Chris Heller >Priority: Minor > > SPARK-2691 only exposed a few options from the DockerInfo message. It would > be reasonable to expose them all, especially given one can now specify > arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-8737) Allow configuration of Docker networking for Mesos
[ https://issues.apache.org/jira/browse/SPARK-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Heller closed SPARK-8737. --- Resolution: Duplicate One thing to note about --net, I believe Spark may have issues when running in BRIDGE mode due to Akka port access. Though this might be mitigated if ports are fixed at runtime. > Allow configuration of Docker networking for Mesos > -- > > Key: SPARK-8737 > URL: https://issues.apache.org/jira/browse/SPARK-8737 > Project: Spark > Issue Type: Improvement > Components: Mesos >Affects Versions: 1.4.0 >Reporter: Martin Tapp >Priority: Minor > > Support specifying host vs bridge networking (docker run's --net) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
[ https://issues.apache.org/jira/browse/SPARK-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14608760#comment-14608760 ] Chris Heller commented on SPARK-8734: - This work will enhance the features added to Spark in SPARK-2691 > Expose all Mesos DockerInfo options to Spark > > > Key: SPARK-8734 > URL: https://issues.apache.org/jira/browse/SPARK-8734 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Chris Heller > > SPARK-2691 only exposed a few options from the DockerInfo message. It would > be reasonable to expose them all, especially given one can now specify > arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-8734) Expose all Mesos DockerInfo options to Spark
Chris Heller created SPARK-8734: --- Summary: Expose all Mesos DockerInfo options to Spark Key: SPARK-8734 URL: https://issues.apache.org/jira/browse/SPARK-8734 Project: Spark Issue Type: Improvement Components: Mesos Reporter: Chris Heller SPARK-2691 only exposed a few options from the DockerInfo message. It would be reasonable to expose them all, especially given one can now specify arbitrary parameters to docker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14608748#comment-14608748 ] Chris Heller commented on SPARK-2691: - Only a limited set of options were exposed in this patch. It wouldn't be too difficult to expose the rest. I can look at adding this. > Allow Spark on Mesos to be launched with Docker > --- > > Key: SPARK-2691 > URL: https://issues.apache.org/jira/browse/SPARK-2691 > Project: Spark > Issue Type: New Feature > Components: Mesos >Affects Versions: 1.0.0 >Reporter: Timothy Chen >Assignee: Chris Heller > Labels: mesos > Fix For: 1.4.0 > > Attachments: spark-docker.patch > > > Currently to launch Spark with Mesos one must upload a tarball and specifiy > the executor URI to be passed in that is to be downloaded on each slave or > even each execution depending coarse mode or not. > We want to make Spark able to support launching Executors via a Docker image > that utilizes the recent Docker and Mesos integration work. > With the recent integration Spark can simply specify a Docker image and > options that is needed and it should continue to work as-is. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-5964) Allow spark-daemon.sh to support foreground operation
Chris Heller created SPARK-5964: --- Summary: Allow spark-daemon.sh to support foreground operation Key: SPARK-5964 URL: https://issues.apache.org/jira/browse/SPARK-5964 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Chris Heller Priority: Minor Add --foreground option to spark-daemon.sh to prevent the process from daemonizing itself. Useful if running under a watchdog which waits on its child process. https://github.com/apache/spark/pull/3881 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[
https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204764#comment-14204764
]
Chris Heller commented on SPARK-2691:
-
Just an update. I've been working on this patch over on GitHub
(https://github.com/apache/spark/pull/3074), and have added support for Docker
in both coarse and fine mode -- as well as the ability to map ports.
[~tarnfeld] {spark.executorEnv} was already available from the Spark
configuration properties prior to my fixes.
> Allow Spark on Mesos to be launched with Docker
> ---
>
> Key: SPARK-2691
> URL: https://issues.apache.org/jira/browse/SPARK-2691
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Reporter: Timothy Chen
>Assignee: Timothy Chen
> Labels: mesos
> Attachments: spark-docker.patch
>
>
> Currently to launch Spark with Mesos one must upload a tarball and specifiy
> the executor URI to be passed in that is to be downloaded on each slave or
> even each execution depending coarse mode or not.
> We want to make Spark able to support launching Executors via a Docker image
> that utilizes the recent Docker and Mesos integration work.
> With the recent integration Spark can simply specify a Docker image and
> options that is needed and it should continue to work as-is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[
https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204764#comment-14204764
]
Chris Heller edited comment on SPARK-2691 at 11/10/14 1:28 PM:
---
Just an update. I've been working on this patch over on GitHub
(https://github.com/apache/spark/pull/3074), and have added support for Docker
in both coarse and fine mode -- as well as the ability to map ports.
[~tarnfeld] {{spark.executorEnv}} was already available from the Spark
configuration properties prior to my fixes.
was (Author: chrisheller):
Just an update. I've been working on this patch over on GitHub
(https://github.com/apache/spark/pull/3074), and have added support for Docker
in both coarse and fine mode -- as well as the ability to map ports.
[~tarnfeld] {spark.executorEnv} was already available from the Spark
configuration properties prior to my fixes.
> Allow Spark on Mesos to be launched with Docker
> ---
>
> Key: SPARK-2691
> URL: https://issues.apache.org/jira/browse/SPARK-2691
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Reporter: Timothy Chen
>Assignee: Timothy Chen
> Labels: mesos
> Attachments: spark-docker.patch
>
>
> Currently to launch Spark with Mesos one must upload a tarball and specifiy
> the executor URI to be passed in that is to be downloaded on each slave or
> even each execution depending coarse mode or not.
> We want to make Spark able to support launching Executors via a Docker image
> that utilizes the recent Docker and Mesos integration work.
> With the recent integration Spark can simply specify a Docker image and
> options that is needed and it should continue to work as-is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[
https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194679#comment-14194679
]
Chris Heller commented on SPARK-2691:
-
Ok here is the patch as a PR: https://github.com/apache/spark/pull/3074
[~tarnfeld] feel free to expand on this patch. I was looking at the code today
and realized the coarse mode support should be trivial (just setting a
ContainerInfo inside the TaskInfo created) -- it just cannot reuse the
fine-grained code path in its current form since that assumes passing of an
ExecutorInfo, but it could easily be generalized over a ContainerInfo instead.
We are not shipping the spark image as an executor URI, instead spark is
bundled in the image. It is just a stock spark needed in the image, a simple
docker file would look like (assuming you have a spark tar ball and libmesos in
your directory with the Dockerfile):
{noformat}
FROM ubuntu
RUN apt-get -y update
RUN apt-get -y install default-jre-headless
RUN apt-get -y install python2.7
ADD spark-1.1.0-bin-hadoop1.tgz /
RUN mv /spark-1.1.0-bin-hadoop1 /spark
COPY libmesos-0.20.1.so /usr/lib/libmesos.so
ENV SPARK_HOME /spark
ENV MESOS_JAVA_NATIVE_LIBRARY /usr/lib/libmesos.so
CMD ps -ef
{noformat}
[~yoeduardoj] one awesome thing, which is actually beyond the scope of docker
support, but still related to mesos would be the ability to support
configuration of what role and attributes in a mesos offer are filtered by
spark -- but this is not relevant just wanted to bring it up while folks are
digging into the mesos backend code.
> Allow Spark on Mesos to be launched with Docker
> ---
>
> Key: SPARK-2691
> URL: https://issues.apache.org/jira/browse/SPARK-2691
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
>Reporter: Timothy Chen
>Assignee: Timothy Chen
> Labels: mesos
> Attachments: spark-docker.patch
>
>
> Currently to launch Spark with Mesos one must upload a tarball and specifiy
> the executor URI to be passed in that is to be downloaded on each slave or
> even each execution depending coarse mode or not.
> We want to make Spark able to support launching Executors via a Docker image
> that utilizes the recent Docker and Mesos integration work.
> With the recent integration Spark can simply specify a Docker image and
> options that is needed and it should continue to work as-is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194637#comment-14194637 ] Chris Heller commented on SPARK-2691: - +1 on passing in environment. great idea. There isn't a pull at the moment, I didn't feel the patch was complete enough for that (the lack of support for coarse mode and the total disregard for mesos pre 0.20, make the patch a little fragile) -- but I'll happily create one if you'd like. What is there has been in use on our cluster for a while now, and I would really love to have this be part of upstream. > Allow Spark on Mesos to be launched with Docker > --- > > Key: SPARK-2691 > URL: https://issues.apache.org/jira/browse/SPARK-2691 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen >Assignee: Timothy Chen > Labels: mesos > Attachments: spark-docker.patch > > > Currently to launch Spark with Mesos one must upload a tarball and specifiy > the executor URI to be passed in that is to be downloaded on each slave or > even each execution depending coarse mode or not. > We want to make Spark able to support launching Executors via a Docker image > that utilizes the recent Docker and Mesos integration work. > With the recent integration Spark can simply specify a Docker image and > options that is needed and it should continue to work as-is. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194632#comment-14194632 ] Chris Heller commented on SPARK-2691: - That seems reasonable. In fact, the volumes field of a ContainerInfo is not part of the DockerInfo structure, but since there is only a DOCKER type of ContainerInfo at the moment, and since the volumes field is described perfectly by the 'docker run -v' syntax, it seems OK to repurpose it here. > Allow Spark on Mesos to be launched with Docker > --- > > Key: SPARK-2691 > URL: https://issues.apache.org/jira/browse/SPARK-2691 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen >Assignee: Timothy Chen > Labels: mesos > Attachments: spark-docker.patch > > > Currently to launch Spark with Mesos one must upload a tarball and specifiy > the executor URI to be passed in that is to be downloaded on each slave or > even each execution depending coarse mode or not. > We want to make Spark able to support launching Executors via a Docker image > that utilizes the recent Docker and Mesos integration work. > With the recent integration Spark can simply specify a Docker image and > options that is needed and it should continue to work as-is. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194513#comment-14194513 ] Chris Heller edited comment on SPARK-2691 at 11/3/14 1:08 PM: -- Here is the patch for the changes needed to support docker images in the fine grained backend. The approach taken here was to just populate the DockerInfo of the container info if some properties were set in the properties file. This has no support for versions of mesos which do not support docker, so it is very incomplete. Additionally there is only support for image name, and volumes. For volumes, you just provide a string, which takes a value similar in form to the argument to 'docker run -v', ie. it is a comma separated list of [host:]container[:mode] options. I think this is sufficient, it parallels the command line, and so should be familiar. I would suggest, for all options of the DockerInfo exposed, to mirror how those options are set on the docker command line. was (Author: chrisheller): Here is the patch for the changes needed to support docker images in the fine grained backend. The approach taken here was to just populate the DockerInfo of the container info if some properties were set in the properties file. This has no support for versions of mesos which do not support docker, so it is very incomplete. Additionally there is only support for image name, and volumes. For volumes, you just provide a string, which takes a value similar in form to the argument to 'docker run -v', ie. it is a comma separated list of [host:]container[:mode] options. I think this is sufficient, it parallels the command line, and so should be familiar. > Allow Spark on Mesos to be launched with Docker > --- > > Key: SPARK-2691 > URL: https://issues.apache.org/jira/browse/SPARK-2691 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen >Assignee: Timothy Chen > Labels: mesos > Attachments: spark-docker.patch > > > Currently to launch Spark with Mesos one must upload a tarball and specifiy > the executor URI to be passed in that is to be downloaded on each slave or > even each execution depending coarse mode or not. > We want to make Spark able to support launching Executors via a Docker image > that utilizes the recent Docker and Mesos integration work. > With the recent integration Spark can simply specify a Docker image and > options that is needed and it should continue to work as-is. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Heller updated SPARK-2691: Attachment: spark-docker.patch Here is the patch for the changes needed to support docker images in the fine grained backend. The approach taken here was to just populate the DockerInfo of the container info if some properties were set in the properties file. This has no support for versions of mesos which do not support docker, so it is very incomplete. Additionally there is only support for image name, and volumes. For volumes, you just provide a string, which takes a value similar in form to the argument to 'docker run -v', ie. it is a comma separated list of [host:]container[:mode] options. I think this is sufficient, it parallels the command line, and so should be familiar. > Allow Spark on Mesos to be launched with Docker > --- > > Key: SPARK-2691 > URL: https://issues.apache.org/jira/browse/SPARK-2691 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen >Assignee: Timothy Chen > Labels: mesos > Attachments: spark-docker.patch > > > Currently to launch Spark with Mesos one must upload a tarball and specifiy > the executor URI to be passed in that is to be downloaded on each slave or > even each execution depending coarse mode or not. > We want to make Spark able to support launching Executors via a Docker image > that utilizes the recent Docker and Mesos integration work. > With the recent integration Spark can simply specify a Docker image and > options that is needed and it should continue to work as-is. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2691) Allow Spark on Mesos to be launched with Docker
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190372#comment-14190372 ] Chris Heller commented on SPARK-2691: - This issue seems to have stalled out, but it is a feature that I would really find interesting to have in Spark (hopefully by the next release). I actually have a patched version of Spark which supports DockerInfo already built. However it only supports fine-grained mode. But it may be a good starting point for a more complete feature. Has anyone started on a fix? Or would it make sense for me to provide the patch? > Allow Spark on Mesos to be launched with Docker > --- > > Key: SPARK-2691 > URL: https://issues.apache.org/jira/browse/SPARK-2691 > Project: Spark > Issue Type: Improvement > Components: Mesos >Reporter: Timothy Chen >Assignee: Timothy Chen > Labels: mesos > > Currently to launch Spark with Mesos one must upload a tarball and specifiy > the executor URI to be passed in that is to be downloaded on each slave or > even each execution depending coarse mode or not. > We want to make Spark able to support launching Executors via a Docker image > that utilizes the recent Docker and Mesos integration work. > With the recent integration Spark can simply specify a Docker image and > options that is needed and it should continue to work as-is. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
