[GitHub] spark pull request: [SPARK-12133][STREAMING] Streaming dynamic all...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/12154#issuecomment-212539770 I understand that, but I want to port this feature to our internal custom 1.6 build, if it's not too much trouble. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12133][STREAMING] Streaming dynamic all...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/12154#issuecomment-212279971 @tdas or @andrewor14 does this depend on any 2.0 APIs, I would like to backport this to 1.5 or 1.6 if possible. need to run multiple concurrent streaming jobs on mesos --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Backport [SPARK-11327] [MESOS] Dispatcher does...
Github user jayv closed the pull request at: https://github.com/apache/spark/pull/12101 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Backport [SPARK-11327] [MESOS] Dispatcher does...
GitHub user jayv opened a pull request: https://github.com/apache/spark/pull/12101 Backport [SPARK-11327] [MESOS] Dispatcher does not respect all args f⦠Backport for https://github.com/apache/spark/pull/10370 @andrewor14 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayv/spark mesos_cluster_params_backport Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12101.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12101 commit 8cd2a247de3a8408bdc0b71a943ec977f3269ba1 Author: Jo Voordeckers <jo.voordeck...@gmail.com> Date: 2016-03-31T19:08:10Z Backport [SPARK-11327] [MESOS] Dispatcher does not respect all args from the Submit request --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11327] [MESOS] Dispatcher does not resp...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-203555239 @dragos rebased! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11327] [MESOS] Dispatcher does not resp...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-199957820 Nice, will you pick this up @dragos or waiting for me? - can't commit any time this week --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11327] [MESOS] Dispatcher does not resp...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-191369895 Sorry for the delay @dragos very busy indeed, maybe this can help https://github.com/jayv/spark/commit/8023da309df2e5440e244e12d449cb8107d15b00 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11327] [MESOS] Dispatcher does not resp...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-184807346 @dragos Unfortunately not, I'll come back to this issue when I've done some more testing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11327] [MESOS] Dispatcher does not resp...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-178241777 Got an implementation that escapes all the things under linux shells, but had to suspend work on this, so was unable to test mesos' native escaping. I'll try to get back to it this week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11327] [MESOS] Dispatcher does not resp...
Github user jayv commented on a diff in the pull request:
https://github.com/apache/spark/pull/10370#discussion_r50805506
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -440,6 +446,9 @@ private[spark] class MesosClusterScheduler(
.mkString(",")
options ++= Seq("--py-files", formattedFiles)
}
+desc.schedulerProperties
+ .filter { case (key, _) => !replicatedOptionsBlacklist.contains(key)
}
+ .foreach { case (key, value) => options ++= Seq("--conf",
s"""$key="$value"""") }
--- End diff --
Ah OK, I can check that, I wrote a bunch of regexes and tests today to
solve escaping for most Unix shells. Was about to push a PR early morning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11327] [MESOS] Dispatcher does not resp...
Github user jayv commented on a diff in the pull request:
https://github.com/apache/spark/pull/10370#discussion_r50884168
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -440,6 +446,9 @@ private[spark] class MesosClusterScheduler(
.mkString(",")
options ++= Seq("--py-files", formattedFiles)
}
+desc.schedulerProperties
+ .filter { case (key, _) => !replicatedOptionsBlacklist.contains(key)
}
+ .foreach { case (key, value) => options ++= Seq("--conf",
s"""$key="$value"""") }
--- End diff --
The meat of it is only 5 lines :) but let's see if we can make it work
without side-effects and for my use-case with args with spaces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11327] [MESOS] Dispatcher does not resp...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-174626690 Today I will test the driver issue you mentioned, we also seem to have a rogue driver spinning up on workers, and work on better escaping. Isn't there code for this in spark-submit or something already? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11327] [MESOS] Dispatcher does not resp...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-172980362 @dragos I don't remember seeing these errors on the 1.5.x branch I tested against,but I may have missed them since I was focussing on the missing args. This looks totally reasonable to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv commented on a diff in the pull request:
https://github.com/apache/spark/pull/10370#discussion_r48311266
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -440,6 +444,9 @@ private[spark] class MesosClusterScheduler(
.mkString(",")
options ++= Seq("--py-files", formattedFiles)
}
+desc.schedulerProperties
+ .filter { case (key, _) => !replicatedOptionsBlacklist.contains(key)
}
+ .foreach { case (key, value) => options ++= Seq("--conf", Seq(key,
"=\"", value, "\"").mkString("")) }
--- End diff --
Yes, the quotes work for me.
In order to pass multiple system properties via eg
`spark.driver.extraJavaOptions`, I passed the arg in quotes on the shell to
spark-submit, but the value itself doesn't have quotes around it and ends up
broken on the commandline for the driver, unless I add escaped quotes inside
the string.
So this fails when executing the driver, but spark-submit accepts the
command:
```
bin/spark-submit [...]
spark.driver.extraJavaOptions="-Dcom.sun.jersey.server.impl.cdi.lookupExtensionInBeanManager=true
-Dorg.jboss.weld.bootstrap.concurrentDeployment=false" [...]
```
This doesn't look very intuitive to me, but it does work without wrapping
values in quotes:
```
bin/spark-submit [...]
spark.driver.extraJavaOptions="\"-Dcom.sun.jersey.server.impl.cdi.lookupExtensionInBeanManager=true
-Dorg.jboss.weld.bootstrap.concurrentDeployment=false\"" [...]
```
String interpolation doesn't support double quotes, but I've since learned
about triple-double-quoted string interpolation, that why I initially went with
`Seq.mkString`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165893840 @tnachen I've tested that patch against branch-1.5 and I'm now able to specify options such as ```--conf spark.executor.userClassPathFirst=true --conf spark.driver.userClassPathFirst=true``` on spark-submit for a cluster deploy via mesos. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
GitHub user jayv opened a pull request: https://github.com/apache/spark/pull/10370 [WIP] [SPARK-11327] [MESOS] Dispatcher does not respect all args from the Submit request Supersedes https://github.com/apache/spark/pull/9752 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayv/spark mesos_cluster_params Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10370.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10370 commit cdde93d45401a2f7a97a46b25387c297dcf21d2f Author: Jo Voordeckers <jo.voordeck...@gmail.com> Date: 2015-11-17T01:18:44Z Messos scheduler does not respect all args from the Submit request --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv closed the pull request at: https://github.com/apache/spark/pull/9752 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-165655382 New PR against master: https://github.com/apache/spark/pull/10370 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv commented on the pull request:
https://github.com/apache/spark/pull/9752#issuecomment-165317051
I wasn't able to make time for this, but I should have time tomorrow.
Sorry for the delay.
- Jo Voordeckers
On Mon, Dec 14, 2015 at 4:51 PM, andrewor14 <notificati...@github.com>
wrote:
> @jayv <https://github.com/jayv> have you had the chance to work on this
> patch? If not, shall one of us take it over?
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/9752#issuecomment-164606612>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv commented on the pull request:
https://github.com/apache/spark/pull/9752#issuecomment-158715518
I will get to it on Monday.
- Jo Voordeckers
On Sat, Nov 21, 2015 at 2:31 PM, Iulian Dragos <notificati...@github.com>
wrote:
> @jayv <https://github.com/jayv> will you have time to update this PR?
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/9752#issuecomment-158687388>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158180964 I needed a patch for our version, so branched off of that. I'll make a new PR. @dragos no docker, puppet installs our spark build on all our mesos slaves in `/opt/spark`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158185915 I would assume so. When I used `spark.*.extraJavaOptions` to specify `-Dfoo=bar` it got applied to both my driver and tasks which it didn't before my patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv commented on a diff in the pull request:
https://github.com/apache/spark/pull/9752#discussion_r45244546
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -422,6 +422,37 @@ private[spark] class MesosClusterScheduler(
desc.schedulerProperties.get("spark.cores.max").map { v =>
options ++= Seq("--total-executor-cores", v)
}
+
+/**
+ * The properties to pass along from Spark Submit to the Mesos
Scheduler.
+ *
http://spark.apache.org/docs/1.5.1/configuration.html#available-properties
+ */
+val propertiesToPropagate = Set(
+ "spark.driver.extraClassPath",
--- End diff --
Agree on the maintenance burden and +1 for including all args minus
`spark.jars` which causes class path issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] Mesos Dispatcher does not ...
Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-157464149 This problem is discussed here: https://issues.apache.org/jira/browse/SPARK-11327 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...
Github user jayv commented on a diff in the pull request:
https://github.com/apache/spark/pull/9752#discussion_r45144080
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -422,6 +422,37 @@ private[spark] class MesosClusterScheduler(
desc.schedulerProperties.get("spark.cores.max").map { v =>
options ++= Seq("--total-executor-cores", v)
}
+
+/**
+ * The properties to pass along from Spark Submit to the Mesos
Scheduler.
+ *
http://spark.apache.org/docs/1.5.1/configuration.html#available-properties
+ */
+val propertiesToPropagate = Set(
+ "spark.driver.extraClassPath",
--- End diff --
My app jar was duplicated in the classpath due to the `spark.jars`
property, then I wondered about which settings you care about when
"_customizing_" a job vs infrastructure implied settings from the config files.
Stripping `spark.jars` would probably be fine, but I'm not familiar enough with
the framework to know of any other potential conflicts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIPMessos scheduler does not respect all args...
GitHub user jayv opened a pull request: https://github.com/apache/spark/pull/9752 [WIPMessos scheduler does not respect all args from the Submit request You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayv/spark mesos_cluster_params Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9752 commit b2025ddc1d565d1cc3036200fc3b3046578f4b02 Author: Jo Voordeckers <jo.voordeck...@gmail.com> Date: 2015-11-17T01:18:44Z Messos scheduler does not respect all args from the Submit request --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
