Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88908858
you have to package or send all the appropriate stuff with your spark jar.
For instance the hadoop configs. How were you running it?
---
If your project is set up fo
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88708773
@tgravescs @srowen @sryza As i have retest again, if we don't populate
hadoop classpath, then in all case it dosen't work. This PR cann't solve this
issue, i will c
Github user Sephiroth-Lin closed the pull request at:
https://github.com/apache/spark/pull/5294
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88590039
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88507012
@Sephiroth-Lin what testing have you done with this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/5294#discussion_r27530621
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -809,7 +809,13 @@ object Client extends Logging {
}
}
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/5294#discussion_r27530532
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -809,7 +809,13 @@ object Client extends Logging {
}
}
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/5294#discussion_r27530361
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -809,7 +809,13 @@ object Client extends Logging {
}
}
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88271630
So I was mostly interested in understanding what the use case was, since
the bug was a little short on details. Tom's explanation makes sense; the
opposite (hadoopA built
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88259688
So yes I could use hadoop provided and then package my own hadoop but you
end up with same scenario as I describe. If I don't package hadoop then I rely
on the version
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88255263
Yeah, that's the idea behind deploying a Spark that doesn't include Hadoop.
In your scenario, if Spark totally works with Hadoop A and B, then
Spark-without-Hadoop should
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88254838
Note, I do understand what you are saying with if there isn't really a use
case we shouldn't include it as it costs in dev. If everyone else disagrees
with my use case
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88254374
Yes that is basically the scenario. Although I would expect it start out
package hadoopA with Spark running on HadoopA, then hadoopB is deployed and
spark with hadoopA
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88247299
I don't know that it's controversial. As in all things, it's a question of
how much of a problem it solves for how many users versus how much burden it
puts on other users
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88239503
Yes it should be an odd case but if you are using it in production and it
suddenly breaks while Hadoop is doing a rolling upgrade then it could be a
major issue. I ha
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88210123
@tgravescs in that case you're running Spark with a slightly different
version of Hadoop classes than is found on the local machine or on the rest of
the cluster. I can im
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88183017
So we had run into an issue where something in hadoop changed that required
me to recompile Spark, whereas if it hadn't included the stuff from hadoop in
my classpath i
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88174985
/cc @tgravescs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88171273
Yeah, unless there's an actual use case for this, it doesn't sound like we
need the change. The classpath is added after Spark's assembly, so if the
assembly includes the
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88165688
@srowen I assume this would be for running assemblies that already include
the Hadoop classes. @Sephiroth-Lin do you mind going into detail about the
situations you need t
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88054697
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pro
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5294#issuecomment-88054581
CC @vanzin The cluster's assembly would generally have Hadoop "provided"
right? so you would want the cluster's classes.
---
If your project is set up for it, you can rep
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/5294
[SPARK-1502][YARN]Add config option to not include yarn/mapred cluster
classpath
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Sephirot
23 matches
Mail list logo
