Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-216407536
@jdesmet , by default if cpu scheduling is not enabled in yarn, what you
saw on yarn's web UI about vcore usage (1 per container) is actually
meaningless, I think
Github user jdesmet commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-216370906
However, memory reported in yarn ui on the containers seems to largely
match with what I declared to use for the spark executors. Also capacity
scheduler does have the
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-216305430
> why we can't report the correct vCores
@jdesmet Spark is not reporting anything, and that's the part you are
confused about. YARN does all its accounting
Github user jdesmet commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-216016788
Humbly, I think I understood what this PR was about. I probably (still) do
not understand some of the reasoning as to why we can't report the correct
vCores even if the
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-216000549
@jdesmet you did not understand what this PR was about. Nothing you're
saying is affected by this PR. Accounting of core usage in YARN is not changed.
Please read the
Github user jdesmet commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-215998052
From a user point of view the closure of this issue as-is is unacceptable.
I cannot understand why one would allow wrong job accounting for the executors
as reported in
Github user jdesmet commented on a diff in the pull request:
https://github.com/apache/spark/pull/9095#discussion_r61675971
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -395,6 +395,10 @@ private[yarn] class YarnAllocator(
val
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147649000
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/9095
[SPARK-11082][YARN] Fix wrong core number when response vcore is less than
requested vcore
This should be guarded out and use response vcore number, this will be
happened when use
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147649014
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147656903
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147656497
[Test build #43639 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43639/console)
for PR 9095 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147656906
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147651353
[Test build #43639 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43639/consoleFull)
for PR 9095 at commit
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147665208
CC @sryza @vanzin seems reasonable to make sure it's actually allocating
what YARN said it could?
Is this really the extent of the assumption though? it seems
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147668124
Gotcha. This is probably my ignorance/misunderstanding then. As long as
this is the only place the fact that the requested amount wasn't the same as
the granted amount.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147667700
@srowen , not sure what exactly you mean?
From what I know in `CoarseGrainedSchedulerBackend`, it will manage the
executors with cores available, this number
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/9095#discussion_r41866396
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -414,7 +418,7 @@ private[yarn] class YarnAllocator(
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/9095#discussion_r41866472
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -395,6 +395,10 @@ private[yarn] class YarnAllocator(
val
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9095#discussion_r41857332
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -395,6 +395,10 @@ private[yarn] class YarnAllocator(
val
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9095#discussion_r41856996
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -395,6 +395,10 @@ private[yarn] class YarnAllocator(
val
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9095#discussion_r41858120
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -395,6 +395,10 @@ private[yarn] class YarnAllocator(
val
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147734624
Sometimes its not up to the user what scheduler they user. Like in our
case cluster admins choose what its running and users just use it. They have
to use whatever
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147727046
But from yarn's side actually only allocated 1 vcores, whereas in the
driver side, it notified with more than 1 cores when executor get registered,
this is not
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147721566
So actually against this change. It breaks backwards compatibility and I
think the current behavior is what we want.
@jerryshao why do you think this is a
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9095#discussion_r41872358
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -395,6 +395,10 @@ private[yarn] class YarnAllocator(
val
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147734023
Actually YARN doesn't allocate any. The only reason it reports 1 is because
cpu scheduling is disabled and its trying to return something reasonable.YARN
does not
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147737702
Yeah, I get it, thanks a lot for your explanation, still from user' point,
it may easily get confused, maybe we should document this difference.
---
If your project
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147727971
If user want to set executor cores more than 1, user should choose dominant
scheduler calculator, that will keep consistent both in spark and yarn side.
---
If your
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147767229
There's related discussion about this in
https://issues.apache.org/jira/browse/SPARK-6050 and the respective PR (#4818).
---
If your project is set up for it, you can
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147769172
yes its really more a YARN problem then a SPARK problem. Ideal the YARN
side wouldn't show cores at all if you aren't using a scheduler that does
cores, but that is
Github user jerryshao closed the pull request at:
https://github.com/apache/spark/pull/9095
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-147898407
Thanks a lot @tgravescs and @vanzin , looks like it is a intention to do
such way, greatly appreciate your explanation, I will close it.
---
If your project is set
33 matches
Mail list logo
