Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/9282
received ClassNotFound error in Yarn-Cluster mode(spark 1.6.2)ï¼_doesn't
reproduce the problem_
The error message is such as below:
```
[2017-10-26 16:53:18,274] ERROR Error while
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
sorry for the late response, IIUC, in MR, this case handled by below
1. AM received the container failed message
2. AM will check whether there are any attempts of the same task is RUNNING
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
@jerryshao thank you for your comment, I will try to find how MR/TEZ handle
this
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
@squito I agree with you that this should be handled by yarn.
In my opinion, this is some form of defensive programming. The Spark
Streaming and structured streaming will both request more
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
My colleague create a
[issue](https://issues.apache.org/jira/browse/YARN-7214) here, I rewrite the
description here.
Spark Streaming (app1) running on Yarn, app1's one container (c1) runs
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
We enabled RM and NM recovery.
If we assume there are 2 containers running on this NM, after 10 minute, RM
detects the failure of NM and relaunches 2 lost containers in other NMs
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
Hi @jerryshao, thank you for your reply.
# Problem
the problem is for long running jobs which run on **yarn with HA** will
request more executors than it requests.
# How
Github user klion26 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19145#discussion_r139125435
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
---
@@ -158,6 +158,7 @@ private[yarn] class YarnAllocator
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
@jerryshao Could you help me to review this pathc?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
Will the same completed message will be reported more than twice, if these
message will not be reported more than twice, then i could use
`completedContainerIdSet.remove(containerId)`
instead
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
@HyukjinKwon @vanzin @srowen @foxish @djvulee @squito Could you please
help to review this pr?
---
-
To unsubscribe, e-mail
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
@HyukjinKwon i am sorry for that, have changed the title form
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user klion26 commented on the issue:
https://github.com/apache/spark/pull/19145
create a issue https://issues.apache.org/jira/browse/SPARK-21933
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user klion26 opened a pull request:
https://github.com/apache/spark/pull/19145
add logic to test whether the complete container has been completed before
## What changes were proposed in this pull request?
Add logic to test whether the complete container has been
Github user klion26 closed the pull request at:
https://github.com/apache/spark/pull/17206
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user klion26 opened a pull request:
https://github.com/apache/spark/pull/17206
Branch 1.6
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch
16 matches
Mail list logo
