[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11721


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-198152244
  
Ping @vazin, what about the current fix, would you please review again, 
thanks a lot.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-198463862
  
That vazin guy must be wondering why he was pinged. :-) Sorry, I've been 
busy with other things, I'll try to take a look today.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-198513672
  
LGTM, merging to master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197209163
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197209169
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53282/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197208742
  
**[Test build #53282 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53282/consoleFull)**
 for PR 11721 at commit 
[`188c83b`](https://github.com/apache/spark/commit/188c83b3317a5dda5d896a9c1a6da9373aeb2091).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197207834
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197207836
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53281/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197207642
  
**[Test build #53281 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53281/consoleFull)**
 for PR 11721 at commit 
[`0c34e0e`](https://github.com/apache/spark/commit/0c34e0e00506998d992e0e5758fae2dadd8296dc).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197168676
  
**[Test build #53282 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53282/consoleFull)**
 for PR 11721 at commit 
[`188c83b`](https://github.com/apache/spark/commit/188c83b3317a5dda5d896a9c1a6da9373aeb2091).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[jerryshao]

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/11721#discussion_r56285494
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -133,12 +133,9 @@ private[spark] class ApplicationMaster(
 System.setProperty("spark.master", "yarn")
 System.setProperty("spark.submit.deployMode", "cluster")
 
-// Propagate the application ID so that 
YarnClusterSchedulerBackend can pick it up.
+// Set this internal configuration to true if it is running on 
cluster mode, this
+// configuration will be checked in SparkContext to avoid misuse 
of yarn cluster mode.
 System.setProperty("spark.yarn.app.id", 
appAttemptId.getApplicationId().toString())
--- End diff --

Using configuration is a simple and safe way to check the misuse of 
configuration, so here I revert back this configuration `spark.yarn.app.id`, 
but only use to check whether app is running on yarn-cluster mode correctly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197167540
  
**[Test build #53281 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53281/consoleFull)**
 for PR 11721 at commit 
[`0c34e0e`](https://github.com/apache/spark/commit/0c34e0e00506998d992e0e5758fae2dadd8296dc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[jerryshao]

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/11721#discussion_r56284175
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -374,12 +374,6 @@ class SparkContext(config: SparkConf) extends Logging 
with ExecutorAllocationCli
   throw new SparkException("An application name must be set in your 
configuration")
 }
 
-// System property spark.yarn.app.id must be set if user code ran by 
AM on a YARN cluster
-if (master == "yarn" && deployMode == "cluster" && 
!_conf.contains("spark.yarn.app.id")) {
--- End diff --

I see, removing this will still get other error messages if trying to run 
cluster mode with such way, but maybe a little odd as you mentioned.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197156547
  
I think `SchedulerExtensionService` can still be worked, since I still 
maintain the full attempt id in `YarnScheduler`, only change to the simple 
counter for Spark scheduler.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-197074328
  
LGTM aside from small comment.

@steveloughran should take a look since IIRC he wanted the full attempt id 
for something.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/11721#discussion_r56263000
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -374,12 +374,6 @@ class SparkContext(config: SparkConf) extends Logging 
with ExecutorAllocationCli
   throw new SparkException("An application name must be set in your 
configuration")
 }
 
-// System property spark.yarn.app.id must be set if user code ran by 
AM on a YARN cluster
-if (master == "yarn" && deployMode == "cluster" && 
!_conf.contains("spark.yarn.app.id")) {
--- End diff --

Hmm, this was introduced so that people don't try `new SparkContext(new 
SparkConf().setMaster("yarn-cluster"))` and then complain when they get really 
odd error messages.

Is there another way to do this check?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-196780449
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-196780452
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53183/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-196780179
  
**[Test build #53183 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53183/consoleFull)**
 for PR 11721 at commit 
[`6339a01`](https://github.com/apache/spark/commit/6339a01574c619cc4a36f95966b8ba65a640f7fe).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11721#issuecomment-196731895
  
**[Test build #53183 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53183/consoleFull)**
 for PR 11721 at commit 
[`6339a01`](https://github.com/apache/spark/commit/6339a01574c619cc4a36f95966b8ba65a640f7fe).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13885][YARN] Fix attempt id regression ...

[jerryshao]

GitHub user jerryshao opened a pull request:

https://github.com/apache/spark/pull/11721

[SPARK-13885][YARN] Fix attempt id regression for Spark running on Yarn

## What changes were proposed in this pull request?

This regression is introduced in #9182, previously attempt id is simply as 
counter "1" or "2". With the change of #9182, it is changed to full name as 
"appattemtp-xxx-1", this will affect all the parts which uses this attempt 
id, like event log file name, history server app url link. So here change it 
back to the counter to keep consistent with previous code.

Also revert back this patch #11518, this patch fix the url link of history 
log according to the new way of attempt id, since here we change back to the 
previous way, so this patch is not necessary, here to revert it.

Also clean "spark.yarn.app.id" and "spark.yarn.app.attemptId", since it is 
useless now.

## How was this patch tested?

Test it with unit test and manually test different scenario:

1. application running in yarn-client mode.
2. application running in yarn-cluster mode.
3. application running in yarn-cluster mode with multiple attempts.

Checked both the event log file name and url link.

CC @vanzin @tgravescs , please help to review, thanks a lot.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerryshao/apache-spark SPARK-13885

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11721.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11721


commit b777533ffd39e91cab6b6a5c9a28364e81aca50e
Author: jerryshao 
Date:   2016-03-15T04:26:26Z

Revert "[SPARK-13675][UI] Fix wrong historyserver url link for application 
running in yarn cluster mode"

This reverts commit 9e86e6efd136182bb00fa925c3818c9baccbd1fc.

commit 6339a01574c619cc4a36f95966b8ba65a640f7fe
Author: jerryshao 
Date:   2016-03-15T05:43:02Z

Change to the correct attempt id




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org