[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[lianhuiwang]

Github user lianhuiwang commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-212227437
  
@HyukjinKwon  @vanzin Thanks. I have updated PR description. But @vanzin 
have merged to master before. So I think it does not matter for this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/12473


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-212226185
  
@vanzin Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-212224734
  
LGTM, merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-212223875
  
@HyukjinKwon the merge scripts clean up "@" references from the PR summary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-212185623
  
(This is super minor but I remember I was told it might be better if those 
cc are added in comments not in the description because PR description is the 
place where to describe the PR.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211722592
  
**[Test build #56188 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56188/consoleFull)**
 for PR 12473 at commit 
[`61b51f2`](https://github.com/apache/spark/commit/61b51f2bcff0b0e45cb237779db72c53641a93d3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211722649
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56188/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211722647
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[lianhuiwang]

Github user lianhuiwang commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211720533
  
@jerryshao Yes, what you said is right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211720423
  
**[Test build #56188 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56188/consoleFull)**
 for PR 12473 at commit 
[`61b51f2`](https://github.com/apache/spark/commit/61b51f2bcff0b0e45cb237779db72c53641a93d3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211717311
  
So my understanding is that actually supporting **different** HDFS other 
than default one, not **multiple** HDFS, is that right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211716533
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56186/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211716518
  
**[Test build #56186 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56186/consoleFull)**
 for PR 12473 at commit 
[`59eb03c`](https://github.com/apache/spark/commit/59eb03cc46ce9526fdf733fe0ded3f8131215cb4).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211716532
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211714034
  
**[Test build #56186 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56186/consoleFull)**
 for PR 12473 at commit 
[`59eb03c`](https://github.com/apache/spark/commit/59eb03cc46ce9526fdf733fe0ded3f8131215cb4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211703497
  
**[Test build #56182 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56182/consoleFull)**
 for PR 12473 at commit 
[`c0374af`](https://github.com/apache/spark/commit/c0374afab5d858e948d69f03dcbf6ca95dfc1dd0).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211703519
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56182/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211703518
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[lianhuiwang]

Github user lianhuiwang commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211702442
  
@vanzin  yes, I update code with your comments.Thanks.
@tgravescs I test this on my spark using spark-shell and spark-submit, I 
update it.Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211702239
  
**[Test build #56182 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56182/consoleFull)**
 for PR 12473 at commit 
[`c0374af`](https://github.com/apache/spark/commit/c0374afab5d858e948d69f03dcbf6ca95dfc1dd0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211502073
  
test says NA, what testing have you done with this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211500988
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211500993
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56064/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211500511
  
**[Test build #56064 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56064/consoleFull)**
 for PR 12473 at commit 
[`0ba4de8`](https://github.com/apache/spark/commit/0ba4de81c1ccbff195d9df63cf4dd582e01ee24f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12473#issuecomment-211487353
  
**[Test build #56064 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56064/consoleFull)**
 for PR 12473 at commit 
[`0ba4de8`](https://github.com/apache/spark/commit/0ba4de81c1ccbff195d9df63cf4dd582e01ee24f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12473#discussion_r60098422
  
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -356,8 +361,8 @@ private[spark] class Client(
 logInfo("Preparing resources for our AM container")
 // Upload Spark and the application JAR to the remote file system if 
necessary,
 // and add them as local resources to the application master.
-val fs = FileSystem.get(hadoopConf)
-val dst = getAppStagingDirPath(sparkConf, fs, appStagingDir)
+val dst = new Path(appStagingBaseDir, appStagingDir)
--- End diff --

You could pass `appStagingDir` as a `Path` to this method and save some 
duplication; same for `setupLaunchEnv` below.

This whole class could use some cleanup in that regard, but these two are 
pretty low-hanging fruit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...

[lianhuiwang]

GitHub user lianhuiwang opened a pull request:

https://github.com/apache/spark/pull/12473

[SPARK-14705][YARN]support Multiple FileSystem for YARN STAGING DIR

## What changes were proposed in this pull request?
In SPARK-13063, It makes the SPARK YARN STAGING DIR as configurable. But it 
only support default FileSystem. If there are many clusters, It can be 
different FileSystem for different cluster in our spark.

## How was this patch tested?
N/A

cc @tgravescs @vanzin @andrewor14 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lianhuiwang/spark SPARK-14705

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12473.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12473


commit 0ba4de81c1ccbff195d9df63cf4dd582e01ee24f
Author: Lianhui Wang 
Date:   2016-04-18T16:34:14Z

init commit




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org