[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-212227437 @HyukjinKwon @vanzin Thanks. I have updated PR description. But @vanzin have merged to master before. So I think it does not matter for this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12473 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-212226185 @vanzin Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-212224734 LGTM, merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-212223875 @HyukjinKwon the merge scripts clean up "@" references from the PR summary. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-212185623 (This is super minor but I remember I was told it might be better if those cc are added in comments not in the description because PR description is the place where to describe the PR.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211722592 **[Test build #56188 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56188/consoleFull)** for PR 12473 at commit [`61b51f2`](https://github.com/apache/spark/commit/61b51f2bcff0b0e45cb237779db72c53641a93d3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211722649 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56188/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211722647 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211720533 @jerryshao Yes, what you said is right. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211720423 **[Test build #56188 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56188/consoleFull)** for PR 12473 at commit [`61b51f2`](https://github.com/apache/spark/commit/61b51f2bcff0b0e45cb237779db72c53641a93d3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211717311 So my understanding is that actually supporting **different** HDFS other than default one, not **multiple** HDFS, is that right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211716533 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56186/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211716518 **[Test build #56186 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56186/consoleFull)** for PR 12473 at commit [`59eb03c`](https://github.com/apache/spark/commit/59eb03cc46ce9526fdf733fe0ded3f8131215cb4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211716532 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211714034 **[Test build #56186 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56186/consoleFull)** for PR 12473 at commit [`59eb03c`](https://github.com/apache/spark/commit/59eb03cc46ce9526fdf733fe0ded3f8131215cb4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211703497 **[Test build #56182 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56182/consoleFull)** for PR 12473 at commit [`c0374af`](https://github.com/apache/spark/commit/c0374afab5d858e948d69f03dcbf6ca95dfc1dd0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211703519 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56182/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211703518 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211702442 @vanzin yes, I update code with your comments.Thanks. @tgravescs I test this on my spark using spark-shell and spark-submit, I update it.Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211702239 **[Test build #56182 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56182/consoleFull)** for PR 12473 at commit [`c0374af`](https://github.com/apache/spark/commit/c0374afab5d858e948d69f03dcbf6ca95dfc1dd0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211502073 test says NA, what testing have you done with this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211500988 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211500993 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56064/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211500511 **[Test build #56064 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56064/consoleFull)** for PR 12473 at commit [`0ba4de8`](https://github.com/apache/spark/commit/0ba4de81c1ccbff195d9df63cf4dd582e01ee24f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12473#issuecomment-211487353 **[Test build #56064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56064/consoleFull)** for PR 12473 at commit [`0ba4de8`](https://github.com/apache/spark/commit/0ba4de81c1ccbff195d9df63cf4dd582e01ee24f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/12473#discussion_r60098422 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -356,8 +361,8 @@ private[spark] class Client( logInfo("Preparing resources for our AM container") // Upload Spark and the application JAR to the remote file system if necessary, // and add them as local resources to the application master. -val fs = FileSystem.get(hadoopConf) -val dst = getAppStagingDirPath(sparkConf, fs, appStagingDir) +val dst = new Path(appStagingBaseDir, appStagingDir) --- End diff -- You could pass `appStagingDir` as a `Path` to this method and save some duplication; same for `setupLaunchEnv` below. This whole class could use some cleanup in that regard, but these two are pretty low-hanging fruit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14705][YARN]support Multiple FileSystem...
GitHub user lianhuiwang opened a pull request: https://github.com/apache/spark/pull/12473 [SPARK-14705][YARN]support Multiple FileSystem for YARN STAGING DIR ## What changes were proposed in this pull request? In SPARK-13063, It makes the SPARK YARN STAGING DIR as configurable. But it only support default FileSystem. If there are many clusters, It can be different FileSystem for different cluster in our spark. ## How was this patch tested? N/A cc @tgravescs @vanzin @andrewor14 You can merge this pull request into a Git repository by running: $ git pull https://github.com/lianhuiwang/spark SPARK-14705 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12473.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12473 commit 0ba4de81c1ccbff195d9df63cf4dd582e01ee24f Author: Lianhui Wang Date: 2016-04-18T16:34:14Z init commit --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org