[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/19885 @jerryshao can you backport this to branch 2.2 as well. thanks --- - To unsubscribe, e-mail:

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 Let me merge to master and branch 2.3. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/19885 @jerryshao and @steveloughran thanks for your comments and review. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 LGTM. @merlintang please fix the PR title, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85931/ Test PASSed. ---

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85931/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85931/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19885 LGTM. Effective use of parameterization --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 @steveloughran @vanzin please help to review again. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85902/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85902/ Test PASSed. ---

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85902/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85897/ Test FAILed. ---

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85897/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85897/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85889/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85889/ Test FAILed. ---

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85889/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/19885 @steveloughran can you review the added system test cases? --- - To unsubscribe, e-mail:

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/19885 My local test is ok. I would set up a system test and update this soon. sorry about this delay. On Tue, Jan 2, 2018 at 3:42 PM, Marcelo Vanzin wrote:

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19885 Any updates? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/19885 I am so sorry for the late of testing function, I would update it soon. On Thu, Dec 14, 2017 at 12:55 PM, UCB AMPLab wrote: > Can one of the admins

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19885 I think everybody is still waiting for the tests to be added to the PR. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19885 I'd recommend the tests are parameterized, generating a separate test for each URI pair, and including the values on a failure. Plan for a future where all you have is a stack trace from

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #84594 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84594/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84594/ Test PASSed. ---

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/19885 I have added this test case for the URI comparing based on Steve's comments. I have tested this in my local vm, it pass the test. meanwhile, for the hdfs://namenode1/path1

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #84594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84594/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 I see. Thanks for the explanation @steveloughran . My concern is that current changes will affect all the filesystems, but we only saw this issue in wasb. So limiting authority comparison to only

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19885 if you make a path of each of these and call getFileSystem() on them, you will end up with two different FS instances in the same JVM. But they'll both be talking to the same namenode using

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 >User info isn't picked up from the URL, it's taken off your Kerberos credentials. If you are running HDFS unkerberized, then UGI takes it from the environment variable HADOOP_USER_NAME.

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19885 @vanzin its too late for this, but I don't see any reason why `FileSystem.getCanonicalUri` should be kept protected. If someone wants to volunteer with the spec changes to filesystem.md &

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19885 User info isn't picked up from the URL, it's taken off your Kerberos credentials. If you are running HDFS unkerberized, then UGI takes it from the environment variable `HADOOP_USER_NAME`.

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 I still have a question about it, URIs for HDFS like `hdfs://us...@nn1.com:8020` and `hdfs://us...@nn1.com:8020` , do we honor userInfo for HDFS filesystems, are they two HDFS clusters, or just

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19885 Switching to comparing URIs should be ok if there's an easy way to canonicalize them. There's `FileSystem.getCanonicalUri`, but it's a protected method. Otherwise it might be tricky to

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19885 Hi. If the comparision is isolated to a method testing URIs, rather than filesystems, it should be straightforward to write a suite of tests for this, with lists of URIs expected to

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #84455 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84455/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84455/ Test PASSed. ---

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #84455 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84455/testReport)** for PR 19885 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 ok to test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 Is this assumption based on the implementation of Hadoop `FileSystem`? I was thinking that wasb is an exception, for other we still keep the original code. @steveloughran would you

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/19885 @jerryshao yes, hdfs://us...@nn1.com:8020 and hdfs://us...@nn1.com:8020 would consider as two filesystem, since the authority information should be taken into consideration. that is why need to

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 @vanzin please help to review, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19885 @merlintang would you please add the problem to your PR description, currently it is a WASB problem in which userInfo is honored to differentiate filesystems. Please add the scenario to the

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/19885 @jerryshao can you review this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional