Repository: spark Updated Branches: refs/heads/branch-2.3 d22379ec2 -> 84922e506
[SPARK-25330][BUILD][BRANCH-2.3] Revert Hadoop 2.7 to 2.7.3 ## What changes were proposed in this pull request? How to reproduce permission issue: ```sh # build spark ./dev/make-distribution.sh --name SPARK-25330 --tgz -Phadoop-2.7 -Phive -Phive-thriftserver -Pyarn tar -zxf spark-2.4.0-SNAPSHOT-bin-SPARK-25330.tar && cd spark-2.4.0-SNAPSHOT-bin-SPARK-25330 export HADOOP_PROXY_USER=user_a bin/spark-sql export HADOOP_PROXY_USER=user_b bin/spark-sql ``` ```java Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=user_b, access=EXECUTE, inode="/tmp/hive-$%7Buser.name%7D/user_b/668748f2-f6c5-4325-a797-fd0a7ee7f4d4":user_b:hadoop:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) ``` The issue occurred in this commit: https://github.com/apache/hadoop/commit/feb886f2093ea5da0cd09c69bd1360a335335c86. This pr revert Hadoop 2.7 to 2.7.3 to avoid this issue. ## How was this patch tested? unit tests and manual tests. Closes #22327 from wangyum/SPARK-25330. Authored-by: Yuming Wang <yumw...@ebay.com> Signed-off-by: Sean Owen <sean.o...@databricks.com> (cherry picked from commit b0ada7dce02d101b6a04323d8185394e997caca4) Signed-off-by: Sean Owen <sean.o...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/84922e50 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/84922e50 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/84922e50 Branch: refs/heads/branch-2.3 Commit: 84922e506e57413a83cea4460a2a1649f2700293 Parents: d22379e Author: Yuming Wang <yumw...@ebay.com> Authored: Thu Sep 6 21:41:13 2018 -0700 Committer: Sean Owen <sean.o...@databricks.com> Committed: Thu Sep 6 21:41:38 2018 -0700 ---------------------------------------------------------------------- assembly/README | 2 +- dev/deps/spark-deps-hadoop-2.7 | 31 +++++++++++++++---------------- docs/building-spark.md | 2 +- pom.xml | 2 +- 4 files changed, 18 insertions(+), 19 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/84922e50/assembly/README ---------------------------------------------------------------------- diff --git a/assembly/README b/assembly/README index affd281..d5dafab 100644 --- a/assembly/README +++ b/assembly/README @@ -9,4 +9,4 @@ This module is off by default. To activate it specify the profile in the command If you need to build an assembly for a different version of Hadoop the hadoop-version system property needs to be set as in this example: - -Dhadoop.version=2.7.7 + -Dhadoop.version=2.7.3 http://git-wip-us.apache.org/repos/asf/spark/blob/84922e50/dev/deps/spark-deps-hadoop-2.7 ---------------------------------------------------------------------- diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7 index 6a3b38b..df2be77 100644 --- a/dev/deps/spark-deps-hadoop-2.7 +++ b/dev/deps/spark-deps-hadoop-2.7 @@ -66,21 +66,21 @@ gson-2.2.4.jar guava-14.0.1.jar guice-3.0.jar guice-servlet-3.0.jar -hadoop-annotations-2.7.7.jar -hadoop-auth-2.7.7.jar -hadoop-client-2.7.7.jar -hadoop-common-2.7.7.jar -hadoop-hdfs-2.7.7.jar -hadoop-mapreduce-client-app-2.7.7.jar -hadoop-mapreduce-client-common-2.7.7.jar -hadoop-mapreduce-client-core-2.7.7.jar -hadoop-mapreduce-client-jobclient-2.7.7.jar -hadoop-mapreduce-client-shuffle-2.7.7.jar -hadoop-yarn-api-2.7.7.jar -hadoop-yarn-client-2.7.7.jar -hadoop-yarn-common-2.7.7.jar -hadoop-yarn-server-common-2.7.7.jar -hadoop-yarn-server-web-proxy-2.7.7.jar +hadoop-annotations-2.7.3.jar +hadoop-auth-2.7.3.jar +hadoop-client-2.7.3.jar +hadoop-common-2.7.3.jar +hadoop-hdfs-2.7.3.jar +hadoop-mapreduce-client-app-2.7.3.jar +hadoop-mapreduce-client-common-2.7.3.jar +hadoop-mapreduce-client-core-2.7.3.jar +hadoop-mapreduce-client-jobclient-2.7.3.jar +hadoop-mapreduce-client-shuffle-2.7.3.jar +hadoop-yarn-api-2.7.3.jar +hadoop-yarn-client-2.7.3.jar +hadoop-yarn-common-2.7.3.jar +hadoop-yarn-server-common-2.7.3.jar +hadoop-yarn-server-web-proxy-2.7.3.jar hk2-api-2.4.0-b34.jar hk2-locator-2.4.0-b34.jar hk2-utils-2.4.0-b34.jar @@ -121,7 +121,6 @@ jersey-media-jaxb-2.22.2.jar jersey-server-2.22.2.jar jets3t-0.9.4.jar jetty-6.1.26.jar -jetty-sslengine-6.1.26.jar jetty-util-6.1.26.jar jline-2.12.1.jar joda-time-2.9.3.jar http://git-wip-us.apache.org/repos/asf/spark/blob/84922e50/docs/building-spark.md ---------------------------------------------------------------------- diff --git a/docs/building-spark.md b/docs/building-spark.md index cd80835..5395ddf 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -67,7 +67,7 @@ Examples: ./build/mvn -Pyarn -DskipTests clean package # Apache Hadoop 2.7.X and later - ./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.7 -DskipTests clean package + ./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean package ## Building With Hive and JDBC Support http://git-wip-us.apache.org/repos/asf/spark/blob/84922e50/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index 3ff0408..090abf2 100644 --- a/pom.xml +++ b/pom.xml @@ -2664,7 +2664,7 @@ <profile> <id>hadoop-2.7</id> <properties> - <hadoop.version>2.7.7</hadoop.version> + <hadoop.version>2.7.3</hadoop.version> <curator.version>2.7.1</curator.version> </properties> </profile> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
