[ https://issues.apache.org/jira/browse/SPARK-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273041#comment-14273041 ]
Apache Spark commented on SPARK-5172: ------------------------------------- User 'srowen' has created a pull request for this issue: https://github.com/apache/spark/pull/3992 > spark-examples-***.jar shades a wrong Hadoop distribution > --------------------------------------------------------- > > Key: SPARK-5172 > URL: https://issues.apache.org/jira/browse/SPARK-5172 > Project: Spark > Issue Type: Bug > Components: Build > Reporter: Shixiong Zhu > Priority: Minor > > Steps to check it: > 1. Download "spark-1.2.0-bin-hadoop2.4.tgz" from > http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz > 2. unzip `spark-examples-1.2.0-hadoop2.4.0.jar`. > 3. There is a file called `org/apache/hadoop/package-info.class` in the jar. > It doesn't exist in hadoop 2.4. > 4. Run "javap -classpath . -private -c -v org.apache.hadoop.package-info" > {code} > Compiled from "package-info.java" > interface org.apache.hadoop.package-info > SourceFile: "package-info.java" > RuntimeVisibleAnnotations: length = 0x24 > 00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A > 00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00 > 11 73 00 12 > minor version: 0 > major version: 50 > Constant pool: > const #1 = Asciz org/apache/hadoop/package-info; > const #2 = class #1; // "org/apache/hadoop/package-info" > const #3 = Asciz java/lang/Object; > const #4 = class #3; // java/lang/Object > const #5 = Asciz package-info.java; > const #6 = Asciz Lorg/apache/hadoop/HadoopVersionAnnotation;; > const #7 = Asciz version; > const #8 = Asciz 1.2.1; > const #9 = Asciz revision; > const #10 = Asciz 1503152; > const #11 = Asciz user; > const #12 = Asciz mattf; > const #13 = Asciz date; > const #14 = Asciz Wed Jul 24 13:39:35 PDT 2013; > const #15 = Asciz url; > const #16 = Asciz > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2; > const #17 = Asciz srcChecksum; > const #18 = Asciz 6923c86528809c4e7e6f493b6b413a9a; > const #19 = Asciz SourceFile; > const #20 = Asciz RuntimeVisibleAnnotations; > { > } > {code} > The version is {{1.2.1}} > It comes because a wrong hbase version settings in examples project. Here is > a part of the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4 > -Dhadoop.version=2.4.0 -pl examples dependency:tree" > {noformat} > [INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile > [INFO] | +- > org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile > [INFO] | +- > org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile > [INFO] | | +- com.sun.jersey:jersey-core:jar:1.8:compile > [INFO] | | +- com.sun.jersey:jersey-json:jar:1.8:compile > [INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile > [INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile > [INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile > [INFO] | | \- com.sun.jersey:jersey-server:jar:1.8:compile > [INFO] | | \- asm:asm:jar:3.3.1:test > [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile > [INFO] | +- > org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile > [INFO] | +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile > [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile > [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile > [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile > [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile > [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile > [INFO] | | \- commons-el:commons-el:jar:1.0:compile > [INFO] | +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile > [INFO] | | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile > [INFO] | | +- org.apache.mina:mina-core:jar:2.0.0-M5:compile > [INFO] | | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile > [INFO] | | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile > [INFO] | +- > com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile > [INFO] | \- junit:junit:jar:4.10:test > [INFO] | \- org.hamcrest:hamcrest-core:jar:1.1:test > {noformat} > If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am > dependency:tree -Dhbase.profile=hadoop2`, the dependency tree is right. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org