[ https://issues.apache.org/jira/browse/SPARK-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu updated SPARK-5172: -------------------------------- Description: Steps to check it: 1. Download "spark-1.2.0-bin-hadoop2.4.tgz" from http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz 2. unzip `spark-examples-1.2.0-hadoop2.4.0.jar`. 3. There is a file called `org/apache/hadoop/package-info.class` in the jar. It doesn't exist in hadoop 2.4. 4. Run "javap -classpath . -private -c -v org.apache.hadoop.package-info" {code} Compiled from "package-info.java" interface org.apache.hadoop.package-info SourceFile: "package-info.java" RuntimeVisibleAnnotations: length = 0x24 00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A 00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00 11 73 00 12 minor version: 0 major version: 50 Constant pool: const #1 = Asciz org/apache/hadoop/package-info; const #2 = class #1; // "org/apache/hadoop/package-info" const #3 = Asciz java/lang/Object; const #4 = class #3; // java/lang/Object const #5 = Asciz package-info.java; const #6 = Asciz Lorg/apache/hadoop/HadoopVersionAnnotation;; const #7 = Asciz version; const #8 = Asciz 1.2.1; const #9 = Asciz revision; const #10 = Asciz 1503152; const #11 = Asciz user; const #12 = Asciz mattf; const #13 = Asciz date; const #14 = Asciz Wed Jul 24 13:39:35 PDT 2013; const #15 = Asciz url; const #16 = Asciz https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2; const #17 = Asciz srcChecksum; const #18 = Asciz 6923c86528809c4e7e6f493b6b413a9a; const #19 = Asciz SourceFile; const #20 = Asciz RuntimeVisibleAnnotations; { } {code} The version is {{1.2.1}} It comes because a wrong hbase version settings in examples project. Here is a port of the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples dependency:tree" {noformat} [INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile [INFO] | +- org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile [INFO] | +- org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile [INFO] | | +- com.sun.jersey:jersey-core:jar:1.8:compile [INFO] | | +- com.sun.jersey:jersey-json:jar:1.8:compile [INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile [INFO] | | \- com.sun.jersey:jersey-server:jar:1.8:compile [INFO] | | \- asm:asm:jar:3.3.1:test [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile [INFO] | +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] | | \- commons-el:commons-el:jar:1.0:compile [INFO] | +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile [INFO] | | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile [INFO] | | +- org.apache.mina:mina-core:jar:2.0.0-M5:compile [INFO] | | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile [INFO] | | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile [INFO] | +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile [INFO] | \- junit:junit:jar:4.10:test [INFO] | \- org.hamcrest:hamcrest-core:jar:1.1:test {noformat} If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am dependency:tree -Dhbase.profile=hadoop2`, the dependency tree is right. was: Steps to check it: 1. Download "spark-1.2.0-bin-hadoop2.4.tgz" from http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz 2. unzip `spark-assembly-1.2.0-hadoop2.4.0.jar`. 3. There is a file called `org/apache/hadoop/package-info.class` in the jar. It doesn't exist in hadoop 2.4. 4. Run "javap -classpath . -private -c -v org.apache.hadoop.package-info" {code} Compiled from "package-info.java" interface org.apache.hadoop.package-info SourceFile: "package-info.java" RuntimeVisibleAnnotations: length = 0x24 00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A 00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00 11 73 00 12 minor version: 0 major version: 50 Constant pool: const #1 = Asciz org/apache/hadoop/package-info; const #2 = class #1; // "org/apache/hadoop/package-info" const #3 = Asciz java/lang/Object; const #4 = class #3; // java/lang/Object const #5 = Asciz package-info.java; const #6 = Asciz Lorg/apache/hadoop/HadoopVersionAnnotation;; const #7 = Asciz version; const #8 = Asciz 1.2.1; const #9 = Asciz revision; const #10 = Asciz 1503152; const #11 = Asciz user; const #12 = Asciz mattf; const #13 = Asciz date; const #14 = Asciz Wed Jul 24 13:39:35 PDT 2013; const #15 = Asciz url; const #16 = Asciz https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2; const #17 = Asciz srcChecksum; const #18 = Asciz 6923c86528809c4e7e6f493b6b413a9a; const #19 = Asciz SourceFile; const #20 = Asciz RuntimeVisibleAnnotations; { } {code} The version is {{1.2.1}} It comes because a wrong hbase version settings in examples project. Here is a port of the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples dependency:tree" {noformat} [INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile [INFO] | +- org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile [INFO] | +- org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile [INFO] | | +- com.sun.jersey:jersey-core:jar:1.8:compile [INFO] | | +- com.sun.jersey:jersey-json:jar:1.8:compile [INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile [INFO] | | \- com.sun.jersey:jersey-server:jar:1.8:compile [INFO] | | \- asm:asm:jar:3.3.1:test [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile [INFO] | +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] | | \- commons-el:commons-el:jar:1.0:compile [INFO] | +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile [INFO] | | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile [INFO] | | +- org.apache.mina:mina-core:jar:2.0.0-M5:compile [INFO] | | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile [INFO] | | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile [INFO] | +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile [INFO] | \- junit:junit:jar:4.10:test [INFO] | \- org.hamcrest:hamcrest-core:jar:1.1:test {noformat} If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am dependency:tree -Dhbase.profile=hadoop2`, the dependency tree is right. > spark-examples-***.jar shades a wrong Hadoop distribution > --------------------------------------------------------- > > Key: SPARK-5172 > URL: https://issues.apache.org/jira/browse/SPARK-5172 > Project: Spark > Issue Type: Bug > Components: Build > Reporter: Shixiong Zhu > Priority: Minor > > Steps to check it: > 1. Download "spark-1.2.0-bin-hadoop2.4.tgz" from > http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz > 2. unzip `spark-examples-1.2.0-hadoop2.4.0.jar`. > 3. There is a file called `org/apache/hadoop/package-info.class` in the jar. > It doesn't exist in hadoop 2.4. > 4. Run "javap -classpath . -private -c -v org.apache.hadoop.package-info" > {code} > Compiled from "package-info.java" > interface org.apache.hadoop.package-info > SourceFile: "package-info.java" > RuntimeVisibleAnnotations: length = 0x24 > 00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A > 00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00 > 11 73 00 12 > minor version: 0 > major version: 50 > Constant pool: > const #1 = Asciz org/apache/hadoop/package-info; > const #2 = class #1; // "org/apache/hadoop/package-info" > const #3 = Asciz java/lang/Object; > const #4 = class #3; // java/lang/Object > const #5 = Asciz package-info.java; > const #6 = Asciz Lorg/apache/hadoop/HadoopVersionAnnotation;; > const #7 = Asciz version; > const #8 = Asciz 1.2.1; > const #9 = Asciz revision; > const #10 = Asciz 1503152; > const #11 = Asciz user; > const #12 = Asciz mattf; > const #13 = Asciz date; > const #14 = Asciz Wed Jul 24 13:39:35 PDT 2013; > const #15 = Asciz url; > const #16 = Asciz > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2; > const #17 = Asciz srcChecksum; > const #18 = Asciz 6923c86528809c4e7e6f493b6b413a9a; > const #19 = Asciz SourceFile; > const #20 = Asciz RuntimeVisibleAnnotations; > { > } > {code} > The version is {{1.2.1}} > It comes because a wrong hbase version settings in examples project. Here is > a port of the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4 > -Dhadoop.version=2.4.0 -pl examples dependency:tree" > {noformat} > [INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile > [INFO] | +- > org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile > [INFO] | +- > org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile > [INFO] | | +- com.sun.jersey:jersey-core:jar:1.8:compile > [INFO] | | +- com.sun.jersey:jersey-json:jar:1.8:compile > [INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile > [INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile > [INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile > [INFO] | | \- com.sun.jersey:jersey-server:jar:1.8:compile > [INFO] | | \- asm:asm:jar:3.3.1:test > [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile > [INFO] | +- > org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile > [INFO] | +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile > [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile > [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile > [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile > [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile > [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile > [INFO] | | \- commons-el:commons-el:jar:1.0:compile > [INFO] | +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile > [INFO] | | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile > [INFO] | | +- org.apache.mina:mina-core:jar:2.0.0-M5:compile > [INFO] | | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile > [INFO] | | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile > [INFO] | +- > com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile > [INFO] | \- junit:junit:jar:4.10:test > [INFO] | \- org.hamcrest:hamcrest-core:jar:1.1:test > {noformat} > If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am > dependency:tree -Dhbase.profile=hadoop2`, the dependency tree is right. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org