Remark, just including the jar built by sbt will produce the same error. i,.e this pig script will fail:
REGISTER /usr/share/osi1/spark-1.0.0/assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop0.20.2-cdh3u4.jar; edgeList0 = LOAD '/user/rfcompton/twitter-mention-networks/bidirectional-network-current/part-r-00001' USING PigStorage() AS (id1:long, id2:long, weight:int); ttt = LIMIT edgeList0 10; DUMP ttt; On Wed, May 28, 2014 at 12:55 PM, Ryan Compton <compton.r...@gmail.com> wrote: > It appears to be Spark 1.0 related. I made a pom.xml with a single > dependency on Spark, registering the resulting jar created the error. > > Spark 1.0 was compiled via $ SPARK_HADOOP_VERSION=0.20.2-cdh3u4 sbt/sbt > assembly > > The pom.xml, as well as some other information, is below. The only > thing that should not be standard is the inclusion of my in-house > repository (it's where I host the spark jar I compiled above). > > <project xmlns="http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > <modelVersion>4.0.0</modelVersion> > > <groupId>com.mycompany.app</groupId> > <artifactId>my-app</artifactId> > <version>1.0-SNAPSHOT</version> > <packaging>jar</packaging> > > <name>my-app</name> > <url>http://maven.apache.org</url> > > <properties> > <maven.compiler.source>1.6</maven.compiler.source> > <maven.compiler.target>1.6</maven.compiler.target> > <encoding>UTF-8</encoding> > <scala.version>2.10.4</scala.version> > </properties> > > <build> > <pluginManagement> > <plugins> > <plugin> > <groupId>net.alchim31.maven</groupId> > <artifactId>scala-maven-plugin</artifactId> > <version>3.1.5</version> > </plugin> > <plugin> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-compiler-plugin</artifactId> > <version>2.0.2</version> > </plugin> > </plugins> > </pluginManagement> > > <plugins> > > <plugin> > <groupId>net.alchim31.maven</groupId> > <artifactId>scala-maven-plugin</artifactId> > <executions> > <execution> > <id>scala-compile-first</id> > <phase>process-resources</phase> > <goals> > <goal>add-source</goal> > <goal>compile</goal> > </goals> > </execution> > <execution> > <id>scala-test-compile</id> > <phase>process-test-resources</phase> > <goals> > <goal>testCompile</goal> > </goals> > </execution> > </executions> > </plugin> > > <!-- Plugin to create a single jar that includes all > dependencies --> > <plugin> > <artifactId>maven-assembly-plugin</artifactId> > <version>2.4</version> > <configuration> > <descriptorRefs> > <descriptorRef>jar-with-dependencies</descriptorRef> > </descriptorRefs> > </configuration> > <executions> > <execution> > <id>make-assembly</id> > <phase>package</phase> > <goals> > <goal>single</goal> > </goals> > </execution> > </executions> > </plugin> > > </plugins> > </build> > > <repositories> > > <!-- needed for cdh build of Spark --> > <repository> > <id>releases</id> > <url>10.10.1.29:8081/nexus/content/repositories/releases</url> > </repository> > > <repository> > <id>cloudera</id> > > <url>https://repository.cloudera.com/artifactory/cloudera-repos</url> > </repository> > > </repositories> > > <dependencies> > > <dependency> > <groupId>org.scala-lang</groupId> > <artifactId>scala-library</artifactId> > <version>${scala.version}</version> > </dependency> > > <!--on node29--> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-assembly</artifactId> > <version>1.0.0-cdh3u4</version> > <classifier>cdh3u4</classifier> > </dependency> > > <!--spark docs says I need hadoop-client, cdh3u3 repo no > longer exists--> > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-client</artifactId> > <version>0.20.2-cdh3u4</version> > </dependency> > > </dependencies> > </project> > > > Here's what I get in the dependency tree: > > [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ my-app --- > [INFO] com.mycompany.app:my-app:jar:1.0-SNAPSHOT > [INFO] +- org.scala-lang:scala-library:jar:2.10.4:compile > [INFO] +- org.apache.spark:spark-assembly:jar:cdh3u4:1.0.0-cdh3u4:compile > [INFO] \- org.apache.hadoop:hadoop-client:jar:0.20.2-cdh3u4:compile > [INFO] \- org.apache.hadoop:hadoop-core:jar:0.20.2-cdh3u4:compile > [INFO] +- com.cloudera.cdh:hadoop-ant:pom:0.20.2-cdh3u4:compile > [INFO] +- xmlenc:xmlenc:jar:0.52:compile > [INFO] +- > org.apache.hadoop.thirdparty.guava:guava:jar:r09-jarjar:compile > [INFO] +- commons-codec:commons-codec:jar:1.4:compile > [INFO] +- commons-net:commons-net:jar:1.4.1:compile > [INFO] | \- (oro:oro:jar:2.0.8:compile - omitted for duplicate) > [INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.5.2:compile > [INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.5.2:compile > [INFO] | \- > (org.codehaus.jackson:jackson-core-asl:jar:1.5.2:compile - omitted for > duplicate) > [INFO] +- commons-el:commons-el:jar:1.0:compile > [INFO] | \- commons-logging:commons-logging:jar:1.0.3:compile > [INFO] +- hsqldb:hsqldb:jar:1.8.0.7:compile > [INFO] \- oro:oro:jar:2.0.8:compile > > > While I don't see slf4j anywhere in there, it does manage to find it's > way into the jar somehow: > rfcompton@node19 /u/s/o/n/my-app> find . -name "*.jar" | xargs -tn1 > jar tvf | grep -i "slf" | grep LocationAware > jar tvf ./target/my-app-1.0-SNAPSHOT.jar > jar tvf ./target/my-app-1.0-SNAPSHOT-jar-with-dependencies.jar > 3259 Mon Mar 25 21:49:34 PDT 2013 > org/apache/commons/logging/impl/SLF4JLocationAwareLog.class > 455 Mon Mar 25 21:49:22 PDT 2013 org/slf4j/spi/LocationAwareLogger.class > 479 Fri Dec 13 16:44:40 PST 2013 > parquet/org/slf4j/spi/LocationAwareLogger.class > > Here's a pig script that will fail with the slf4j error: > > REGISTER > /usr/share/osi1/nonhome/my-app/target/my-app-1.0-SNAPSHOT-jar-with-dependencies.jar; > > edgeList0 = LOAD > '/user/rfcompton/twitter-mention-networks/bidirectional-network-current/part-r-00001' > USING PigStorage() AS (id1:long, id2:long, weight:int); > > ttt = LIMIT edgeList0 10; > DUMP ttt; > > (the error) > rfcompton@node19 /u/s/o/n/my-app> pig src/main/pig/testSparkJar.pig > 2014-05-28 12:43:58,076 [main] INFO org.apache.pig.Main - Apache Pig > version 0.12.1 (r1585011) compiled Apr 05 2014, 01:41:34 > 2014-05-28 12:43:58,078 [main] INFO org.apache.pig.Main - Logging > error messages to: > /usr/share/osi1/nonhome/my-app/pig_1401306238074.log > 2014-05-28 12:43:58,722 [main] INFO org.apache.pig.impl.util.Utils - > Default bootup file /home/isl/rfcompton/.pigbootup not found > 2014-05-28 12:43:59,195 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to hadoop file system at: hdfs://master:8020/ > 2014-05-28 12:43:59,811 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to map-reduce job tracker at: node4:8021 > 2014-05-28 12:44:00,987 [main] ERROR org.apache.pig.tools.grunt.Grunt > - ERROR 2998: Unhandled internal error. > org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V > Details at logfile: /usr/share/osi1/nonhome/my-app/pig_1401306238074.log > > > To confirm this is 1.0 related, I modified the pom.xml build with > 0.9.1 and saw no problems from pig. Looking into the 0.9.1 jar > revealed less dependence on slf4j (i.e > "parquet/org/slf4j/spi/LocationAwareLogger.class" appeared in > spark1.0). > > (after recompiling for 0.9.1) > rfcompton@node19 /u/s/o/n/my-app> find . -name "*.jar" | xargs -tn1 > jar tvf | grep -i "slf" | grep LocationAware > jar tvf ./target/my-app-1.0-SNAPSHOT.jar > jar tvf ./target/my-app-1.0-SNAPSHOT-jar-with-dependencies.jar > 455 Mon Mar 25 21:49:22 PDT 2013 org/slf4j/spi/LocationAwareLogger.class > > > On Tue, May 27, 2014 at 2:53 PM, Sean Owen <so...@cloudera.com> wrote: >> Spark uses 1.7.5, and you should probably see 1.7.{4,5} in use through >> Hadoop. But those are compatible. >> >> That method appears to have been around since 1.3. What version does Pig >> want? >> >> I usually do "mvn -Dverbose dependency:tree" to see both what the >> final dependencies are, and what got overwritten, to diagnose things >> like this. >> >> My hunch is that something is depending on an old slf4j in your build >> and it's overwriting Spark et al. >> >> On Tue, May 27, 2014 at 10:45 PM, Ryan Compton <compton.r...@gmail.com> >> wrote: >>> I use both Pig and Spark. All my code is built with Maven into a giant >>> *-jar-with-dependencies.jar. I recently upgraded to Spark 1.0 and now >>> all my pig scripts fail with: >>> >>> Caused by: java.lang.RuntimeException: Could not resolve error that >>> occured when launching map reduce job: java.lang.NoSuchMethodError: >>> org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V >>> at >>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$JobControlThreadExceptionHandler.uncaughtException(MapReduceLauncher.java:598) >>> at java.lang.Thread.dispatchUncaughtException(Thread.java:1874) >>> >>> >>> Did Spark 1.0 change the version of slf4j? I can't seem to find it via >>> mvn dependency:tree
