Looks like a hadoop classpath problem since it can't find the guava jar file. Does this command return anything?
pig -x local -secretDebugCmd | sed 's/:/\n/g' | grep guava scott. On Thu, May 3, 2012 at 11:56 AM, Nicholas Kolegraff <[email protected]> wrote: > Hi Everyone, > This doesn't seem to be a *pig* error but I'm trying to do a join in pig > when I get it. This use to work just fine but I did an update on some > packages that left hadoop+pig alone .. is this a JVM thing? > > This is my pig version: > Apache Pig version 0.8.1-cdh3u3 (rexported) > > This is my hadoop version: > Hadoop 0.20.2-cdh3u1 > > (it use to work with both of these versions) > > Here is what I am trying to do: > for i in {1..10000}; do echo $(expr $RANDOM % 1000) >> a.txt; done > for i in {1..10000}; do echo $(expr $RANDOM % 1000) >> b.txt; done > > pig_join.pig: > A = load 'a.txt'; > B = load 'b.txt'; > > C = JOIN A by $0, B by $0; > D = FOREACH C GENERATE $0; > > DUMP D; > > > Here is what I get: Any Ideas? > 2012-05-03 11:31:19,622 [main] INFO > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > script: HASH_JOIN > 2012-05-03 11:31:19,622 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > pig.usenewlogicalplan is set to true. New logical plan will be used. > 2012-05-03 11:31:19,744 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: D: > Store(file:/tmp/temp1192087216/tmp-1872578945:org.apache.pig.impl.io.InterStorage) > - scope-16 Operator Key: scope-16) > 2012-05-03 11:31:19,753 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > File concatenation threshold: 100 optimistic? false > 2012-05-03 11:31:19,774 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer > - Rewrite: POPackage->POForEach to POJoinPackage > 2012-05-03 11:31:19,783 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size before optimization: 1 > 2012-05-03 11:31:19,783 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size after optimization: 1 > 2012-05-03 11:31:19,801 [main] INFO > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with > processName=JobTracker, sessionId= > 2012-05-03 11:31:19,812 [main] INFO > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added > to the job > 2012-05-03 11:31:19,835 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 > 2012-05-03 11:31:21,580 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - Setting up single store job > 2012-05-03 11:31:21,590 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=113322 > 2012-05-03 11:31:21,590 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - Neither PARALLEL nor default parallelism is set for this job. Setting > number of reducers to 1 > 2012-05-03 11:31:21,659 [main] INFO > org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics > with processName=JobTracker, sessionId= - already initialized > 2012-05-03 11:31:21,659 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 1 map-reduce job(s) waiting for submission. > 2012-05-03 11:31:21,694 [Thread-3] INFO > org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library > 2012-05-03 11:31:21,843 [Thread-3] INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths > to process : 1 > 2012-05-03 11:31:21,843 [Thread-3] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths to process : 1 > 2012-05-03 11:31:21,852 [Thread-3] WARN > org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library is > available > 2012-05-03 11:31:21,852 [Thread-3] INFO > org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library > loaded > 2012-05-03 11:31:21,854 [Thread-3] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths (combined) to process : 1 > 2012-05-03 11:31:21,860 [Thread-3] INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths > to process : 1 > 2012-05-03 11:31:21,860 [Thread-3] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths to process : 1 > 2012-05-03 11:31:21,860 [Thread-3] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths (combined) to process : 1 > 2012-05-03 11:31:22,160 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - HadoopJobId: job_local_0001 > 2012-05-03 11:31:22,160 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 0% complete > 2012-05-03 11:31:22,168 [Thread-4] INFO org.apache.hadoop.util.ProcessTree > - setsid exited with exit code 0 > 2012-05-03 11:31:22,171 [Thread-4] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1badd463 > 2012-05-03 11:31:22,195 [Thread-4] INFO org.apache.hadoop.mapred.MapTask - > io.sort.mb = 100 > 2012-05-03 11:31:22,269 [Thread-4] INFO org.apache.hadoop.mapred.MapTask - > data buffer = 79691776/99614720 > 2012-05-03 11:31:22,269 [Thread-4] INFO org.apache.hadoop.mapred.MapTask - > record buffer = 262144/327680 > 2012-05-03 11:31:22,320 [Thread-4] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader > - Created input record counter: Input records from a.txt > 2012-05-03 11:31:22,685 [Thread-4] INFO org.apache.hadoop.mapred.MapTask - > Starting flush of map output > 2012-05-03 11:31:22,701 [Thread-4] WARN > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > java.lang.NoClassDefFoundError: > org/apache/hadoop/thirdparty/guava/common/primitives/UnsignedBytes > at > org.apache.hadoop.io.FastByteComparisons$LexicographicalComparerHolder$UnsafeComparer.compareTo(FastByteComparisons.java:226) > at > org.apache.hadoop.io.FastByteComparisons$LexicographicalComparerHolder$UnsafeComparer.compareTo(FastByteComparisons.java:113) > at > org.apache.hadoop.io.FastByteComparisons.compareTo(FastByteComparisons.java:42) > at > org.apache.hadoop.io.WritableComparator.compareBytes(WritableComparator.java:150) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler$PigWritableComparator.compare(JobControlCompiler.java:828) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:968) > at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) > at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1254) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1155) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:582) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:649) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.thirdparty.guava.common.primitives.UnsignedBytes > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 14 more
