Hi everyone,
I am having some weird classpath issues with a UDF that returns a custom tuple.
My custom tuple has an arraylist of custom objects. It looks like:
class MyTuple
private ArrayList<MyClass> list;
When the UDF is called, everything works fine: the tuples are created and the
UDF returns successfully. But then in the next step when I try to dump the
results of the UDF I see class not found exceptions. That is, MyClass can be
found during UDF execution, but it cannot be found while dumping the UDF
results. I tried setting PIG_CLASSPATH, but didn't work. The only working
solution so far is to add my jar file to hadoop's classpath. I wonder whether
this is a bug (PIG_CLASSPATH should work) or I am not setting something
correctly. Any help will be greatly appreciated.
Thanks,
Nezih
Here is my pig script:
register 'my_jar.jar';
x = load '/etc/passwd' using PigStorage(':') as (username:chararray, f1:
chararray, f2: chararray, f3:chararray, f4:chararray);
parsed = foreach x generate username, MyParser(*);
dump parsed;
Here is the stack trace:
Caused by: java.io.IOException: Could not find class MyClass, while attempting
to de-serialize it
at org.apache.pig.data.BinInterSedes.readWritable(BinInterSedes.java:293)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:422)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:318)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:349)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:318)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:349)
at
org.apache.pig.impl.io.InterRecordReader.nextKeyValue(InterRecordReader.java:113)
at org.apache.pig.impl.io.InterStorage.getNext(InterStorage.java:77)
at
org.apache.pig.impl.io.ReadToEndLoader.getNextHelper(ReadToEndLoader.java:246)
at org.apache.pig.impl.io.ReadToEndLoader.getNext(ReadToEndLoader.java:226)
at
org.apache.pig.backend.hadoop.executionengine.HJob$1.hasNext(HJob.java:111)
... 12 more
