[
https://issues.apache.org/jira/browse/PIG-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037742#comment-14037742
]
Mona Chitnis commented on PIG-3972:
-----------------------------------
This is currently expected behavior in Pig. If the bag entry is null/empty, the
inner schema is unknown and hence Pig will not accept it. The suggested
approach is to use builtin func IsEmpty in a conditional operator before the
FLATTEN. I have tested it on a similar script myself.
Using same example as above, make this change for it to work:
{code}
A = LOAD 'test1.txt' AS (mybag:bag{t:tuple(title:chararray,name:chararray,
id:chararray)});
NOEMPTY = foreach A generate ((mybag is null or IsEmpty(mybag))
? {('unknown','unknown','unknown')} : mybag) as mybag;
A1 = FOREACH NOEMPTY generate flatten(mybag) as
(title:chararray,name:chararray, id:chararray);
B = LOAD 'test2.txt' AS (id:chararray, content:chararray);
C = JOIN A1 BY id LEFT OUTER, B BY id;
dump C;
{code}
This is a better way of handling empty/null bags rather than introducing a
bunch of nulls which the extra Foreach - Generate step does.
If your bag has tuples of mixed types, you can also use something like
{('unknown',(int)0,(double)0)} to match the schema.
> java.lang.IndexOutOfBoundsException when flatten meets an empty row
> -------------------------------------------------------------------
>
> Key: PIG-3972
> URL: https://issues.apache.org/jira/browse/PIG-3972
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.12.0, 0.13.0
> Reporter: Bing Jiang
> Assignee: Mona Chitnis
>
> $ cat test1.txt
> {(,A1111,A),(,B222,B),(,C333,C)}
> $ cat test2.txt
> A Helloworld
> B Pig
> C Hive
> $ cat tt.pig
> A = LOAD 'test1.txt' AS (mybag:bag{t:tuple(title:chararray,name:chararray,
> id:chararray)});
> A1 = FOREACH A generate flatten(mybag) as (title:chararray,name:chararray,
> id:chararray);
> B = LOAD 'test2.txt' AS (id:chararray, content:chararray);
> C = JOIN A1 BY id LEFT OUTER, B BY id;
> dump C;
> $ pig -x local -f tt.pig
> ....
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:604)
> at java.util.ArrayList.get(ArrayList.java:382)
> at org.apache.pig.data.DefaultTuple.get(DefaultTuple.java:115)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getValueTuple(POPackage.java:350)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getNext(POPackage.java:273)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:425)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:416)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:256)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:636)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:396)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:441)
> [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Ooops! Some job has failed! Specify -stop_on_failu
> re if you want Pig to stop immediately on failure.
> ==============
> If we change the code into this one:
> $ cat tt.pig
> A = LOAD 'test1.txt' AS (mybag:bag{t:tuple(title:chararray,name:chararray,
> id:chararray)});
> A1 = FOREACH A generate flatten(mybag) as (title:chararray,name:chararray,
> id:chararray);
> A1 = FOREACH A1 generate title,name,id;
> B = LOAD 'test2.txt' AS (id:chararray, content:chararray);
> C = JOIN A1 BY id LEFT OUTER, B BY id;
> dump C;
> The job succeed, and here is the result of execution.
> ========
> (,A1111,A,A,Helloworld)
> (,B222,B,B,Pig)
> (,C333,C,C,Hive)
> (,,,,)
> ========
--
This message was sent by Atlassian JIRA
(v6.2#6252)
