Pig 0.9 new logical plan throws class cast exception
----------------------------------------------------

                 Key: PIG-1835
                 URL: https://issues.apache.org/jira/browse/PIG-1835
             Project: Pig
          Issue Type: Bug
    Affects Versions: 0.9.0
            Reporter: Vivek Padmanabhan


I have the below script which is throwing class cast exception while doing SUM. 
Even though all the fields are properly typed, while computing sum in m_agg0 
and m_agg02 the record from tuple is coming as java.lang.Long instead of Double.

The problem is happening in Pig 0.9. It works fine with 0.9 if I flag off new 
logical plan by -Dpig.usenewlogicalplan=false. 

{code}
A0 = load 'inputA' using PigStorage('\t') as ( group_id, r_id:long, 
is_phase2:int, roi_value:double,roi_cost:double,ecpm, prob:double,pixel_id, 
pixel_type,
val:long,f3, f4,type:long, amount:double,item_id:long);

A0 = foreach A0 generate r_id, is_phase2, ((val==257 or val==258)? 1: 0) as 
imps,
        ((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or 
val==258)? item_id: 0) as a_item_id,
        ((val==257 or val==258)? roi_value: 0.0) as roi_value,((val==257 or 
val==258)? roi_cost: 0.0) as roi_cost,
        ((val==257 or val==513)? ecpm: 0.0) as ecpm, ((val==257 or val==513)? 
prob: 0.0) as prob,
        ((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or 
val==513)? item_id: 0) as pub_line_id,((val==257 or val==513)? type: 0) as 
pub_pt;
-------------------------------------------------------------------------------------------------------------------------------------------------
B0 = load 'inputB' using PigStorage('\t') as ( group_id:long, r_id:long, 
roi_value:double,roi_cost:double,receive_time, 
host_name,site_id,rm_has_cookies,rm_pearl_id, 
f1,f2,pixel_id:long,pixel_type:int, xcookie,val:long,f3, 
f4,type:long,amount:double,item_id:long);

B0 = foreach B0 generate r_id, ((val==257 or val==258)? 1: 0) as B0,((val==257 
or val==258)? amount: 0.0) as a_out,
        ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or 
val==258)? roi_value: 0.0) as roi_value,
        ((val==257 or val==258)? roi_cost: 0.0) as roi_cost, ((val==257 or 
val==513)? amount: 0.0) as pub_rev,
        ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or 
val==513)? type: 0) as pub_pt;
------------------------------------------------------------------------------------------------------------------------------------------------
C0 = load 'inputC' using PigStorage('\t') as (  group_id:long, r_id:long, 
roi_value:double, roi_cost:double, receive_time:long, host_name:chararray, 
site_id:long, rm_has_cookies:int,rm_pearl_id:long,f1,f2, pixel_id:long, 
pixel_type:int,rm_is_post_click:int, 
rm_conversion_id,xcookie:chararray,val:long,f3:long,f4:long,type:long,amount:double,item_id:long);

C0 = foreach C0 generate   r_id,((val==257 or val==258)? 1: 0) as C0, 
((val==257 or val==258)? amount: 0.0) as a_out,
        ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or 
val==513)? amount: 0.0) as pub_rev,
        ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or 
val==513)? type: 0) as pub_pt;
------------------------------------------------------------------------------------------------------------------------------------------------
m_all = cogroup   A0 by (r_id) outer, B0 by (r_id) outer, C0  by (r_id) outer ;
m_agg01 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) 
as conv_pub_rev;
store m_agg01 into 'out1' USING PigStorage(',');

m_all = cogroup   A0 by (r_id) outer, B0 by (r_id) outer, C0  by (r_id) outer ;
m_agg02 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) 
as conv_pub_rev;
store m_agg02 into 'out2' USING PigStorage(',');
{code}



The below are the inputs to the script (all single record and tab seperated)


inputA
------
1111    1111    1       1.1     1.1     1.1     1.1     1111    1       1111    
1111    1111    1111    1.1     1111

inputB
------
1111    1111    1.1     1.1     1111    a1      1111    1       1111    b1      
1       1111    1       c1      1111    1111    1111    1111    1.1     1111

inputC
------
1111    1111    1.1     1.1     1111    a1      1111    1       1111    b1      
1       1111    1       1       1111    c1      1111    1111    1111    1111    
1.1     1111


Exception from reducers 
_______________________
org.apache.pig.backend.executionengine.ExecException: ERROR 2103: Problem while 
computing sum of doubles.
        at org.apache.pig.builtin.DoubleSum.sum(DoubleSum.java:147)
        at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:46)
        at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:41)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:230)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:302)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.getNext(POBinCond.java:140)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:446)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:346)
        at 
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:289)
        at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:455)
        at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:423)
        at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:403)
        at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:258)
        at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
        at 
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:242)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062)
        at org.apache.hadoop.mapred.Child.main(Child.java:236)
Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to 
java.lang.Double
        at org.apache.pig.builtin.DoubleSum.sum(DoubleSum.java:140)
        ... 20 more



-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Reply via email to