Pig 0.9 new logical plan throws class cast exception
----------------------------------------------------
Key: PIG-1835
URL: https://issues.apache.org/jira/browse/PIG-1835
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Vivek Padmanabhan
I have the below script which is throwing class cast exception while doing SUM.
Even though all the fields are properly typed, while computing sum in m_agg0
and m_agg02 the record from tuple is coming as java.lang.Long instead of Double.
The problem is happening in Pig 0.9. It works fine with 0.9 if I flag off new
logical plan by -Dpig.usenewlogicalplan=false.
{code}
A0 = load 'inputA' using PigStorage('\t') as ( group_id, r_id:long,
is_phase2:int, roi_value:double,roi_cost:double,ecpm, prob:double,pixel_id,
pixel_type,
val:long,f3, f4,type:long, amount:double,item_id:long);
A0 = foreach A0 generate r_id, is_phase2, ((val==257 or val==258)? 1: 0) as
imps,
((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or
val==258)? item_id: 0) as a_item_id,
((val==257 or val==258)? roi_value: 0.0) as roi_value,((val==257 or
val==258)? roi_cost: 0.0) as roi_cost,
((val==257 or val==513)? ecpm: 0.0) as ecpm, ((val==257 or val==513)?
prob: 0.0) as prob,
((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or
val==513)? item_id: 0) as pub_line_id,((val==257 or val==513)? type: 0) as
pub_pt;
-------------------------------------------------------------------------------------------------------------------------------------------------
B0 = load 'inputB' using PigStorage('\t') as ( group_id:long, r_id:long,
roi_value:double,roi_cost:double,receive_time,
host_name,site_id,rm_has_cookies,rm_pearl_id,
f1,f2,pixel_id:long,pixel_type:int, xcookie,val:long,f3,
f4,type:long,amount:double,item_id:long);
B0 = foreach B0 generate r_id, ((val==257 or val==258)? 1: 0) as B0,((val==257
or val==258)? amount: 0.0) as a_out,
((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or
val==258)? roi_value: 0.0) as roi_value,
((val==257 or val==258)? roi_cost: 0.0) as roi_cost, ((val==257 or
val==513)? amount: 0.0) as pub_rev,
((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or
val==513)? type: 0) as pub_pt;
------------------------------------------------------------------------------------------------------------------------------------------------
C0 = load 'inputC' using PigStorage('\t') as ( group_id:long, r_id:long,
roi_value:double, roi_cost:double, receive_time:long, host_name:chararray,
site_id:long, rm_has_cookies:int,rm_pearl_id:long,f1,f2, pixel_id:long,
pixel_type:int,rm_is_post_click:int,
rm_conversion_id,xcookie:chararray,val:long,f3:long,f4:long,type:long,amount:double,item_id:long);
C0 = foreach C0 generate r_id,((val==257 or val==258)? 1: 0) as C0,
((val==257 or val==258)? amount: 0.0) as a_out,
((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or
val==513)? amount: 0.0) as pub_rev,
((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or
val==513)? type: 0) as pub_pt;
------------------------------------------------------------------------------------------------------------------------------------------------
m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ;
m_agg01 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev))
as conv_pub_rev;
store m_agg01 into 'out1' USING PigStorage(',');
m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ;
m_agg02 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev))
as conv_pub_rev;
store m_agg02 into 'out2' USING PigStorage(',');
{code}
The below are the inputs to the script (all single record and tab seperated)
inputA
------
1111 1111 1 1.1 1.1 1.1 1.1 1111 1 1111
1111 1111 1111 1.1 1111
inputB
------
1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1
1 1111 1 c1 1111 1111 1111 1111 1.1 1111
inputC
------
1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1
1 1111 1 1 1111 c1 1111 1111 1111 1111
1.1 1111
Exception from reducers
_______________________
org.apache.pig.backend.executionengine.ExecException: ERROR 2103: Problem while
computing sum of doubles.
at org.apache.pig.builtin.DoubleSum.sum(DoubleSum.java:147)
at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:46)
at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:41)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:230)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:302)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.getNext(POBinCond.java:140)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:446)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:346)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:289)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:455)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:423)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:403)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:258)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414)
at org.apache.hadoop.mapred.Child$4.run(Child.java:242)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062)
at org.apache.hadoop.mapred.Child.main(Child.java:236)
Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to
java.lang.Double
at org.apache.pig.builtin.DoubleSum.sum(DoubleSum.java:140)
... 20 more
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
