Hi Guys,
When I am running my Pig Script, I am doing OK with small data sets. But if I
use the same script for big data sets, I got this Exception:
ERROR 0: Exception while executing (Name: Kff: Filter[bag] - scope-142 Operator
Key: scope-142): org.apache.pig.backend.executionengine.ExecException: ERROR 0:
Exception while executing [PORelationToExprProject (Name:
RelationToExpressionProject[bag][*] - scope-134 Operator Key: scope-134)
children: null at []]: java.lang.ClassCastException:
org.apache.pig.data.BinSedesTuple cannot be cast to
org.apache.pig.pen.util.ExampleTuple
And I am using the following script:
F = LOAD './q' AS (ori_query:chararray, t:chararray, w:chararray,
wordCnts:long);
G = JOIN F by t, E by word;
H = FOREACH G GENERATE F::ori_query as ori_query, F::t as word, F::w as weight,
F::wordCnts as wordCnts, E::proID as proID, E::proScore as proScore;
I = FOREACH H GENERATE ori_query, proID, wordCnts,
que_udfs.getQueryScore(weight, proScore) as queryScore;
J = COGROUP I BY (ori_query, proID);
K = FOREACH J GENERATE group.ori_query as ori_query, group.proID as proID,
I.wordCnts as wordCnts, I.queryScore as queryScore, SIZE(I.queryScore) as
matchCnts;
K_fil = FILTER K BY (ori_query is not NULL) AND (proID is not NULL);
Kgetter = FOREACH K_fil {
count = limit wordCnts 1;
GENERATE ori_query, proID, flatten(count) as count, queryScore,
matchCnts;
}
Kff = FILTER Kgetter BY (count is not NULL) and (matchCnts is not NULL) and
(count == matchCnts) and (queryScore is not NULL);
E has this data format:
8801 102588050 0.555555555556
ZERO 102588050 0.0
s 102588050 0.555555555556
zero 102588050 0.555555555556
US 102588050 0.0376254180602
hello 102588050 0.0376254180602
people 102588050 0.0376254180602
any 102588050 0.0376254180602
F has this data format:
8801 8801s 4.285714285714286 2
hello Media hello 5.714285714285715 2
ZERO mate s ZERO 7.127210931539155 3
ZERO mate s mate 0.6432799241636031 3
ZERO mate s s 2.229509144297241 3
US US 10.0 1
any people any 0.1620255817609147 2
any people people 9.837974418239085 2
Does anyone has any idea why this Exception is happening?
Many thanks in advance!
Best Regards,
Chloe
