pig-0.8.1: some unit test case failed with open source JDK
----------------------------------------------------------
Key: PIG-2281
URL: https://issues.apache.org/jira/browse/PIG-2281
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.1
Environment: open source jdk
ant: 1.8.2
Reporter: fang fang chen
Fix For: 0.8.1
Based on my analysis, I think this is because HashMap does not guarantee
ordering.
1.
[junit] Running org.apache.pig.test.TestDataModel
[junit] Tests run: 22, Failures: 1, Errors: 0, Time elapsed: 0.382 sec
Caused by different output of HashMap.toString() with open source JDK and sun
jdk. Based on the discussion with Thejas Nair(pig committer) the output in open
source jdk is also correct.
Detail:
Testcase: testTupleToString took 0.002 sec
FAILED
toString expected:<...ad a little
lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a
little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
junit.framework.ComparisonFailure: toString expected:<...ad a little
lamb)},[[hello#world,goodbye#all]],42,5000000000,3.14...> but was:<...ad a
little lamb)},[[goodbye#all,hello#world]],42,5000000000,3.14...>
at org.apache.pig.test.TestDataModel.testTupleToString(TestDataModel.java:269)
public void testTupleToString() throws Exception {
Tuple t = giveMeOneOfEach();
assertEquals("toString", "((3,3.0),{(4),(mary had a little
lamb)},[hello#world,goodbye#all],42,5000000000,3.1415927,2.99792458E8,true,hello,goodbye,)",
t.toString());//line 269
}
comment:
private Tuple giveMeOneOfEach() throws Exception {
TupleFactory tf = TupleFactory.getInstance();
Tuple t1 = tf.newTuple(11);
Tuple t2 = tf.newTuple(2);
t2.set(0, new Integer(3));
t2.set(1, new Float(3.0));
DataBag bag = BagFactory.getInstance().newDefaultBag();
bag.add(tf.newTuple(new Integer(4)));
bag.add(tf.newTuple(new String("mary had a little lamb")));
Map<String, Object> map = new HashMap<String, Object>(2);
map.put(new String("hello"), new String("world"));
map.put(new String("goodbye"), new String("all"));
t1.set(0, t2);
t1.set(1, bag);
t1.set(2, map);//when run t.toString(), HashMap.toString() will be
invoked
t1.set(3, new Integer(42));
t1.set(4, new Long(5000000000L));
t1.set(5, new Float(3.141592654));
t1.set(6, new Double(2.99792458e8));
t1.set(7, new Boolean(true));
t1.set(8, new DataByteArray("hello"));
t1.set(9, new String("goodbye"));
return t1;
}
}
2.
[junit] Running org.apache.pig.test.TestLogToPhyCompiler
[junit] Tests run: 23, Failures: 1, Errors: 0, Time elapsed: 1.16 sec
Maybe caused by different output of HashMap.keySet() with open source JDK and
sun jdk.
Detail:
Failure information:
Testcase: testSplit took 0.226 sec
FAILED
Plan not match
junit.framework.AssertionFailedError: Plan not match
at
org.apache.pig.test.TestLogToPhyCompiler.testSplit(TestLogToPhyCompiler.java:444)
public void testSplit() throws VisitorException, IOException {
String query = "split (load 'a') into x if $0 < '7', y if $0 > '7';";
LogicalPlan plan = buildPlan(query);
log.info("ff test plan:"+plan);
PhysicalPlan pp = buildPhysicalPlan(plan);
log.info("ff test pp:"+pp);
int MAX_SIZE = 100000;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
pp.explain(baos);
baos.write((int)'\n');
String compiledPlan = baos.toString();
compiledPlan = compiledPlan.replaceAll("Load(.*)","Load()");
if(generate){
FileOutputStream fos = new
FileOutputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
fos.write(baos.toByteArray());
return;
}
FileInputStream fis1 = new
FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split1.gld");
FileInputStream fis2 = new
FileInputStream("test/org/apache/pig/test/data/GoldenFiles/Split2.gld");
byte[] b1 = new byte[MAX_SIZE];
byte[] b2 = new byte[MAX_SIZE];
int len = fis1.read(b1);
int test = fis2.read(b2);
//System.out.println("Length of first plan = " + len + " of second = "
+ test);
String goldenPlan1 = new String(b1, 0, len);
String goldenPlan2 = new String(b2, 0, len);
goldenPlan1 = goldenPlan1.replaceAll("Load(.*)","Load()");
goldenPlan2 = goldenPlan2.replaceAll("Load(.*)","Load()");
System.out.println();
System.out.println(compiledPlan);
System.out.println("-------------");
if(compiledPlan.compareTo(goldenPlan1) == 0 ||
compiledPlan.compareTo(goldenPlan2) == 0) {
// good
}
else {
System.out.println("Expected plan1=") ;
System.out.println(goldenPlan1) ;
System.out.println("Expected plan2=") ;
System.out.println(goldenPlan1) ;
System.out.println("Actual plan=") ;
System.out.println(compiledPlan) ;
System.out.println("**END**") ;
fail("Plan not match") ;//line 444
}
}
comment:
variable compiledPlan initialize invoke the following methods:
pp.explain(baos);// explain(OutputStream out) method
baos.write((int)'\n');
String compiledPlan = baos.toString();
explain(OutputStream out)explain(OutputStream out, boolean
verbose)print(OutputStream printer)depthFirstPP()getLeaves()
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet
method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
baos variable output:
> open source jdk:
> x: Filter[tuple] - Test-Plan-Builder-240
> | |
> | Less Than[boolean] - Test-Plan-Builder-243
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-241
> | |
> | |---Constant(7) - Test-Plan-Builder-242
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
>
> y: Filter[tuple] - Test-Plan-Builder-244
> | |
> | Greater Than[boolean] - Test-Plan-Builder-247
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-245
> | |
> | |---Constant(7) - Test-Plan-Builder-246
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
> sun jdk:
> y: Filter[tuple] - Test-Plan-Builder-240
> | |
> | Greater Than[boolean] - Test-Plan-Builder-243
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-241
> | |
> | |---Constant(7) - Test-Plan-Builder-242
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
>
> x: Filter[tuple] - Test-Plan-Builder-244
> | |
> | Less Than[boolean] - Test-Plan-Builder-247
> | |
> | |---Project[bytearray][0] - Test-Plan-Builder-245
> | |
> | |---Constant(7) - Test-Plan-Builder-246
> |
> |---Split - Test-Plan-Builder-239
> |
> |---229: Load()
3.
[junit] Running org.apache.pig.test.TestMRCompiler
[junit] Tests run: 25, Failures: 1, Errors: 0, Time elapsed: 0.729 sec
Maybe, caused by different output of HashMap.keySet() with OPEN SOURCE JDK and
sun jdk.
Detail:
Testcase: testSortUDF1 took 0.02 sec
FAILED
null expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) -
-18:
| ...> but
was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
| ...>
junit.framework.ComparisonFailure: null
expected:<...---MapReduce(20,SUM,[COUNT,TestMRCompiler$WeirdComparator]) - -18:
| ...> but
was:<...---MapReduce(20,SUM,[TestMRCompiler$WeirdComparator,COUNT]) - -18:
| ...>
at org.apache.pig.test.TestMRCompiler.run(TestMRCompiler.java:1056)
at
org.apache.pig.test.TestMRCompiler.testSortUDF1(TestMRCompiler.java:790)
private void run(PhysicalPlan pp, String expectedFile) throws Exception {
String compiledPlan, goldenPlan = null;
int MAX_SIZE = 100000;
MRCompiler comp = new MRCompiler(pp, pc);
comp.compile();
MROperPlan mrp = comp.getMRPlan();
PlanPrinter ppp = new PlanPrinter(mrp);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ppp.print(baos);//see "comments"
compiledPlan = baos.toString();//compiledPlan's initialize is based on
baos
if(generate ){
FileOutputStream fos = new FileOutputStream(expectedFile);
fos.write(baos.toByteArray());
return;
}
FileInputStream fis = new FileInputStream(expectedFile);
byte[] b = new byte[MAX_SIZE];
int len = fis.read(b);
goldenPlan = new String(b, 0, len);
if (goldenPlan.charAt(len-1) == '\n')
goldenPlan = goldenPlan.substring(0, len-1);
pp.explain(System.out);
System.out.println();
System.out.println("<<<" + compiledPlan + ">>>");
System.out.println("-------------");
System.out.println("Golden");
System.out.println("<<<" + goldenPlan + ">>>");
System.out.println("-------------");
assertEquals(goldenPlan, compiledPlan);//line 1056
}
comment:
ppp.print(baos) invokes method as following:
print(OutputStream printer) depthFirstPP()getLeaves()
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and
keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
4.
[junit] Running org.apache.pig.test.TestMergeJoinOuter
[junit] Tests run: 5, Failures: 1, Errors: 0, Time elapsed: 132.66 sec
Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun
jdk.
Testcase: testCompilation took 0.443 sec
FAILED
junit.framework.AssertionFailedError:
at
org.apache.pig.test.TestMergeJoinOuter.testCompilation(TestMergeJoinOuter.java:116)
Iterator<MapReduceOper> itr = mrPlan.iterator();// see comments
MapReduceOper oper = itr.next();
assertTrue(oper.reducePlan.isEmpty());//line 116
comments:
iterator() method:
public Iterator<E> iterator() {
return mOps.keySet().iterator();//mOps is HashMap structure, and keySet
method output different with OPEN SOURCE JDK compared with SUN JDK.
}
with same mrPlan:
MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) -
scope-39:
Reduce Plan Empty
| C:
Store(hdfs://localhost.localdomain:34390/user/root/out:org.apache.pig.builtin.PigStorage)
- scope-38
| |
| |---C: New For Each(true,true)[tuple] - scope-37
| | |
| | Project[bag][1] - scope-31
| | |
| | POBinCond[bag] - scope-36
| | |
| | |---Project[bag][2] - scope-32
| | |
| | |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - scope-34
| | | |
| | | |---Project[bag][2] - scope-33
| | |
| | |---Constant({(,,)}) - scope-35
| |
| |---C: MergeCogroup[tuple] - scope-30
| |
| |---A:
Load(hdfs://localhost.localdomain:34390/user/root/data1:org.apache.pig.test.TestMapSideCogroup$DummyCollectableLoader)
- scope-22
|
|---MapReduce(1) - scope-41:
|
Store(hdfs://localhost.localdomain:34390/tmp/temp-1456742965/tmp2077335416:org.apache.pig.impl.io.InterStorage)
- scope-48
| |
| |---New For Each(true)[bag] - scope-47
| | |
| | Project[tuple][1] - scope-46
| |
| |---Package[tuple]{tuple} - scope-45
| Local Rearrange[tuple]{tuple}(false) - scope-44
| | |
| | Project[tuple][*] - scope-43
| |
variable itr is in different order:
open source jdk:
(Name: MapReduce(1) - scope-41(itr.next())
(Name:
MapReduce(-1,IsEmpty,TestMapSideCogroup$DummyCollectableLoader,PigStorage) -
scope-39:
sun jdk:
(Name:
MapReduce(-1,TestMapSideCogroup$DummyCollectableLoader,IsEmpty,PigStorage) -
scope-39: (itr.next())
(Name: MapReduce(1) - scope-41:
5.
[junit] Running org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor
[junit] Tests run: 25, Failures: 3, Errors: 0, Time elapsed: 1.081 sec
Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun
jdk.
(1).
Testcase: testSimplePlan took 0.295 sec
FAILED
expected:<class
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
but was:<class
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
junit.framework.AssertionFailedError: expected:<class
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
but was:<class
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
at
org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testSimplePlan(TestNewPlanLogToPhyTranslationVisitor.java:131)
public void testSimplePlan() throws Exception {
LogicalPlanTester lpt = new LogicalPlanTester(pc);
lpt.buildPlan("a = load 'd.txt';");
lpt.buildPlan("b = filter a by $0==NULL;");
LogicalPlan plan = lpt.buildPlan("store b into 'empty';");
org.apache.pig.newplan.logical.relational.LogicalPlan newLogicalPlan =
migratePlan(plan);
PhysicalPlan phyPlan = translatePlan(newLogicalPlan);
assertEquals( 3, phyPlan.size() );
assertEquals( 1, phyPlan.getRoots().size() );
assertEquals( 1, phyPlan.getLeaves().size() );
PhysicalOperator load = phyPlan.getRoots().get(0);
assertEquals( POLoad.class, load.getClass() );
assertTrue( ((POLoad)load).getLFile().getFileName().contains("d.txt")
);
// Check for Filter
PhysicalOperator fil = phyPlan.getSuccessors(load).get(0);
assertEquals( POFilter.class, fil.getClass() );
PhysicalPlan filPlan = ((POFilter)fil).getPlan();
assertEquals( 2, filPlan.getRoots().size() );
assertEquals( 1, filPlan.getLeaves().size() );
PhysicalOperator eq = filPlan.getLeaves().get(0);
assertEquals( EqualToExpr.class, eq.getClass() );
PhysicalOperator prj1 = filPlan.getRoots().get(0);
assertEquals( POProject.class, prj1.getClass() );//line 131
assertEquals( 0, ((POProject)prj1).getColumn() );
PhysicalOperator constExp = filPlan.getRoots().get(1);
assertEquals( ConstantExpression.class, constExp.getClass() );
assertEquals( null, ((ConstantExpression)constExp).getValue() );
// Check for Store
PhysicalOperator stor = phyPlan.getSuccessors(fil).get(0);
assertEquals( POStore.class, stor.getClass() );
assertTrue(
((POStore)stor).getSFile().getFileName().contains("empty"));
}
comment:
public List<E> getRoots() {
if (mRoots.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet
method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mToEdges.get(op) == null) {
mRoots.add(op);
}
}
}
return mRoots;
(2).
Testcase: testJoinPlan took 0.062 sec
FAILED
expected:<class
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
but was:<class
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
junit.framework.AssertionFailedError: expected:<class
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ConstantExpression>
but was:<class
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject>
at
org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testJoinPlan(TestNewPlanLogToPhyTranslationVisitor.java:201)
public void testJoinPlan() throws Exception {
LogicalPlanTester lpt = new LogicalPlanTester(pc);
lpt.buildPlan("a = load 'd1.txt' as (id, c);");
lpt.buildPlan("b = load 'd2.txt'as (id, c);");
lpt.buildPlan("c = join a by id, b by c;");
lpt.buildPlan("d = filter c by a::id==NULL AND b::c==NULL;");
LogicalPlan plan = lpt.buildPlan("store d into 'empty';");
// check basics
org.apache.pig.newplan.logical.relational.LogicalPlan newPlan =
migratePlan(plan);
PhysicalPlan physicalPlan = translatePlan(newPlan);
assertEquals(9, physicalPlan.size());
assertEquals(physicalPlan.getRoots().size(), 2);
// Check Load and LocalRearrange and GlobalRearrange
PhysicalOperator LoR =
(PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(0)).get(0);
assertEquals( POLocalRearrange.class, LoR.getClass() );
POLocalRearrange Lor = (POLocalRearrange) LoR;
PhysicalOperator prj3 = Lor.getPlans().get(0).getLeaves().get(0);
assertEquals( POProject.class, prj3.getClass() );
assertEquals(0, ((POProject)prj3).getColumn() );
PhysicalOperator inp1 = Lor.getInputs().get(0);
assertEquals( POLoad.class, inp1.getClass() );
assertTrue( ((POLoad)inp1).getLFile().getFileName().contains("d1.txt")
);
PhysicalOperator LoR1 =
(PhysicalOperator)physicalPlan.getSuccessors(physicalPlan.getRoots().get(1)).get(0);
assertEquals( POLocalRearrange.class, LoR1.getClass() );
POLocalRearrange Lor1 = (POLocalRearrange) LoR1;
PhysicalOperator prj4 = Lor1.getPlans().get(0).getLeaves().get(0);
assertEquals( POProject.class, prj4.getClass() );
assertEquals(1, ((POProject)prj4).getColumn() );
PhysicalOperator inp2 = Lor1.getInputs().get(0);
assertEquals( POLoad.class, inp2.getClass() );
assertTrue( ((POLoad)inp2).getLFile().getFileName().contains("d2.txt")
);
PhysicalOperator GoR =
(PhysicalOperator)physicalPlan.getSuccessors(LoR).get(0);
assertEquals( POGlobalRearrange.class, GoR.getClass() );
PhysicalOperator Pack =
(PhysicalOperator)physicalPlan.getSuccessors(GoR).get(0);
assertEquals( POPackage.class, Pack.getClass() );
// Check for ForEach
PhysicalOperator ForE =
(PhysicalOperator)physicalPlan.getSuccessors(Pack).get(0);
assertEquals( POForEach.class, ForE.getClass() );
PhysicalOperator prj5 =
((POForEach)ForE).getInputPlans().get(0).getLeaves().get(0);
assertEquals( POProject.class, prj5.getClass() );
assertEquals( 1, ((POProject)prj5).getColumn() );
PhysicalOperator prj6 =
((POForEach)ForE).getInputPlans().get(1).getLeaves().get(0);
assertEquals( POProject.class, prj6.getClass() );
assertEquals( 2, ((POProject)prj6).getColumn() );
// Filter Operator
PhysicalOperator fil =
(PhysicalOperator)physicalPlan.getSuccessors(ForE).get(0);
assertEquals( POFilter.class, fil.getClass() );
PhysicalPlan filPlan = ((POFilter)fil).getPlan();
List<PhysicalOperator> filRoots = filPlan.getRoots();
assertEquals( ConstantExpression.class, filRoots.get(1).getClass()
);//line 201
ConstantExpression ce1 = (ConstantExpression) filRoots.get(1);
assertEquals( null, ce1.getValue() );
assertEquals( ConstantExpression.class, filRoots.get(3).getClass() );
ConstantExpression ce2 = (ConstantExpression) filRoots.get(3);
assertEquals( null, ce2.getValue() );
assertEquals( POProject.class, filRoots.get(0).getClass() );
POProject prj1 = (POProject) filRoots.get(0);
assertEquals( 3, prj1.getColumn() );
assertEquals( POProject.class, filRoots.get(2).getClass() );
POProject prj2 = (POProject) filRoots.get(2);
assertEquals( 0, prj2.getColumn() );
// Check Store Operator
PhysicalOperator stor =
(PhysicalOperator)physicalPlan.getSuccessors(fil).get(0);
assertEquals( POStore.class, stor.getClass() );
assertTrue( ((POStore)stor).getSFile().getFileName().contains("empty")
);
}
comment:
public List<E> getRoots() {
if (mRoots.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { mOps is HashMap structure, and keySet
method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mToEdges.get(op) == null) {
mRoots.add(op);
}
}
}
return mRoots;
}
(3).
Testcase: testMultiStore took 0.083 sec
FAILED
expected:<1> but was:<0>
junit.framework.AssertionFailedError: expected:<1> but was:<0>
at
org.apache.pig.test.TestNewPlanLogToPhyTranslationVisitor.testMultiStore(TestNewPlanLogToPhyTranslationVisitor.java:255)
PhysicalOperator prj2 = Lor1.getPlans().get(0).getLeaves().get(0);
assertEquals(1, ((POProject)prj2).getColumn() );//line 255
comment:
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and
keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
6.
[junit] Running org.apache.pig.test.TestPruneColumn
[junit] Tests run: 67, Failures: 4, Errors: 0, Time elapsed: 528.047 sec
Maybe caused by different output of HashMap.keySet() with OPEN SOURCE JDK and
sun jdk.
(1).
Testcase: testMapKey2 took 6.291 sec
FAILED
null
junit.framework.AssertionFailedError: null
at
org.apache.pig.test.TestPruneColumn.testMapKey2(TestPruneColumn.java:1213)
public void testMapKey2() throws Exception{
pigServer.registerQuery("A = load '"+
Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as
(a0:int, a1:map[]);");
pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");// see
comment3
pigServer.registerQuery("C = foreach B generate $0#'key2', $1;");
Iterator<Tuple> iter = pigServer.openIterator("C");//see comment1
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("2"));
assertTrue(t.get(1).toString().equals("1"));
assertTrue(iter.hasNext());
t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("4"));
assertTrue(t.get(1).toString().equals("2"));
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0",
"Map key required for A: $1->[key2, key1]"}));// line 1213 see
comment2
}
comments1:
pigServer.openIterator("C") invokes the following method to save pigServer
information to filename:
store(String id, String filename, String func)
comment2:
public boolean checkLogFileMessage(String[] messages)
{
BufferedReader reader = null;
try {
reader = new BufferedReader(new
FileReader(logFile));//logFile=filename
List<String> logMessages=new ArrayList<String>();
String line;
while ((line=reader.readLine())!=null)
{
logMessages.add(line);
}
// Check if all messages appear in the log
for (int i=0;i<messages.length;i++)
{
boolean found = false;
for (int j=0;j<logMessages.size();j++)
if (logMessages.get(j).contains(messages[i])) {
found = true;
break;
}
if (!found)
return false;
}
// Check no other log besides messages
for (int i=0;i<logMessages.size();i++) {
boolean found = false;
for (int j=0;j<messages.length;j++) {
if (logMessages.get(i).contains(messages[j])) {
found = true;
break;
}
}
if (!found) {
if (logMessages.get(i).contains("Columns pruned for")||
logMessages.get(i).contains("Map key required
for")) {
return false;
}
}
}
return true;
}
catch (IOException e) {
return false;
}
}
comment3:
the content in filename is different, because pigServer(invoke
HashMap.keySet()) is different.
pigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and
keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
(2)
Testcase: testMapKey3 took 6.319 sec
FAILED
null
junit.framework.AssertionFailedError: null
at
org.apache.pig.test.TestPruneColumn.testMapKey3(TestPruneColumn.java:1229)
public void testMapKey3() throws Exception {
pigServer.registerQuery("A = load '"+
Util.generateURI(tmpFile3.toString(), pigServer.getPigContext()) + "' as
(a0:int, a1:map[]);");
pigServer.registerQuery("B = foreach A generate a1, a1#'key1';");
pigServer.registerQuery("C = group B all;");
Iterator<Tuple> iter = pigServer.openIterator("C");
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("all"));
assertTrue(t.get(1).toString().equals("{([key2#2,key1#1],1),([key2#4,key1#2],2)}"));
//line 1229
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A:
$0"}));
}
comment:
variable "t" initialize process:
titerpigServer.registerQuery(...)getSingleLeafPlanOutputOp()getLeaves()
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and
keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
different t.get(1).toString():
OPEN SOURCE JDK:
{([key1#1,key2#2],1),([key1#2,key2#4],2)}
SUN JDK:
{([key2#2,key1#1],1),([key2#4,key1#2],2)}
(3).
Testcase: testMapKeyInSplit1 took 6.3 sec
FAILED
null
junit.framework.AssertionFailedError: null
at
org.apache.pig.test.TestPruneColumn.testMapKeyInSplit1(TestPruneColumn.java:1303)
public void testMapKeyInSplit1() throws Exception {
pigServer.registerQuery("A = load '"+
Util.generateURI(tmpFile12.toString(), pigServer.getPigContext()) + "' as
(m:map[]);");
pigServer.registerQuery("B = foreach A generate m#'key1' as key1;");
pigServer.registerQuery("C = foreach A generate m#'key2' as key2;");
pigServer.registerQuery("D = join B by key1, C by key2;");
Iterator<Tuple> iter = pigServer.openIterator("D");
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.size()==2);
assertTrue(t.get(0).toString().equals("2"));
assertTrue(t.get(1).toString().equals("2"));
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Map key required for A:
$0->[key2, key1]"}));//line 1303
}
comment: same with (1).
(4).
Testcase: testSharedSchemaObject took 6.327 sec
FAILED
null
junit.framework.AssertionFailedError: null
at
org.apache.pig.test.TestPruneColumn.testSharedSchemaObject(TestPruneColumn.java:1626)
public void testSharedSchemaObject() throws Exception {
pigServer.registerQuery("A = load '"+
Util.generateURI(tmpFile10.toString(), pigServer.getPigContext()) + "' AS (a0,
a1:map[], a2);");
pigServer.registerQuery("B = foreach A generate a1;");
pigServer.registerQuery("C = limit B 10;");
Iterator<Tuple> iter = pigServer.openIterator("C");
assertTrue(iter.hasNext());
Tuple t = iter.next();
assertTrue(t.toString().equals("([2#1,1#1])"));
assertFalse(iter.hasNext());
assertTrue(checkLogFileMessage(new String[]{"Columns pruned for A: $0,
$2"}));// line 1626
}
comment: same with (2).
7.
[junit] Running org.apache.pig.test.TestUnionOnSchema
[junit] Tests run: 21, Failures: 1, Errors: 0, Time elapsed: 196.841 sec
Testcase: testUnionOnSchemaScopedColumnNameNeg took 0.008 sec
FAILED
Expected exception message matching 'Found more than one match: l1::i, l2::i'
but got 'Error during parsing. Found more than one match: l2::i, l1::i'
at
org.apache.pig.test.TestUnionOnSchema.checkSchemaEx(TestUnionOnSchema.java:604)
at
org.apache.pig.test.TestUnionOnSchema.testUnionOnSchemaScopedColumnNameNeg(TestUnionOnSchema.java:370)
8..
[junit] Running org.apache.pig.test.TestPushDownForeachFlatten
[junit] Tests run: 37, Failures: 0, Errors: 8, Time elapsed: 1.455 sec
Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun
jdk.
(1)
Testcase: testForeachUnion took 0.039 sec
Caused an ERROR
Expected LOForEach, got LOUnion
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected
LOForEach, got LOUnion
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at
org.apache.pig.test.TestPushDownForeachFlatten.testForeachUnion(TestPushDownForeachFlatten.java:275)
public void testForeachUnion() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age,
preference);");
LogicalPlan lp = planTester.buildPlan("D = union B, C;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);// see comment
assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 275
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
comment:
LOLoad load = (LOLoad) lp.getRoots().get(0);//
public List<E> getRoots() {
if (mRoots.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet
method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mToEdges.get(op) == null) {
mRoots.add(op);
}
}
}
return mRoots;
}
(2)
Testcase: testForeachCogroup took 0.038 sec
Caused an ERROR
Expected LOForEach, got LOCogroup
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected
LOForEach, got LOCogroup
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at
org.apache.pig.test.TestPushDownForeachFlatten.testForeachCogroup(TestPushDownForeachFlatten.java:295)
public void testForeachCogroup() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age, gpa);");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age,
preference);");
LogicalPlan lp = planTester.buildPlan("D = cogroup B by $0, C by $0;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);
assertTrue(!pushDownForeach.check(lp.getSuccessors(load)));//line 295
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
comment: same with (1)
(3)
Testcase: testForeachCross took 0.035 sec
Caused an ERROR
Expected LOForEach, got LOCross
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected
LOForEach, got LOCross
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at
org.apache.pig.test.TestPushDownForeachFlatten.testForeachCross(TestPushDownForeachFlatten.java:427)
public void testForeachCross() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age,
gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age,
preference);");
planTester.buildPlan("D = cross B, C;");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);
LOLimit limit = (LOLimit) lp.getLeaves().get(0);
LOCross cross = (LOCross)lp.getPredecessors(limit).get(0);
LOForEach foreach = (LOForEach) lp.getPredecessors(cross).get(0);
Schema limitSchema = limit.getSchema();
assertTrue(pushDownForeach.check(lp.getSuccessors(load)));
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == true);
assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
pushDownForeach.transform(lp.getSuccessors(load));//line 427
planTester.rebuildSchema(lp);
for(Boolean b: foreach.getFlatten()) {
assertEquals(b.booleanValue(), false);
}
LOForEach newForeach = (LOForEach)lp.getSuccessors(cross).get(0);
List<Boolean> newForeachFlatten = newForeach.getFlatten();
Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
for(Integer key: remap.keySet()) {
Integer value = remap.get(key);
assertEquals(newForeachFlatten.get(value).booleanValue(), true);
}
assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
}
comment: same with (1)
(4)
Testcase: testForeachFlattenAddedColumnCross took 0.034 sec
Caused an ERROR
Expected LOForEach, got LOCross
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected
LOForEach, got LOCross
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at
org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnCross(TestPushDownForeachFlatten.java:545)
public void testForeachFlattenAddedColumnCross() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age,
gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
planTester.buildPlan("C = load 'anotherfile' as (name, age,
preference:(course_name, instructor));");
planTester.buildPlan("D = cross B, C;");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad loada = (LOLoad) lp.getRoots().get(0);
assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 545
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
comment: same with (1)
(5)
Testcase: testForeachFRJoin took 0.027 sec
Caused an ERROR
Expected LOForEach, got LOJoin
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected
LOForEach, got LOJoin
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at
org.apache.pig.test.TestPushDownForeachFlatten.testForeachFRJoin(TestPushDownForeachFlatten.java:619)
public void testForeachFRJoin() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age,
gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age,
preference);");
planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);
LOLimit limit = (LOLimit) lp.getLeaves().get(0);
LOJoin frjoin = (LOJoin)lp.getPredecessors(limit).get(0);
LOForEach foreach = (LOForEach) lp.getPredecessors(frjoin).get(0);
Schema limitSchema = limit.getSchema();
assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 619
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == true);
assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
pushDownForeach.transform(lp.getSuccessors(load));
planTester.rebuildSchema(lp);
for(Boolean b: foreach.getFlatten()) {
assertEquals(b.booleanValue(), false);
}
LOForEach newForeach = (LOForEach)lp.getSuccessors(frjoin).get(0);
List<Boolean> newForeachFlatten = newForeach.getFlatten();
Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
for(Integer key: remap.keySet()) {
Integer value = remap.get(key);
assertEquals(newForeachFlatten.get(value).booleanValue(), true);
}
assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
}
comment: same with (1)
(6)
Testcase: testForeachFlattenAddedColumnFRJoin took 0.026 sec
Caused an ERROR
Expected LOForEach, got LOJoin
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected
LOForEach, got LOJoin
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at
org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnFRJoin(TestPushDownForeachFlatten.java:738)
public void testForeachFlattenAddedColumnFRJoin() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age,
gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
planTester.buildPlan("C = load 'anotherfile' as (name, age,
preference:(course_name, instructor));");
planTester.buildPlan("D = join B by $0, C by $0 using \"replicated\";");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad loada = (LOLoad) lp.getRoots().get(0);
assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 738
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
comment: same with (1)
(7)
Testcase: testForeachInnerJoin took 0.026 sec
Caused an ERROR
Expected LOForEach, got LOJoin
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected
LOForEach, got LOJoin
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at
org.apache.pig.test.TestPushDownForeachFlatten.testForeachInnerJoin(TestPushDownForeachFlatten.java:812)
public void testForeachInnerJoin() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age,
gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten($2);");
planTester.buildPlan("C = load 'anotherfile' as (name, age,
preference:(course_name, instructor));");
planTester.buildPlan("D = join B by $0, C by $0;");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad load = (LOLoad) lp.getRoots().get(0);
LOLimit limit = (LOLimit) lp.getLeaves().get(0);
LOJoin join = (LOJoin)lp.getPredecessors(limit).get(0);
LOForEach foreach = (LOForEach) lp.getPredecessors(join).get(0);
Schema limitSchema = limit.getSchema();
assertTrue(pushDownForeach.check(lp.getSuccessors(load)));//line 812
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == true);
assertTrue(pushDownForeach.getFlattenedColumnMap() != null);
pushDownForeach.transform(lp.getSuccessors(load));
planTester.rebuildSchema(lp);
for(Boolean b: foreach.getFlatten()) {
assertEquals(b.booleanValue(), false);
}
LOForEach newForeach = (LOForEach)lp.getSuccessors(join).get(0);
List<Boolean> newForeachFlatten = newForeach.getFlatten();
Map<Integer, Integer> remap = pushDownForeach.getFlattenedColumnMap();
for(Integer key: remap.keySet()) {
Integer value = remap.get(key);
assertEquals(newForeachFlatten.get(value).booleanValue(), true);
}
assertTrue(Schema.equals(limitSchema, limit.getSchema(), false, true));
}
comment: same with (1)
(8)
Testcase: testForeachFlattenAddedColumnInnerJoin took 0.021 sec
Caused an ERROR
Expected LOForEach, got LOJoin
org.apache.pig.impl.plan.optimizer.OptimizerException: ERROR 2005: Expected
LOForEach, got LOJoin
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.getOperator(PushDownForeachFlatten.java:338)
at
org.apache.pig.impl.logicalLayer.optimizer.PushDownForeachFlatten.check(PushDownForeachFlatten.java:101)
at
org.apache.pig.test.TestPushDownForeachFlatten.testForeachFlattenAddedColumnInnerJoin(TestPushDownForeachFlatten.java:931)
public void testForeachFlattenAddedColumnInnerJoin() throws Exception {
planTester.buildPlan("A = load 'myfile' as (name, age,
gpa:(letter_grade, point_score));");
planTester.buildPlan("B = foreach A generate $0, $1, flatten(1);");
planTester.buildPlan("C = load 'anotherfile' as (name, age,
preference:(course_name, instructor));");
planTester.buildPlan("D = join B by $0, C by $0;");
LogicalPlan lp = planTester.buildPlan("E = limit D 10;");
planTester.setPlan(lp);
planTester.setProjectionMap(lp);
planTester.rebuildSchema(lp);
PushDownForeachFlatten pushDownForeach = new PushDownForeachFlatten(lp);
LOLoad loada = (LOLoad) lp.getRoots().get(0);
assertTrue(!pushDownForeach.check(lp.getSuccessors(loada)));//line 931
assertTrue(pushDownForeach.getSwap() == false);
assertTrue(pushDownForeach.getInsertBetween() == false);
assertTrue(pushDownForeach.getFlattenedColumnMap() == null);
}
9.
[junit] Running org.apache.pig.test.TestTypeCheckingValidator
[junit] Tests run: 120, Failures: 0, Errors: 1, Time elapsed: 15.047 sec
Caused by different output of HashMap.keySet() with OPEN SOURCE JDK and sun
jdk. Based on the discussion with Thejas Nair(pig committer) the output in OPEN
SOURCE jdk is also correct.
Detail:
Testcase: testMapLookupLineage took 0.012 sec
Caused an ERROR
org.apache.pig.impl.logicalLayer.LOAdd incompatible with
org.apache.pig.impl.logicalLayer.LOCast
java.lang.ClassCastException: org.apache.pig.impl.logicalLayer.LOAdd
incompatible with org.apache.pig.impl.logicalLayer.LOCast
at
org.apache.pig.test.TestTypeCheckingValidator.testMapLookupLineage(TestTypeCheckingValidator.java:5397)
public void testMapLookupLineage() throws Throwable {
planTester.buildPlan("a = load 'a' using BinStorage() as (field1,
field2: float, field3: chararray );") ;
planTester.buildPlan("b = foreach a generate field1#'key1' as map1;") ;
LogicalPlan plan = planTester.buildPlan("c = foreach b generate
map1#'key2' + 1 ;") ;
// validate
CompilationMessageCollector collector = new
CompilationMessageCollector() ;
TypeCheckingValidator typeValidator = new TypeCheckingValidator() ;
typeValidator.validate(plan, collector) ;
printMessageCollector(collector) ;
printTypeGraph(plan) ;
planTester.printPlan(plan, TypeCheckingTestUtil.getCurrentMethodName());
if (collector.hasError()) {
throw new AssertionError("Expect no error") ;
}
LOForEach foreach = (LOForEach)plan.getLeaves().get(0);// see comement1
LogicalPlan foreachPlan = foreach.getForEachPlans().get(0);
LogicalOperator exOp = foreachPlan.getRoots().get(0);//see comment2
// the root would be the project and there would be cast
// to map between the project and LOMapLookup
LOCast cast1 = (LOCast)foreachPlan.getSuccessors(exOp).get(0);//line
5397
assertTrue(cast1.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
LOMapLookup map = (LOMapLookup)foreachPlan.getSuccessors(cast1).get(0);
LOCast cast = (LOCast)foreachPlan.getSuccessors(map).get(0);
assertTrue(cast.getLoadFuncSpec().getClassName().startsWith("BinStorage"));
}
comment1:
foreachPlan initialize process, this will cause the different foreachPlan.
foreachPlanforeachplan.getLeaves
public List<E> getLeaves() {
if (mLeaves.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) { //mOps is HashMap structure, and
keySet method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mFromEdges.get(op) == null) {
mLeaves.add(op);
}
}
}
return mLeaves;
}
comment2:
exOpforeachPlan.getRoots(), this is the root cause
public List<E> getRoots() {
if (mRoots.size() == 0 && mOps.size() > 0) {
for (E op : mOps.keySet()) {//mOps is HashMap structure, and keySet
method output different with OPEN SOURCE JDK compared with SUN JDK.
if (mToEdges.get(op) == null) {
mRoots.add(op);
}
}
}
return mRoots;
}
SUN JDK output:
foreachPlan:
Add scope-12 FieldSchema: int Type: int
|
|---Const scope-15( 1 ) FieldSchema: int Type: int
|
|---Cast scope-19 FieldSchema: int Type: int
|
|---MapLookup scope-14 FieldSchema: bytearray Type: bytearray
|
|---Cast scope-18 FieldSchema: map Type: map
|
|---Project scope-13 Projections: [0] Overloaded: false
FieldSchema: map1: bytearray Type: bytearray
Input: b: ForEach scope-6
getRoots method process:
keySet() method:
Cast scope-19
Cast scope-18
Add scope-12
Project scope-13 Projections: [0] Overloaded: false(add to mRoots)
MapLookup scope-14
Const scope-15( 1 )(add to mRoots)
foreachPlan.getRoots():(Name: Project scope-13 Projections: [0] Overloaded:
false Operator Key: scope-13)
(Name: Const scope-15( 1 ) Operator
Key: scope-15)
exOp:(Name: Project scope-13 Projections: [0] Overloaded: false Operator Key:
scope-13)
OPEN SOURCE JDK output:
foreachPlan:
Add scope-13 FieldSchema: int Type: int
|
|---Const scope-12( 1 ) FieldSchema: int Type: int
|
|---Cast scope-19 FieldSchema: int Type: int
|
|---MapLookup scope-15 FieldSchema: bytearray Type: bytearray
|
|---Cast scope-18 FieldSchema: map Type: map
|
|---Project scope-14 Projections: [0] Overloaded: false
FieldSchema: map1: bytearray Type: bytearray
Input: b: ForEach scope-6
getRoots method process:
keySet() method:
Cast scope-18
Cast scope-19
Const scope-12( 1 )(add to mRoots)
Add scope-13
Project scope-14 Projections: [0] Overloaded: false(add to mRoots)
MapLookup scope-15
foreachPlan.getRoots():(Name: Const scope-12( 1 ) Operator Key: scope-12)
(Name: Project scope-14 Projections:
[0] Overloaded: false Operator Key: scope-14)//output in different order
compared with SUN JDK output
exOp:(Name: Const scope-12( 1 ) Operator Key: scope-12)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
