[
https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787716#action_12787716
]
Daniel Dai commented on PIG-1132:
---------------------------------
Yes, it could happen for any logical operators after Load. The key issue for
this problem is OperatorPlan.insertBetween assume logical plan in a consistent
state. What I want to do here is inserting a LOForEach after LOLoad if the
loader do not support pruning. I will need to achieve this in two steps:
1. insert a LOForEach after LoLoad
2. prune columns the logical operator below
I have to leave the logical plan in an inconsistent state in the middle. If I
insert LOForEach first, then the underlying logical operator has not been
pruned yet; If I prune underlying logical operator first, then there is a
schema gap between underlying logical operator and LOLoad. Both situations are
not allowed for insertBetween. I have to use a trick to get around this problem.
> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>
> Key: PIG-1132
> URL: https://issues.apache.org/jira/browse/PIG-1132
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1132-1.patch, PIG-1132-2.patch
>
>
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement
> column for column: 1 in the new predecessor at
> org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
> at
> org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
> at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
> at
> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
> at
> org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at
> org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
> ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
