[
https://issues.apache.org/jira/browse/PIG-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371646#comment-15371646
]
Ryan Blue commented on PIG-4940:
--------------------------------
I attached a patch with my changes. The case that we hit is a "not
map_column#'prop' is null" expression. The map access can't be pushed down, but
the code assumes that anything under a unary expression can be.
I looked at writing a test, but I don't see a test suite for predicate
push-down in the FilterExpression code already there so it will take me a while
to wrap my head around what needs to be done. I'm not very familiar with the
Pig internals necessary to cleanly test this. If someone can help with the
setup, I can get a test case written pretty quickly. Otherwise, it may take me
some time so I'm posting the patch now so you can get started looking at it in
the mean time.
> Predicate push-down filtering unary expressions can be pushed.
> --------------------------------------------------------------
>
> Key: PIG-4940
> URL: https://issues.apache.org/jira/browse/PIG-4940
> Project: Pig
> Issue Type: Bug
> Reporter: Ryan Blue
> Assignee: Ryan Blue
> Attachments: PIG-4940.1.patch
>
>
> While testing predicate push-down, I ran into the following error:
> {code:title=Pig Exception}
> ERROR 0: Unsupported conversion of LogicalExpression to Expression: Map
> at
> org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:389)
> at
> org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:401)
> at
> org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:378)
> at
> org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:401)
> at
> org.apache.pig.newplan.FilterExtractor.getExpression(FilterExtractor.java:380)
> at
> org.apache.pig.newplan.FilterExtractor.visit(FilterExtractor.java:109)
> at
> org.apache.pig.newplan.PredicatePushDownFilterExtractor.visit(PredicatePushDownFilterExtractor.java:70)
> at
> org.apache.pig.newplan.logical.rules.PredicatePushdownOptimizer$PredicatePushDownTransformer.transform(PredicatePushdownOptimizer.java:146)
> at
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> ... 19 more
> {code}
> The problem is that the code is trying to push a map access operation, that
> isn't supported. The cause appears to be the logic in
> {{checkPushDown(UnaryExpression)}} that separates expressions that can be
> pushed from expressions that must be run by Pig. This function assumes that
> any expression under {{IsNullExpression}} or {{NotExpression}} can be pushed
> and adds the unary node's child expression to the pushdown expression without
> calling {{checkPushDown}} on it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
