[
https://issues.apache.org/jira/browse/PIG-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774305#comment-13774305
]
Aniket Mokashi commented on PIG-3461:
-------------------------------------
bq. Can you make the old visitor default and the new one optional in 0.12? I
think that's safer given we haven't tested the new code enough. Do you agree?
I prefer having the new optimizer as default. That way we can make sure
everyone migrates to the new code instead hiding that against a flag. Reverting
to old behavior should only happen if there bugs/corner cases.
bq. Can you annotate the old visitor class (and its unit test) as deprecated so
that we can delete them perhaps in 0.13?
Definitely.
bq. Can you make FilterExtractor extend PlanVisitor? It doesn't seem necessary,
but I find it helpful to view it in the type hierarchy of PlanVisitor in IDE.
FilterExtractor is not a PlanVisitor actually. It can have applications even
outside of the optimizer framework and hence I do not want to tie it to visitor
framework. The code needs to be strictly recursive (aka depth first) instead of
the visitor pattern. PColFilterExtractor also extends PlanVisitor for namesake
and does not use any of its features.
bq. Please remove trailing whites paces.
Will do.
bq. Is there any filter expression that used to be pushed down is no longer
pushed down.
That should not be the case.
bq. The "1 == 0" case in your comment is pushed down now while it wasn't
before, so I see it more as an improvement than a backward incompatibility.
True. I agree with you. However, I have seen people use filter a by 1==0 as a
no-op load-and-drop data operation. That won't work anymore with Loader's that
implement LoadMetadata. There are no known incompatibilities with this fix.
But, since more stuff is getting pushed to the loaders, you might need to make
sure your loaders have the correct logic for all cases (eg- 1==0, long
expressions etc.)
> Rewrite PartitionFilterOptimizer to make it work for all the cases
> ------------------------------------------------------------------
>
> Key: PIG-3461
> URL: https://issues.apache.org/jira/browse/PIG-3461
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.11.1
> Reporter: Aniket Mokashi
> Assignee: Aniket Mokashi
> Fix For: 0.12
>
> Attachments: PIG-3461-2.patch, PIG-3461-4.patch
>
>
> Current algorithm for Partition Filter pushdown identification fails in
> several corner cases. We need to rewrite its logic so that it works in all
> cases and does the maximum possible filter pushdown.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
