[
https://issues.apache.org/jira/browse/PIG-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081948#comment-14081948
]
Julien Le Dem commented on PIG-3760:
------------------------------------
FYI in Parquet the filter is not a hint and it will be applied to records after
the metadata
> Predicate pushdown for columnar file formats
> --------------------------------------------
>
> Key: PIG-3760
> URL: https://issues.apache.org/jira/browse/PIG-3760
> Project: Pig
> Issue Type: New Feature
> Reporter: Andrew Musselman
> Fix For: 0.14.0
>
>
> From the conversation on dev@pig:
> "Partition pruning for ORC is not addressed in PIG-3558. We will need
> to do partition pruning for both ORC and Parquet in a new ticket.
> Curently there is no interface to deal with this kind of pushdown
> (LoadMetadata.setPartitionFilter push the filter to loader, but remove
> the filter statement, for ORC/Parquet, filter is a hint, and we need
> to do the filter again in Pig even it is pushed to loader), we will
> need to define a new interface for that. You are welcome to initiate
> the work. I know Aniket is also interested in doing that, so be sure
> the talk with him about this work.
> Thanks,
> Daniel
> On Mon, Feb 10, 2014 at 11:42 AM, Andrew Musselman
> <andrew.mussel...@gmail.com> wrote:
> > I had a chat with a couple people last week about a feature request for
> > Pig: in a "where" or "filter" clause, when loading an ORC file, to skip
> > directly to the right offset instead of scanning the whole file."
--
This message was sent by Atlassian JIRA
(v6.2#6252)
