[
https://issues.apache.org/jira/browse/PIG-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980249#action_12980249
]
Ashutosh Chauhan commented on PIG-1717:
---------------------------------------
Sounds good. I was wondering if you already know about your partitioning keys
and their values (from path string) then using hadoop path globbing (of
FileInputFormat) will be the fastest way to get partition pruning without the
need of all the extra code and requiring Pig to push filters to the loaders
etc. But looks like in your usecase you still need to get the filter
expressions which may occur later on in pig scripts. Option B sounds ok as
short term solution.
> pig needs to call setPartitionFilter if schema is null but getPartitionKeys
> is not
> ----------------------------------------------------------------------------------
>
> Key: PIG-1717
> URL: https://issues.apache.org/jira/browse/PIG-1717
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.9.0
> Reporter: Gerrit Jansen van Vuuren
> Assignee: Gerrit Jansen van Vuuren
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: PIG-1717.patch
>
>
> I'm writing a loader that works with hive style partitioning e.g.
> /logs/type1/daydate=2010-11-01
> The loader does not know the schema upfront and this is something that the
> user adds in the script using the AS clause.
> The problem is that this user defined schema is not available to the loader,
> so the loader cannot return any schema, the Loader does know what the
> partition keys are and pig needs in some way to know about these partition
> keys.
> Currently if the schema is null pig never calls the
> LoadMetaData:getPartitionKeys method or the setPartitionFilter method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
