[
https://issues.apache.org/jira/browse/PIG-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765172#action_12765172
]
Dmitriy V. Ryaboy commented on PIG-966:
---------------------------------------
Alan, thanks for the explanation on the kinds of pushdowns you are envisioning.
This makes sense, although I have a feeling that if we get this complex with
pushdowns, it may be more appropriate to start thinking of interfaces that
expose different access paths, rather than "pushdownable operations".
Starting to think perhaps you are right in wanting to make this a single
interface instead of multiple ones like I suggested.
A couple more thoughts on the LoadPushdown interface.
getFeatures() should probably return a Set, not a List, as duplicates don't
really make sense and we want fast contains() calls on the returned object.
The new idea is just a small tweak on your design that aims to avoid the
OperatorPlan issue.
Maintain a Set of LogicalOperator classes (as in, LOProject.class) to indicate
acceptable operators, and provide an pushOperator(LogicalOperator op) method,
which can be called multiple times. If the order of operators matters, it
should be up to whoever is calling this method to do so in the right order.
This does force LoadFunc implementations to understand Pig operator classes,
and in the case of Filter it does have to deal with an inner LogicalPlan, but I
think those classes are mostly ok. If someone is advanced enough to want to
implement pushdowns, they can handle those interfaces. There is the danger of
the interfaces changing, of course, but, well, that consideration hasn't
stopped Hadoop... and we are setting a precedent by breaking the LoadFunc
interface right now anyway :-).
Too simple?
> Proposed rework for LoadFunc, StoreFunc, and Slice/r interfaces
> ---------------------------------------------------------------
>
> Key: PIG-966
> URL: https://issues.apache.org/jira/browse/PIG-966
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Alan Gates
> Assignee: Alan Gates
>
> I propose that we rework the LoadFunc, StoreFunc, and Slice/r interfaces
> significantly. See http://wiki.apache.org/pig/LoadStoreRedesignProposal for
> full details
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
