[
https://issues.apache.org/jira/browse/PIG-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antonio Magnaghi updated PIG-32:
--------------------------------
Attachment: patch2007_12_26.diff
The basic functionalities are now in place. Whit this patch, it is now possible
to run queries locally and on Hadoop clusters with some limitations as listed
below. Have added some tests (TestAbstractionExecutionLayer.java)
In this patch:
- Pig (Front-End) has been completely modified to utilize the Abstraction layer
APIs
- Two back-ends (Local and Hadoop) implement the DataStaorage and
ExecutionEnginer APIs
- Logical Layer: Removed LORead, and IntermediateResult operators
- Local Execution Engine:
- Provided support for Split in Local plans (before it was not supported)
- Re-use of materialized results
- (working on providing ability to re-use results materialized in memory
as well, this will just require to extend LocalResult types that are stored in
the table ot materialized results)
- Hadoop Execution Engine:
- Basic support in place to run Pig queries on Hadoop.
- (need to provide support for reuse of materialized results and split
construct)
In addition, I'm working on adding further end-to-end testing and turning old
tests back on (currently most of them are commented out to get Pig to build).
(Note: some files/directory have been deleted or moved. I have marked them as
deleted in svn and a commit command from my side would probably take care of
this. However, I am not sure svn diff is properly showing that, so I may need
to coordinate with one of committers when the patch is applied to the branch.)
> Abstraction Layer to decouple Pig from Back-End
> -----------------------------------------------
>
> Key: PIG-32
> URL: https://issues.apache.org/jira/browse/PIG-32
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Reporter: Antonio Magnaghi
> Assignee: Antonio Magnaghi
> Attachments: DataStorage.diff, DataStorage20071212.diff,
> patch2007_12_26.diff
>
>
> I'm opening a new issue to track the development work to support an
> abstraction layer for Pig as defined at
> http://wiki.apache.org/pig/PigAbstractionLayer
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
