[ https://issues.apache.org/jira/browse/OOZIE-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476611#comment-13476611 ]
Mohammad Kamrul Islam commented on OOZIE-561: --------------------------------------------- High-level Requirements: • Allow users to specify table/partition based data dependencies used in HCatalog. • User should be able to specify the existing directory based data dependency as well. • User can use both types of dataset (such directory and table-partition based) for the same coordinator. • Single data source should be from the same type (either table-partition or directory). • For table-partition based dataset, enable user to utilize the existing concept defined through EL functions such as current(), latest(), future(). • Oozie should allow to utilize metadata from multiple HCatalog servers. • User can optionally provide the HCatalog server end-points. Oozie should have a default HCatalog server. • Oozie will allow passing the input data destinations as db/table/partition(filter) that could easily be used in MR job and pig script. • Oozie will allow passing the output data destination as db/table/partition(filter) that could easily be used in MR job and pig script. • Include “remove partition “ like statement into <prepare> block. • Include HCatalog Action. (Future) • In condition expression ((e.g. <case> statement) ), support the same functionality provided for directory-based system. Provide equivalent new EL function with table-partition support based logic. (Future) • Support non-timed based data dependency (Asynchronous data processing) (Future) The future items will not be implemented as part of this JIRA. However, it should be considered in our design consideration. > Integrate Oozie with HCatalog > ----------------------------- > > Key: OOZIE-561 > URL: https://issues.apache.org/jira/browse/OOZIE-561 > Project: Oozie > Issue Type: New Feature > Reporter: Santhosh Srinivasan > Assignee: Mona Chitnis > > With the incubation of HCatalog, we have a mechanism to abstract data and > storage on HDFS. A natural progression for Oozie is to interact with HCatalog > to facilitate the interplay between MapReduce, Pig and Hive. In addition, the > support for notification in HCatalog will alleviate (and not eliminate) the > need to poll HDFS for data sets represented as tables and partitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira