[
https://issues.apache.org/jira/browse/OOZIE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102679#comment-14102679
]
Ryota Egashira commented on OOZIE-1976:
---------------------------------------
agree with the idea. another suggestion is
- OR or n out of m case, user should be able to specify priority among
multiple dataset (e.g, n out of m, if all of m are available, which n out of
m to pick), for instance, the order in workflow.xml can be used as priority
for OR case, but could be more sophisticated way.
- EL function also need to be changed or new EL to be added. when we use
AND/OR operator across multiple <data-in>, probably need some other EL to list
all input data (other than coord:dataIn)
> Specifying coordinator input datasets in more logical ways
> ----------------------------------------------------------
>
> Key: OOZIE-1976
> URL: https://issues.apache.org/jira/browse/OOZIE-1976
> Project: Oozie
> Issue Type: New Feature
> Components: coordinator
> Affects Versions: trunk
> Reporter: Mona Chitnis
> Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1976-rough-design.pdf
>
>
> All dataset instances specified as input to coordinator, currently work on
> AND logic i.e. ALL of them should be available for workflow to start. We
> should enhance this to include more logical ways of specifying availability
> criteria e.g.
> * OR between instances
> * minimum N out of K instances
> * delta datasets (process data incrementally)
> Use-cases for this:
> * Different datasets are BCP, and workflow can run with either, whichever
> arrives earlier.
> * Data is not guaranteed, and while $coord:latest allows skipping to
> available ones, workflow will never trigger unless mentioned number of
> instances are found.
> * Workflow is like a ‘refining’ algorithm which should run after minimum
> required datasets are ready, and should only process the delta for efficiency.
> This JIRA is to discuss the design and then the review the implementation for
> some or all of the above features.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
