On Wed, 2015-04-22 at 12:36 +0000, Chen, Ken wrote: > o more complex workflows (job dependencies, DAGs, etc. Do we rely on > Oozie, or something else? > >> Huichun is now figuring this. I am not whether you guys already have > some detail ideas about this? If needed we can contribute some effort. If no > details are ready, we can help draw a draft version first.
I just made a note on the pad https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions Maybe the right approach here is to develop a mapping notation that can be expressed as a JSON object (like the proposed job interface mapping). If we can develop an abstract way to describe relationships between jobs, then the individual EDP engines can implement it. For the Oozie EDP engine, maybe it uses Oozie features in workflows. For Spark, or Storm, maybe it uses some existing opensource coordinator or one is written. The key idea would be to make job coordination part of the EDP engine, with a well defined set of objects to describe the relationships. What do you think? Just a rough idea. Maybe there is a better way. __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
