Hi, Oozie team at Y! is going to implement the non-polling based data dependency using HCatalog metadata system. This is expected to impact a lot of performance issues. We created a JIRA and upload the high-level requirements and design. A quick presentation slides are also attached.
JIRA: https://issues.apache.org/jira/browse/OOZIE-561 Please give your valuable suggestions and comments. We are about to start the development. Regards, Mohammad