The point is valid. Oozie has a good amount of specific HDFS sauce baked in, and not in a good way. I was contemplating DataFileDependencies that use HivePartition information. It was not easy to do.
On Tue, Dec 19, 2017 at 10:38 AM, Serega Sheypak <serega.shey...@gmail.com> wrote: > Don't push NiFi :) Please don't :) > > 2017-12-19 16:16 GMT+01:00 Artem Ervits <artemerv...@gmail.com>: > > > you're probably better off using Apache Nifi at that point, Kafka Nifi is > > better integrated today than Oozie and Kafka. Probably wrong mailing list > > to push Nifi but it's a common use case in that realm. > > > > Thanks > > > > On Mon, Dec 18, 2017 at 7:04 PM, Serega Sheypak < > serega.shey...@gmail.com> > > wrote: > > > > > Hi, I know default coordinator functionality, but it's limited (almost) > > to > > > HDFS. > > > Kafka (any other pub/sub or queue like rabbitMQ, whatever-MQ) makes > > > integration contract much more flexible. > > > I could have traceability, debuggability, transparency, throttling, > > > concurrency of oozie and push coordinator job on demand. And I'm not > > > limited to strict HDFS path pattern. > > > > > > > > > 2017-12-18 18:14 GMT+01:00 Andras Piros <andras.pi...@cloudera.com>: > > > > > > > Hi Serega, > > > > > > > > not to my knowledge. Would be interested on your use case, though. > > > > > > > > Would start w/ *Coordinator Input Events / Datasets > > > > <https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec. > > > > html#a5._Dataset>* > > > > . > > > > > > > > Andras > > > > > > > > On Sat, Dec 16, 2017 at 2:54 PM, Serega Sheypak < > > > serega.shey...@gmail.com> > > > > wrote: > > > > > > > > > Hi, did anyone try to integrate oozie coordinator with kafka? > > > > > use case: > > > > > > > > > > System publishes message to kafka topic (sample message) > > > > > - cluster: hdfs://prod-cluster > > > > > - path: /my/input/data > > > > > - format: avro > > > > > > > > > > Oozie coordinator listens to kafka topic, consumes message and > starts > > > > > workflow. > > > > > > > > > > > > > > >
