Re: Oozie coordinator and kafka

2017-12-20 Thread Serega Sheypak
> @Serega @Artem do you have ideas where Oozie HDFS path handling is way > inflexible? I'm using oozie for last 5 years. It's inflexible. I've explained why in initial message. The whole idea with this input / output events is wy too complex and over engineered.

Re: Oozie coordinator and kafka

2017-12-20 Thread Andras Piros
@Edward can you create upstream Oozie JIRA for that HivePartition using stuff? Maybe providing your bits as patch ;) @Serega @Artem do you have ideas where Oozie HDFS path handling is way inflexible? Thanks, Andras On Tue, Dec 19, 2017 at 8:42 PM, Edward Capriolo wrote: > The point is valid.

Re: Oozie coordinator and kafka

2017-12-19 Thread Edward Capriolo
The point is valid. Oozie has a good amount of specific HDFS sauce baked in, and not in a good way. I was contemplating DataFileDependencies that use HivePartition information. It was not easy to do. On Tue, Dec 19, 2017 at 10:38 AM, Serega Sheypak wrote: > Don't push NiFi :) Please don't :) > >

Re: Oozie coordinator and kafka

2017-12-19 Thread Artem Ervits
like I said wrong mailing list and I don't want to steal attention from Oozie. Let's move along. On Tue, Dec 19, 2017 at 10:40 AM, Ilya Karpov wrote: > Why not? > > > 19 дек. 2017 г., в 18:38, Serega Sheypak > написал(а): > > > > Don't push NiFi :) Please don't :) > > > > 2017-12-19 16:16 GMT+0

Re: Oozie coordinator and kafka

2017-12-19 Thread Ilya Karpov
Why not? > 19 дек. 2017 г., в 18:38, Serega Sheypak > написал(а): > > Don't push NiFi :) Please don't :) > > 2017-12-19 16:16 GMT+01:00 Artem Ervits : > >> you're probably better off using Apache Nifi at that point, Kafka Nifi is >> better integrated today than Oozie and Kafka. Probably wrong

Re: Oozie coordinator and kafka

2017-12-19 Thread Serega Sheypak
Don't push NiFi :) Please don't :) 2017-12-19 16:16 GMT+01:00 Artem Ervits : > you're probably better off using Apache Nifi at that point, Kafka Nifi is > better integrated today than Oozie and Kafka. Probably wrong mailing list > to push Nifi but it's a common use case in that realm. > > Thanks

Re: Oozie coordinator and kafka

2017-12-19 Thread Artem Ervits
you're probably better off using Apache Nifi at that point, Kafka Nifi is better integrated today than Oozie and Kafka. Probably wrong mailing list to push Nifi but it's a common use case in that realm. Thanks On Mon, Dec 18, 2017 at 7:04 PM, Serega Sheypak wrote: > Hi, I know default coordinat

Re: Oozie coordinator and kafka

2017-12-18 Thread Serega Sheypak
Hi, I know default coordinator functionality, but it's limited (almost) to HDFS. Kafka (any other pub/sub or queue like rabbitMQ, whatever-MQ) makes integration contract much more flexible. I could have traceability, debuggability, transparency, throttling, concurrency of oozie and push coordinator

Re: Oozie coordinator and kafka

2017-12-18 Thread Andras Piros
Hi Serega, not to my knowledge. Would be interested on your use case, though. Would start w/ *Coordinator Input Events / Datasets * . Andras On Sat, Dec 16, 2017 at 2:54 PM, Serega Sheypak wrote: > Hi, did anyone

Oozie coordinator and kafka

2017-12-16 Thread Serega Sheypak
Hi, did anyone try to integrate oozie coordinator with kafka? use case: System publishes message to kafka topic (sample message) - cluster: hdfs://prod-cluster - path: /my/input/data - format: avro Oozie coordinator listens to kafka topic, consumes message and starts workflow.