Hi, For (1) and perhaps even for (2) where distribution/filtering on scale is required, I would look at using Apache Storm with kafka.
For (3) , it seems you just need REST services wrapping kafka consumers/producers. I would start with usual suspects like jersey. regards On Tue, Mar 24, 2015 at 12:06 PM, Valentin <[email protected]> wrote: > > Hi guys, > > we have three Kafka use cases for which we have written our own PoC > implementations, > but where I am wondering whether there might be any fitting open source > solution/tool/framework out there. > Maybe someone of you has some ideas/pointers? :) > > 1) Message routing/distribution/filter tool > We need to copy messages from a set of input topics to a set of output > topics > based on their message key values. Each message in an input topic will go > to 0 to N output topics, > each output topic will receive messages from 0 to N input topics. > So basically the tool acts as a message routing component in our system. > Example configuration: > <input topic A>:<output topic K>:<key value 1>,<key value 2>,<key value 3> > <input topic A>:<output topic L>:<key value 2>,<key value 4> > <input topic B>:<output topic K>:<key value 5>,<key value 6> > ... > It would also be interesting to define distribution/filter rules based on > regular expressions on the message key or message body. > > 2) Kafka-to-REST Push service > We need to consume messages from a set of topics, translate them into REST > web service calls > and forward the data to existing, non-Kafka-aware systems with REST APIs > that way. > > 3) HTTP REST API for consumers and producers > We need to expose the simple consumer and the producer functionalities via > REST web service calls, > with authentication and per-topic-authorization on REST API level and TLS > for transport encryption. > Offset tracking is done by the connected systems, not by the > broker/zookeeper/REST API. > We expect a high message volume in the future here, so performance would > be a key concern. > > Greetings > Valentin > -- http://khangaonkar.blogspot.com/
