If you are augmenting streaming data with dimensional data, or if you can transform your data with a map, filter or join operation Streams will be a good option. If your system is stateless and the transformations are not interdependent, you may want to look into using one of the queue technologies like AcitveMQ.
On Tue, Mar 13, 2018 at 10:00 AM Sameer Rahmani <lxsame...@gmail.com> wrote: > Thanks Jacob. I'm using kafka as a distributed queue and my data pipeline > is a several components which connected together via a stream abstraction. > each component has a input and output stream. Basically the source of > this pipeline can be anything and the output can be anything to. > > On Tue, Mar 13, 2018 at 1:27 PM, Jacob Sheck <shec0...@gmail.com> wrote: > > > Sameer when you say that you need to "consume from and produce to a > topic" > > to me that seems like a good fit for Kafka Streams. Streaming your data > > out of Kafka for a transform and back in has some fundamental costs and > > operational challenges involved. Are the events in your stream > stateless? > > If it isn't stateless streams will ensure a consistent playback of events > > if needed. Without knowing more about your pipeline it is hard to make > > recommendations. Are you possibly using Kafka as a distributed queue? > > > > On Tue, Mar 13, 2018 at 6:29 AM Sameer Rahmani <lxsame...@gmail.com> > > wrote: > > > > > Hi folks, > > > I need to consume from and produce to a topic. I have my own data > > pipeline > > > to process the data. > > > So I was wondering beside the stores and StreamDSL what does Kafka > > Streams > > > brings to the table > > > that might be useful to me ? > > > > > >
