Nice read. We just started using kafka, and have multiple cases which need some kind of stream processing. So we most likely will start testing/using it as soon as it will be released, adding stream processing containers to our docker landscape.
On Fri, Mar 11, 2016 at 2:42 AM Jay Kreps <j...@confluent.io> wrote: > Hey David, > > Yeah I think the similarity to Spark (and Flink and RxJava) is the stream > api style in the DSL. That is totally the way to go for stream processing. > We tried really hard to make that work early on when we were doing Samza, > but we really didn't understand the whole iterator/observable distinction > and the experiment wasn't very successful. We ended up doing a process() > callback in Samza which I think is just much much less readable. One of the > nice things about Kafka Streams is I think we really got this right. The > API is split into two layers--a kind of infrastructure layer which is based > on modeling data flow DAGs, in some sense all stream processing boils down > to this, though it is not necessarily the most readable way to express it. > This layer is documented here ( > > http://docs.confluent.io/2.1.0-alpha1/streams/developer-guide.html#streams-developer-guide-processor-api > ). > Then on top of that you can layer any kind of DSL or language you like. The > KStreams layer is our take on a readable DSL. > > As for RxJava, it is super cool. We looked at it a little bit as a > potential alternative language versus doing a custom DSL in KStreams. There > is enough that is unique to distributed stream processing, including the > whole table/stream distinction, the details of the partitioning model and > when data is committed, etc that we felt trying to glue something on top > would end up being a bit limiting. That said, I think there is > reactive-streams integration for Kafka, though I have no experience with > it: > https://github.com/akka/reactive-kafka > > Cheers, > > -Jay > > On Thu, Mar 10, 2016 at 3:26 PM, David Buschman <david.busch...@timeli.io> > wrote: > > > Very interesting, looks a lot like many operations from Spark were > brought > > across. > > > > Any plans to integrate with the reactive-stream protocol for > > interoperability with libraries akka-stream and RxJava? > > > > Thanks, > > DaVe. > > > > David Buschman > > d...@timeli.io > > > > > > > > > On Mar 10, 2016, at 2:26 PM, Jay Kreps <j...@confluent.io> wrote: > > > > > > Hey all, > > > > > > Lot's of people have probably seen the ongoing work on Kafka Streams > > > happening. There is no real way to design a system like this in a > vacuum, > > > so we put up a blog, some snapshot docs, and something you can download > > and > > > use easily to get feedback: > > > > > > > > > http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple > > > > > > We'd love comments or thoughts from anyone... > > > > > > -Jay > > > > >
