Hey David, Yeah I think the similarity to Spark (and Flink and RxJava) is the stream api style in the DSL. That is totally the way to go for stream processing. We tried really hard to make that work early on when we were doing Samza, but we really didn't understand the whole iterator/observable distinction and the experiment wasn't very successful. We ended up doing a process() callback in Samza which I think is just much much less readable. One of the nice things about Kafka Streams is I think we really got this right. The API is split into two layers--a kind of infrastructure layer which is based on modeling data flow DAGs, in some sense all stream processing boils down to this, though it is not necessarily the most readable way to express it. This layer is documented here ( http://docs.confluent.io/2.1.0-alpha1/streams/developer-guide.html#streams-developer-guide-processor-api). Then on top of that you can layer any kind of DSL or language you like. The KStreams layer is our take on a readable DSL.
As for RxJava, it is super cool. We looked at it a little bit as a potential alternative language versus doing a custom DSL in KStreams. There is enough that is unique to distributed stream processing, including the whole table/stream distinction, the details of the partitioning model and when data is committed, etc that we felt trying to glue something on top would end up being a bit limiting. That said, I think there is reactive-streams integration for Kafka, though I have no experience with it: https://github.com/akka/reactive-kafka Cheers, -Jay On Thu, Mar 10, 2016 at 3:26 PM, David Buschman <david.busch...@timeli.io> wrote: > Very interesting, looks a lot like many operations from Spark were brought > across. > > Any plans to integrate with the reactive-stream protocol for > interoperability with libraries akka-stream and RxJava? > > Thanks, > DaVe. > > David Buschman > d...@timeli.io > > > > > On Mar 10, 2016, at 2:26 PM, Jay Kreps <j...@confluent.io> wrote: > > > > Hey all, > > > > Lot's of people have probably seen the ongoing work on Kafka Streams > > happening. There is no real way to design a system like this in a vacuum, > > so we put up a blog, some snapshot docs, and something you can download > and > > use easily to get feedback: > > > > > http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple > > > > We'd love comments or thoughts from anyone... > > > > -Jay > >
