It might be possible to do all the transformations in #2 inside Kafka Connect. Connect has a simple one message at a time transformation capability called Single Message Transforms (SMT). There are built in Transformation functions that you can declaratively add to any existing connector via configuration properties and without coding. If the built in functions are insufficient you can write your own SMT functions in Java.
-hans > On Dec 21, 2017, at 7:19 AM, Bill Bejeck <b...@confluent.io> wrote: > > Hi Mads, > > Great question and yes your use case here is an excellent fit for Kafka > Streams and Kafka Connect. > > For step 2 you could use a KStram#flatMap operation to split it up into > multiple rows. > > Regarding a Cassandra connector, there is an existing one: > > 1. For some background try > > https://www.confluent.io/blog/kafka-connect-cassandra-sink-the-perfect-match/ > > 2. To download the connector go to > https://www.confluent.io/product/connectors/ > > As for writing directly to Cassandra, you *could*, in theory, do so with a > KStream.process or KStream.foreach call. > > But you'd have to code that yourself which includes error handling, retry > logic etc. Additionally, it's usually not recommended to write out > directly to external systems from Kafka Streams. > > I'd say it's better to leverage Kafka Connect for that. > > HTH, > Bill > > > On Thu, Dec 21, 2017 at 5:49 AM, Mads Tandrup < > mads.tand...@schneider-electric.com> wrote: > >> Hi >> >> Sorry for the simple question. I’m just starting to learn about Kafka >> streams and connect and I’m struggling to understand the exact difference >> and which one to use. I’m coming from Apache Storm so forgive me if I make >> false assumptions. >> >> I have a use case where I have a Kafka topic with some messages. What I >> need to do: >> 1. Read the messages >> 2. Split and map the message into a number of rows >> 3. Write the rows to Cassandra >> >> It seems the first 2 steps are a natural fit for Kafka Streams. But it >> seems the way to write to Cassandra is to use Kafka Connect. >> Is that correctly understood? >> >> Is there any way to connect Kafka Streams and Kafka Connect without >> writing it to a new kafka topic? Since the transformation in step 2 is so >> simple it seems a waste to write it to disk. >> >> Is there any other way I should consider? >> >> Best regards, >> Mads >> >>
