Re: Seeking advice on Kafka Streams and Kafka Connect

[Manoj Khangaonkar]

Hi

I am not a big fan of kafka connect.

I had use case for kafka messages that needed to be written to MongoDb. The
available third party connectors were less than ideal.

To me a well written Kafka consumer is  simpler and better longer term
solution instead of an additional moving part and additional programming
model of Kafka connect.

Keep it simple with topics , producers and consumers.

regards


On Thu, Dec 21, 2017 at 2:49 AM, Mads Tandrup <
mads.tand...@schneider-electric.com> wrote:

> Hi
>
> Sorry for the simple question. I’m just starting to learn about Kafka
> streams and connect and I’m struggling to understand the exact difference
> and which one to use. I’m coming from Apache Storm so forgive me if I make
> false assumptions.
>
> I have a use case where I have a Kafka topic with some messages. What I
> need to do:
> 1. Read the messages
> 2. Split and map the message into a number of rows
> 3. Write the rows to Cassandra
>
> It seems the first 2 steps are a natural fit for Kafka Streams. But it
> seems the way to write to Cassandra is to use Kafka Connect.
> Is that correctly understood?
>
> Is there any way to connect Kafka Streams and Kafka Connect without
> writing it to a new kafka topic? Since the transformation in step 2 is so
> simple it seems a waste to write it to disk.
>
> Is there any other way I should consider?
>
> Best regards,
> Mads
>
>


-- 
http://khangaonkar.blogspot.com/

Re: Seeking advice on Kafka Streams and Kafka Connect

[Hans Jespersen]

It might be possible to do all the transformations in #2 inside Kafka Connect. 
Connect has a simple one message at a time transformation capability called 
Single Message Transforms (SMT). There are built in Transformation functions 
that you can declaratively add to any existing connector via configuration 
properties and without coding. If the built in functions are insufficient you 
can write your own SMT functions in Java.

-hans

> On Dec 21, 2017, at 7:19 AM, Bill Bejeck  wrote:
> 
> Hi Mads,
> 
> Great question and yes your use case here is an excellent fit for Kafka
> Streams and Kafka Connect.
> 
> For step 2 you could use a KStram#flatMap operation to split it up into
> multiple rows.
> 
> Regarding a Cassandra connector, there is an existing one:
> 
>   1. For some background try
>   
> https://www.confluent.io/blog/kafka-connect-cassandra-sink-the-perfect-match/
> 
>   2. To download the connector go to
>   https://www.confluent.io/product/connectors/
> 
> As for writing directly to Cassandra, you *could*, in theory, do so with a
> KStream.process or KStream.foreach call.
> 
> But you'd have to code that yourself which includes error handling, retry
> logic etc.   Additionally, it's usually not recommended to write out
> directly to external systems from Kafka Streams.
> 
> I'd say it's better to leverage Kafka Connect for that.
> 
> HTH,
> Bill
> 
> 
> On Thu, Dec 21, 2017 at 5:49 AM, Mads Tandrup <
> mads.tand...@schneider-electric.com> wrote:
> 
>> Hi
>> 
>> Sorry for the simple question. I’m just starting to learn about Kafka
>> streams and connect and I’m struggling to understand the exact difference
>> and which one to use. I’m coming from Apache Storm so forgive me if I make
>> false assumptions.
>> 
>> I have a use case where I have a Kafka topic with some messages. What I
>> need to do:
>> 1. Read the messages
>> 2. Split and map the message into a number of rows
>> 3. Write the rows to Cassandra
>> 
>> It seems the first 2 steps are a natural fit for Kafka Streams. But it
>> seems the way to write to Cassandra is to use Kafka Connect.
>> Is that correctly understood?
>> 
>> Is there any way to connect Kafka Streams and Kafka Connect without
>> writing it to a new kafka topic? Since the transformation in step 2 is so
>> simple it seems a waste to write it to disk.
>> 
>> Is there any other way I should consider?
>> 
>> Best regards,
>> Mads
>> 
>> 

Re: Seeking advice on Kafka Streams and Kafka Connect

[Bill Bejeck]

Hi Mads,

Great question and yes your use case here is an excellent fit for Kafka
Streams and Kafka Connect.

For step 2 you could use a KStram#flatMap operation to split it up into
multiple rows.

Regarding a Cassandra connector, there is an existing one:

   1. For some background try
   https://www.confluent.io/blog/kafka-connect-cassandra-sink-the-perfect-match/

   2. To download the connector go to
   https://www.confluent.io/product/connectors/

As for writing directly to Cassandra, you *could*, in theory, do so with a
KStream.process or KStream.foreach call.

But you'd have to code that yourself which includes error handling, retry
logic etc.   Additionally, it's usually not recommended to write out
directly to external systems from Kafka Streams.

I'd say it's better to leverage Kafka Connect for that.

HTH,
Bill


On Thu, Dec 21, 2017 at 5:49 AM, Mads Tandrup <
mads.tand...@schneider-electric.com> wrote:

> Hi
>
> Sorry for the simple question. I’m just starting to learn about Kafka
> streams and connect and I’m struggling to understand the exact difference
> and which one to use. I’m coming from Apache Storm so forgive me if I make
> false assumptions.
>
> I have a use case where I have a Kafka topic with some messages. What I
> need to do:
> 1. Read the messages
> 2. Split and map the message into a number of rows
> 3. Write the rows to Cassandra
>
> It seems the first 2 steps are a natural fit for Kafka Streams. But it
> seems the way to write to Cassandra is to use Kafka Connect.
> Is that correctly understood?
>
> Is there any way to connect Kafka Streams and Kafka Connect without
> writing it to a new kafka topic? Since the transformation in step 2 is so
> simple it seems a waste to write it to disk.
>
> Is there any other way I should consider?
>
> Best regards,
> Mads
>
>

Seeking advice on Kafka Streams and Kafka Connect

[Mads Tandrup]

Hi

Sorry for the simple question. I’m just starting to learn about Kafka streams 
and connect and I’m struggling to understand the exact difference and which one 
to use. I’m coming from Apache Storm so forgive me if I make false assumptions.

I have a use case where I have a Kafka topic with some messages. What I need to 
do:
1. Read the messages
2. Split and map the message into a number of rows
3. Write the rows to Cassandra

It seems the first 2 steps are a natural fit for Kafka Streams. But it seems 
the way to write to Cassandra is to use Kafka Connect.
Is that correctly understood?

Is there any way to connect Kafka Streams and Kafka Connect without writing it 
to a new kafka topic? Since the transformation in step 2 is so simple it seems 
a waste to write it to disk.

Is there any other way I should consider?

Best regards,
Mads