Hi RK,
You may find this blog post helpful for incorporating Streams with Connect:
https://yokota.blog/2018/04/23/embedding-kafka-connect-in-kafka-streams-ksql/
I've also cc'ed Robert, the author of this blog post if you are interested
in his code and want to learn more.
Guozhang
On Tue, Oct
Thank you that's helpful.
So that means resource management of streams needs to a separate concern
than that of Kafka connect
On Thu, Oct 25, 2018 at 3:10 PM Matthias J. Sax
wrote:
> Streams is not designed to be run inside Connect, and this won't work.
>
> What you can do is, to import the
Streams is not designed to be run inside Connect, and this won't work.
What you can do is, to import the data via connect into a "staging
topic" and then read this "staging topic" with a Kafka Streams
application and apply the transformations etc to write the data into the
actual target topics.
Thank you Ryanne for the answer, My question is can I run Streams app
inside Connect, what I mean is since Streams is a library, can I build a
custom Source connector and use Streams in there . or using Streams in
transforms??
We want to containerize Kafka connect as in wrap it in a docker, and
Dhurandar, definitely! Connect and Streams are both agnostic to how their
workers are run. They aren't really platforms per se. You just need to spin
up one or more workers and they do their thing. So a Streams app doesn't
run "inside" Connect, but you can certainly have Connect and Streams
We have a requirement to do Single row transformations, basic joins,
deduping and routing from the source to the destination Kafka topics.
We want to use Kafka-connect as the platform which is running Kafka stream
inside. Has anyone used Kafkastreams inside Kafka connect ??
Is this pattern ok
