Re: running kafka streams inside kafka connect
Hi RK, You may find this blog post helpful for incorporating Streams with Connect: https://yokota.blog/2018/04/23/embedding-kafka-connect-in-kafka-streams-ksql/ I've also cc'ed Robert, the author of this blog post if you are interested in his code and want to learn more. Guozhang On Tue, Oct 30, 2018 at 8:54 PM RK Sing wrote: > Thank you that's helpful. > > So that means resource management of streams needs to a separate concern > than that of Kafka connect > > On Thu, Oct 25, 2018 at 3:10 PM Matthias J. Sax > wrote: > > > Streams is not designed to be run inside Connect, and this won't work. > > > > What you can do is, to import the data via connect into a "staging > > topic" and then read this "staging topic" with a Kafka Streams > > application and apply the transformations etc to write the data into the > > actual target topics. > > > > > > -Matthias > > > > On 10/25/18 2:34 PM, RK Sing wrote: > > > Thank you Ryanne for the answer, My question is can I run Streams app > > > inside Connect, what I mean is since Streams is a library, can I build > a > > > custom Source connector and use Streams in there . or using Streams in > > > transforms?? > > > We want to containerize Kafka connect as in wrap it in a docker, and > > since > > > all the Kafka-connect configs are stored in Kafka itself, so the > > container > > > can be stateless, we can spin up containers as and when needed, we want > > > this approach to fetch data from microservices and golden gate oracle. > > etc. > > > > > > > > > > > > > > > On Thu, Oct 25, 2018 at 2:24 PM Ryanne Dolan > > wrote: > > > > > >> Dhurandar, definitely! Connect and Streams are both agnostic to how > > their > > >> workers are run. They aren't really platforms per se. You just need to > > spin > > >> up one or more workers and they do their thing. So a Streams app > doesn't > > >> run "inside" Connect, but you can certainly have Connect and Streams > > >> workers talk to each other through Kafka topics. > > >> > > >> Ryanne > > >> > > >> On Thu, Oct 25, 2018 at 3:34 PM RK Sing > > wrote: > > >> > > >>> We have a requirement to do Single row transformations, basic joins, > > >>> deduping and routing from the source to the destination Kafka topics. > > >>> > > >>> We want to use Kafka-connect as the platform which is running Kafka > > >> stream > > >>> inside. Has anyone used Kafkastreams inside Kafka connect ?? > > >>> > > >>> Is this pattern ok for Kafka-connect? > > >>> > > >>> regards > > >>> dhurandar > > >>> > > >> > > > > > > > > -- -- Guozhang
Re: running kafka streams inside kafka connect
Thank you that's helpful. So that means resource management of streams needs to a separate concern than that of Kafka connect On Thu, Oct 25, 2018 at 3:10 PM Matthias J. Sax wrote: > Streams is not designed to be run inside Connect, and this won't work. > > What you can do is, to import the data via connect into a "staging > topic" and then read this "staging topic" with a Kafka Streams > application and apply the transformations etc to write the data into the > actual target topics. > > > -Matthias > > On 10/25/18 2:34 PM, RK Sing wrote: > > Thank you Ryanne for the answer, My question is can I run Streams app > > inside Connect, what I mean is since Streams is a library, can I build a > > custom Source connector and use Streams in there . or using Streams in > > transforms?? > > We want to containerize Kafka connect as in wrap it in a docker, and > since > > all the Kafka-connect configs are stored in Kafka itself, so the > container > > can be stateless, we can spin up containers as and when needed, we want > > this approach to fetch data from microservices and golden gate oracle. > etc. > > > > > > > > > > On Thu, Oct 25, 2018 at 2:24 PM Ryanne Dolan > wrote: > > > >> Dhurandar, definitely! Connect and Streams are both agnostic to how > their > >> workers are run. They aren't really platforms per se. You just need to > spin > >> up one or more workers and they do their thing. So a Streams app doesn't > >> run "inside" Connect, but you can certainly have Connect and Streams > >> workers talk to each other through Kafka topics. > >> > >> Ryanne > >> > >> On Thu, Oct 25, 2018 at 3:34 PM RK Sing > wrote: > >> > >>> We have a requirement to do Single row transformations, basic joins, > >>> deduping and routing from the source to the destination Kafka topics. > >>> > >>> We want to use Kafka-connect as the platform which is running Kafka > >> stream > >>> inside. Has anyone used Kafkastreams inside Kafka connect ?? > >>> > >>> Is this pattern ok for Kafka-connect? > >>> > >>> regards > >>> dhurandar > >>> > >> > > > >
Re: running kafka streams inside kafka connect
Streams is not designed to be run inside Connect, and this won't work. What you can do is, to import the data via connect into a "staging topic" and then read this "staging topic" with a Kafka Streams application and apply the transformations etc to write the data into the actual target topics. -Matthias On 10/25/18 2:34 PM, RK Sing wrote: > Thank you Ryanne for the answer, My question is can I run Streams app > inside Connect, what I mean is since Streams is a library, can I build a > custom Source connector and use Streams in there . or using Streams in > transforms?? > We want to containerize Kafka connect as in wrap it in a docker, and since > all the Kafka-connect configs are stored in Kafka itself, so the container > can be stateless, we can spin up containers as and when needed, we want > this approach to fetch data from microservices and golden gate oracle. etc. > > > > > On Thu, Oct 25, 2018 at 2:24 PM Ryanne Dolan wrote: > >> Dhurandar, definitely! Connect and Streams are both agnostic to how their >> workers are run. They aren't really platforms per se. You just need to spin >> up one or more workers and they do their thing. So a Streams app doesn't >> run "inside" Connect, but you can certainly have Connect and Streams >> workers talk to each other through Kafka topics. >> >> Ryanne >> >> On Thu, Oct 25, 2018 at 3:34 PM RK Sing wrote: >> >>> We have a requirement to do Single row transformations, basic joins, >>> deduping and routing from the source to the destination Kafka topics. >>> >>> We want to use Kafka-connect as the platform which is running Kafka >> stream >>> inside. Has anyone used Kafkastreams inside Kafka connect ?? >>> >>> Is this pattern ok for Kafka-connect? >>> >>> regards >>> dhurandar >>> >> > signature.asc Description: OpenPGP digital signature
Re: running kafka streams inside kafka connect
Thank you Ryanne for the answer, My question is can I run Streams app inside Connect, what I mean is since Streams is a library, can I build a custom Source connector and use Streams in there . or using Streams in transforms?? We want to containerize Kafka connect as in wrap it in a docker, and since all the Kafka-connect configs are stored in Kafka itself, so the container can be stateless, we can spin up containers as and when needed, we want this approach to fetch data from microservices and golden gate oracle. etc. On Thu, Oct 25, 2018 at 2:24 PM Ryanne Dolan wrote: > Dhurandar, definitely! Connect and Streams are both agnostic to how their > workers are run. They aren't really platforms per se. You just need to spin > up one or more workers and they do their thing. So a Streams app doesn't > run "inside" Connect, but you can certainly have Connect and Streams > workers talk to each other through Kafka topics. > > Ryanne > > On Thu, Oct 25, 2018 at 3:34 PM RK Sing wrote: > > > We have a requirement to do Single row transformations, basic joins, > > deduping and routing from the source to the destination Kafka topics. > > > > We want to use Kafka-connect as the platform which is running Kafka > stream > > inside. Has anyone used Kafkastreams inside Kafka connect ?? > > > > Is this pattern ok for Kafka-connect? > > > > regards > > dhurandar > > >
Re: running kafka streams inside kafka connect
Dhurandar, definitely! Connect and Streams are both agnostic to how their workers are run. They aren't really platforms per se. You just need to spin up one or more workers and they do their thing. So a Streams app doesn't run "inside" Connect, but you can certainly have Connect and Streams workers talk to each other through Kafka topics. Ryanne On Thu, Oct 25, 2018 at 3:34 PM RK Sing wrote: > We have a requirement to do Single row transformations, basic joins, > deduping and routing from the source to the destination Kafka topics. > > We want to use Kafka-connect as the platform which is running Kafka stream > inside. Has anyone used Kafkastreams inside Kafka connect ?? > > Is this pattern ok for Kafka-connect? > > regards > dhurandar >
running kafka streams inside kafka connect
We have a requirement to do Single row transformations, basic joins, deduping and routing from the source to the destination Kafka topics. We want to use Kafka-connect as the platform which is running Kafka stream inside. Has anyone used Kafkastreams inside Kafka connect ?? Is this pattern ok for Kafka-connect? regards dhurandar
