can Kafka streams support ordering across 2 different topics when consuming from multiple source topics?

2023-03-20 Thread Pushkar Deole
Hi All, We have a kafka streams application that consumes from 2 different topics say topic A and topic B. The application uses data of telephone call on those topics and each call has a call id which is used as key to send events to those 2 topics. e.g. for a telephone call, the 1st event related

Mirrormaker2 automated consumer offset sync for mirromaker2 as remote consumer

2023-03-20 Thread Cai Elvis
Hi team we are doing kafka cluster migration from kafka1 to kafka2, at the same time we also have remote MM2 to replicate topic A from kafka1, as you know after migrating from kafka1 to kafak2, remote MM2 will also be migrated, and the issue happened here, for topic configured with retention time

Re: MirrorMaker 2.0 question

2023-03-20 Thread Miguel Ángel Fernández Fernández
I have two clusters up on the same machine with docker-compose services: zookeeper-lab: image: "bitnami/zookeeper:3.8.1" restart: always environment: ZOO_PORT_NUMBER: 2183 ALLOW_ANONYMOUS_LOGIN: "yes" ports: - "2183:2183" - "2886:2888" - "3886:3888" kafk

Re: MirrorMaker 2.0 question

2023-03-20 Thread Chris Egerton
Mi Miguel, How many nodes are you running MM2 with? Just one? Separately, do you notice anything at ERROR level in the logs? Cheers, Chris On Mon, Mar 20, 2023 at 5:35 PM Miguel Ángel Fernández Fernández < miguelangelprogramac...@gmail.com> wrote: > Hello, > > I'm doing some tests with Mirror

MirrorMaker 2.0 question

2023-03-20 Thread Miguel Ángel Fernández Fernández
Hello, I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple of kafka clusters, I think everything is set up correctly. However, when I run /bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties the result I get is the creation of the topics mm2-configs.A.internal, mm2-of

Kafka Connect Startup Hook

2023-03-20 Thread Jan Baudisch (extern)
Hello, can someone please give me a hint how to execute two lines of code upon Kafka Connect Startup, like: final JaegerTracer tracer = Configuration.fromEnv().getTracer(); GlobalTracer.register(tracer); I implemented using a custom (Fake-)Connector, but there is much overhead, because you als

Re: AW: Kafka Connect Startup Hook

2023-03-20 Thread Hutson, Paul
unsubscribe On 20/03/2023, 10:30, "Jan Baudisch (extern)" mailto:jan.baudisch.ext...@bdess.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hello Jakub,

AW: Kafka Connect Startup Hook

2023-03-20 Thread Jan Baudisch (extern)
Hello Jakub, thank you for you quick answer. We solved it by implementing a ConfigProvider, like described here: https://docs.confluent.io/kafka-connectors/self-managed/userguide.html#configprovider-interface public class TracingConfigProvider implements ConfigProvider { @Override publ

Re: Kafka Connect Startup Hook

2023-03-20 Thread Jakub Scholz
In Strimzi, we use a Java agent to register the tracer ( https://github.com/strimzi/strimzi-kafka-operator/tree/main/tracing-agent/ if you wanna check the source code). Jakub On Mon, Mar 20, 2023 at 9:18 AM Jan Baudisch (extern) < jan.baudisch.ext...@bdess.com> wrote: > Hello, > > can someone pl

Kafka Connect Startup Hook

2023-03-20 Thread Jan Baudisch (extern)
Hello, can someone please give me a hint how to execute two lines of code upon Kafka Connect Startup, like: final JaegerTracer tracer = Configuration.fromEnv().getTracer(); GlobalTracer.register(tracer); I implemented using a custom (Fake-)Connector, but there is much overhead, because you als