Re: can Kafka streams support ordering across 2 different topics when consuming from multiple source topics?

[Matthias J. Sax]

In general there is no ordering guarantee between topics. So it might 
depend a lot ofnthe details of your use case.


For example, if you know that it will be always two event, you could 
buffer the first one in a state-store, and wait for the second one to 
arrive and decide in which order to forward both events downstream for 
actual processing.



HTH, Matthias


On 3/20/23 11:57 PM, Pushkar Deole wrote:

Hi All,

We have a kafka streams application that consumes from 2 different topics
say topic A and topic B. The application uses data of telephone call on
those topics and each call has a call id which is used as key to send
events to those 2 topics. e.g. for a telephone call, the 1st event related
to that call is sent to A with call id however subsequent event for that
same call might go to topic B again with call id as key.

*At times, we need to process those 2 events in an order, which is not
possible with the current topology that we are using*. *Can someone suggest
if this is possible to achieve with streams?*
The topology is as below:

Topic A has 6 partitions
Topics B has 6 partitions
Call id used as key on both topics
Kafka streams application has 3 instances that consumes from both of the
topics as source topics.
Each streams application instance has 2 stream threads thus total 6 stream
threads across 3 instances of streams application cater to 6 partitions of
inputs topics.

Re: MirrorMaker 2.0 question

[Miguel Ángel Fernández Fernández]

I got it to work by raising a kafka connect from which to launch the mm2.
Silly question.

El lun, 20 mar 2023 a las 23:09, Miguel Ángel Fernández Fernández (<
miguelangelprogramac...@gmail.com>) escribió:

> I have two clusters up on the same machine with docker-compose
>
> services:
>   zookeeper-lab:
> image: "bitnami/zookeeper:3.8.1"
> restart: always
> environment:
>   ZOO_PORT_NUMBER: 2183
>   ALLOW_ANONYMOUS_LOGIN: "yes"
> ports:
> - "2183:2183"
> - "2886:2888"
> - "3886:3888"
>   kafka-confluent:
> image: "confluentinc/cp-kafka:6.2.0"
> restart: always
> environment:
> - KAFKA_ZOOKEEPER_CONNECT=localhost:2183
> - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:29092
> network_mode: "host"
> volumes:
>   -
> "/home/miguelangel/Documents/KafkaLab/mm2.properties:/var/lib/kafka/data/mm2.properties"
>
> and mm2 is running on one of them.
>
> Regarding the log, I don't see anything that seems suspicious to me. Not
> ERROR logs.
>
> [2023-03-20 22:03:14,290] WARN The configuration '
> metrics.context.connect.group.id' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:380)
> [2023-03-20 22:03:14,290] INFO Created topic (name=mm2-configs.B.internal,
> numPartitions=1, replicationFactor=1, replicasAssignments=null,
> configs={cleanup.policy=compact}) on brokers at 127.0.0.1:29092
> (org.apache.kafka.connect.util.TopicAdmin:389)
> [2023-03-20 22:03:14,290] WARN The configuration 'group.id' was supplied
> but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:380)
> [2023-03-20 22:03:14,290] WARN The configuration 'status.storage.topic'
> was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:380)
>
> Cheers,
>
> Miguel Ángel
>
>
> El lun, 20 mar 2023 a las 22:44, Chris Egerton ()
> escribió:
>
>> Mi Miguel,
>>
>> How many nodes are you running MM2 with? Just one?
>>
>> Separately, do you notice anything at ERROR level in the logs?
>>
>> Cheers,
>>
>> Chris
>>
>> On Mon, Mar 20, 2023 at 5:35 PM Miguel Ángel Fernández Fernández <
>> miguelangelprogramac...@gmail.com> wrote:
>>
>> > Hello,
>> >
>> > I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple
>> of
>> > kafka clusters, I think everything is set up correctly. However, when I
>> run
>> >
>> > /bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties
>> >
>> > the result I get is the creation of the topics
>> >
>> > mm2-configs.A.internal,
>> > mm2-offsets.A.internal,
>> > mm2-status.A.internal
>> >
>> > but not of the specific topics found in cluster A (topic1).
>> >
>> > The mm2.properties configuration is basically a copy of
>> >
>> >
>> >
>> https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
>> >
>> > modifying bootstrap servers
>> >
>> > A.bootstrap.servers = 127.0.0.1:29092
>> > B.bootstrap.servers = 127.0.0.1:29093
>> >
>> > Can anyone advise me with this problem? Is it possible that some
>> > configuration in mm2.properties is missing? Are there any real
>> requirements
>> > regarding the configuration of the kafka clusters that may be
>> overlooked?
>> >
>> > Thank you for your help,
>> > Miguel Ángel
>> >
>>
>