Re: ConsumeKafka to PublishKafka doesn't keep the order of the messages in the destination topic

[Willem Pretorius]

Hello Edi

NiFi is amazing for many use cases but this feels like swimming upstream 
so to speak.


Have you considered Kafka Connect?

Define a Sink Connector to dump source Kafka topic to disk/s3 and then 
define a Source Connector to read it back to a destination topic.
Have not played with the S3 connector yet but it does seem to support 
for Kafka partitions.



On 2023/12/15 11:26, edi mari wrote:
Hi Mark,
I tried the combination of FIFO and setting the back pressure to 10k, 
but it didn't preserve the order.

Thanks
Edi

On Wed, Dec 13, 2023 at 3:47 PM Mark Payne <marka...@hotmail.com> wrote:

    Hey Edi,

    By default, nifi doesn’t preserve ordering but you can have it do
    so by updating the connection’s configuration and adding the First
    In First Out Prioritizer.

    Also of note you will want to keep the backpressure threshold set
    to 10,000 objects rather than increasing it as shown in the image.

    Thanks
    Mark


    Sent from my iPhone

    On Dec 13, 2023, at 8:19 AM, edi mari <edim2...@gmail.com> wrote:

    

    Hello ,
    I'm using NIFI v1.20.0 to replicate 250 million messages between
    Kafka topics.
    The problem is that NIFI replicates messages in a non-sequential
    order, resulting in the destination topic storing messages
    differently than the source topic.

    for example
    *source topic - partition 0*
    offset:5 key:a value:v1
    offset:6 key:a value:v2
    offset:7 key:a value:v3

    *destination topic - partition 0*
    offset:5 key:a value:v2
    offset:6 key:a value:v1
    offset:7 key:a value:v3

    The topics are configured with a cleanup policy: compact.

    I'm using ConsumeKafka and PublishKafka processors to replicate
    topics.

    <image.png>

    <image.png>

    <image.png>

    <image.png>

    <image.png>

    Thanks
    Edi