Hi Mark ,
I appreciate your advice. Upon examination, implementing the queue as a
"First In First Out" prioritizer and configuring the load balancing
strategy to "Partition by attribute" with the "kafka.partition" attribute
has proven effective in maintaining the order.
Thanks
Edi
On Fri, Dec 15, 2023 at 9:53 PM Mark Payne <marka...@hotmail.com> wrote:
> Edi,
>
> Looking at your config again, you’ll want to also ensure that on the
> Publisher, you set the partition to `${kafka.partition}` so that it goes to
> the same partition on the destination system. You’ll also went to ensure
> that you set “Failure Strategy” to “Rollback” - otherwise any failure would
> route to ‘failure’ relationship and change the ordering. You’ll also need
> to limit the concurrent tasks on the publisher to 1 concurrent task, to
> ensure that you’re not sending multiple FlowFiles out of order.
>
> Thanks
> -Mark
>
>
> On Dec 15, 2023, at 2:26 AM, edi mari <edim2...@gmail.com> wrote:
>
> Hi Mark,
> I tried the combination of FIFO and setting the back pressure to 10k, but
> it didn't preserve the order.
>
> Thanks
> Edi
>
> On Wed, Dec 13, 2023 at 3:47 PM Mark Payne <marka...@hotmail.com> wrote:
>
>> Hey Edi,
>>
>> By default, nifi doesn’t preserve ordering but you can have it do so by
>> updating the connection’s configuration and adding the First In First Out
>> Prioritizer.
>>
>> Also of note you will want to keep the backpressure threshold set to
>> 10,000 objects rather than increasing it as shown in the image.
>>
>> Thanks
>> Mark
>>
>>
>> Sent from my iPhone
>>
>> On Dec 13, 2023, at 8:19 AM, edi mari <edim2...@gmail.com> wrote:
>>
>>
>>
>> Hello ,
>> I'm using NIFI v1.20.0 to replicate 250 million messages between Kafka
>> topics.
>> The problem is that NIFI replicates messages in a non-sequential order,
>> resulting in the destination topic storing messages differently than the
>> source topic.
>>
>> for example
>> *source topic - partition 0*
>> offset:5 key:a value:v1
>> offset:6 key:a value:v2
>> offset:7 key:a value:v3
>>
>> *destination topic - partition 0*
>> offset:5 key:a value:v2
>> offset:6 key:a value:v1
>> offset:7 key:a value:v3
>>
>> The topics are configured with a cleanup policy: compact.
>>
>> I'm using ConsumeKafka and PublishKafka processors to replicate topics.
>>
>> <image.png>
>>
>> <image.png>
>>
>> <image.png>
>>
>> <image.png>
>>
>> <image.png>
>>
>> Thanks
>> Edi
>>
>>
>
