Hi Mark, I tried the combination of FIFO and setting the back pressure to 10k, but it didn't preserve the order.
Thanks Edi On Wed, Dec 13, 2023 at 3:47 PM Mark Payne <marka...@hotmail.com> wrote: > Hey Edi, > > By default, nifi doesn’t preserve ordering but you can have it do so by > updating the connection’s configuration and adding the First In First Out > Prioritizer. > > Also of note you will want to keep the backpressure threshold set to > 10,000 objects rather than increasing it as shown in the image. > > Thanks > Mark > > > Sent from my iPhone > > On Dec 13, 2023, at 8:19 AM, edi mari <edim2...@gmail.com> wrote: > > > > Hello , > I'm using NIFI v1.20.0 to replicate 250 million messages between Kafka > topics. > The problem is that NIFI replicates messages in a non-sequential order, > resulting in the destination topic storing messages differently than the > source topic. > > for example > *source topic - partition 0* > offset:5 key:a value:v1 > offset:6 key:a value:v2 > offset:7 key:a value:v3 > > *destination topic - partition 0* > offset:5 key:a value:v2 > offset:6 key:a value:v1 > offset:7 key:a value:v3 > > The topics are configured with a cleanup policy: compact. > > I'm using ConsumeKafka and PublishKafka processors to replicate topics. > > <image.png> > > <image.png> > > <image.png> > > <image.png> > > <image.png> > > Thanks > Edi > >
