Mark, thanks for the detailed information. It’s clear to us now. Boris, also
thanks for your input.
--
Kind Regards
Dries Van Autreve
From: Mark Payne
Reply to: "users@nifi.apache.org"
Date: Thursday, 1 April 2021 at 18:11
To: "users@nifi.apache.org"
Subject: Re: Strict order of flow files
thanks Mark, very nice feature indeed, but we cannot upgrade
anymore because of new ZK version requirement. but it is another story and
issue specific to us (have to stay with CDH 6.2)
On Thu, Apr 1, 2021 at 12:11 PM Mark Payne wrote:
> Boris,
>
> To be clear, the options to rollback on failure
Boris,
To be clear, the options to rollback on failure instead of routing to a failure
relationship were added specifically for this use case of consuming CDC events
(typically from Kafka). But they were only recently added, in either 1.12 or
1.13. That should make things simpler.
Thanks
our use was exactly that - CDC events that we had to apply in the order.
FirstInFirstOutPrioritizer was not enough because some flowfiles will fail
but Nifi will process next one. All examples I've seen had all kind of
issues but mostly enforcing order. So our only choice was our custom FIFO
Dries,
The short answer is that, depending on your source, destination, and what
you’re doing in between, it is sometimes (but not always) possible. But not
particularly simple.
The longer version:
NiFi doesn’t strive to provide strict ordering guarantees. Rather, it strives
to provide data
Hi All.
I'm developing a NIFI flow to fetch csv files from a S3 bucket and write
those data to Apache kudu. I need to understand a few things regarding this
flow.
1. Does ListS3 processor list down S3 objects in alphabetical order? Or
is there any other specific order it uses to list down
We ended up building a simple groovy processor that will use mysql db to
queue up flowfiles. If a flowfile A fails, flowfile B would sit in a queue
until we address an issue with flowfile A. We also used back pressure
feature to slow down upstream Kafka consumers.
After playing with wait/notify
We work with a trigger mechanism using wait/notify to ensure that a sequence
will be maintained. Basically a new flowfile will only be worked upon if a
corresponding special triggerflowfile has been received. As soon as the
flowfile has been 'finished' the flow itself creates the trigger (kind
Hello Harald
Thanks for the swift reply.
If I read about this feature, I see in the user guide:
"This means that in a 5-node cluster, for example, there may be up to 5
incoming FlowFiles being processed simultaneously."
In the tooltip: "Only a single FlowFile is to be allowed to enter the
I suppose the feature 'Ability to specify group level flow file concurrency -
for instance run a single flow file end to end before running another for
traditional job handling' available from Version 1.12 upward should be helpful
here (have not tried myself yet)
-Ursprüngliche
Hello all
We recently started using NiFi and we were wondering if strict order of
processing flow files in a cluster could be guaranteed by NiFi.
One of the use cases is as following: messages arrive in a specific order, go
through a simple flow with some basic transformations and are written
11 matches
Mail list logo
