Hello everyone,
I was wondering if it were possible for excessive use of the provenance api
would cause flowfile processing to slow down and even come to a halt? My
test setup queries the provenance API to see if all flowfiles that
descended from a given input file have completed processing. This
Eric,
short version: Provenance queries can absolutely take away CPU time from
the flow.
longer version:
They get the same priority as any other thread in nifi. Once prov queries
are being executed they use CPU. I would strongly advise against any
blending of the flow execution with provenance
Hi Joe,
Thanks for the explanation, is there a Jira ticket for for a job management
mechanism? Is this a priority for a coming release?
Is there a lag between events occurring and them becoming searchable, if so
what settings help control this lag?
Thanks,
Eric
On Tue, Nov 3, 2020 at 2:35 PM Jo
Hello,
I agree it's not the best idea to use the provenance data constantly to
check when a test file is finally done being processed. Do you know if
anything came out in 1.12.0 or 1.12.1 which would cause provenance queries
to have a greater impact? We recently upgraded nifi among other changes a
Eric
Nope Im not aware of anything specific.
Thanks
Joe
On Wed, Nov 4, 2020 at 9:54 AM Eric Secules wrote:
> Hello,
>
> I agree it's not the best idea to use the provenance data constantly to
> check when a test file is finally done being processed. Do you know if
> anything came out in 1.12.0
Not sure if this would be helpful for you, but I wonder if you could use
the new process group concurrency controls introduced in 1.12.0 to help
determine when processing is complete. You could set a process group to
allow 1 flow file in at a time and then batch output to wait for all flow
files to
I dont think this solution would work for me. I am wondering whether the
rollover time has any affect on when I am able to query the events. Does
the provenance log file need to roll over before it's indexed?
On Wed, Nov 4, 2020 at 11:21 AM Bryan Bende wrote:
> Not sure if this would be helpful
