Eric Nope Im not aware of anything specific.
Thanks Joe On Wed, Nov 4, 2020 at 9:54 AM Eric Secules <esecu...@gmail.com> wrote: > Hello, > > I agree it's not the best idea to use the provenance data constantly to > check when a test file is finally done being processed. Do you know if > anything came out in 1.12.0 or 1.12.1 which would cause provenance queries > to have a greater impact? We recently upgraded nifi among other changes and > I'm trying to figure out what's the cause for many of our tests suddenly > timing out. > > Thanks, > Eric > > On Tue., Nov. 3, 2020, 6:18 p.m. Eric Secules, <esecu...@gmail.com> wrote: > >> Hi Joe, >> >> Thanks for the explanation, is there a Jira ticket for for a job >> management mechanism? Is this a priority for a coming release? >> >> Is there a lag between events occurring and them becoming searchable, if >> so what settings help control this lag? >> >> Thanks, >> Eric >> >> On Tue, Nov 3, 2020 at 2:35 PM Joe Witt <joe.w...@gmail.com> wrote: >> >>> Eric, >>> >>> short version: Provenance queries can absolutely take away CPU time from >>> the flow. >>> >>> longer version: >>> They get the same priority as any other thread in nifi. Once prov >>> queries are being executed they use CPU. I would strongly advise against >>> any blending of the flow execution with provenance queries. What you're >>> trying to do though is a great idea and aligns to what Mark Payne has >>> talked about previous as a job management mechanism. This can/should be >>> done without provenance itself. >>> >>> >>> Thanks >>> >>> >>> >>> On Tue, Nov 3, 2020 at 3:31 PM Eric Secules <esecu...@gmail.com> wrote: >>> >>>> Hello everyone, >>>> >>>> I was wondering if it were possible for excessive use of the provenance >>>> api would cause flowfile processing to slow down and even come to a halt? >>>> My test setup queries the provenance API to see if all flowfiles that >>>> descended from a given input file have completed processing. This leads to >>>> a lot of requests to the provenance API and I fear it may be too much for >>>> the test VM to handle (4 vCPU and 16GB RAM). Do provenance query threads >>>> take precedence over scheduled task threads? >>>> >>>> Thanks, >>>> Eric >>>> >>>
