Scott et al, There are a number of options for monitoring flows, including backpressure and even backpressure prediction:
1) The REST API for metrics. As you point out, it's subject to the same authz/authn as any other NiFi operation and doesn't sound like it will work out for you. 2) The Prometheus scrape target via the REST API. The issue would be the same as #1 I presume. 3) PrometheusReportingTask. This is similar to the REST scrape target but isn't subject to the usual NiFi authz/authn stuff, however it does support SSL/TLS for a secure solution (and is also a "pull" approach despite it being a reporting task) 4) QueryNiFiReportingTask. This is not included with the NiFi distribution but can be downloaded separately, the latest version (1.14.0) is at [1]. I believe this is what Andrew was referring to when he mentioned being able to run SQL queries over the information, you can do something like "SELECT * FROM CONNECTION_STATUS_PREDICTIONS WHERE predictedTimeToBytesBackpressureMillis < 10000". This can be done either as a push or pull depending on the Record Sink you choose. A SiteToSiteReportingRecordSink, KafkaRecordSink, or LoggingRecordSink results in a push (to NiFi, Kafka, or nifi-app.log respectively), where a PrometheusRecordSink results in a pull the same as #2 and #3. There's even a ScriptedRecordSink where you can write your own script to put the results where you want them. 5) The other reporting tasks. These have been mentioned frequently in this thread so no need for elaboration here :) Regards, Matt [1] https://repository.apache.org/content/repositories/releases/org/apache/nifi/nifi-sql-reporting-nar/1.14.0/ On Wed, Jul 21, 2021 at 5:58 PM scott <[email protected]> wrote: > > Great comments all. I agree with the architecture comment about push > monitoring. I've been monitoring applications for more than 2 decades now, > but sometimes you have to work around the limitations of the situation. It > would be really nice if NiFi had this logic built-in, and frankly I'm > surprised it is not yet. I can't be the only one who has had to deal with > queues filling up, causing problems downstream. NiFi certainly knows that the > queues fill up, they change color and execute back-pressure logic. If it > would just do something simple like write a log/error message to a log file > when this happens, I would be good. > I have looked at the new metrics and reporting tasks but still haven't found > the right thing to do to get notified when any queue in my instance fills up. > Are there any examples of using them for a similar task you can share? > > Thanks, > Scott > > On Wed, Jul 21, 2021 at 11:29 AM [email protected] <[email protected]> > wrote: >> >> In general, it is a bad architecture to do monitoring via pull request. You >> should always push. I recommend a look at the book "The Art of Monitoring" >> by James Turnbull. >> >> I also recommend the very good articles by Pierre Villard on the subject of >> NiFi monitoring at >> https://pierrevillard.com/2017/05/11/monitoring-nifi-introduction/. >> >> Hope this helps. >> >> Mit freundlichen Grüßen / best regards >> Kay-Uwe Moosheimer >> >> Am 21.07.2021 um 16:45 schrieb Andrew Grande <[email protected]>: >> >> >> Can't you leverage some of the recent nifi features and basically run sql >> queries over NiFi metrics directly as part of the flow? Then act on it with >> a full flexibility of the flow. Kinda like a push design. >> >> Andrew >> >> On Tue, Jul 20, 2021, 2:31 PM scott <[email protected]> wrote: >>> >>> Hi all, >>> I'm trying to setup some monitoring of all queues in my NiFi instance, to >>> catch before queues become full. One solution I am looking at is to use the >>> API, but because I have a secure NiFi that uses LDAP, it seems to require a >>> token that expires in 24 hours or so. I need this to be an automated >>> solution, so that is not going to work. Has anyone else tackled this >>> problem with a secure LDAP enabled cluster? >>> >>> Thanks, >>> Scott
