Richard, In addition to the flow that loads those files and performs whatever follow-on transformation and routing, you can have a parallel flow which monitors the directory and uses cron scheduling to run at a specific time (0900 daily) and is configured to expect a threshold number of files and in a specific naming pattern. This separate flow can route to email processors, etc. to alert relevant teams. If ListHDFS does not meet your requirements for monitoring, ExecuteScript is very versatile in performing non-standard actions within NiFi.
You should also take a look at the monitoring capabilities of the NiFi UI/API such as data provenance, queue interaction, and statistics/monitoring. These features provide live monitoring for your administrators/data flow managers. You can read from these resources/queries via external monitoring tools if necessary as well. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Dec 6, 2016, at 9:29 AM, Richard Duarte <richard.c.dua...@gmail.com> wrote: > > Hello, > > I'm working on a project that I'd like to use Nifi for, however I'm trying to > figure out if there's a way to alert on the availability of data being > provided correctly to an agreement. > > Example: > 1. Team X has agreed to provide files in directory Y in HDFS > 2. Team X has agreed to provide those files once per day by 9 AM PST > 3. Team X has agreed to name those files following naming convention > [x]-[date]-[abc] > > > Is there any way for Nifi to alert on Team X violating any of those > agreements? I.E. can we get Nifi to alert on the files not being there, the > files not being there in time, and/or the files not following the correct > naming conventions? > > If not, are there any other tools that tie into Nifi to provide help on the > Data SLA? > > Thanks, > Richard
signature.asc
Description: Message signed with OpenPGP using GPGMail
