Richard, Piggy backing on Andy's comment, you could also use getHDFS to seek files based on the date/time or whatever parameters then use the failure relationship to post to Slack or send an email.
On Tue, Dec 6, 2016 at 10:23 AM, Andy LoPresto <alopre...@apache.org> wrote: > Richard, > > In addition to the flow that loads those files and performs whatever > follow-on transformation and routing, you can have a parallel flow which > monitors the directory and uses cron scheduling to run at a specific time > (0900 daily) and is configured to expect a threshold number of files and in > a specific naming pattern. This separate flow can route to email > processors, etc. to alert relevant teams. If ListHDFS does not meet your > requirements for monitoring, ExecuteScript is very versatile in performing > non-standard actions within NiFi. > > You should also take a look at the monitoring capabilities of the NiFi > UI/API such as data provenance, queue interaction, and > statistics/monitoring. These features provide live monitoring for your > administrators/data flow managers. You can read from these > resources/queries via external monitoring tools if necessary as well. > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Dec 6, 2016, at 9:29 AM, Richard Duarte <richard.c.dua...@gmail.com> > wrote: > > Hello, > > I'm working on a project that I'd like to use Nifi for, however I'm trying > to figure out if there's a way to alert on the availability of data being > provided correctly to an agreement. > > Example: > 1. Team X has agreed to provide files in directory Y in HDFS > 2. Team X has agreed to provide those files once per day by 9 AM PST > 3. Team X has agreed to name those files following naming convention > [x]-[date]-[abc] > > > Is there any way for Nifi to alert on Team X violating any of those > agreements? I.E. can we get Nifi to alert on the files not being there, > the files not being there in time, and/or the files not following the > correct naming conventions? > > If not, are there any other tools that tie into Nifi to provide help on > the Data SLA? > > Thanks, > Richard > > > -- [image: Payoff, Inc.] <http://www.payoff.com/> Jeremy Farbota Software Engineer, Data jfarb...@payoff.com <em...@payoff.com> • (217) 898-8110 <(949)+430-0630> I'm a Storyteller. Discover your Financial Personality! <https://www.payoff.com/quiz> [image: Facebook] <https://www.facebook.com/payoff> [image: Twitter] <https://www.twitter.com/payoff> [image: Linkedin] <https://www.linkedin.com/company/payoff-com>
