Thank your for the info. I guess I will try a small PoC and see how it goes.
On Mon, May 15, 2017 at 8:13 PM, Pierre Villard <pierre.villard...@gmail.com> wrote: > Hi Andy, > > Regarding bulletins, you can use the new SiteToSiteBulletinReportingTask [1] > to achieve what you are expecting (in NiFi 1.2.0). For provenance, you can > definitely use the REST API but could also be interested by the > SiteToSiteProvenanceReportingTask [2] with the addition of NIFI-3859 [3]. > You may also be interested by this article [4]. > > From a high-level perspective all what you described is perfectly doable. > Obviously, it'd be necessary to go into the details of what are the > monitoring tools you want to integrate with but, again, NiFi sounds totally > suitable to your use case. > > [1] > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-site-to-site-reporting-nar/1.2.0/org.apache.nifi.reporting.SiteToSiteBulletinReportingTask/index.html > [2] > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-site-to-site-reporting-nar/1.2.0/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html > [3] https://issues.apache.org/jira/browse/NIFI-3859 > [4] > https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/ > > Pierre > > > > 2017-05-15 19:54 GMT+02:00 Andy Yar <andyya...@gmail.com>: >> >> Hello, >> I'd like to know if NiFi is suitable for my slightly unusual use case. >> >> Basically, the system should route files/events/etc. from various >> sources, validate, process them and route somewhere. However, I need >> to do this in a reliable way. This means, the system needs to have >> extensive monitoring/reprocessing capabilities. Various errors can >> occur in the system - an outbound destination gets unavailable, a >> malformed event doesn't match schema, an exception occurred during >> processing, etc. In my case, the system has to allow discovering + >> fixing + reprocessing of any failed event. >> >> I've figured that it is possible to route every processor's "failure" >> state to a dead-letter-queue, fire up a notification (Slack, API call, >> etc.), reroute the DLQ to the input again and so on. It seems to be a >> bit complicated with adding a few retry steps before sending to the >> DLQ everywhere but OK. >> >> However, I'm a bit worried about the monitoring side. For instance, is >> NiFi, in its current state, able/intended to persist Bulletin Board >> messages? I can imagine that after a operator sees an error on the >> Bulletin Board he/she traces the flow file, fixes the issue and marks >> the incident as solved. Currently the BB messages seems to be a bit >> temporary. I guess the intended way is to create a custom Reporting >> task in order to handle this. >> >> The Data Provenance itself is great, but still I can't imagine it >> being usable without a major customization. In my case I would like to >> monitor mostly a set of "inbound oriented" processors. So it would be >> possible to list eg. all current events on inbound processors in a >> (pseudo) realtime way. Again, I guess custom API/tasks could handle >> this. >> >> To summarize: is NiFi usable for this particular use case? >> >> Thanks for any answers and also for your effort with this project! >> >> A. > >
