I would recommend using the site to site task instead of your own because it will give you a very scalable way to asynchronously handle your provenance tracking. There's nothing preventing you from having the tracking instance of NiFi be responsible for publishing the events to your tracking system. A lot of people do this with Elastic and Kibana with very good results.
On Fri, Oct 18, 2019 at 12:01 PM Tim Dean <tim.d...@gmail.com> wrote: > I would like to implement some custom monitoring logic that captures > certain information from the provenance repository. It would be similar in > some ways to the existing SiteToSiteProvenanceReporting task, but it will > not be sending information to another NiFi node but instead sending things > to an in-house tracking service that we use. > > There is a little information about writing custom reporting tasks in the > NiFi developer documentation, but it’s pretty limited in what it covers. I > want to make sure I capture all the events that come in, and I want to be > resilient to situations when the NiFi node is rebooted, or certain > components are stopped temporarily, etc. > > To get an idea of how to best accomplish this I’ve been looking at the > source code for SiteToSiteProvenanceReportingTask on GitHub. It appears to > push a delegate a lot of the details of event capturing to the > org.apache.nifi.reporting.util.provenance.ProvenanceEventConsumer class. > I’ve poked around that class to see what it does, and I think I could use > it In my own custom reporting class to make things easier. However, that > class doesn’t seem to be documented publicly anywhere, and isn’t mentioned > in the developer guide. I am concerned that it might be necessary to treat > this as a private class that may change from release to release. > > How would you recommend that I proceed to create my own provenance event > reporting task? Should I mimic SiteToSiteProvenanceReportingTask and built > it on top of the ProvenanceEventConsumer class? Should I instead pull out > the parts of ProvenanceEventConsumer that I need and create my own version > of that class that I can control? > Any other approaches I should consider instead of the path I’m heading > down now? > >
