Re: Implementing a custom provenance reporting task

2019-10-30 Thread Tim Dean
Thanks Matt - and sorry for the delayed response I think this approach would work well for me, so I look forward to trying it out once it has been released. -Tim > On Oct 18, 2019, at 1:49 PM, Matt Burgess wrote: > > Tim, > > The "secondary flow" issue is something I wanted to address as

Re: Implementing a custom provenance reporting task

2019-10-18 Thread Matt Burgess
Tim, The "secondary flow" issue is something I wanted to address as well, so I decoupled the formatting of data from the transmission of that data into a new paradigm/service I'm calling a RecordServiceSink. A ReportingTask can use RecordServiceSink to allow the user to choose where the reporting

Re: Implementing a custom provenance reporting task

2019-10-18 Thread Tim Dean
Thanks Mike, I understand what you are saying but I am really trying to avoid having a secondary flow in NiFi if I can avoid it. It seems like NiFi was designed to allow this kind of custom reporting task and that I should ideally be able to do this without using the NiFi-provided S2S

Re: Implementing a custom provenance reporting task

2019-10-18 Thread Mike Thomsen
I would recommend using the site to site task instead of your own because it will give you a very scalable way to asynchronously handle your provenance tracking. There's nothing preventing you from having the tracking instance of NiFi be responsible for publishing the events to your tracking

Implementing a custom provenance reporting task

2019-10-18 Thread Tim Dean
I would like to implement some custom monitoring logic that captures certain information from the provenance repository. It would be similar in some ways to the existing SiteToSiteProvenanceReporting task, but it will not be sending information to another NiFi node but instead sending things to