Is there a plugin hook that can be written to allow one to see the current state of all flows?
One thought I have is that this process could look for a configured property, like volatile.delete.strategy that a hook on startup could delete this based on given factors. One use case would be that something being ingested if is still in the pipeline for over a given period after a restart it may be considered stale. I could think of other use cases similar to this. > On 12 Jan 2018, at 15:48, 尹文才 <batman...@gmail.com> wrote: > > Hi Andrew, sorry I didn't follow your idea, could you please elaborate with > more details? > What I want to do is to be able to clear all the FlowFiles when NiFi dies > unexpectedly and restarts itself. > > Regards, > Ben > > 2018-01-12 12:44 GMT+08:00 Andrew Grande <apere...@gmail.com>: > >> Perhaps you could delete the repository directories when you need to >> restart with no data? >> >>> On Thu, Jan 11, 2018, 9:16 PM 尹文才 <batman...@gmail.com> wrote: >>> >>> Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned, >> if I >>> switch to use VolatileFlowFileRepository, will NiFi swap out all the >> other >>> FlowFiles to disk if a queue is already full? >>> Is it just simply keeping all FlowFiles in memory? >>> >>> Regards, >>> Ben >>> >>> 2018-01-12 12:07 GMT+08:00 尹文才 <batman...@gmail.com>: >>> >>>> Thanks Mark, my case is that I'm using NiFi to do some ETL work and >> it's >>>> possible that NiFi dies unexpectedly due to lack of system resources. >>> After >>>> NiFi restarts itself, >>>> I will re-extract all the data from database and re-perform all the >>>> operations, so I need to clear all possible FlowFiles that might exist >> in >>>> any queue. >>>> >>>> Regards, >>>> Ben >>>> >>>> 2018-01-12 11:49 GMT+08:00 Mark Payne <marka...@hotmail.com>: >>>> >>>>> Ben, >>>>> >>>>> I have to admit - that’s kind of an odd request :) I’m curious what >> the >>>>> use case is, if you can share? >>>>> >>>>> Regardless, the easiest way would be to update nifi.properties so that >>>>> the FlowFile repo that is used is the VolatileFlowFileRepository. This >>>>> would avoid writing the FlowFile state to disk, so ok restart you will >>> lose >>>>> all FlowFiles. The content will still be present, but nifi will delete >>> it >>>>> all on startup because there is no FlowFile associated with it. >>>>> >>>>> I’m on my phone right now so can’t easily tell you the exact name of >> the >>>>> property to change but you’ll probably find it pretty quickly. The >> Admin >>>>> Guide may well explain the different repositories as well. >>>>> >>>>> Thanks >>>>> -Mark >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Jan 11, 2018, at 10:31 PM, 尹文才 <batman...@gmail.com> wrote: >>>>>> >>>>>> Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi >> is >>>>>> restarted, but I don't know the correct way to do this. I checked >> all >>>>>> NiFi's guide documentation, >>>>>> it seems there're 2 possible solutions: >>>>>> 1. write a custom notification service: a notification service could >>> be >>>>>> notified when NiFi is restarted and then inside the service, delete >>> all >>>>> the >>>>>> files inside content_repository, flowfile_repository and >>>>>> provenance_repository. >>>>>> I know there're now 2 existing services: email and http. But I'm >> not >>>>>> quite sure how to correctly write one and deploy it into my NiFi >>>>>> environment, is there a tutorial on writing one notification >> service? >>>>>> >>>>>> 2. I know from the developer guide that by using the annotation >>>>> @Shutdown >>>>>> in a custom processor, the method could be called when NiFi is >>>>> successfully >>>>>> shut down. The problem with this approach is the method could >>>>>> not be guaranteed to be called when NiFi dies unexpectedly. >>>>>> >>>>>> Does anyone know what is the correct way to implement it? Thanks. >>>>>> >>>>>> Regards, >>>>>> Ben >>>>> >>>> >>>> >>> >>
