Re: Stopping processor after MAX number of retries
There's an example template on the Example Dataflow Templates page [1] called Retry_Count_Loop.xml [2], not sure what components it uses though. Regards, Matt [1] https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates [2] https://cwiki.apache.org/confluence/download/attachments/57904847/Retry_Count_Loop.xml?version=1=1433271239000=v2 On Mon, Mar 1, 2021 at 1:07 PM James McMahon wrote: > > Any recommended examples of successful DistributedMapCache implementations? > > On Fri, Feb 26, 2021 at 11:30 AM Andrew Grande wrote: >> >> I saw it several times and I have a strong conviction this is an >> anti-pattern. A dataflow must not mess with start/stop state of processors >> or process groups. >> >> Instead, a flow is always running and one puts a conditional check to either >> not get the data in or reroute/deny it. >> >> If you need to coordinate between processors, use wait/notify. >> >> If you need to coordinate across nodes, consider DistributedMapCache with a >> variety of implementations. >> >> Finally, can use external stores directly if need to coordinate with other >> systems. >> >> These become part of the flow design. >> >> Andrew >> >> On Fri, Feb 26, 2021, 7:42 AM Tomislav Novosel >> wrote: >>> >>> Hi guys, >>> >>> >>> >>> I want to stop the processor after exceeding maximum number of retries. >>> >>> For that I'm using RetryFlowFile processor, after 5 times of retry, it >>> routes >>> >>> flowfile to retries_exceeded. >>> >>> >>> >>> When that kicks in, I want to stop the processor which was retried 5 times. >>> >>> >>> >>> What is the best approach? I have few ones: >>> >>> >>> >>> Execute shell script which sends request to nifi-api to set processor state >>> to STOPPED >>> Put InvokeHTTP processor to send request >>> >>> >>> >>> The downside is, what if processor-id changes, e.g. deploying to another >>> env or nifi restart, not sure about that. >>> >>> Also, it is nifi cluster with authentication and SSL, so it complicates the >>> things. >>> >>> >>> >>> Maybe someone has much simpler approach, with backpressure or something. >>> >>> >>> >>> Regards, >>> >>> Tom
Re: Stopping processor after MAX number of retries
Any recommended examples of successful DistributedMapCache implementations? On Fri, Feb 26, 2021 at 11:30 AM Andrew Grande wrote: > I saw it several times and I have a strong conviction this is an > anti-pattern. A dataflow must not mess with start/stop state of processors > or process groups. > > Instead, a flow is always running and one puts a conditional check to > either not get the data in or reroute/deny it. > > If you need to coordinate between processors, use wait/notify. > > If you need to coordinate across nodes, consider DistributedMapCache with > a variety of implementations. > > Finally, can use external stores directly if need to coordinate with other > systems. > > These become part of the flow design. > > Andrew > > On Fri, Feb 26, 2021, 7:42 AM Tomislav Novosel < > tomislav.novo...@clearpeaks.com> wrote: > >> Hi guys, >> >> >> >> I want to stop the processor after exceeding maximum number of retries. >> >> For that I'm using RetryFlowFile processor, after 5 times of retry, it >> routes >> >> flowfile to retries_exceeded. >> >> >> >> When that kicks in, I want to stop the processor which was retried 5 >> times. >> >> >> >> What is the best approach? I have few ones: >> >> >> >>- Execute shell script which sends request to nifi-api to set >>processor state to STOPPED >>- Put InvokeHTTP processor to send request >> >> >> >> The downside is, what if processor-id changes, e.g. deploying to another >> env or nifi restart, not sure about that. >> >> Also, it is nifi cluster with authentication and SSL, so it complicates >> the things. >> >> >> >> Maybe someone has much simpler approach, with backpressure or something. >> >> >> >> Regards, >> >> Tom >> >
Re: Stopping processor after MAX number of retries
I saw it several times and I have a strong conviction this is an anti-pattern. A dataflow must not mess with start/stop state of processors or process groups. Instead, a flow is always running and one puts a conditional check to either not get the data in or reroute/deny it. If you need to coordinate between processors, use wait/notify. If you need to coordinate across nodes, consider DistributedMapCache with a variety of implementations. Finally, can use external stores directly if need to coordinate with other systems. These become part of the flow design. Andrew On Fri, Feb 26, 2021, 7:42 AM Tomislav Novosel < tomislav.novo...@clearpeaks.com> wrote: > Hi guys, > > > > I want to stop the processor after exceeding maximum number of retries. > > For that I'm using RetryFlowFile processor, after 5 times of retry, it > routes > > flowfile to retries_exceeded. > > > > When that kicks in, I want to stop the processor which was retried 5 times. > > > > What is the best approach? I have few ones: > > > >- Execute shell script which sends request to nifi-api to set >processor state to STOPPED >- Put InvokeHTTP processor to send request > > > > The downside is, what if processor-id changes, e.g. deploying to another > env or nifi restart, not sure about that. > > Also, it is nifi cluster with authentication and SSL, so it complicates > the things. > > > > Maybe someone has much simpler approach, with backpressure or something. > > > > Regards, > > Tom >
Stopping processor after MAX number of retries
Hi guys, I want to stop the processor after exceeding maximum number of retries. For that I'm using RetryFlowFile processor, after 5 times of retry, it routes flowfile to retries_exceeded. When that kicks in, I want to stop the processor which was retried 5 times. What is the best approach? I have few ones: * Execute shell script which sends request to nifi-api to set processor state to STOPPED * Put InvokeHTTP processor to send request The downside is, what if processor-id changes, e.g. deploying to another env or nifi restart, not sure about that. Also, it is nifi cluster with authentication and SSL, so it complicates the things. Maybe someone has much simpler approach, with backpressure or something. Regards, Tom
