Thanks, Shawn – I had seen your post to the mailing list and that’s where I got the gist. It’s really close to what I need to do, but it seems to be designed for a scenario where the data is arriving on a given interval. In my case, I could have multiple messages arriving at the Wait processor simultaneously. In my testing, if 3 messages are queued, all 3 will be passed through the gate immediately once the gate is open. What I would really want is for each message to close the gate, and for the gate to stay closed until that message has been processed. I’m racking my brain trying to think of a way to do that. I’ve also tried to implement the 0/1 gate that’s mentioned in the documentation, but I’m not sure how it works. If I set the target count to 0, all wait files pass through immediately. I tried setting the ‘Releasable FlowFile Count’ to 0 as well, but that didn’t work either.
Thanks, Dave From: Shawn Weeks <swe...@weeksconsulting.us> Sent: Saturday, May 18, 2019 8:46 AM To: users@nifi.apache.org Subject: Re: Open / Close Gate examples? See this example as I had a lot of questions about wait and notify earlier and this helped a lot. https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fijokarumawak%2F9e1a4855934f2bb9661f88ca625bd244&data=02%7C01%7Cdgallagher%40cleverdevices.com%7Cc2bb5f1ef6b442815ce908d6db8ec4ab%7Cc24ec497266d4560bcda1d93096d9d18%7C0%7C0%7C636937803587114486&sdata=5EhJf0%2FvM%2F4VPKe0z916zcHyVx5UpgzCttu2yUZbCbk%3D&reserved=0> Thanks Shawn Sent from my iPhone On May 17, 2019, at 1:59 PM, David Gallagher <dgallag...@cleverdevices.com<mailto:dgallag...@cleverdevices.com>> wrote: Hi Joe – thanks for answering. I can actually live with the message ack at the moment. But, I can’t have two messages at once because the underlying process will deadlock if two flows try to use it at the same time. This is a problem because my nifi node might be brought up after having been shut off for a few days, at which point it will read multiple messages off the queue and attempt to process them. The problem I’m having is getting flow files to wait for each other. If I use a Notify processor to set a release signal as indicated in the linked gist, it does release the Wait as designed. But, the next flowfile then enters the queue and it will get released when the Notify is triggered again by the GenerateFlowFile. What I really want to have happen is for each flowfile to block the next flowfile from entering until it has exited the process. I’m not really sure how to set that up successfully… Thanks, Dave From: Joe Witt <joe.w...@gmail.com<mailto:joe.w...@gmail.com>> Sent: Friday, May 17, 2019 12:33 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Open / Close Gate examples? Dave Using Wait/Notify would ensure you only have one message in flight at a time (or it can/should). But the message will be ack'd before processed. For Kafka and some of these message queue mechanisms if we want to offer a 'do not ack until the whole flow is done' behavior we should update the processors to allow for that and actually this is a great use case for the nifi-fn/stateless work that SamH in the community has been talking about. Thanks On Fri, May 17, 2019 at 12:30 PM David Gallagher <dgallag...@cleverdevices.com<mailto:dgallag...@cleverdevices.com>> wrote: Hi – I have a requirement to read one message off RabbitMQ and fully process that message before I take another message off the queue. I found some great guidance to setting up a ‘traffic light’ pattern here (https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244<https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fijokarumawak%2F9e1a4855934f2bb9661f88ca625bd244&data=02%7C01%7Cdgallagher%40cleverdevices.com%7Cc2bb5f1ef6b442815ce908d6db8ec4ab%7Cc24ec497266d4560bcda1d93096d9d18%7C0%7C0%7C636937803587114486&sdata=5EhJf0%2FvM%2F4VPKe0z916zcHyVx5UpgzCttu2yUZbCbk%3D&reserved=0>), but it won’t work for my process because the Notify will get triggered by the GenerateFlowFile before the long-running process triggers, which will put two or more messages into the process at once. The documentation for Notify makes reference to using a 0 value in Signal Counter Delta to simulate ‘open-close-gate’ flow control, which sounds like what I’m looking for. Can someone point me in the right direction? Thanks, Dave
