RE: [EXT] Re: Wait only if flagged?

2017-11-13 Thread Peter Wicks (pwicks)
Matt,

I played around with your idea. I haven't been able to get it to work.

First flow file comes in, goes out the wait relationship. Now how do we stop 
the next FlowFile from going out the wait relationship and entering the flow? 
Well we'd have to use Notify along with some scheduling on Wait so multiple 
FlowFile's don't come through right away; so I put a Notify processor right 
after Wait. Now the second FlowFile enters, goes out the success relationship 
because there is a signal count, then immediately goes down wait since the 
signal is clear...

If this use case seems common enough, I was thinking that the solution might be 
to have add an option on Wait that lets you choose whether you want to "Wait 
for signal" or "Wait only if signaled". This would restore the relationships 
back to their proper uses. Then you'd use the Notify property 'Signal Counter 
Delta' mode of 0 at the end of the flow.  Flow would be something like Wait 
(Wait only if signaled)->Notify (Signal to 1)->... Processors ...->Notify 
(Signal to 0). Or perhaps as part of the feature, "Wait only if signaled" mode 
would do the notify step internally.

Would still like to find a clean/easy way to do this though.

Thanks,
 Peter

-Original Message-
From: Matt Burgess [mailto:mattyb...@apache.org] 
Sent: Tuesday, November 14, 2017 08:53
To: users@nifi.apache.org
Subject: [EXT] Re: Wait only if flagged?

Peter,

I haven't tried this, but my knee-jerk reaction is to switch the roles
of the "wait" and "success" relationships. Maybe you can send the
"wait" relationship downstream and route the "success" one back to
Wait. Then when the flag is "cleared", the flow files will start going
to the "success" relationship which is routed back to wait.

Regards,
Matt

On Mon, Nov 13, 2017 at 7:21 PM, Peter Wicks (pwicks)  wrote:
> I have a database flow, which is a sequence of 4 processors. For database
> performance reasons I need to make sure only one file is in this section of
> the flow at a time. Not just one file per queue/processor, but for the whole
> section.
>
>
>
> I feel like I should be able to use Wait/Notify to do this, but Wait/Notify
> seem to do the opposite. I want to allow a file to go into the flow unless
> there is a flag. If there is a flag I want the FlowFile to wait until the
> flag is cleared.
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
>  Peter


Re: [EXT] Re: Wait only if flagged?

2017-11-13 Thread Koji Kawamura
Hi Peter,

I think I have done similar thing before. This flow template may be
useful in your case, too.
https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244

Thanks,
Koji

On Tue, Nov 14, 2017 at 12:58 PM, Peter Wicks (pwicks)
 wrote:
> Matt,
>
> I played around with your idea. I haven't been able to get it to work.
>
> First flow file comes in, goes out the wait relationship. Now how do we stop 
> the next FlowFile from going out the wait relationship and entering the flow? 
> Well we'd have to use Notify along with some scheduling on Wait so multiple 
> FlowFile's don't come through right away; so I put a Notify processor right 
> after Wait. Now the second FlowFile enters, goes out the success relationship 
> because there is a signal count, then immediately goes down wait since the 
> signal is clear...
>
> If this use case seems common enough, I was thinking that the solution might 
> be to have add an option on Wait that lets you choose whether you want to 
> "Wait for signal" or "Wait only if signaled". This would restore the 
> relationships back to their proper uses. Then you'd use the Notify property 
> 'Signal Counter Delta' mode of 0 at the end of the flow.  Flow would be 
> something like Wait (Wait only if signaled)->Notify (Signal to 1)->... 
> Processors ...->Notify (Signal to 0). Or perhaps as part of the feature, 
> "Wait only if signaled" mode would do the notify step internally.
>
> Would still like to find a clean/easy way to do this though.
>
> Thanks,
>  Peter
>
> -Original Message-
> From: Matt Burgess [mailto:mattyb...@apache.org]
> Sent: Tuesday, November 14, 2017 08:53
> To: users@nifi.apache.org
> Subject: [EXT] Re: Wait only if flagged?
>
> Peter,
>
> I haven't tried this, but my knee-jerk reaction is to switch the roles
> of the "wait" and "success" relationships. Maybe you can send the
> "wait" relationship downstream and route the "success" one back to
> Wait. Then when the flag is "cleared", the flow files will start going
> to the "success" relationship which is routed back to wait.
>
> Regards,
> Matt
>
> On Mon, Nov 13, 2017 at 7:21 PM, Peter Wicks (pwicks)  
> wrote:
>> I have a database flow, which is a sequence of 4 processors. For database
>> performance reasons I need to make sure only one file is in this section of
>> the flow at a time. Not just one file per queue/processor, but for the whole
>> section.
>>
>>
>>
>> I feel like I should be able to use Wait/Notify to do this, but Wait/Notify
>> seem to do the opposite. I want to allow a file to go into the flow unless
>> there is a flag. If there is a flag I want the FlowFile to wait until the
>> flag is cleared.
>>
>>
>>
>> Thoughts?
>>
>>
>>
>> Thanks,
>>
>>  Peter