Re: [External] Re: Having a processor wait for all inputs

[Michael John LEWIS]

Hi,

Thanks Andy/Lee, will take a look.

— Mike


From: Lee Laim >
Reply-To: "users@nifi.apache.org" 
>
Date: Thursday, March 16, 2017 at 10:01 AM
To: "users@nifi.apache.org" 
>
Subject: [External] Re: Having a processor wait for all inputs

Hi Michael,

NiFi offers several ways of accomplishing this.   One way is to use the merge 
content processor with a correlation attribute that each of the 3 inputs share. 
By setting max and minimum flowfiles to 3, the processor will wait for exactly 
3 inputs with a given attribute before combining them and releasing them to the 
next processor.
This is assuming that each of the 3 inputs for a given task will only show up 
once.

I'm sure others will chime in with different methods but this will get some 
gears turning.

Have fun!
Lee


On Mar 15, 2017, at 7:38 PM, Michael John LEWIS 
> wrote:

Hi,

I’ve just started taking a look at Nifi as it seems perfect for a project I’m 
working on right now.

The one thing I’m trying to solve is this – I have a process (say a batch job 
or similar) that I want to invoke,
only once all the inputs have arrived ( a simple dependency graph ).

i.e. I have 3 files and I don’t want to run a process after each file arrival, 
but once all 3 files have arrived.

Is this easily achievable in Nifi ? Or would I have to think about custom 
processor?

Thanks in advance,

Mike


CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Re: Having a processor wait for all inputs

[Andy LoPresto]

Hi Mike,

Depending on the specifics of the inputs (source, format, correlating 
attributes, etc.) it sounds like you have a few good options to achieve this.

1. MergeContent [1] — This processor is very old (read: stable & widely-used, 
not legacy) and is used to merge content from multiple incoming flowfiles into 
batches according to a variety of strategies. There are many threads on these 
lists and articles/Stack Overflow/HCC posts about it, as it is very versatile.
2. Wait/Notify [2] — New in NiFi 1.2.0 (not yet released but available 
currently on the master branch), Joe Gresock and Koji Kawamura have contributed 
some very useful processors which allow you to wait for an arbitrary number of 
“signals” to invoke further processors. See the PRs [3][4][5] and excellent 
explanation [6].
3. ExecuteScript — If neither of those options work for you, although I suspect 
they will, you can write custom code in any of 6 languages (Ruby, Python, Lua, 
Javascript, Groovy, or Java) and run it with the ExecuteScript or 
InvokeScriptedProcessor to perform arbitrary operations without having to build 
a complete NAR and custom processors. Of course, you can still do that if you 
desire.

Hope that helps. Please let us know if you have any other questions.

[1] 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html
 

[2] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/Wait.java
 

[3] https://github.com/apache/nifi/pull/1329 

[4] https://github.com/apache/nifi/pull/1420 

[5] https://github.com/apache/nifi/pull/1490 

[6] https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Mar 15, 2017, at 6:38 PM, Michael John LEWIS  wrote:
> 
> Hi,
> 
> I’ve just started taking a look at Nifi as it seems perfect for a project I’m 
> working on right now.
> 
> The one thing I’m trying to solve is this – I have a process (say a batch job 
> or similar) that I want to invoke,
> only once all the inputs have arrived ( a simple dependency graph ).
> 
> i.e. I have 3 files and I don’t want to run a process after each file 
> arrival, but once all 3 files have arrived.
> 
> Is this easily achievable in Nifi ? Or would I have to think about custom 
> processor?
> 
> Thanks in advance,
> 
> Mike
> 
> 
> CONFIDENTIAL NOTE:
> 
> The information contained in this email is intended only for the use of the 
> individual or entity named above and may contain information that is 
> privileged, confidential and exempt from disclosure under applicable law. If 
> the reader of this message is not the intended recipient, you are hereby 
> notified that any dissemination, distribution or copying of this 
> communication is strictly prohibited. If you have received this message in 
> error, please immediately notify the sender and delete the mail. Thank you.



signature.asc
Description: Message signed with OpenPGP using GPGMail

Having a processor wait for all inputs

[Michael John LEWIS]

Hi,

I’ve just started taking a look at Nifi as it seems perfect for a project I’m 
working on right now.

The one thing I’m trying to solve is this – I have a process (say a batch job 
or similar) that I want to invoke,
only once all the inputs have arrived ( a simple dependency graph ).

i.e. I have 3 files and I don’t want to run a process after each file arrival, 
but once all 3 files have arrived.

Is this easily achievable in Nifi ? Or would I have to think about custom 
processor?

Thanks in advance,

Mike


CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Re: RuleEngine Processor

[Matt Burgess]

Uwe,

I will do my best to look at this but it will probably be next week. It seems 
very interesting and my only excuse is being swamped by other things. Thank you 
for sharing this, hopefully others in the community will give it a go as well.

Regards,
Matt


> On Mar 15, 2017, at 6:12 PM, Uwe Geercken  wrote:
> 
> Hello,
> 
> I have worked quite a bit on my RuleEngine processor and would appreciate, if 
> somebody takes some time to try it and to look at the source code and give me 
> some feedback.
> 
> You can download it at: https://github.com/uwegeercken/nifi_processors
> 
> It is my first bundle of processors, so I guess there are many improvements I 
> could make - but I need your expertise and opinion.
> 
> Also - quite important - I have renamed the processor to "ExecuteRuleEngine". 
> I believe that fits better to the general naming convention that is used for 
> the processors.
> 
> I still need to work a lot on the documentation, too.
> 
> Greetings,
> 
> Uwe

RuleEngine Processor

[Uwe Geercken]

Hello,

I have worked quite a bit on my RuleEngine processor and would appreciate, if 
somebody takes some time to try it and to look at the source code and give me 
some feedback.

You can download it at: https://github.com/uwegeercken/nifi_processors

It is my first bundle of processors, so I guess there are many improvements I 
could make - but I need your expertise and opinion.

Also - quite important - I have renamed the processor to "ExecuteRuleEngine". I 
believe that fits better to the general naming convention that is used for the 
processors.

I still need to work a lot on the documentation, too.

Greetings,

Uwe

Re: Nifi 1.1.0 cluster on Docker Swarm

[Raf Huys]

Great.
I have an overlay network listed in docker-compose, sure.
I want to narrow down the issue, so if the Nifi configuration is working,
that's already a big step.
Thank you for your time.

On Wed, Mar 15, 2017 at 3:28 PM, Jeremy Dyer  wrote:

> Raf - Ok so good news and bad news. Good news its working for me. Bad news
> its working for me =) Here is the complete list of things that I changed.
> Hopefully this can at least really help narrow down what is causing the
> issue.
>
> - I ran on a single machine. All that was available to me while at the
> airport.
> - I added a "network" section to the end of the docker-compose.yml file. I
> think you might already have that and this was just a snippet in your gist?
> - I removed the COPY from the Dockerfile around the custom processors
> since I don't have those.
>
> In my mind the most likely issue is something around Docker swarm
> networking.
>
> On Wed, Mar 15, 2017 at 9:35 AM, Jeremy Dyer  wrote:
>
>> Thanks for sharing! Give me a little bit to test out and I will get back
>> to you. I obviously will not have your custom processors so will have to
>> make that small adjustment in the Dockerfile but it is highly unlikely that
>> is the actual issue anyway.
>>
>> On Wed, Mar 15, 2017 at 9:04 AM, Raf Huys  wrote:
>>
>>>  Awesome, we wrap https://github.com/mkobit/docker-nifi to add some custom 
>>> processors:
>>>
>>>
>>> Tx
>>>
>>>
>>>
>>> FROM mkobit/nifi:1.1.0-openjdk-8COPY . /opt/nifi/WORKDIR /opt/nifi/RUN cp 
>>> dependencies/*.nar  $NIFI_HOME/libEXPOSE 8080CMD  tail -F 
>>> ${NIFI_HOME}/logs/nifi-app.log & ${NIFI_HOME}/bin/nifi.sh run
>>>
>>>
>>> On Wed, Mar 15, 2017 at 1:59 PM, Jeremy Dyer  wrote:
>>>
 Raf - Glad to help you troubleshoot this. Are you using the official
 NiFi docker image? If not could you please share your Dockerfile so I can
 try out your configuration locally

 On Wed, Mar 15, 2017 at 6:48 AM, Raf Huys  wrote:

> I'm having difficulty getting my 2-node Nifi cluster up and running
> with a separate 3-node Zookeeper ensemble. Part of the issue might be that
> it's being ran as a Docker Swarm cluster (with an overlay network).
>
> In particular, I cannot connect to the UI.
>
> Configuration and logs can be found here: https://gist.github.com/
> Raffiki/0b84d916a2475aba94cae97a7f4fff10
>
>
> The logs seem to indicate that the cluster was successfully
> established however (`NiFi has started. The UI is available at the
> following URLs:...`)
> So not being able to connect to the UI might be a Docker network
> configuration issue. However: I cannot curl Nifi on 8080 on the container
> itself. So I'm stuck.
>
> What would help me out, is a doublecheck on the `nifi.properties` file
> for obvious errors. The nifi-properties file on the second node is similar
> (except for hostnames)
>
>
> Tx
>
> Raf Huys
>


>>>
>>>
>>> --
>>> Mvg,
>>>
>>> Raf Huys
>>>
>>
>>
>


-- 
Mvg,

Raf Huys

Re: Nifi 1.1.0 cluster on Docker Swarm

[Jeremy Dyer]

Raf - Ok so good news and bad news. Good news its working for me. Bad news
its working for me =) Here is the complete list of things that I changed.
Hopefully this can at least really help narrow down what is causing the
issue.

- I ran on a single machine. All that was available to me while at the
airport.
- I added a "network" section to the end of the docker-compose.yml file. I
think you might already have that and this was just a snippet in your gist?
- I removed the COPY from the Dockerfile around the custom processors since
I don't have those.

In my mind the most likely issue is something around Docker swarm
networking.

On Wed, Mar 15, 2017 at 9:35 AM, Jeremy Dyer  wrote:

> Thanks for sharing! Give me a little bit to test out and I will get back
> to you. I obviously will not have your custom processors so will have to
> make that small adjustment in the Dockerfile but it is highly unlikely that
> is the actual issue anyway.
>
> On Wed, Mar 15, 2017 at 9:04 AM, Raf Huys  wrote:
>
>>  Awesome, we wrap https://github.com/mkobit/docker-nifi to add some custom 
>> processors:
>>
>>
>> Tx
>>
>>
>>
>> FROM mkobit/nifi:1.1.0-openjdk-8COPY . /opt/nifi/WORKDIR /opt/nifi/RUN cp 
>> dependencies/*.nar  $NIFI_HOME/libEXPOSE 8080CMD  tail -F 
>> ${NIFI_HOME}/logs/nifi-app.log & ${NIFI_HOME}/bin/nifi.sh run
>>
>>
>> On Wed, Mar 15, 2017 at 1:59 PM, Jeremy Dyer  wrote:
>>
>>> Raf - Glad to help you troubleshoot this. Are you using the official
>>> NiFi docker image? If not could you please share your Dockerfile so I can
>>> try out your configuration locally
>>>
>>> On Wed, Mar 15, 2017 at 6:48 AM, Raf Huys  wrote:
>>>
 I'm having difficulty getting my 2-node Nifi cluster up and running
 with a separate 3-node Zookeeper ensemble. Part of the issue might be that
 it's being ran as a Docker Swarm cluster (with an overlay network).

 In particular, I cannot connect to the UI.

 Configuration and logs can be found here: https://gist.github.com/
 Raffiki/0b84d916a2475aba94cae97a7f4fff10


 The logs seem to indicate that the cluster was successfully established
 however (`NiFi has started. The UI is available at the following
 URLs:...`)
 So not being able to connect to the UI might be a Docker network
 configuration issue. However: I cannot curl Nifi on 8080 on the container
 itself. So I'm stuck.

 What would help me out, is a doublecheck on the `nifi.properties` file
 for obvious errors. The nifi-properties file on the second node is similar
 (except for hostnames)


 Tx

 Raf Huys

>>>
>>>
>>
>>
>> --
>> Mvg,
>>
>> Raf Huys
>>
>
>

Re: Upgrading NiFi, preserving process groups

[Mark Payne]

Jim,

Yes - if you copy over the flow.xml.gz then that's all that you should need to 
do in order to preserve your flow.
If you are running NiFi 0.x though you will also want to copy over your 
conf/templates directory. In the 1.x baseline,
Templates are included in the flow.xml also.

Thanks
-Mark

> On Mar 15, 2017, at 6:26 AM, James McMahon  wrote:
> 
> Good morning. Are there any specific steps I need to take to ensure I 
> preserve all my Process Groups when I upgrade my version of NiFi? Does the 
> step by which I save and copy flow.xml.gz into the conf subdirectory of the 
> new version ensure that my Process Groups transfer to the upgraded version 
> instance? Thank you for your help.

Re: Nifi 1.1.0 cluster on Docker Swarm

[Raf Huys]

 Awesome, we wrap https://github.com/mkobit/docker-nifi to add some
custom processors:


Tx



FROM mkobit/nifi:1.1.0-openjdk-8COPY . /opt/nifi/WORKDIR /opt/nifi/RUN
cp dependencies/*.nar  $NIFI_HOME/libEXPOSE 8080CMD  tail -F
${NIFI_HOME}/logs/nifi-app.log & ${NIFI_HOME}/bin/nifi.sh run


On Wed, Mar 15, 2017 at 1:59 PM, Jeremy Dyer  wrote:

> Raf - Glad to help you troubleshoot this. Are you using the official NiFi
> docker image? If not could you please share your Dockerfile so I can try
> out your configuration locally
>
> On Wed, Mar 15, 2017 at 6:48 AM, Raf Huys  wrote:
>
>> I'm having difficulty getting my 2-node Nifi cluster up and running with
>> a separate 3-node Zookeeper ensemble. Part of the issue might be that it's
>> being ran as a Docker Swarm cluster (with an overlay network).
>>
>> In particular, I cannot connect to the UI.
>>
>> Configuration and logs can be found here: https://gist.github.com/
>> Raffiki/0b84d916a2475aba94cae97a7f4fff10
>>
>>
>> The logs seem to indicate that the cluster was successfully established
>> however (`NiFi has started. The UI is available at the following
>> URLs:...`)
>> So not being able to connect to the UI might be a Docker network
>> configuration issue. However: I cannot curl Nifi on 8080 on the container
>> itself. So I'm stuck.
>>
>> What would help me out, is a doublecheck on the `nifi.properties` file
>> for obvious errors. The nifi-properties file on the second node is similar
>> (except for hostnames)
>>
>>
>> Tx
>>
>> Raf Huys
>>
>
>


-- 
Mvg,

Raf Huys

Upgrading NiFi, preserving process groups

[James McMahon]

Good morning. Are there any specific steps I need to take to ensure I
preserve all my Process Groups when I upgrade my version of NiFi? Does the
step by which I save and copy flow.xml.gz into the conf subdirectory of the
new version ensure that my Process Groups transfer to the upgraded version
instance? Thank you for your help.