Aw: Re: Re: new Nifi Processors

[Uwe Geercken]

Thanks Andrew,

 

Matt also pointed me to the same direction.

 

But my question is, what if I switch to the Apache license model and some other software or distribution wants to package my software and they use a different licence model. Will I have a similar problem there? Do you know?

 

I will spend some time on the topic on the weekend...

 

Rgds,

 

Uwe

 

Gesendet: Donnerstag, 02. März 2017 um 05:06 Uhr
Von: "Andrew Grande" 
An: users@nifi.apache.org
Betreff: Re: Re: new Nifi Processors


Basically the GPL license puts restrictions on how one can distribute in practical terms. Meaning your work may live under GPL license as long as it's not part of the official package. End users will have to download your NAR themselves.

Andrew
 


On Wed, Mar 1, 2017, 8:43 AM Matt Burgess  wrote:

Uwe,

Sorry for misspeaking, by "official Apache NiFi repo" I meant the
Apache NiFi codebase (the "built-in" processors, e.g.). For the
licensing part, if you distribute something with GPL binary
dependencies, I believe the entire distribution must be licensed as
GPL or something GPL-compatible.  This is not a bad thing, but due to
Apache licensing, such a processor could not be accepted as-is into
the NiFi codebase. Even LGPL binary dependencies are not allowed.
However as you have made your processor available via your own repo,
the community is free to download and use your processor under the
terms of your license.  However if someone packaged up a NiFi
distribution with a GPL-licensed processor (for example), they would
not be allowed to distribute that package under an Apache 2.0 license;
rather I believe the whole package would have to be licensed under the
GPL.

I am no licensing expert by any means, but I have had experience with
these kinds of things, both for NiFi and other extensible open-source
projects.

Regards,
Matt

On Wed, Mar 1, 2017 at 7:01 AM, Uwe Geercken  wrote:
> Matt,
>
> I did not know there is an official Apache Nifi repo. If you send me a link, I will have a look.
>
> Also, is there an official way of tagging, annotating or otherwise documenting the license model for a processor? At which point in the code, documentation do I have to place license information?
>
> I will check if the Apache license fits to my personal ideas of how my software should be protected. I am not a license expert, so I will have to spend some time to understand what that means. Also I need to check what it means for the software (and current users) if I change the license model.
>
> Anyway, this is still a first version of the processors. So they will mature over time and I hope at that point the extension registry is there.
>
> In general - as you know Matt - I am creating open source software (since 2000). I believe in the idea of open source and of sharing for the benefit of all of us.
>
> If I can, I will adjust whatever is necessary, so that the license is not a hurdle for using the processors. Nifi is a really great product and I still remember my first impression when I saw it.
>
> Greetings,
>
> Uwe
>
>> Gesendet: Mittwoch, 01. März 2017 um 03:56 Uhr
>> Von: "Matt Burgess" 
>> An: users@nifi.apache.org
>> Betreff: Re: new Nifi Processors
>>
>> Uwe G has made his processors available (thank you!) via his own repo
>> vs the official Apache NiFi repo; this may be directly related to your
>> point about licensing.  Having said that, he is of course at liberty
>> to license those separate processors as he sees fit (assuming it is
>> also in accordance with the licenses he has employed).  Apache NiFi
>> welcomes to its codebase Apache-friendly contributions (FAQ [1]), but
>> alternatively and even before an Extension Registry [2] is supported,
>> authors can make their NiFi processors and such available under the
>> appropriate licenses.  If there are commercial (or other) entities
>> looking to package such extensions with the official Apache NiFi
>> distribution, they would be subject to the same terms of the License &
>> Notice (L&N) of Apache NiFi as well as whatever extensions are added.
>>
>> Regards,
>> Matt
>>
>> [1] https://www.apache.org/legal/resolved.html
>> [2] https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
>>
>>
>> On Tue, Feb 28, 2017 at 9:33 PM, Angry Duck Studio
>>  wrote:
>> > Hi, Uwe,
>> >
>> > These look useful. However, typically custom processors are either Apache
>> > 2.0 or MIT licensed. These don't seem to specify a license, but your
>> > business rule engine (jare) seems to be GPL 3.0 licensed. I'm not sure that
>> > fits with most uses of NiFi.
>> >
>> > Can you please clarify?
>> >
>> > Thanks
>> >
>> > -Matt
>> >
>> > On Tue, Feb 28, 2017 at 4:47 PM, Uwe Geercken  wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >> I just wanted to let you know, that I have created four processors fo

Re: ExecuteStreamCommand Zombie

[Oleg Zhurakousky]

Olav

The fact that the processor does not detect the dead process that was initiated 
by it does appear to be a bug. Would you mind raising JIRA - 
https://issues.apache.org/jira/browse/NIFI so we can take a look.

Cheers
Oleg
On Mar 1, 2017, at 7:29 PM, Olav Jordens 
mailto:olav.jord...@2degreesmobile.co.nz>> 
wrote:


Hi Nifi lovers,

I have just seen this situation for the first time. I run a .sh script using 
ExecuteStreamCommand which usually takes a few minutes. Today, I noticed that 
the resulting flow had not produced anything for almost a day. Then I checked 
on my Linux box and saw that the .sh script had been running for almost a day. 
Something must have been wrong here, so killed the process from the command 
line. However, the Nifi processor will not stop. I eventually copied and 
replaced the components of the flow and all is working again, but the Zombie 
still won’t die in Nifi. I am sure that if I restart the Nifi process, all will 
be resolved, but thought that you may wish to see this screenshot, and consider 
if ExecuteStreamCommand could detect the process being killed and then stop 
running. Here is an image of a portion of the flow – on the left the zombie, 
and on the right, the copy-pasted components working fine.



Thanks!
Olav



Olav Jordens
Senior ETL Developer
Two Degrees Mobile Limited
===
(M) 022 620 2429
(P) 09 919 7000
www.2degreesmobile.co.nz

Two Degrees Mobile Limited | 47-49 George Street | Newmarket | Auckland | New 
Zealand |
PO Box 8355 | Symonds Street | Auckland 1150 | New Zealand | Fax +64 9 919 7001


   




Disclaimer
The e-mail and any files transmitted with it are confidential and may contain 
privileged or copyright information. If you are not the intended recipient you 
must not copy, distribute, or use this e-mail or the information contained in 
it for any purpose other than to notify us of the error. If you have received 
this message in error, please notify the sender immediately, by email or phone 
(+64 9 919 7000) and delete this email from your system. Any views expressed in 
this message are those of the individual sender, except where the sender 
specifically states them to be the views of Two Degrees Mobile Limited. We do 
not guarantee that this material is free from viruses or any other defects 
although due care has been taken to minimize the risk

Re: How to specify priority attributes for seperate flowfiles?

[Bryan Bende]

So in your example you are saying that 10 files get placed in a
directory, and inside each of those 10 files the data is already
ordered the way you want, but you want to ensure the 10 files get
processed in a specific order?

If that is true, what determines the order of the 10 files? is it
based on the order they were written to the directory? or is there
something in the filename that indicates which file comes first? In
order for NiFi to prioritize these files, there has to be something
that tells NiFi what the priority is.

On Wed, Mar 1, 2017 at 11:56 PM, prabhu Mahendran
 wrote:
> As you suggested, setting 3 UpdateAttribute may be tedious. Suppose I have
> more than 10 flowfiles setting 10 updateattribute processor is lengthy one.
> This case also not possible for dynamically generating flowfiles.
>
>
>
> How to set priority attribute for the flowfiles from Getfile? Suppose I get
> 10 files in the Getfile processor, based on my priority I have ordered the
> flowfile each line in the files till PutSQL. Here without considering the
> order, based on the filecreation time, data is moved without my ordered
> records. For this case only I decided with the PriorityAttributePrioritizer
> and used UpdateAttribute processor.
>
>
>
> I can able to set the priority attribute for each line in the file, but not
> each files from GetFile. Can you suggest any solution?
>
>
>
>
> On Wed, Mar 1, 2017 at 7:18 PM, Bryan Bende  wrote:
>>
>> I just responded to this question on stackoverflow:
>>
>>
>> https://stackoverflow.com/questions/42528993/how-to-specify-priority-attributes-for-seperate-flowfiles
>>
>> Thanks,
>>
>> Bryan
>>
>> On Wed, Mar 1, 2017 at 5:19 AM, prabhu Mahendran
>>  wrote:
>> > I need to use PrioritizeAttributePrioritizer in NiFi.
>> >
>> > i have observed that prioritizers in below reference.
>> > https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings
>> >
>> > if i receive 10 flowfiles then i need to set the priority value for
>> > every
>> > flow file to be unique.
>> >
>> > After that specify queue configuration must be
>> > PrioritizeAttributePrioritizer.
>> >
>> > Then processing flowfiles based on priority value.
>> >
>> > How can i set priority value for seperate flow files or which
>> > prioritizer in
>> > Nifi to be work for my case?
>
>

Re: Re: Re: new Nifi Processors

[Matt Burgess]

Uwe,

If your NAR can be licensed under the Apache Software License 2.0,
then you shouldn't run into any issues with other folks want to
package your software, they can even package it up and license it
under a commercial (paid) license; the ASL 2.0 is pretty permissive.
There are some patent protections in there, as well as some rules
about explicitly specifying any code you've changed, and rules about
use of the project name (like you can't sell a version of NiFi with
your additional NAR and call it "NiFi++").

Regards,
Matt

On Thu, Mar 2, 2017 at 7:10 AM, Uwe Geercken  wrote:
> Thanks Andrew,
>
> Matt also pointed me to the same direction.
>
> But my question is, what if I switch to the Apache license model and some
> other software or distribution wants to package my software and they use a
> different licence model. Will I have a similar problem there? Do you know?
>
> I will spend some time on the topic on the weekend...
>
> Rgds,
>
> Uwe
>
> Gesendet: Donnerstag, 02. März 2017 um 05:06 Uhr
> Von: "Andrew Grande" 
> An: users@nifi.apache.org
> Betreff: Re: Re: new Nifi Processors
>
> Basically the GPL license puts restrictions on how one can distribute in
> practical terms. Meaning your work may live under GPL license as long as
> it's not part of the official package. End users will have to download your
> NAR themselves.
>
> Andrew
>
>
> On Wed, Mar 1, 2017, 8:43 AM Matt Burgess  wrote:
>>
>> Uwe,
>>
>> Sorry for misspeaking, by "official Apache NiFi repo" I meant the
>> Apache NiFi codebase (the "built-in" processors, e.g.). For the
>> licensing part, if you distribute something with GPL binary
>> dependencies, I believe the entire distribution must be licensed as
>> GPL or something GPL-compatible.  This is not a bad thing, but due to
>> Apache licensing, such a processor could not be accepted as-is into
>> the NiFi codebase. Even LGPL binary dependencies are not allowed.
>> However as you have made your processor available via your own repo,
>> the community is free to download and use your processor under the
>> terms of your license.  However if someone packaged up a NiFi
>> distribution with a GPL-licensed processor (for example), they would
>> not be allowed to distribute that package under an Apache 2.0 license;
>> rather I believe the whole package would have to be licensed under the
>> GPL.
>>
>> I am no licensing expert by any means, but I have had experience with
>> these kinds of things, both for NiFi and other extensible open-source
>> projects.
>>
>> Regards,
>> Matt
>>
>> On Wed, Mar 1, 2017 at 7:01 AM, Uwe Geercken  wrote:
>> > Matt,
>> >
>> > I did not know there is an official Apache Nifi repo. If you send me a
>> > link, I will have a look.
>> >
>> > Also, is there an official way of tagging, annotating or otherwise
>> > documenting the license model for a processor? At which point in the code,
>> > documentation do I have to place license information?
>> >
>> > I will check if the Apache license fits to my personal ideas of how my
>> > software should be protected. I am not a license expert, so I will have to
>> > spend some time to understand what that means. Also I need to check what it
>> > means for the software (and current users) if I change the license model.
>> >
>> > Anyway, this is still a first version of the processors. So they will
>> > mature over time and I hope at that point the extension registry is there.
>> >
>> > In general - as you know Matt - I am creating open source software
>> > (since 2000). I believe in the idea of open source and of sharing for the
>> > benefit of all of us.
>> >
>> > If I can, I will adjust whatever is necessary, so that the license is
>> > not a hurdle for using the processors. Nifi is a really great product and I
>> > still remember my first impression when I saw it.
>> >
>> > Greetings,
>> >
>> > Uwe
>> >
>> >> Gesendet: Mittwoch, 01. März 2017 um 03:56 Uhr
>> >> Von: "Matt Burgess" 
>> >> An: users@nifi.apache.org
>> >> Betreff: Re: new Nifi Processors
>> >>
>> >> Uwe G has made his processors available (thank you!) via his own repo
>> >> vs the official Apache NiFi repo; this may be directly related to your
>> >> point about licensing.  Having said that, he is of course at liberty
>> >> to license those separate processors as he sees fit (assuming it is
>> >> also in accordance with the licenses he has employed).  Apache NiFi
>> >> welcomes to its codebase Apache-friendly contributions (FAQ [1]), but
>> >> alternatively and even before an Extension Registry [2] is supported,
>> >> authors can make their NiFi processors and such available under the
>> >> appropriate licenses.  If there are commercial (or other) entities
>> >> looking to package such extensions with the official Apache NiFi
>> >> distribution, they would be subject to the same terms of the License &
>> >> Notice (L&N) of Apache NiFi as well as whatever extensions are added.
>> >>
>> >> Regards,
>> >> Matt
>> >>
>> >> [1] https://www.apache.org/legal/resolved

Re: new Nifi Processors

[Matt Foley]

Hi Uwe,

Most everything about Apache licensing, and how it relates to other licenses 
such as GPL (from the Apache viewpoint) is in these two faqs:

http://www.apache.org/foundation/license-faq.html

https://www.apache.org/legal/resolved

 

--Matt

 

From: Uwe Geercken 
Reply-To: "users@nifi.apache.org" 
Date: Thursday, March 2, 2017 at 4:10 AM
To: "users@nifi.apache.org" 
Subject: Aw: Re: Re: new Nifi Processors

 

Thanks Andrew,

 

Matt also pointed me to the same direction.

 

But my question is, what if I switch to the Apache license model and some other 
software or distribution wants to package my software and they use a different 
licence model. Will I have a similar problem there? Do you know?

 

I will spend some time on the topic on the weekend...

 

Rgds,

 

Uwe

  

Gesendet: Donnerstag, 02. März 2017 um 05:06 Uhr
Von: "Andrew Grande" 
An: users@nifi.apache.org
Betreff: Re: Re: new Nifi Processors

Basically the GPL license puts restrictions on how one can distribute in 
practical terms. Meaning your work may live under GPL license as long as it's 
not part of the official package. End users will have to download your NAR 
themselves.

Andrew

  

On Wed, Mar 1, 2017, 8:43 AM Matt Burgess  wrote:

Uwe,

Sorry for misspeaking, by "official Apache NiFi repo" I meant the
Apache NiFi codebase (the "built-in" processors, e.g.). For the
licensing part, if you distribute something with GPL binary
dependencies, I believe the entire distribution must be licensed as
GPL or something GPL-compatible.  This is not a bad thing, but due to
Apache licensing, such a processor could not be accepted as-is into
the NiFi codebase. Even LGPL binary dependencies are not allowed.
However as you have made your processor available via your own repo,
the community is free to download and use your processor under the
terms of your license.  However if someone packaged up a NiFi
distribution with a GPL-licensed processor (for example), they would
not be allowed to distribute that package under an Apache 2.0 license;
rather I believe the whole package would have to be licensed under the
GPL.

I am no licensing expert by any means, but I have had experience with
these kinds of things, both for NiFi and other extensible open-source
projects.

Regards,
Matt

On Wed, Mar 1, 2017 at 7:01 AM, Uwe Geercken  wrote:
> Matt,
>
> I did not know there is an official Apache Nifi repo. If you send me a link, 
> I will have a look.
>
> Also, is there an official way of tagging, annotating or otherwise 
> documenting the license model for a processor? At which point in the code, 
> documentation do I have to place license information?
>
> I will check if the Apache license fits to my personal ideas of how my 
> software should be protected. I am not a license expert, so I will have to 
> spend some time to understand what that means. Also I need to check what it 
> means for the software (and current users) if I change the license model.
>
> Anyway, this is still a first version of the processors. So they will mature 
> over time and I hope at that point the extension registry is there.
>
> In general - as you know Matt - I am creating open source software (since 
> 2000). I believe in the idea of open source and of sharing for the benefit of 
> all of us.
>
> If I can, I will adjust whatever is necessary, so that the license is not a 
> hurdle for using the processors. Nifi is a really great product and I still 
> remember my first impression when I saw it.
>
> Greetings,
>
> Uwe
>
>> Gesendet: Mittwoch, 01. März 2017 um 03:56 Uhr
>> Von: "Matt Burgess" 
>> An: users@nifi.apache.org
>> Betreff: Re: new Nifi Processors
>>
>> Uwe G has made his processors available (thank you!) via his own repo
>> vs the official Apache NiFi repo; this may be directly related to your
>> point about licensing.  Having said that, he is of course at liberty
>> to license those separate processors as he sees fit (assuming it is
>> also in accordance with the licenses he has employed).  Apache NiFi
>> welcomes to its codebase Apache-friendly contributions (FAQ [1]), but
>> alternatively and even before an Extension Registry [2] is supported,
>> authors can make their NiFi processors and such available under the
>> appropriate licenses.  If there are commercial (or other) entities
>> looking to package such extensions with the official Apache NiFi
>> distribution, they would be subject to the same terms of the License &
>> Notice (L&N) of Apache NiFi as well as whatever extensions are added.
>>
>> Regards,
>> Matt
>>
>> [1] https://www.apache.org/legal/resolved.html
>> [2] 
>> https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
>>
>>
>> On Tue, Feb 28, 2017 at 9:33 PM, Angry Duck Studio
>>  wrote:
>> > Hi, Uwe,
>> >
>> > These look useful. However, typically custom processors are either Apache
>> > 2.0 or MIT licensed. These don't seem to specify a license, but your
>>

Re: Aw: Re: new Nifi Processors

[Russell Bateman]

I too find GPL to be confusing enough that I (and I am not alone) 
consider it to be poisonous fruit that I simply do not touch.


I earn my living working as an employee developing software for 
companies that sell it, or rights to use it, for money and do not 
publish their product source code any more than Coca Cola gives away the 
recipe for its flagship beverage.


As I understand it, if I consume a JAR that falls under GPL in my work, 
even if I only consume the JAR's functionality and do not modify it, 
however small and "insignificant" that JAR's contribution may be to the 
whole, that use opens my employer to a lawsuit because my source code is 
in essence and in fact built atop that GPL'd component.


Maybe my interpretation is born of irrational paranoia, but it's how it 
looks to me. To me, GPL means software to be used by academics and 
people who develop software for their health only.


Thoughts? (Yeah, this isn't really the forum for it.)


On 03/01/2017 04:40 AM, Uwe Geercken wrote:

Hello,
what would be the appropriate way to license the processors? Is it an 
annotation, a seperate license file or something else?

To the GPL3: This is what wikipedia says:

The *GNU General Public License* (*GNU GPL* or *GPL*) is a widely used 
free software license 
, which 
guarantees end users  the 
freedom to run, study, share and modify the software.^[6] 
 
The license was originally written by Richard Stallman 
 of the Free Software 
Foundation  
(FSF) for the GNU Project , 
and grants the recipients of a computer program 
 the rights of the 
Free Software Definition 
.^[7] 
 
The GPL is a copyleft  
license, which means that derivative work 
 can only be 
distributed under the same license terms. This is in distinction to 
permissive free software licenses 
, of 
which the BSD licenses  
and the MIT License  are 
widely used examples. GPL was the first copyleft license for general use.


Historically, the GPL license family has been one of the most popular 
software licenses in the free and open-source software 
 
domain.^[6] 
 
^[8] 
 
^[9] 
 
^[10] 
 
^[11] 
 
^[12] 
 
^[13] 
 
Prominent free software programs licensed under the GPL include the 
Linux kernel  and the GNU 
Compiler Collection 
 (GCC). David 
A. Wheeler  argues 
that the copyleft provided by the GPL was crucial to the success of 
Linux -based systems, 
giving the programmers who contributed to the kernel the assurance 
that their work would benefit the whole world and remain free, rather 
than being exploited by software companies that would not have to give 
anything back to the community.^[14] 



The ruleengine is under GPL3. So users acan freely use, embed or share 
it. It is only derivative work, that needs to be distributed under the 
same lisence terms. So what would be the problem with the Nifi 
processor? Can somebody explain that to me.
Also, I would be glad to have a quick explanation of what the core 
differences or advantages are of Apache 2.0 versus GPL3. That would 
help me understand the issue better.

Greetings and thanks for feedback.
Uwe
*Gesendet:* Mittwoch, 01. März 2017 um 03:33 Uhr
*Von:* "Angry Duck Studio" 
*An:* users@nifi.apache.org
*Betreff:* Re: new Nifi Processors
Hi, Uwe,
These look useful. However, typically custom processors are either 
Apache 2.0 or MIT licensed. These don't se

HandleHttpRequest API Port Exposure

[Ryan H]

Hello All,

I  am exposing an API using HandleHttpRequest on my local nifi instance.
The HandleHttpRequest processor requires a Listening port that I need to
provide. If I enter 80 in that field, the processor fails when it starts
saying “unable to initialize the server”. Which is expected as the
webserver uses that port to serve the canvas. So if we provide any other
random number then it works fine.

When I promote the above API on the nifi cluster that is hosted on our AWS
farm, then we are unable to invoke this API. As only 80 and 443 are opened
on AWS.

How do we overcome this problem?

Cheers,

Ryan H

Re: HandleHttpRequest API Port Exposure

[Joe Witt]

Ryan,

We had a very similar question just a bit ago.

https://lists.apache.org/thread.html/1289510c6d69176907273be08f1d1548182af6bdec0c4f51dee1c38b@%3Cdev.nifi.apache.org%3E

Thanks
Joe

On Thu, Mar 2, 2017 at 3:32 PM, Ryan H
 wrote:
> Hello All,
>
> I  am exposing an API using HandleHttpRequest on my local nifi instance. The
> HandleHttpRequest processor requires a Listening port that I need to
> provide. If I enter 80 in that field, the processor fails when it starts
> saying “unable to initialize the server”. Which is expected as the webserver
> uses that port to serve the canvas. So if we provide any other random number
> then it works fine.
>
> When I promote the above API on the nifi cluster that is hosted on our AWS
> farm, then we are unable to invoke this API. As only 80 and 443 are opened
> on AWS.
>
> How do we overcome this problem?
>
> Cheers,
>
> Ryan H

Re: NiFi 1.1.1 Secure Cluster Setup Issue: HTTPS Wrong Hostname wrong should be:

[Ryan H]

Hi Andy,

I wanted to follow up with my solution to the error I was receiving with
the "HTTPS Wrong Hostname" issue. The root of the problem was that I was
using an actual IP address for the certificate generation as well as in the
config files. The fix was to create a hostname on my server and use the
Hostname instead of the IP address. This corrected the problem and allowed
me to continue with the Secure Cluster Setup.

Thanks for the Help!!

Cheers,

Ryan H.

On Wed, Mar 1, 2017 at 1:11 PM, Ryan H 
wrote:

> Hi Andy,
>
> Thanks you as well for a quick response.
>
> Your Questions:
> * Are you trying to run multiple nodes on the same IP address?
> -Yes, for the purpose of the initial set up and "proof of concept" I am
> running more than one NiFi instance on the same machine (and same IP
> address). Once that was working for me, I would move to multiple machines
> (ideally with a single node on each machine which I believe is what is
> recommended)
>
> * What is the output of examining the keytool command against the
> conf/keystore.jks?
> * Does the certificate used for NiFi contain the exact hostname as the CN?
> An IP address? Any SAN entries?
> -It did have the same CN, but it was an IP address and not the hostname
> which looked to be the root of the problem.
> -No SAN entries, but the ability to add them with the NiFi-Toolkit will
> surely be beneficial in the next stages of my development.
>
> * Are you running NiFi as root? Starting a process to listen on 443 (or
> any port below 1024) requires root.
> -Yes
>
> * Is the cluster port unique on each node?
> -Yes, each node is configured with different ports. (80, 81, 82, etc)
>
> * Did you run the TLS toolkit once to generate 3 unique certificates and
> configurations, 3 separate times, or once with one certificate generated
> which you shared across multiple nodes?
> -I've tried pretty much each permutation of this. The most recent was
> running the following commands:
>
> When trying to cluster 3 nodes on same machine:
> ./bin/tls-toolkit.sh standalone -n 'my.ip.address(3)' -C 'CN=admin,
> OU=NIFI' -o './mydir'
>
> When trying for just a single node as a cluster:
> ./bin/tls-toolkit.sh standalone -n 'my.ip.address' -C 'CN=admin, OU=NIFI'
> -o './mydir'
>
> * Are the ports available in whatever AWS firewall you’re using?
> -Yes, this is set up correctly as far as I can tell.
>
> Bryan B responded just a few minutes before your response and he suggested
> that using an IP address may not work; which looked to be the root of the
> problem. I instead used the servers hostname instead and I got past the
> problem. I said that I would follow up with my full solution once
> everything seems to be stable so any other members that may run in to the
> issue have another documented solution.
>
> Thanks for the help and direction! I will follow up in the next day with
> my solution.
>
> Cheers,
>
> Ryan H.
>
>
> On Wed, Mar 1, 2017 at 9:36 AM, Andy LoPresto 
> wrote:
>
>> Hi Ryan,
>>
>> Sorry you are encountering problems. You’ve clearly done research on the
>> issue and we always want to make this process easier and more
>> straightforward.
>>
>> The user log is showing you can successfully authenticate with the client
>> certificate. I have a couple questions:
>>
>> * Are you trying to run multiple nodes on the same IP address?
>> * What is the output of examining the keytool command against the
>> conf/keystore.jks?
>> * Does the certificate used for NiFi contain the exact hostname as the
>> CN? An IP address? Any SAN entries?
>> * Are you running NiFi as root? Starting a process to listen on 443 (or
>> any port below 1024) requires root.
>> * Is the cluster port unique on each node?
>> * Did you run the TLS toolkit once to generate 3 unique certificates and
>> configurations, 3 separate times, or once with one certificate generated
>> which you shared across multiple nodes?
>> * Are the ports available in whatever AWS firewall you’re using?
>>
>> From the stacktrace, it appears the issue is during the request
>> replication from the node whose UI you are hitting to the other nodes in
>> the cluster. The “HTTPS hostname wrong: should be” message is almost always
>> related to the certificate hostname not matching the host/IP that the
>> service is running.
>>
>> Can you try this command from the terminal on a node that works (“node
>> 1”) to another node (“node 2”)?
>>
>> * host - the hostname/IP of “node 2”
>> * port - the port of “node 2”
>> * path_to_your_cert - the certificate containing the public key of the
>> client key pair in PEM encoding (see command below)
>> * path_to_your_key - the client private key in PEM encoding
>> * path_to_your_CA_cert - the certificate containing the public key of the
>> CA (for server certificate path verification) in PEM encoding
>>
>> $ openssl s_client -connect  -debug -state -cert
>>  -key  -CAfile
>> 
>>
>> The response from this should be extensive, but the interesting portion
>> is the end — you should see the 

Re: HandleHttpRequest API Port Exposure

[Ryan H]

Hi Joe,

Thanks for the quick response. I will follow that thread to see what the
outcome is.

Cheers,

Ryan H.

On Thu, Mar 2, 2017 at 3:36 PM, Joe Witt  wrote:

> Ryan,
>
> We had a very similar question just a bit ago.
>
> https://lists.apache.org/thread.html/1289510c6d69176907273be08f1d15
> 48182af6bdec0c4f51dee1c38b@%3Cdev.nifi.apache.org%3E
>
> Thanks
> Joe
>
> On Thu, Mar 2, 2017 at 3:32 PM, Ryan H
>  wrote:
> > Hello All,
> >
> > I  am exposing an API using HandleHttpRequest on my local nifi instance.
> The
> > HandleHttpRequest processor requires a Listening port that I need to
> > provide. If I enter 80 in that field, the processor fails when it starts
> > saying “unable to initialize the server”. Which is expected as the
> webserver
> > uses that port to serve the canvas. So if we provide any other random
> number
> > then it works fine.
> >
> > When I promote the above API on the nifi cluster that is hosted on our
> AWS
> > farm, then we are unable to invoke this API. As only 80 and 443 are
> opened
> > on AWS.
> >
> > How do we overcome this problem?
> >
> > Cheers,
> >
> > Ryan H
>

Aw: Re: Re: Re: new Nifi Processors

[Uwe Geercken]

Thanks for all your resposes and help.

One last question (at least for the moment ;-) ):

Should I place a license file in the nar file? Or is it enough to place it in 
the code? Any conventions from the Apache side?

Rgds,

Uwe


> Gesendet: Donnerstag, 02. März 2017 um 15:53 Uhr
> Von: "Matt Burgess" 
> An: users@nifi.apache.org
> Betreff: Re: Re: Re: new Nifi Processors
>
> Uwe,
> 
> If your NAR can be licensed under the Apache Software License 2.0,
> then you shouldn't run into any issues with other folks want to
> package your software, they can even package it up and license it
> under a commercial (paid) license; the ASL 2.0 is pretty permissive.
> There are some patent protections in there, as well as some rules
> about explicitly specifying any code you've changed, and rules about
> use of the project name (like you can't sell a version of NiFi with
> your additional NAR and call it "NiFi++").
> 
> Regards,
> Matt
> 
> On Thu, Mar 2, 2017 at 7:10 AM, Uwe Geercken  wrote:
> > Thanks Andrew,
> >
> > Matt also pointed me to the same direction.
> >
> > But my question is, what if I switch to the Apache license model and some
> > other software or distribution wants to package my software and they use a
> > different licence model. Will I have a similar problem there? Do you know?
> >
> > I will spend some time on the topic on the weekend...
> >
> > Rgds,
> >
> > Uwe
> >
> > Gesendet: Donnerstag, 02. März 2017 um 05:06 Uhr
> > Von: "Andrew Grande" 
> > An: users@nifi.apache.org
> > Betreff: Re: Re: new Nifi Processors
> >
> > Basically the GPL license puts restrictions on how one can distribute in
> > practical terms. Meaning your work may live under GPL license as long as
> > it's not part of the official package. End users will have to download your
> > NAR themselves.
> >
> > Andrew
> >
> >
> > On Wed, Mar 1, 2017, 8:43 AM Matt Burgess  wrote:
> >>
> >> Uwe,
> >>
> >> Sorry for misspeaking, by "official Apache NiFi repo" I meant the
> >> Apache NiFi codebase (the "built-in" processors, e.g.). For the
> >> licensing part, if you distribute something with GPL binary
> >> dependencies, I believe the entire distribution must be licensed as
> >> GPL or something GPL-compatible.  This is not a bad thing, but due to
> >> Apache licensing, such a processor could not be accepted as-is into
> >> the NiFi codebase. Even LGPL binary dependencies are not allowed.
> >> However as you have made your processor available via your own repo,
> >> the community is free to download and use your processor under the
> >> terms of your license.  However if someone packaged up a NiFi
> >> distribution with a GPL-licensed processor (for example), they would
> >> not be allowed to distribute that package under an Apache 2.0 license;
> >> rather I believe the whole package would have to be licensed under the
> >> GPL.
> >>
> >> I am no licensing expert by any means, but I have had experience with
> >> these kinds of things, both for NiFi and other extensible open-source
> >> projects.
> >>
> >> Regards,
> >> Matt
> >>
> >> On Wed, Mar 1, 2017 at 7:01 AM, Uwe Geercken  wrote:
> >> > Matt,
> >> >
> >> > I did not know there is an official Apache Nifi repo. If you send me a
> >> > link, I will have a look.
> >> >
> >> > Also, is there an official way of tagging, annotating or otherwise
> >> > documenting the license model for a processor? At which point in the 
> >> > code,
> >> > documentation do I have to place license information?
> >> >
> >> > I will check if the Apache license fits to my personal ideas of how my
> >> > software should be protected. I am not a license expert, so I will have 
> >> > to
> >> > spend some time to understand what that means. Also I need to check what 
> >> > it
> >> > means for the software (and current users) if I change the license model.
> >> >
> >> > Anyway, this is still a first version of the processors. So they will
> >> > mature over time and I hope at that point the extension registry is 
> >> > there.
> >> >
> >> > In general - as you know Matt - I am creating open source software
> >> > (since 2000). I believe in the idea of open source and of sharing for the
> >> > benefit of all of us.
> >> >
> >> > If I can, I will adjust whatever is necessary, so that the license is
> >> > not a hurdle for using the processors. Nifi is a really great product 
> >> > and I
> >> > still remember my first impression when I saw it.
> >> >
> >> > Greetings,
> >> >
> >> > Uwe
> >> >
> >> >> Gesendet: Mittwoch, 01. März 2017 um 03:56 Uhr
> >> >> Von: "Matt Burgess" 
> >> >> An: users@nifi.apache.org
> >> >> Betreff: Re: new Nifi Processors
> >> >>
> >> >> Uwe G has made his processors available (thank you!) via his own repo
> >> >> vs the official Apache NiFi repo; this may be directly related to your
> >> >> point about licensing.  Having said that, he is of course at liberty
> >> >> to license those separate processors as he sees fit (assuming it is
> >> >> also in accordance with the licenses he h

Re: Re: Re: Re: new Nifi Processors

[Joe Witt]

Uwe

To progress toward a pull request for inclusion in the nifi community
we will need a LICENSE/NOTICE to end up in the nar bundle itself.  You
can see many examples of this in other nars such as [1]

If you don't need to edit the LICENSE you can not provide it and a
default one will be provided.  Same for NOTICE.  The learning curve on
proper licensing is not pleasant and we're in all the license struggle
together so we can help get it right.

You're doing some really cool work here.  Will be awesome if we can
work with you to get the things you'd like contributed into formal
contribution/PR status.  Certainly you don't have to do that and can
instead just publish them outside the nifi community.  We're happy to
help either way.

[1] 
https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-ignite-bundle/nifi-ignite-nar/src/main/resources/META-INF

Thanks
Joe

On Thu, Mar 2, 2017 at 3:56 PM, Uwe Geercken  wrote:
> Thanks for all your resposes and help.
>
> One last question (at least for the moment ;-) ):
>
> Should I place a license file in the nar file? Or is it enough to place it in 
> the code? Any conventions from the Apache side?
>
> Rgds,
>
> Uwe
>
>
>> Gesendet: Donnerstag, 02. März 2017 um 15:53 Uhr
>> Von: "Matt Burgess" 
>> An: users@nifi.apache.org
>> Betreff: Re: Re: Re: new Nifi Processors
>>
>> Uwe,
>>
>> If your NAR can be licensed under the Apache Software License 2.0,
>> then you shouldn't run into any issues with other folks want to
>> package your software, they can even package it up and license it
>> under a commercial (paid) license; the ASL 2.0 is pretty permissive.
>> There are some patent protections in there, as well as some rules
>> about explicitly specifying any code you've changed, and rules about
>> use of the project name (like you can't sell a version of NiFi with
>> your additional NAR and call it "NiFi++").
>>
>> Regards,
>> Matt
>>
>> On Thu, Mar 2, 2017 at 7:10 AM, Uwe Geercken  wrote:
>> > Thanks Andrew,
>> >
>> > Matt also pointed me to the same direction.
>> >
>> > But my question is, what if I switch to the Apache license model and some
>> > other software or distribution wants to package my software and they use a
>> > different licence model. Will I have a similar problem there? Do you know?
>> >
>> > I will spend some time on the topic on the weekend...
>> >
>> > Rgds,
>> >
>> > Uwe
>> >
>> > Gesendet: Donnerstag, 02. März 2017 um 05:06 Uhr
>> > Von: "Andrew Grande" 
>> > An: users@nifi.apache.org
>> > Betreff: Re: Re: new Nifi Processors
>> >
>> > Basically the GPL license puts restrictions on how one can distribute in
>> > practical terms. Meaning your work may live under GPL license as long as
>> > it's not part of the official package. End users will have to download your
>> > NAR themselves.
>> >
>> > Andrew
>> >
>> >
>> > On Wed, Mar 1, 2017, 8:43 AM Matt Burgess  wrote:
>> >>
>> >> Uwe,
>> >>
>> >> Sorry for misspeaking, by "official Apache NiFi repo" I meant the
>> >> Apache NiFi codebase (the "built-in" processors, e.g.). For the
>> >> licensing part, if you distribute something with GPL binary
>> >> dependencies, I believe the entire distribution must be licensed as
>> >> GPL or something GPL-compatible.  This is not a bad thing, but due to
>> >> Apache licensing, such a processor could not be accepted as-is into
>> >> the NiFi codebase. Even LGPL binary dependencies are not allowed.
>> >> However as you have made your processor available via your own repo,
>> >> the community is free to download and use your processor under the
>> >> terms of your license.  However if someone packaged up a NiFi
>> >> distribution with a GPL-licensed processor (for example), they would
>> >> not be allowed to distribute that package under an Apache 2.0 license;
>> >> rather I believe the whole package would have to be licensed under the
>> >> GPL.
>> >>
>> >> I am no licensing expert by any means, but I have had experience with
>> >> these kinds of things, both for NiFi and other extensible open-source
>> >> projects.
>> >>
>> >> Regards,
>> >> Matt
>> >>
>> >> On Wed, Mar 1, 2017 at 7:01 AM, Uwe Geercken  wrote:
>> >> > Matt,
>> >> >
>> >> > I did not know there is an official Apache Nifi repo. If you send me a
>> >> > link, I will have a look.
>> >> >
>> >> > Also, is there an official way of tagging, annotating or otherwise
>> >> > documenting the license model for a processor? At which point in the 
>> >> > code,
>> >> > documentation do I have to place license information?
>> >> >
>> >> > I will check if the Apache license fits to my personal ideas of how my
>> >> > software should be protected. I am not a license expert, so I will have 
>> >> > to
>> >> > spend some time to understand what that means. Also I need to check 
>> >> > what it
>> >> > means for the software (and current users) if I change the license 
>> >> > model.
>> >> >
>> >> > Anyway, this is still a first version of the processors. So they will
>> >> > mature over time and I hope at t

Processor using Apache Thrift ?

[Márcio Faria]

Hi Andy,

Apologies both for this delayed response and the missing subject of the 
original message.

Thank you for your suggestions. 

Option # 1, I'm afraid, is not applicable. You see, working with Apache Thrift 
[1], similarly to Protocol Buffers [2], involves writing specifications of data 
types and service interfaces in a platform-independent "definition language", 
then "compiling" them into, e.g., Java, Python, or C++ code. The final result 
is a client program or library that can read/write only messages complying with 
those specific definitions. There is no generic client.
Option # 2 would be my first choice then, but Chris Herrera, who also replied 
to this thread, recommended Option #3, and for good reasons.
I know that binary protocols in general offer better serialization performance, 
but there are many valid use cases where employing more verbose formats like 
JSON or even XML (argh!) is fine, or at least good enough. I just wish that 
working with Thrift in NiFi was as easy as working with Avro. 
[1] http://thrift.apache.org/
[2] https://developers.google.com/protocol-buffers/


Thank you,
Marcio

On Tuesday, February 14, 2017 10:31 PM, Andy LoPresto  
wrote:



Hello Márcio,

I have not used Thrift, but in reading the introduction from the link you 
provided, my first reactions would be as follows:

1. To communicate with a service defined by Thrift, I’d look at InvokeHTTP and 
see if it covers your needs. 
2. If not, I would try downloading the Thrift client and invoking it via Groovy 
in an ExecuteScript processor.
3. If that is not sufficient, you would have to build a custom processor and 
include Thrift as a dependency. 

Someone else may have more experience, but I would try in that order (from 
easiest to hardest). Good luck. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Feb 14, 2017, at 6:24 PM, Márcio Faria  wrote:
>
>Dear NiFi users,
>
>Does anybody have an example of a NiFi component talking to a Apache Thrift 
>(https://thrift.apache.org/) -based webservice?
>
>Thank you,
>
>Marcio
>

Re: Processor using Apache Thrift ?

[Márcio Faria]

Hi Chris,
Thank you for sharing your experience using Thrift with NiFi, and please accept 
my apologies for this delayed response. 
Writing a new processor is also what I'm now considering to do. The Maven 
archetype will be a good start.
Thanks again,
Marcio

On Tuesday, February 14, 2017 10:41 PM, Chris Herrera 
 wrote:

Try that again sending from the right address this time :).

On Feb 14, 2017, at 9:40 PM, Chris Herrera  wrote:
>
Hi Marcio,

I played around with this a bit and eventually needed to build a custom 
processor that essentially used thrift as a dependancy. Originally I tried it 
through ExecuteScript but it took me more time than just writing a new 
processor. 
There is a really good example of doing this here: 
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-MavenProcessorArchetype
 using the maven archtype. 

Regards,
Chris

On Feb 14, 2017, at 8:24 PM, Márcio Faria  wrote:
>
>Dear NiFi users,
>
>Does anybody have an example of a NiFi component talking to a Apache Thrift 
>(https://thrift.apache.org/) -based webservice?
>
>Thank you,
>
>Marcio

Loss tolerance in NiFi

[Andre]

All,

A good number of NiFi related material makes the following statement about
NiFi:

Loss tolerant vs guaranteed delivery

I have not historically used NiFi in a loss tolerant environment but after
investigating the documentation it isn't clear to me how NiFi implements
loss tolerance?

Are we referring to Clustering? Flow expiration? To the ability to discard
flowfiles that get piped to the REL_FAILURE relationships? etc?

Would anyone mind shedding some light about what are we referring to when
presenting the "Loss tolerant vs guaranteed delivery"

I thank you in advance

Re: How to specify priority attributes for seperate flowfiles?

[prabhu Mahendran]

Yes, exactly you got my point.



Consider the filename contains date, how to prioritze the files from the
directory to come first based on the date(oldest date comes first to the
latest date comes last)?



Issue faced here: Consider I have 2 files in the directory, after the
GetFile->SplitText->ExtractText, I used priority attribute in
UpdateAttribute. Now each file is initalized with priority 1...10. For
file1, each records has 1 to 10 priority value, similarly for file2, each
records has 1 to 10 priority value. Actually I want input files to be
prioritized based on date in the filename?  So that finally, oldest date
records will be processed first and then the latest date records.



On Thu, Mar 2, 2017 at 6:39 PM, Bryan Bende  wrote:

> So in your example you are saying that 10 files get placed in a
> directory, and inside each of those 10 files the data is already
> ordered the way you want, but you want to ensure the 10 files get
> processed in a specific order?
>
> If that is true, what determines the order of the 10 files? is it
> based on the order they were written to the directory? or is there
> something in the filename that indicates which file comes first? In
> order for NiFi to prioritize these files, there has to be something
> that tells NiFi what the priority is.
>
> On Wed, Mar 1, 2017 at 11:56 PM, prabhu Mahendran
>  wrote:
> > As you suggested, setting 3 UpdateAttribute may be tedious. Suppose I
> have
> > more than 10 flowfiles setting 10 updateattribute processor is lengthy
> one.
> > This case also not possible for dynamically generating flowfiles.
> >
> >
> >
> > How to set priority attribute for the flowfiles from Getfile? Suppose I
> get
> > 10 files in the Getfile processor, based on my priority I have ordered
> the
> > flowfile each line in the files till PutSQL. Here without considering the
> > order, based on the filecreation time, data is moved without my ordered
> > records. For this case only I decided with the
> PriorityAttributePrioritizer
> > and used UpdateAttribute processor.
> >
> >
> >
> > I can able to set the priority attribute for each line in the file, but
> not
> > each files from GetFile. Can you suggest any solution?
> >
> >
> >
> >
> > On Wed, Mar 1, 2017 at 7:18 PM, Bryan Bende  wrote:
> >>
> >> I just responded to this question on stackoverflow:
> >>
> >>
> >> https://stackoverflow.com/questions/42528993/how-to-
> specify-priority-attributes-for-seperate-flowfiles
> >>
> >> Thanks,
> >>
> >> Bryan
> >>
> >> On Wed, Mar 1, 2017 at 5:19 AM, prabhu Mahendran
> >>  wrote:
> >> > I need to use PrioritizeAttributePrioritizer in NiFi.
> >> >
> >> > i have observed that prioritizers in below reference.
> >> > https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings
> >> >
> >> > if i receive 10 flowfiles then i need to set the priority value for
> >> > every
> >> > flow file to be unique.
> >> >
> >> > After that specify queue configuration must be
> >> > PrioritizeAttributePrioritizer.
> >> >
> >> > Then processing flowfiles based on priority value.
> >> >
> >> > How can i set priority value for seperate flow files or which
> >> > prioritizer in
> >> > Nifi to be work for my case?
> >
> >
>

Re: How to specify priority attributes for seperate flowfiles?

[Andre]

Hi,

There's an existing JIRA ticket(NIFI-470) requesting a way to allow a DFM
to fine tune how GetFile build it's queues and control how to prioritise
the consumption of files.

Would that be what you are looking after?

Cheers

On 3 Mar 2017 15:55, "prabhu Mahendran"  wrote:

Yes, exactly you got my point.



Consider the filename contains date, how to prioritze the files from the
directory to come first based on the date(oldest date comes first to the
latest date comes last)?



Issue faced here: Consider I have 2 files in the directory, after the
GetFile->SplitText->ExtractText, I used priority attribute in
UpdateAttribute. Now each file is initalized with priority 1...10. For
file1, each records has 1 to 10 priority value, similarly for file2, each
records has 1 to 10 priority value. Actually I want input files to be
prioritized based on date in the filename?  So that finally, oldest date
records will be processed first and then the latest date records.



On Thu, Mar 2, 2017 at 6:39 PM, Bryan Bende  wrote:

> So in your example you are saying that 10 files get placed in a
> directory, and inside each of those 10 files the data is already
> ordered the way you want, but you want to ensure the 10 files get
> processed in a specific order?
>
> If that is true, what determines the order of the 10 files? is it
> based on the order they were written to the directory? or is there
> something in the filename that indicates which file comes first? In
> order for NiFi to prioritize these files, there has to be something
> that tells NiFi what the priority is.
>
> On Wed, Mar 1, 2017 at 11:56 PM, prabhu Mahendran
>  wrote:
> > As you suggested, setting 3 UpdateAttribute may be tedious. Suppose I
> have
> > more than 10 flowfiles setting 10 updateattribute processor is lengthy
> one.
> > This case also not possible for dynamically generating flowfiles.
> >
> >
> >
> > How to set priority attribute for the flowfiles from Getfile? Suppose I
> get
> > 10 files in the Getfile processor, based on my priority I have ordered
> the
> > flowfile each line in the files till PutSQL. Here without considering the
> > order, based on the filecreation time, data is moved without my ordered
> > records. For this case only I decided with the
> PriorityAttributePrioritizer
> > and used UpdateAttribute processor.
> >
> >
> >
> > I can able to set the priority attribute for each line in the file, but
> not
> > each files from GetFile. Can you suggest any solution?
> >
> >
> >
> >
> > On Wed, Mar 1, 2017 at 7:18 PM, Bryan Bende  wrote:
> >>
> >> I just responded to this question on stackoverflow:
> >>
> >>
> >> https://stackoverflow.com/questions/42528993/how-to-specify-
> priority-attributes-for-seperate-flowfiles
> >>
> >> Thanks,
> >>
> >> Bryan
> >>
> >> On Wed, Mar 1, 2017 at 5:19 AM, prabhu Mahendran
> >>  wrote:
> >> > I need to use PrioritizeAttributePrioritizer in NiFi.
> >> >
> >> > i have observed that prioritizers in below reference.
> >> > https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings
> >> >
> >> > if i receive 10 flowfiles then i need to set the priority value for
> >> > every
> >> > flow file to be unique.
> >> >
> >> > After that specify queue configuration must be
> >> > PrioritizeAttributePrioritizer.
> >> >
> >> > Then processing flowfiles based on priority value.
> >> >
> >> > How can i set priority value for seperate flow files or which
> >> > prioritizer in
> >> > Nifi to be work for my case?
> >
> >
>

Re: How to specify priority attributes for seperate flowfiles?

[prabhu Mahendran]

This task(NIFI-470) suits to some of the workflow. If I set concurrent task
to 10, records runs in parallel so that each file gets shuffled as I can
see in the List Queue.



If we get order of files from the Getfile, How I can ensure the data from
each file is properly moved to destination(consider SQL) in same order with
respect to concurrent task also?



I need flow like this: Consider file1 has 10 records and it should be
priortized from the value 1 to 10, then next file2 records should start
with the priority value 11 to so on.. Filename can be in the order of the
date from the getfile processor. Here I can ensure each ordered files are
moved in the same order into SQL.



Will this be achieved in the ticket or any suggestion for this?

On Fri, Mar 3, 2017 at 11:37 AM, Andre  wrote:

> Hi,
>
> There's an existing JIRA ticket(NIFI-470) requesting a way to allow a DFM
> to fine tune how GetFile build it's queues and control how to prioritise
> the consumption of files.
>
> Would that be what you are looking after?
>
> Cheers
>
>
> On 3 Mar 2017 15:55, "prabhu Mahendran"  wrote:
>
> Yes, exactly you got my point.
>
>
>
> Consider the filename contains date, how to prioritze the files from the
> directory to come first based on the date(oldest date comes first to the
> latest date comes last)?
>
>
>
> Issue faced here: Consider I have 2 files in the directory, after the
> GetFile->SplitText->ExtractText, I used priority attribute in
> UpdateAttribute. Now each file is initalized with priority 1...10. For
> file1, each records has 1 to 10 priority value, similarly for file2, each
> records has 1 to 10 priority value. Actually I want input files to be
> prioritized based on date in the filename?  So that finally, oldest date
> records will be processed first and then the latest date records.
>
>
>
> On Thu, Mar 2, 2017 at 6:39 PM, Bryan Bende  wrote:
>
>> So in your example you are saying that 10 files get placed in a
>> directory, and inside each of those 10 files the data is already
>> ordered the way you want, but you want to ensure the 10 files get
>> processed in a specific order?
>>
>> If that is true, what determines the order of the 10 files? is it
>> based on the order they were written to the directory? or is there
>> something in the filename that indicates which file comes first? In
>> order for NiFi to prioritize these files, there has to be something
>> that tells NiFi what the priority is.
>>
>> On Wed, Mar 1, 2017 at 11:56 PM, prabhu Mahendran
>>  wrote:
>> > As you suggested, setting 3 UpdateAttribute may be tedious. Suppose I
>> have
>> > more than 10 flowfiles setting 10 updateattribute processor is lengthy
>> one.
>> > This case also not possible for dynamically generating flowfiles.
>> >
>> >
>> >
>> > How to set priority attribute for the flowfiles from Getfile? Suppose I
>> get
>> > 10 files in the Getfile processor, based on my priority I have ordered
>> the
>> > flowfile each line in the files till PutSQL. Here without considering
>> the
>> > order, based on the filecreation time, data is moved without my ordered
>> > records. For this case only I decided with the
>> PriorityAttributePrioritizer
>> > and used UpdateAttribute processor.
>> >
>> >
>> >
>> > I can able to set the priority attribute for each line in the file, but
>> not
>> > each files from GetFile. Can you suggest any solution?
>> >
>> >
>> >
>> >
>> > On Wed, Mar 1, 2017 at 7:18 PM, Bryan Bende  wrote:
>> >>
>> >> I just responded to this question on stackoverflow:
>> >>
>> >>
>> >> https://stackoverflow.com/questions/42528993/how-to-specify-
>> priority-attributes-for-seperate-flowfiles
>> >>
>> >> Thanks,
>> >>
>> >> Bryan
>> >>
>> >> On Wed, Mar 1, 2017 at 5:19 AM, prabhu Mahendran
>> >>  wrote:
>> >> > I need to use PrioritizeAttributePrioritizer in NiFi.
>> >> >
>> >> > i have observed that prioritizers in below reference.
>> >> > https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings
>> >> >
>> >> > if i receive 10 flowfiles then i need to set the priority value for
>> >> > every
>> >> > flow file to be unique.
>> >> >
>> >> > After that specify queue configuration must be
>> >> > PrioritizeAttributePrioritizer.
>> >> >
>> >> > Then processing flowfiles based on priority value.
>> >> >
>> >> > How can i set priority value for seperate flow files or which
>> >> > prioritizer in
>> >> > Nifi to be work for my case?
>> >
>> >
>>
>
>
>