Re: NiFi Devs: powered by nifi page
Hi Joe, I have tried to make it easier by preparing pull request https://github.com/apache/nifi-site/pull/12 I hope I understood the page structure well :) Thanks, Michal Klempa On Tue, Feb 14, 2017 at 8:47 PM, Joe Witt wrote: > your/you're...either way it is live :-) > > On Tue, Feb 14, 2017 at 2:47 PM, Joe Witt wrote: >> Thanks Ronnie. You're info is now live. >> >> On Tue, Feb 14, 2017 at 2:31 PM, Ronnie Dove wrote: >>> Company Name: Dovestech (linking to http://www.dovestech.com) >>> Industry: Cyber Security >>> Summary: Dovestech's cyber security visualization product, ThreatPop (link: >>> http://www.dovestech.com/threatpop), uses Apache NiFi to enrich and >>> normalize millions of cyber security related events into a central database >>> which allows customers to interact with cyber security events through game >>> engine visualization technology. >>> >>> On Tue, Feb 14, 2017 at 2:08 PM, Joe Witt wrote: >>> NiFi Users I just realized we have a 'powered by nifi' page. It looks a little...light :-). So wanted to reach out and offer to anyone interested that if you reply back on this thread with your company/organization that you'd like referenced on there I'd be happy to put in the change to the site for you. https://nifi.apache.org/powered-by-nifi.html We are aware of a very large user base and obviously can see quite a bit of this on the Internet but I don't think we can/should put that unless folks volunteer to have this on the page. So please let us know if you're interested in your company/organizational use of NiFi being mentioned there. Thanks Joe >>> >>> >>> >>> -- >>> Ronnie Dove
Re: NiFi Devs: powered by nifi page
Michal, Thanks for providing this. I merged your PR and pushed to the site, so Slovak Telekom is live on the list now. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 15, 2017, at 12:09 AM, Michal Klempa wrote: > > Hi Joe, > I have tried to make it easier by preparing pull request > https://github.com/apache/nifi-site/pull/12 > I hope I understood the page structure well :) > > Thanks, Michal Klempa > > On Tue, Feb 14, 2017 at 8:47 PM, Joe Witt wrote: >> your/you're...either way it is live :-) >> >> On Tue, Feb 14, 2017 at 2:47 PM, Joe Witt wrote: >>> Thanks Ronnie. You're info is now live. >>> >>> On Tue, Feb 14, 2017 at 2:31 PM, Ronnie Dove wrote: Company Name: Dovestech (linking to http://www.dovestech.com) Industry: Cyber Security Summary: Dovestech's cyber security visualization product, ThreatPop (link: http://www.dovestech.com/threatpop), uses Apache NiFi to enrich and normalize millions of cyber security related events into a central database which allows customers to interact with cyber security events through game engine visualization technology. On Tue, Feb 14, 2017 at 2:08 PM, Joe Witt wrote: > NiFi Users > > I just realized we have a 'powered by nifi' page. It looks a > little...light :-). So wanted to reach out and offer to anyone > interested that if you reply back on this thread with your > company/organization that you'd like referenced on there I'd be happy > to put in the change to the site for you. > > https://nifi.apache.org/powered-by-nifi.html > > We are aware of a very large user base and obviously can see quite a > bit of this on the Internet but I don't think we can/should put that > unless folks volunteer to have this on the page. So please let us > know if you're interested in your company/organizational use of NiFi > being mentioned there. > > Thanks > Joe > -- Ronnie Dove signature.asc Description: Message signed with OpenPGP using GPGMail
Something is Wrong When I Use PUTHDFS processor
when i want to put the dataflow together into hdfs with the same filename .it can not work by using append.My version is NIFI 1.1.what shoud i do? c...@kingnet.com
Re: Something is Wrong When I Use PUTHDFS processor
Can you provide the error message and stacktrace from nifi-app.log? On Wed, Feb 15, 2017 at 5:29 AM, c...@kingnet.com wrote: > when i want to put the dataflow together into hdfs with the same filename .it > can not work by using append.My version is NIFI 1.1.what shoud i do? > > > > c...@kingnet.com
Re: Starting apache NiFi as a windows service
Andre, can you share any details on how you’re approaching this? Are you using an existing service wrapper? The javapackager tool added in Java 8? Etc. On Feb 14, 2017, 8:50 PM -0600, Andre , wrote: > Chris, > > Not that I am aware of but I am working on this atm. Should have something > baked until the weekend. > > On Wed, Feb 15, 2017 at 12:25 PM, Chris Bulleri wrote: > > > Are there any instructions for starting the latest apache NiFi as a windows > > service? Server 2012 in particular. > > > > > > Chris Bulleri > > 2HB Inc. > > Principal/CTO > > ch...@2hb.com > > Voice: 410.872.2800 x717 > >
Re: [NIFI-1657] Heads-up: NiFi's travis-ci is going "multi-lingual"
Definitely helpful and it looks like we have picked up a test failure with the way in which generated JSON numbers are formatted. Created a ticket to address: https://issues.apache.org/jira/browse/NIFI-3483 On Wed, Feb 15, 2017 at 2:25 AM, Andy LoPresto wrote: > Andre & Pierre, > > Thanks for tackling this. As NiFi continues to grow internationally, we > hope it will eventually touch down everywhere, and this is a big step > forward. > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Feb 9, 2017, at 2:02 PM, Andre wrote: > > devs, > > In order to improve our ability to detect locale related bugs during > development, Pierre and I have been working on getting the surefire-plugin > to support user defined languages and to add some of these languages to the > travis-ci environment matrix. > > It took a bit of trial and error but it seems like we are finally on the > way to merge. > > Once the PR gets merged, travis-ci will start building using the following > locales: > > * en_US; > * ja_JP; > * pt_BR, and > * fr_FR > > the locale can also be set on you local development environment - without > changes to your overall operating system - by using the following > properties: > > mvn -Pcontrib-check verify -Dsurefire.user.language=hu > -Dsurefire.user.region=HU > > > We know that at this stage some of the tests will fail but hopefully the > community will soon manage to address those. > > Cheers > > >
Re: Moving ZIP and TAR.GZ "formats" into an explicit profile
All, This is a reminder that the dir-only profile has been merged to the master branch. So for those trying to save SSD writes or simply skip a few CPU cycles archiving the nifi assembly during development, you can now use mvn -Pdir-only to skip the pain. I also would like to point out that since commit 7e97946c359bfcb62ef39a2ce37083554501f1b0, parallel test runs now seem to be far more stable. If things continue stable, we may later on re-introduce parallel test runs to travis. Cheers On Fri, Feb 3, 2017 at 11:04 PM, Andre wrote: > devs, > > Thank you for the comments. > > NIFI-3434 has been raised and submitted as PR#1472. The commit introduces > generateArchives (default and [hopefully] mimicking current behavior) and > dir-only (skipping the archive generation). > > Once again thank you for your comments. > > On Fri, Feb 3, 2017 at 10:34 AM, Andy LoPresto > wrote: > >> +1 to Russell’s point and thanks Andre for bringing up the topic. I’d >> hesitate to change the default now because people probably are depending on >> the default, but I wouldn’t object to a shift in the default at a major >> release. >> >> Andy LoPresto >> alopre...@apache.org >> *alopresto.apa...@gmail.com * >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Feb 2, 2017, at 8:27 AM, Russell Bateman >> wrote: >> >> Maven profiles could be used to sort this out, one for just the zip, >> another for just the tarball and a third, maybe the default, to continue >> working the way it has been working. >> >> On 02/02/2017 09:09 AM, Aldrin Piri wrote: >> >> I think this could be useful. Only caveat is that I'm sure there are >> folks >> in the community that have automated processes that make use of these >> binaries. >> >> From the dev standpoint, I could see a profile that disables the assembly >> from happening such that the build occurs as it does now unless folks >> explicitly want to avoid it. Regardless of implementation, can see why it >> would be helpful. >> >> On Thu, Feb 2, 2017 at 9:29 AM, Andre wrote: >> >> devs, >> >> Currently calling 'mvn clean install' creates a ZIP, a TAR.GZ and a >> directory containing the same code. This leads to wasted disk space and a >> lot of wasted disk writes (something that a lot of folks using SSDs prefer >> to avoid). >> >> Would anyone oppose the idea of moving the ZIP and TAR.GZ assemblies into >> a >> "release" profile (or whatever name we agree to). This way we could >> maintain the directory "format" (which I suspect most of us use during >> development), while still providing a convenient way of creating the ZIP >> and TAR.GZ packages. >> >> >> Depending on the feedback I will be happy to raise the JIRA and work on >> it. >> >> Cheers >> >> >> >> >
Re: Moving ZIP and TAR.GZ "formats" into an explicit profile
Andre - thanks for this. I just used it. Much happiness. I cannot wait until we get some of this registry work advanced further. Breaking apart NiFi builds to get extensions and core framework aspects separated will result in much happiness for travis-ci and all of us. Thanks joe On Wed, Feb 15, 2017 at 5:12 PM, Andre wrote: > All, > > This is a reminder that the dir-only profile has been merged to the master > branch. So for those trying to save SSD writes or simply skip a few CPU > cycles archiving the nifi assembly during development, you can now use mvn > -Pdir-only to skip the pain. > > I also would like to point out that since > commit 7e97946c359bfcb62ef39a2ce37083554501f1b0, parallel test runs now > seem to be far more stable. If things continue stable, we may later on > re-introduce parallel test runs to travis. > > Cheers > > > > On Fri, Feb 3, 2017 at 11:04 PM, Andre wrote: > >> devs, >> >> Thank you for the comments. >> >> NIFI-3434 has been raised and submitted as PR#1472. The commit introduces >> generateArchives (default and [hopefully] mimicking current behavior) and >> dir-only (skipping the archive generation). >> >> Once again thank you for your comments. >> >> On Fri, Feb 3, 2017 at 10:34 AM, Andy LoPresto >> wrote: >> >>> +1 to Russell’s point and thanks Andre for bringing up the topic. I’d >>> hesitate to change the default now because people probably are depending on >>> the default, but I wouldn’t object to a shift in the default at a major >>> release. >>> >>> Andy LoPresto >>> alopre...@apache.org >>> *alopresto.apa...@gmail.com * >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> >>> On Feb 2, 2017, at 8:27 AM, Russell Bateman >>> wrote: >>> >>> Maven profiles could be used to sort this out, one for just the zip, >>> another for just the tarball and a third, maybe the default, to continue >>> working the way it has been working. >>> >>> On 02/02/2017 09:09 AM, Aldrin Piri wrote: >>> >>> I think this could be useful. Only caveat is that I'm sure there are >>> folks >>> in the community that have automated processes that make use of these >>> binaries. >>> >>> From the dev standpoint, I could see a profile that disables the assembly >>> from happening such that the build occurs as it does now unless folks >>> explicitly want to avoid it. Regardless of implementation, can see why it >>> would be helpful. >>> >>> On Thu, Feb 2, 2017 at 9:29 AM, Andre wrote: >>> >>> devs, >>> >>> Currently calling 'mvn clean install' creates a ZIP, a TAR.GZ and a >>> directory containing the same code. This leads to wasted disk space and a >>> lot of wasted disk writes (something that a lot of folks using SSDs prefer >>> to avoid). >>> >>> Would anyone oppose the idea of moving the ZIP and TAR.GZ assemblies into >>> a >>> "release" profile (or whatever name we agree to). This way we could >>> maintain the directory "format" (which I suspect most of us use during >>> development), while still providing a convenient way of creating the ZIP >>> and TAR.GZ packages. >>> >>> >>> Depending on the feedback I will be happy to raise the JIRA and work on >>> it. >>> >>> Cheers >>> >>> >>> >>> >>
Discuss: Process Group as a Process / "Single Threaded" Process Group
I've been throwing around some ideas for how to migrate processes that would work great in NiFi if you only processed one FlowFile at a time, but if there are multiple Flow Files in the Flow processing through various Processes it would cause issues. One of the concepts I've been playing with is a Process Group that acts more like a 1 threaded processor. One (and only one) FLowFile goes in to the Process Group, 0+ FlowFiles come out (Output port would be optional, kind of like deciding whether or not to auto end a relationship or continue it, but more manual); but no more than 1 FlowFile is ever pulled in at a time. The processor would have an input port, and so long as the Process Group has 1+ Flow Files in it, no new files will flow into the Process Group through the input port. I've been digging around the code, and it doesn't look to difficult to pull off. - New property on ProcessGroup, accompanying UI/validation o Validation would probably need to verify there was 1 Input Port if enabled. - Updated Logic in LocalPort's onTrigger code to check the parent container and see if it has this setting enabled, and if it does how many total FlowFile's are currently in queue. Only transfer in a new FlowFile if 0 flow files in queue. Thoughts? One specific scenario I've come across is a database table that will be truncated at the start of a flow, populated with a FlowFile's contents, updated by a second step to remove duplicates, and then merged into the final table using a third step. Of course if I could use a separate table for each flow file this wouldn't be an issue, but I've run into constraints where I'm limited to a single table.
Re: Discuss: Process Group as a Process / "Single Threaded" Process Group
Peter We have a feature proposal on the nifi wiki for process groups as functions. I think we could update that to cover your case as well https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups with the notion of a single input to the 'function/group' at a time. I guess it has a little different intent than you're needing though in that you're not saying you need it to be referenceable like a function but rather act more like a gate. I wonder if the Wait/Notify processors now on master would help your case. The request as asked is a bit outside the spirit of flow based programming fundamentals that NiFi is built on so I'm hopeful we can find another path that gets you the result you need. Thanks Joe On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) wrote: > I've been throwing around some ideas for how to migrate processes that would > work great in NiFi if you only processed one FlowFile at a time, but if there > are multiple Flow Files in the Flow processing through various Processes it > would cause issues. > > One of the concepts I've been playing with is a Process Group that acts more > like a 1 threaded processor. One (and only one) FLowFile goes in to the > Process Group, 0+ FlowFiles come out (Output port would be optional, kind of > like deciding whether or not to auto end a relationship or continue it, but > more manual); but no more than 1 FlowFile is ever pulled in at a time. The > processor would have an input port, and so long as the Process Group has 1+ > Flow Files in it, no new files will flow into the Process Group through the > input port. I've been digging around the code, and it doesn't look to > difficult to pull off. > > > - New property on ProcessGroup, accompanying UI/validation > > o Validation would probably need to verify there was 1 Input Port if > enabled. > > - Updated Logic in LocalPort's onTrigger code to check the parent > container and see if it has this setting enabled, and if it does how many > total FlowFile's are currently in queue. Only transfer in a new FlowFile if 0 > flow files in queue. > > > Thoughts? > > One specific scenario I've come across is a database table that will be > truncated at the start of a flow, populated with a FlowFile's contents, > updated by a second step to remove duplicates, and then merged into the final > table using a third step. Of course if I could use a separate table for each > flow file this wouldn't be an issue, but I've run into constraints where I'm > limited to a single table. >
Re: Discuss: Process Group as a Process / "Single Threaded" Process Group
Hi Peter, Joe Recently, I've worked on NIFI-3452 "Add Wait processor Wait Mode property". Using Wait/Notify combined with Back-pressure FlowFile count 1, it's possible to limit only one FlowFile to be processed in a part of flow. It's already merged into the master. Example is available here: https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67 https://issues.apache.org/jira/browse/NIFI-3452 It may not be an ideal solution, but it will work for your use case. Thanks, Koji On Thu, Feb 16, 2017 at 1:31 PM, Joe Witt wrote: > Peter > > We have a feature proposal on the nifi wiki for process groups as > functions. I think we could update that to cover your case as well > https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups > with the notion of a single input to the 'function/group' at a time. > I guess it has a little different intent than you're needing though in > that you're not saying you need it to be referenceable like a function > but rather act more like a gate. I wonder if the Wait/Notify > processors now on master would help your case. > > The request as asked is a bit outside the spirit of flow based > programming fundamentals that NiFi is built on so I'm hopeful we can > find another path that gets you the result you need. > > Thanks > Joe > > On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) > wrote: >> I've been throwing around some ideas for how to migrate processes that would >> work great in NiFi if you only processed one FlowFile at a time, but if >> there are multiple Flow Files in the Flow processing through various >> Processes it would cause issues. >> >> One of the concepts I've been playing with is a Process Group that acts more >> like a 1 threaded processor. One (and only one) FLowFile goes in to the >> Process Group, 0+ FlowFiles come out (Output port would be optional, kind of >> like deciding whether or not to auto end a relationship or continue it, but >> more manual); but no more than 1 FlowFile is ever pulled in at a time. The >> processor would have an input port, and so long as the Process Group has 1+ >> Flow Files in it, no new files will flow into the Process Group through the >> input port. I've been digging around the code, and it doesn't look to >> difficult to pull off. >> >> >> - New property on ProcessGroup, accompanying UI/validation >> >> o Validation would probably need to verify there was 1 Input Port if >> enabled. >> >> - Updated Logic in LocalPort's onTrigger code to check the parent >> container and see if it has this setting enabled, and if it does how many >> total FlowFile's are currently in queue. Only transfer in a new FlowFile if >> 0 flow files in queue. >> >> >> Thoughts? >> >> One specific scenario I've come across is a database table that will be >> truncated at the start of a flow, populated with a FlowFile's contents, >> updated by a second step to remove duplicates, and then merged into the >> final table using a third step. Of course if I could use a separate table >> for each flow file this wouldn't be an issue, but I've run into constraints >> where I'm limited to a single table. >>
RE: Discuss: Process Group as a Process / "Single Threaded" Process Group
Thanks Joe, I'll read through it; and give Koji's method a shot too. -Original Message- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Wednesday, February 15, 2017 9:31 PM To: dev@nifi.apache.org Subject: Re: Discuss: Process Group as a Process / "Single Threaded" Process Group Peter We have a feature proposal on the nifi wiki for process groups as functions. I think we could update that to cover your case as well https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups with the notion of a single input to the 'function/group' at a time. I guess it has a little different intent than you're needing though in that you're not saying you need it to be referenceable like a function but rather act more like a gate. I wonder if the Wait/Notify processors now on master would help your case. The request as asked is a bit outside the spirit of flow based programming fundamentals that NiFi is built on so I'm hopeful we can find another path that gets you the result you need. Thanks Joe On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) wrote: > I've been throwing around some ideas for how to migrate processes that would > work great in NiFi if you only processed one FlowFile at a time, but if there > are multiple Flow Files in the Flow processing through various Processes it > would cause issues. > > One of the concepts I've been playing with is a Process Group that acts more > like a 1 threaded processor. One (and only one) FLowFile goes in to the > Process Group, 0+ FlowFiles come out (Output port would be optional, kind of > like deciding whether or not to auto end a relationship or continue it, but > more manual); but no more than 1 FlowFile is ever pulled in at a time. The > processor would have an input port, and so long as the Process Group has 1+ > Flow Files in it, no new files will flow into the Process Group through the > input port. I've been digging around the code, and it doesn't look to > difficult to pull off. > > > - New property on ProcessGroup, accompanying UI/validation > > o Validation would probably need to verify there was 1 Input Port if > enabled. > > - Updated Logic in LocalPort's onTrigger code to check the parent > container and see if it has this setting enabled, and if it does how many > total FlowFile's are currently in queue. Only transfer in a new FlowFile if 0 > flow files in queue. > > > Thoughts? > > One specific scenario I've come across is a database table that will be > truncated at the start of a flow, populated with a FlowFile's contents, > updated by a second step to remove duplicates, and then merged into the final > table using a third step. Of course if I could use a separate table for each > flow file this wouldn't be an issue, but I've run into constraints where I'm > limited to a single table. >
RE: Discuss: Process Group as a Process / "Single Threaded" Process Group
This is good stuff Koji, there is so much new stuff going into NiFi it's hard to keep up. I'm a little disappointed to see that you have to use a Distributed Map Cache. This makes sense for many scenarios, but I wonder what other options there are (of course with extra coding), specifically for non-clustered environments. Any ideas? -Original Message- From: Koji Kawamura [mailto:ijokaruma...@gmail.com] Sent: Wednesday, February 15, 2017 9:35 PM To: dev@nifi.apache.org Subject: Re: Discuss: Process Group as a Process / "Single Threaded" Process Group Hi Peter, Joe Recently, I've worked on NIFI-3452 "Add Wait processor Wait Mode property". Using Wait/Notify combined with Back-pressure FlowFile count 1, it's possible to limit only one FlowFile to be processed in a part of flow. It's already merged into the master. Example is available here: https://gist.github.com/ijokarumawak/85a3d77297ea94614e9f3f2a9dabca67 https://issues.apache.org/jira/browse/NIFI-3452 It may not be an ideal solution, but it will work for your use case. Thanks, Koji On Thu, Feb 16, 2017 at 1:31 PM, Joe Witt wrote: > Peter > > We have a feature proposal on the nifi wiki for process groups as > functions. I think we could update that to cover your case as well > https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Proces > s+Groups with the notion of a single input to the 'function/group' at > a time. > I guess it has a little different intent than you're needing though in > that you're not saying you need it to be referenceable like a function > but rather act more like a gate. I wonder if the Wait/Notify > processors now on master would help your case. > > The request as asked is a bit outside the spirit of flow based > programming fundamentals that NiFi is built on so I'm hopeful we can > find another path that gets you the result you need. > > Thanks > Joe > > On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) > wrote: >> I've been throwing around some ideas for how to migrate processes that would >> work great in NiFi if you only processed one FlowFile at a time, but if >> there are multiple Flow Files in the Flow processing through various >> Processes it would cause issues. >> >> One of the concepts I've been playing with is a Process Group that acts more >> like a 1 threaded processor. One (and only one) FLowFile goes in to the >> Process Group, 0+ FlowFiles come out (Output port would be optional, kind of >> like deciding whether or not to auto end a relationship or continue it, but >> more manual); but no more than 1 FlowFile is ever pulled in at a time. The >> processor would have an input port, and so long as the Process Group has 1+ >> Flow Files in it, no new files will flow into the Process Group through the >> input port. I've been digging around the code, and it doesn't look to >> difficult to pull off. >> >> >> - New property on ProcessGroup, accompanying UI/validation >> >> o Validation would probably need to verify there was 1 Input Port if >> enabled. >> >> - Updated Logic in LocalPort's onTrigger code to check the parent >> container and see if it has this setting enabled, and if it does how many >> total FlowFile's are currently in queue. Only transfer in a new FlowFile if >> 0 flow files in queue. >> >> >> Thoughts? >> >> One specific scenario I've come across is a database table that will be >> truncated at the start of a flow, populated with a FlowFile's contents, >> updated by a second step to remove duplicates, and then merged into the >> final table using a third step. Of course if I could use a separate table >> for each flow file this wouldn't be an issue, but I've run into constraints >> where I'm limited to a single table. >>
Re: Discuss: Process Group as a Process / "Single Threaded" Process Group
definitely Koji's route makes sense to me. If the wait/notify procs need to leverage a different/more abstract CS that could be workable. On Wed, Feb 15, 2017 at 11:43 PM, Peter Wicks (pwicks) wrote: > Thanks Joe, I'll read through it; and give Koji's method a shot too. > > -Original Message- > From: Joe Witt [mailto:joe.w...@gmail.com] > Sent: Wednesday, February 15, 2017 9:31 PM > To: dev@nifi.apache.org > Subject: Re: Discuss: Process Group as a Process / "Single Threaded" Process > Group > > Peter > > We have a feature proposal on the nifi wiki for process groups as functions. > I think we could update that to cover your case as well > https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups > with the notion of a single input to the 'function/group' at a time. > I guess it has a little different intent than you're needing though in that > you're not saying you need it to be referenceable like a function but rather > act more like a gate. I wonder if the Wait/Notify processors now on master > would help your case. > > The request as asked is a bit outside the spirit of flow based programming > fundamentals that NiFi is built on so I'm hopeful we can find another path > that gets you the result you need. > > Thanks > Joe > > On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) > wrote: >> I've been throwing around some ideas for how to migrate processes that would >> work great in NiFi if you only processed one FlowFile at a time, but if >> there are multiple Flow Files in the Flow processing through various >> Processes it would cause issues. >> >> One of the concepts I've been playing with is a Process Group that acts more >> like a 1 threaded processor. One (and only one) FLowFile goes in to the >> Process Group, 0+ FlowFiles come out (Output port would be optional, kind of >> like deciding whether or not to auto end a relationship or continue it, but >> more manual); but no more than 1 FlowFile is ever pulled in at a time. The >> processor would have an input port, and so long as the Process Group has 1+ >> Flow Files in it, no new files will flow into the Process Group through the >> input port. I've been digging around the code, and it doesn't look to >> difficult to pull off. >> >> >> - New property on ProcessGroup, accompanying UI/validation >> >> o Validation would probably need to verify there was 1 Input Port if >> enabled. >> >> - Updated Logic in LocalPort's onTrigger code to check the parent >> container and see if it has this setting enabled, and if it does how many >> total FlowFile's are currently in queue. Only transfer in a new FlowFile if >> 0 flow files in queue. >> >> >> Thoughts? >> >> One specific scenario I've come across is a database table that will be >> truncated at the start of a flow, populated with a FlowFile's contents, >> updated by a second step to remove duplicates, and then merged into the >> final table using a third step. Of course if I could use a separate table >> for each flow file this wouldn't be an issue, but I've run into constraints >> where I'm limited to a single table. >>
Re: Discuss: Process Group as a Process / "Single Threaded" Process Group
I searched JIRA for new ControllerService suggestion that works as a MapCacheClient and can be used locally, so I created one, Add LocalMapCacheService https://issues.apache.org/jira/browse/NIFI-3488 On Thu, Feb 16, 2017 at 1:46 PM, Joe Witt wrote: > definitely Koji's route makes sense to me. If the wait/notify procs > need to leverage a different/more abstract CS that could be workable. > > On Wed, Feb 15, 2017 at 11:43 PM, Peter Wicks (pwicks) > wrote: >> Thanks Joe, I'll read through it; and give Koji's method a shot too. >> >> -Original Message- >> From: Joe Witt [mailto:joe.w...@gmail.com] >> Sent: Wednesday, February 15, 2017 9:31 PM >> To: dev@nifi.apache.org >> Subject: Re: Discuss: Process Group as a Process / "Single Threaded" Process >> Group >> >> Peter >> >> We have a feature proposal on the nifi wiki for process groups as functions. >> I think we could update that to cover your case as well >> https://cwiki.apache.org/confluence/display/NIFI/Reference-able+Process+Groups >> with the notion of a single input to the 'function/group' at a time. >> I guess it has a little different intent than you're needing though in that >> you're not saying you need it to be referenceable like a function but rather >> act more like a gate. I wonder if the Wait/Notify processors now on master >> would help your case. >> >> The request as asked is a bit outside the spirit of flow based programming >> fundamentals that NiFi is built on so I'm hopeful we can find another path >> that gets you the result you need. >> >> Thanks >> Joe >> >> On Wed, Feb 15, 2017 at 11:18 PM, Peter Wicks (pwicks) >> wrote: >>> I've been throwing around some ideas for how to migrate processes that >>> would work great in NiFi if you only processed one FlowFile at a time, but >>> if there are multiple Flow Files in the Flow processing through various >>> Processes it would cause issues. >>> >>> One of the concepts I've been playing with is a Process Group that acts >>> more like a 1 threaded processor. One (and only one) FLowFile goes in to >>> the Process Group, 0+ FlowFiles come out (Output port would be optional, >>> kind of like deciding whether or not to auto end a relationship or continue >>> it, but more manual); but no more than 1 FlowFile is ever pulled in at a >>> time. The processor would have an input port, and so long as the Process >>> Group has 1+ Flow Files in it, no new files will flow into the Process >>> Group through the input port. I've been digging around the code, and it >>> doesn't look to difficult to pull off. >>> >>> >>> - New property on ProcessGroup, accompanying UI/validation >>> >>> o Validation would probably need to verify there was 1 Input Port if >>> enabled. >>> >>> - Updated Logic in LocalPort's onTrigger code to check the parent >>> container and see if it has this setting enabled, and if it does how many >>> total FlowFile's are currently in queue. Only transfer in a new FlowFile if >>> 0 flow files in queue. >>> >>> >>> Thoughts? >>> >>> One specific scenario I've come across is a database table that will be >>> truncated at the start of a flow, populated with a FlowFile's contents, >>> updated by a second step to remove duplicates, and then merged into the >>> final table using a third step. Of course if I could use a separate table >>> for each flow file this wouldn't be an issue, but I've run into constraints >>> where I'm limited to a single table. >>>
