There are 2 efforts, with somewhat different focus. You are already aware of the community-driven nipyapi, but there's also an official module Ientioned before, will be included with the 1.6 release.
Andrew On Fri, Mar 2, 2018, 8:59 AM Boris Tyukin <bo...@boristyukin.com> wrote: > Hi Andrew, > > thanks for the idea. I've been playing with nipyapi recently so might give > this a try. > > Thanks > > On Thu, Mar 1, 2018 at 7:32 PM, Andrew Grande <apere...@gmail.com> wrote: > >> Boris, >> >> Here's an idea youncould explore _today_. >> >> Assume your dev and prod flows live in different bucket/registry >> instance. Given that you are trying out NiFi 1.6, you should be able to >> extract the versioned flow from DEV and process it to change the >> concurrency level for PROD before committing it to the prod registry >> instance. Any script which understands json would do. nifi-toolkit-cli will >> take care of extracting and moving flow versions. >> >> It's not ideal (yes, would like concurrency to be a customizable flow >> var), and it assumes an explicit process to promote between environments, >> but technically it is possible already. The user experience can be improved >> in the future. >> >> Andrew >> >> >> On Thu, Mar 1, 2018, 1:52 PM Kevin Doran <kdo...@apache.org> wrote: >> >>> I think you could put it under either project. Ultimately, if we go with >>> that approach, most (all?) of the logic/enhancement would be in the NiFi >>> code base during save version / import flow / change version operations, so >>> probably best to create it there. >>> >>> >>> >>> Glad you are finding NiFi useful. >>> >>> >>> >>> Cheers, >>> >>> Kevin >>> >>> >>> >>> *From: *Boris Tyukin <bo...@boristyukin.com> >>> *Reply-To: *<users@nifi.apache.org> >>> *Date: *Thursday, March 1, 2018 at 13:44 >>> *To: *<users@nifi.apache.org> >>> *Subject: *Re: setting processor concurrency based on the >>> development/production environment >>> >>> >>> >>> thanks Bryan and Kevin. I will be happy to open a jira - would it be a >>> NiFi jira or NiFi registry? >>> >>> >>> >>> I like the approach that Bryan suggested. >>> >>> >>> >>> I guess for now I will just color code the processors that need to be >>> changed in production. >>> >>> >>> >>> P.S. I really, really like where NiFi is going...I've looked at >>> StreamSets and Cask, but for my purposes, I was looking for a tool when I >>> can process various tables without creating a flow per table. I was able to >>> create a very simple flow in NiFi, that will handle 25 tables. My next >>> project is to handle 600 tables in near real-time. I just could see how I >>> would do that with StreamSets or Cask, when you have to create a pipeline >>> per table. I was only being able to do something similar with Apache >>> Airflow, but airflow cannot do things in near real-time. The concept of >>> FlowFiles with attributes is a genius idea, and I am blown away with all >>> the possibilities to extend the functionality of NiFi with custom >>> processors and Groovy scripts. Awesome job, guys. >>> >>> >>> >>> On Thu, Mar 1, 2018 at 1:29 PM, Kevin Doran <kdo...@apache.org> wrote: >>> >>> Hi Boris, >>> >>> Good point regarding concurrent tasks; thanks for sharing! >>> >>> This is a great candidate for something that one should be able to >>> create environment-specific values for, as Bryan suggests. I agree we >>> should create a NiFi JIRA to track this enhancement. >>> >>> Thanks, >>> Kevin >>> >>> >>> On 3/1/18, 11:44, "Bryan Bende" <bbe...@gmail.com> wrote: >>> >>> Hello, >>> >>> Glad you are having success with NiFi + NiFi Registry! >>> >>> You brought up an interesting point about the concurrent tasks... >>> >>> I think we may want to consider making the concurrent tasks work >>> similar to variables, in that we capture the concurrent tasks that >>> the >>> flow was developed with and would use it initially, but then if you >>> have modified this value in the target environment it would not >>> trigger a local change and would be retained across upgrades so that >>> you don't have to reset it. >>> >>> For now you could probably always leave the versioned flow with the >>> lower value of 2, then once you are in prod you bump it to 4 until >>> the >>> next upgrade is available, you then revert the local changes, do the >>> upgrade, and put it back to 4, but its not ideal because it shows a >>> local change the entire time. >>> >>> I don't think there is much you can do differently right now, but I >>> think this is a valid case to create a JIRA for. >>> >>> Thanks, >>> >>> Bryan >>> >>> On Thu, Mar 1, 2018 at 11:29 AM, Boris Tyukin <bo...@boristyukin.com> >>> wrote: >>> > Hello NiFi community, >>> > >>> > started using NiFi recently and fell in love with it! We run 1.6 >>> NiFi alone >>> > with new NiFi registry and I am trying to figure out how to >>> promote NiFi >>> > flow, created in VM environment to our cluster. >>> > >>> > One of the things is "Concurrent Tasks" processor parameter. I >>> bump it to 2 >>> > or 4 for some processors in my flow, when I develop / test it in >>> VM. >>> > >>> > Then we deploy this flow to a beefy cluster node (with 48 cores) >>> and want to >>> > change concurrency to let's say 8 or 10 or 12 for some processors. >>> > >>> > Then I work on a new version/make some changes in my VM, and need >>> to be more >>> > shy with concurrency so set it back to 2 or 4. >>> > >>> > Then the story repeats... >>> > >>> > Is there a better way than to manually set this parameter? I do >>> not believe >>> > I can use a variable there and have to type the actual number of >>> tasks. >>> > >>> > >>> > Thanks >>> > Boris >>> >>> >>> >>> >> >
