Hi Andrew, thanks for the idea. I've been playing with nipyapi recently so might give this a try.
Thanks On Thu, Mar 1, 2018 at 7:32 PM, Andrew Grande <apere...@gmail.com> wrote: > Boris, > > Here's an idea youncould explore _today_. > > Assume your dev and prod flows live in different bucket/registry instance. > Given that you are trying out NiFi 1.6, you should be able to extract the > versioned flow from DEV and process it to change the concurrency level for > PROD before committing it to the prod registry instance. Any script which > understands json would do. nifi-toolkit-cli will take care of extracting > and moving flow versions. > > It's not ideal (yes, would like concurrency to be a customizable flow > var), and it assumes an explicit process to promote between environments, > but technically it is possible already. The user experience can be improved > in the future. > > Andrew > > > On Thu, Mar 1, 2018, 1:52 PM Kevin Doran <kdo...@apache.org> wrote: > >> I think you could put it under either project. Ultimately, if we go with >> that approach, most (all?) of the logic/enhancement would be in the NiFi >> code base during save version / import flow / change version operations, so >> probably best to create it there. >> >> >> >> Glad you are finding NiFi useful. >> >> >> >> Cheers, >> >> Kevin >> >> >> >> *From: *Boris Tyukin <bo...@boristyukin.com> >> *Reply-To: *<users@nifi.apache.org> >> *Date: *Thursday, March 1, 2018 at 13:44 >> *To: *<users@nifi.apache.org> >> *Subject: *Re: setting processor concurrency based on the >> development/production environment >> >> >> >> thanks Bryan and Kevin. I will be happy to open a jira - would it be a >> NiFi jira or NiFi registry? >> >> >> >> I like the approach that Bryan suggested. >> >> >> >> I guess for now I will just color code the processors that need to be >> changed in production. >> >> >> >> P.S. I really, really like where NiFi is going...I've looked at >> StreamSets and Cask, but for my purposes, I was looking for a tool when I >> can process various tables without creating a flow per table. I was able to >> create a very simple flow in NiFi, that will handle 25 tables. My next >> project is to handle 600 tables in near real-time. I just could see how I >> would do that with StreamSets or Cask, when you have to create a pipeline >> per table. I was only being able to do something similar with Apache >> Airflow, but airflow cannot do things in near real-time. The concept of >> FlowFiles with attributes is a genius idea, and I am blown away with all >> the possibilities to extend the functionality of NiFi with custom >> processors and Groovy scripts. Awesome job, guys. >> >> >> >> On Thu, Mar 1, 2018 at 1:29 PM, Kevin Doran <kdo...@apache.org> wrote: >> >> Hi Boris, >> >> Good point regarding concurrent tasks; thanks for sharing! >> >> This is a great candidate for something that one should be able to create >> environment-specific values for, as Bryan suggests. I agree we should >> create a NiFi JIRA to track this enhancement. >> >> Thanks, >> Kevin >> >> >> On 3/1/18, 11:44, "Bryan Bende" <bbe...@gmail.com> wrote: >> >> Hello, >> >> Glad you are having success with NiFi + NiFi Registry! >> >> You brought up an interesting point about the concurrent tasks... >> >> I think we may want to consider making the concurrent tasks work >> similar to variables, in that we capture the concurrent tasks that the >> flow was developed with and would use it initially, but then if you >> have modified this value in the target environment it would not >> trigger a local change and would be retained across upgrades so that >> you don't have to reset it. >> >> For now you could probably always leave the versioned flow with the >> lower value of 2, then once you are in prod you bump it to 4 until the >> next upgrade is available, you then revert the local changes, do the >> upgrade, and put it back to 4, but its not ideal because it shows a >> local change the entire time. >> >> I don't think there is much you can do differently right now, but I >> think this is a valid case to create a JIRA for. >> >> Thanks, >> >> Bryan >> >> On Thu, Mar 1, 2018 at 11:29 AM, Boris Tyukin <bo...@boristyukin.com> >> wrote: >> > Hello NiFi community, >> > >> > started using NiFi recently and fell in love with it! We run 1.6 >> NiFi alone >> > with new NiFi registry and I am trying to figure out how to promote >> NiFi >> > flow, created in VM environment to our cluster. >> > >> > One of the things is "Concurrent Tasks" processor parameter. I bump >> it to 2 >> > or 4 for some processors in my flow, when I develop / test it in VM. >> > >> > Then we deploy this flow to a beefy cluster node (with 48 cores) >> and want to >> > change concurrency to let's say 8 or 10 or 12 for some processors. >> > >> > Then I work on a new version/make some changes in my VM, and need >> to be more >> > shy with concurrency so set it back to 2 or 4. >> > >> > Then the story repeats... >> > >> > Is there a better way than to manually set this parameter? I do not >> believe >> > I can use a variable there and have to type the actual number of >> tasks. >> > >> > >> > Thanks >> > Boris >> >> >> >> >
