Hi Ami, I'm testing our NiFi application using a blackbox approach. We have a test harness that stages input files and waits for the corresponding outputs, then validates the content of the output. NiFi runs in a docker container (see https://hub.docker.com/r/apache/nifi/) and you can automate starting it up and orchestrating a test harness for your application in any competent CI/CD pipeline.
Jorge, I don't understand your comment. NiFi registry can be synced to a git repo as a backup mechanism, but you can't simply make branches, work and merge them back, because even the smallest change in a flow will create a massive diff in the Registry's backup git repo. It would be great if NiFi had a way of highlighting the differences between flow versions on the canvas and allow you to choose which elements to keep and which to discard when resolving a merge conflict. -Eric On Thu, May 14, 2020 at 12:00 PM Jorge Machado <jom...@me.com> wrote: > I think we could improve Nifi by hooking up to GitHub somehow. So that we > don’t need the registry… > > On 14. May 2020, at 18:49, Ami Goldenberg <ami.g...@gmail.com> wrote: > > Hi Eric, > Would love to know, what kind of tests do you write/run for NiFi? We were > just researching this topic. > > On Thu, May 14, 2020 at 6:38 PM Eric Secules <esecu...@gmail.com> wrote: > >> Hi Michal, >> >> I'm also using a single registry for development and production. It >> doesn't help with collaborating on the same process group as there is way >> for it to reconcile merge conflicts. Instead, the registry will earn you >> that you're about to overwrite someone else's changes. Another pain of >> concurrent development is there's no concept of a PR and no visual diff of >> your local changes making review difficult. I've backed our registry up to >> git I've set up a CI pipeline in Azure Devops which runs our tests every >> time a new version of a process group is checked in. It's better than >> nothing but I'd rather nifi implemented git flow. Developing as a team on >> nifi and nifi registry is like your whole team developing and pushing >> directly to master. >> >> >> -Eric >> >> On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <jom...@me.com> wrote: >> >>> Hi, >>> >>> Managing xml is always hard I think. Last time I need to do something >>> similar we used https://nifi.apache.org/registry.html >>> Works pretty well It was already 2 Years ago. Maybe now there is >>> something better >>> >>> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external < >>> michal.sl...@dhl.com> wrote: >>> >>> Hello, >>> >>> may I ask you for recommendations for development and CI/CD in NiFi? >>> Pls let me describe our situation…I am a developer from DHL currently >>> working on project including NiFi. It is part of our core and it is >>> responsible for handling incoming data streams, data transormation and >>> put it into various elastic search indexes (and queues). >>> >>> Up to now development was quite straightforward as only one developer >>> did it. But we have extended our team recently and now we face problems how >>> to correctly maintain development for more developers working in >>> parallel and then CI/CD of it as we have classical dev/test/uat/project >>> env. structure. >>> >>> Pls do you have any recommadation how to achive it? For now in general >>> is enought. Its good to mention that currently we work with NiFi version >>> 1.8…tried to upgrade to 1.9. but some of components failed so the >>> upgrade was postponed. But with new features in 1.10. and 1.11. we head >>> to uprade to these versions. >>> >>> Maybe if we can arrange a call it would be great! >>> >>> With regards, >>> Michal Sláma >>> >>> >>> >>> *This message is from DHL Information Services (Europe) s.r.o. and may >>> contain confidential business information. It is intended solely for the >>> use of the individual to whom it is addressed. If you are not the intended >>> recipient please contact the sender and delete this message and any >>> attachment from your system. Unauthorized publication, use, dissemination, >>> forwarding, printing or copying of this email and its attachments is >>> strictly prohibited.* >>> >>> >>> >
