Andrew, I do agree with you point. I think Ryan's saying we need to do a better job of calling that out in the docs/guiding folks to the best path so they don't try combining older practices with ones that are meant to totally replace it.
Is there more to it Ryan? Thanks On Mon, May 7, 2018 at 8:50 AM, Andrew Grande <apere...@gmail.com> wrote: > The NiFi CLI has a dedicated command to move a flow version snapshot between > registry instances. > > What I find out of line in Ryan's description is they are using templates > and registry for the version control. Templates were created with no > registry in sight, and, frankly, should not be used anymore now. > > If registries are setup, use the tools meant for the registry (and come for > any help here). > > Cheers! > Andrew > > > On Mon, May 7, 2018, 8:30 AM Ryan H <ryan.howell.developm...@gmail.com> > wrote: >> >> Hi Kevin, >> >> Thanks for the multiple options as it relates to FDLC; good articles. The >> workaround for our current error was to manually search/destroy all >> references to the registry in the created template xml file >> (<versionControlInformation> tag in the template). Then when the template is >> imported into a the new environment, the issue was resolved as all of the >> bad references were not ported into the new environment. >> >> It seems that the version control information (registry references) >> shouldn't be included when creating templates or there should be an option >> to include/exclude when creating the template. I would guess that there are >> a lot of use cases where a template is being created to be used in >> environments other than the one which it was created in. While it may not >> fully align with best practices/suggested methods for FDLC, this does cause >> issues if done this way. It may just be the case that this is documented as >> being and issue and the recommendation is to follow one of the suggested >> practices to avoid issues such as this one. >> >> Cheers, >> >> Ryan >> >> On Mon, May 7, 2018 at 7:10 AM, Kevin Doran <kdo...@apache.org> wrote: >>> >>> Hi Ryan, >>> >>> >>> >>> I’ve never tried creating a template from a process group that was saved >>> to a NiFi Registry, so I haven’t run into this exact error. However, there >>> are users that cannot connect multiple environments (e.g., dev and >>> production) to the same NiFi Registry and therefore have the need you’ve >>> described of having to move flow definitions without a shared NiFi Registry >>> instance. >>> >>> >>> >>> From what you’ve described and the error you’ve encountered, I’ve >>> gathered that the approach you are using to migrate flows across >>> environments is: >>> >>> >>> >>> 1. [Env A] Design / test a flow in NiFi, saving/loading from >>> environment’s NiFi Registry as needed >>> >>> 2. [Env A] Export to template >>> >>> 3. [Env B] Import template into NiFI >>> >>> 4. [Env B] Save to environment’s NiFi Registry >>> >>> >>> >>> Is that more or less correct? If so, instead of this workflow, would it >>> be possible to take the following path to migrate the flow version, which I >>> have seen others use: >>> >>> >>> >>> [Env A] NiFi >> [Env A] NiFi Registry >> [Env B] NiFi Registry >> [Env >>> B] NiFi >>> >>> >>> >>> In this FDLC workflow, templates are never used, because in Env B the >>> Registry is populated first and then imported into NiFi using the import >>> from Registry feature. >>> >>> >>> >>> If this sounds like it could work for your use case, and you would be >>> able to avoid templates entirely (i.e., there is no other reason you need >>> them other than migrating flows across environments), then there are lots of >>> great write ups by others on how to achieve this workflow. The trick is in >>> the Registry A to Registry B step, and for that there are a few approaches >>> and tools that others have documented. >>> >>> >>> >>> A great write up about this “one Registry per environment” FDLC approach >>> is included in a blog post by Pierre Villard, “Automate workflow deployment >>> in Apache NiFi with the NiFi Registry” [1]. As Pierre describes, you can use >>> the NiFi CLI [1], included in the NiFi Toolkit as a separate executable >>> since NiFi 1.6.0 [2] as a tool to migrate flow version snapshots from one >>> NiFi Registry to another. Specifically, the CLI tools “registry >>> export-flow-version” and “registry import-flow-version” commands. >>> >>> >>> >>> An alternative to using the NiFi CLI would be NiPyApi [4], a NiFi client >>> SDK that makes the REST API for NiFi and NiFi Registry easier to work with >>> from Python automation scripts. There’s a good write up on using this to >>> interact with flows in NiFi Registry instances from Timothy Spann [5]. >>> >>> >>> >>> I hope this helps you workaround the issue you are running into. It >>> certainly does not solve the specific error you have documented, but perhaps >>> it is a viable alternative that avoids that exception. >>> >>> >>> >>> Best, >>> Kevin >>> >>> >>> >>> [1] >>> https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/ >>> >>> [2] >>> https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli >>> >>> [3] https://nifi.apache.org/download.html >>> >>> [4] https://github.com/Chaffelson/nipyapi >>> >>> [5] >>> https://community.hortonworks.com/articles/177301/big-data-devops-apache-nifi-flow-versioning-and-au.html >>> >>> >>> >>> >>> >>> From: Ryan H <ryan.howell.developm...@gmail.com> >>> Reply-To: <users@nifi.apache.org> >>> Date: Sunday, May 6, 2018 at 21:05 >>> To: <users@nifi.apache.org> >>> Subject: Error Reference When Creating Template From PG Under NiFi >>> Registry Version Control >>> >>> >>> >>> Hi All, >>> >>> We have found what seems to be an issue when creating a template of a >>> Process Group that is under version control with NiFi Registry. What seems >>> to happen is that there is a hard reference to the Registry Instance from >>> the Env that the template was created from within the template xml file. >>> When importing the template into a new Env the reference to the registry >>> goes with the template and looks to stay with the new process group, is >>> written to the flow.xml.gz, and causes an issue when clustered (the nodes >>> end up not being able to sync after a while because of the error and goes >>> into a disconnected state). >>> >>> I'll say upfront that this somewhat defeats the purpose of having the >>> registry and I'm guessing that there will be suggestions that we should >>> either go the template FLDC route or go the registry FLDC route. For this >>> purpose we cannot share the registry instance between the two environments >>> and may be the case that we should not use the registry at this time. >>> >>> Should this be an error? Is a better way other than going in a >>> searching/destroying all the old registry references out of the template and >>> flow xml files? Should we manually change the registry instances in each of >>> the environments to be the same guid id? >>> >>> >>> >>> As always any help or direction is greatly appreciated. >>> >>> Cheers, >>> >>> Ryan Howell >>> >>> >>> template.xml snippet (created template to download, which will then be >>> uploaded into new env) (seems that the error is automatically created >>> knowing that it is going to be a problem importing): >>> ... >>> <versionControlInformation> >>> >>> <bucketId>1f26b7fd-8987-491d-a2ac-4f87292ab29b</bucketId> >>> <bucketName>Dev-Test</bucketName> >>> <flowDescription></flowDescription> >>> >>> <flowId>c8e54d77-ce35-4be7-aff6-339a96b58e7d</flowId> >>> <flowName>MySuperCoolFlow</flowName> >>> >>> <registryId>4ee39816-0161-1000-ffff-ffffc4085349</registryId> >>> >>> <registryName>4ee39816-0161-1000-ffff-ffffc4085349</registryName> >>> <state>SYNC_FAILURE</state> >>> <stateExplanation>Unable to synchronize Process >>> Group with Flow Registry because Process Group was placed under Version >>> Control using Flow Registry with identifier >>> 4ee39816-0161-1000-ffff-ffffc4085349 but cannot find any Flow Registry with >>> this identifier</stateExplanation> >>> <version>1</version> >>> </versionControlInformation> >>> </processGroups> >>> <processGroups> >>> <id>9d90f6ea-f2a1-3d65-0000-000000000000</id> >>> >>> <parentGroupId>d8cf7c44-48c7-3c7d-0000-000000000000</parentGroupId> >>> <position> >>> <x>422.23612851041094</x> >>> <y>-1527.1966721061617</y> >>> </position> >>> >>> <versionedComponentId>322cb8f0-cec6-30f9-9565-6948120c96d6</versionedComponentId> >>> <comments></comments> >>> <contents> >>> <connections> >>> <id>80c3a35a-27ec-33a2-0000-000000000000</id> >>> >>> <parentGroupId>9d90f6ea-f2a1-3d65-0000-000000000000</parentGroupId> >>> >>> <versionedComponentId>d77ff912-6127-39f2-9c7a-2eb3c7dde817</versionedComponentId> >>> <backPressureDataSizeThreshold>1 >>> GB</backPressureDataSizeThreshold> >>> >>> <backPressureObjectThreshold>10000</backPressureObjectThreshold> >>> <destination> >>> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> >>> <id>14b964e1-0775-3e6f-0000-000000000000</id> >>> <type>PROCESSOR</type> >>> >>> <versionedComponentId>66873d14-b329-3322-b845-d144571ac93e</versionedComponentId> >>> </destination> >>> <flowFileExpiration>0 >>> sec</flowFileExpiration> >>> <labelIndex>1</labelIndex> >>> <name></name> >>> >>> <selectedRelationships>success</selectedRelationships> >>> <source> >>> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> >>> <id>f26999d5-adbe-33ea-0000-000000000000</id> >>> <type>PROCESSOR</type> >>> >>> <versionedComponentId>c928e355-f4b0-3673-ba8e-9d7f0cf57d5f</versionedComponentId> >>> </source> >>> >> >
