[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[ https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846729#comment-16846729 ] Bryan Bende commented on NIFI-5029: --- Here is a new Jira to track the issue being discussed here in the comments: https://issues.apache.org/jira/browse/NIFI-6314 > CLI - Handling of embedded versioned process groups during export/import > > > Key: NIFI-5029 > URL: https://issues.apache.org/jira/browse/NIFI-5029 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC, Tools and Build >Reporter: Pierre Villard >Priority: Major > Labels: SDLC > > I'm in a situation where, in my dev environment, I have a versioned process > group A in bucket bA containing an embedded process group B from bucket bB. > Both A and B are versioned in the NiFi Registry of my dev environment. > I'm using the CLI to export both A and B from my dev Registry, and then > importing A and B in my prod Registry. The issue is that in the json > representing A, there is a reference to B containing the url of the dev > Registry, the bucket ID and the workflow ID representing B. > To import B in my prod registry, I first created a bucket bB and a workflow > B. New UUIDs have been generated for both in the prod registry. Then I did > the import of the JSON representing B. > Now I manually update the JSON representing A to set the UUIDs related to B > with the new values of my prod registry. Then I created a bucket and a > workflow for A in my prod registry, and did the import. > Problem occurs when I try to import the process group A in my prod NiFi. > It'll fail with an error looking like this: > {code:java} > #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080 > > Using a positional back-reference for 'A' > ERROR: Error executing command 'pg-change-version' : Error updating version > control information: The Flow Registry with ID > 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket > 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow > 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code} > I guess there are multiple ways of handling it. It could be on the CLI side > or on the Registry side (could be related to NIFIREG-148). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[ https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846357#comment-16846357 ] George Knaggs commented on NIFI-5029: - Thanks Doug and Bryan for looking into issue from the UI that I reported. I was not able to replicate it outside our HDF install of Nifi 1.7.1. and Nifi Registry 0.3. I tried to replicate using docker images of Nifi and Nifi Registry and everything worked perfectly. We’ve since decided to stop using HDF and move to straight install from Apache distribution. > CLI - Handling of embedded versioned process groups during export/import > > > Key: NIFI-5029 > URL: https://issues.apache.org/jira/browse/NIFI-5029 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC, Tools and Build >Reporter: Pierre Villard >Priority: Major > Labels: SDLC > > I'm in a situation where, in my dev environment, I have a versioned process > group A in bucket bA containing an embedded process group B from bucket bB. > Both A and B are versioned in the NiFi Registry of my dev environment. > I'm using the CLI to export both A and B from my dev Registry, and then > importing A and B in my prod Registry. The issue is that in the json > representing A, there is a reference to B containing the url of the dev > Registry, the bucket ID and the workflow ID representing B. > To import B in my prod registry, I first created a bucket bB and a workflow > B. New UUIDs have been generated for both in the prod registry. Then I did > the import of the JSON representing B. > Now I manually update the JSON representing A to set the UUIDs related to B > with the new values of my prod registry. Then I created a bucket and a > workflow for A in my prod registry, and did the import. > Problem occurs when I try to import the process group A in my prod NiFi. > It'll fail with an error looking like this: > {code:java} > #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080 > > Using a positional back-reference for 'A' > ERROR: Error executing command 'pg-change-version' : Error updating version > control information: The Flow Registry with ID > 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket > 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow > 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code} > I guess there are multiple ways of handling it. It could be on the CLI side > or on the Registry side (could be related to NIFIREG-148). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[ https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846188#comment-16846188 ] Doug Kalemba commented on NIFI-5029: thanks [~bende]. i really appreciate the help and verification. please let me know what the new ticket is, i would love to watch it > CLI - Handling of embedded versioned process groups during export/import > > > Key: NIFI-5029 > URL: https://issues.apache.org/jira/browse/NIFI-5029 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC, Tools and Build >Reporter: Pierre Villard >Priority: Major > Labels: SDLC > > I'm in a situation where, in my dev environment, I have a versioned process > group A in bucket bA containing an embedded process group B from bucket bB. > Both A and B are versioned in the NiFi Registry of my dev environment. > I'm using the CLI to export both A and B from my dev Registry, and then > importing A and B in my prod Registry. The issue is that in the json > representing A, there is a reference to B containing the url of the dev > Registry, the bucket ID and the workflow ID representing B. > To import B in my prod registry, I first created a bucket bB and a workflow > B. New UUIDs have been generated for both in the prod registry. Then I did > the import of the JSON representing B. > Now I manually update the JSON representing A to set the UUIDs related to B > with the new values of my prod registry. Then I created a bucket and a > workflow for A in my prod registry, and did the import. > Problem occurs when I try to import the process group A in my prod NiFi. > It'll fail with an error looking like this: > {code:java} > #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080 > > Using a positional back-reference for 'A' > ERROR: Error executing command 'pg-change-version' : Error updating version > control information: The Flow Registry with ID > 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket > 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow > 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code} > I guess there are multiple ways of handling it. It could be on the CLI side > or on the Registry side (could be related to NIFIREG-148). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[ https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846185#comment-16846185 ] Bryan Bende commented on NIFI-5029: --- Thanks, I was able to reproduce now and got the same behavior with revert putting the expected values back. Definitely something we want to look into. I will probably create another Jira and link it to this one to track these issues separately (CLI vs this issue we're discussing). > CLI - Handling of embedded versioned process groups during export/import > > > Key: NIFI-5029 > URL: https://issues.apache.org/jira/browse/NIFI-5029 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC, Tools and Build >Reporter: Pierre Villard >Priority: Major > Labels: SDLC > > I'm in a situation where, in my dev environment, I have a versioned process > group A in bucket bA containing an embedded process group B from bucket bB. > Both A and B are versioned in the NiFi Registry of my dev environment. > I'm using the CLI to export both A and B from my dev Registry, and then > importing A and B in my prod Registry. The issue is that in the json > representing A, there is a reference to B containing the url of the dev > Registry, the bucket ID and the workflow ID representing B. > To import B in my prod registry, I first created a bucket bB and a workflow > B. New UUIDs have been generated for both in the prod registry. Then I did > the import of the JSON representing B. > Now I manually update the JSON representing A to set the UUIDs related to B > with the new values of my prod registry. Then I created a bucket and a > workflow for A in my prod registry, and did the import. > Problem occurs when I try to import the process group A in my prod NiFi. > It'll fail with an error looking like this: > {code:java} > #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080 > > Using a positional back-reference for 'A' > ERROR: Error executing command 'pg-change-version' : Error updating version > control information: The Flow Registry with ID > 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket > 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow > 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code} > I guess there are multiple ways of handling it. It could be on the CLI side > or on the Registry side (could be related to NIFIREG-148). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[ https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846176#comment-16846176 ] Doug Kalemba commented on NIFI-5029: thanks [~bende] for responding. perhaps you could add a couple of more steps, to more replicate the issue i was seeing... # NIFI#1: change a value in either the properties or settings in a processor in PGB # NIFI#1: commit changes to PGB # NIFI#1: commit changes to PGA # NIFI#2: change version of PGA to newest version # NIFI#2: look for a "Local changes have been made" indication on the PGB. I have found that when i do the above, PGB will tell me that it is the new version, but i will get the "local changes" message and the values/properties that i have changed will be the old ones. HOWEVER, adding/removing processors seems to work fine. PGB does change version, but the previous versions's properties/settings are restored and a "local change" message showing the restoring of those values is presented. > CLI - Handling of embedded versioned process groups during export/import > > > Key: NIFI-5029 > URL: https://issues.apache.org/jira/browse/NIFI-5029 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC, Tools and Build >Reporter: Pierre Villard >Priority: Major > Labels: SDLC > > I'm in a situation where, in my dev environment, I have a versioned process > group A in bucket bA containing an embedded process group B from bucket bB. > Both A and B are versioned in the NiFi Registry of my dev environment. > I'm using the CLI to export both A and B from my dev Registry, and then > importing A and B in my prod Registry. The issue is that in the json > representing A, there is a reference to B containing the url of the dev > Registry, the bucket ID and the workflow ID representing B. > To import B in my prod registry, I first created a bucket bB and a workflow > B. New UUIDs have been generated for both in the prod registry. Then I did > the import of the JSON representing B. > Now I manually update the JSON representing A to set the UUIDs related to B > with the new values of my prod registry. Then I created a bucket and a > workflow for A in my prod registry, and did the import. > Problem occurs when I try to import the process group A in my prod NiFi. > It'll fail with an error looking like this: > {code:java} > #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080 > > Using a positional back-reference for 'A' > ERROR: Error executing command 'pg-change-version' : Error updating version > control information: The Flow Registry with ID > 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket > 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow > 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code} > I guess there are multiple ways of handling it. It could be on the CLI side > or on the Registry side (could be related to NIFIREG-148). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[ https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846163#comment-16846163 ] Bryan Bende commented on NIFI-5029: --- [~kalemba] thanks for reporting your issue, this ticket was really about how the CLI handles embedded process groups, which that part admittedly doesn't work, but from using a single registry with multiple NiFis, I wasn't aware of any major issues. Here is a test I just did using two local NiFi instances and a single registry, running latest code (1.10.0-SNAPSHOT NiFI and 0.4.0 registry)... # NiFi-1 Create PGA # NiFI-1 Create PGB inside PGA # NiFI-1 Create some processors inside PGB # NIFI-1 Start version control PGB # NIFI-1 Start version control PGA # NIFI-2 Import a new PG and select PGA v1 (at this point the same exact flow is now in both NiFi's) # NIFI-1 Go into PGB and add additional processor # NIFI-1 Commit changes on PGB # NIFI-1 Commit changes on PGA # NIFI-2 Change version on PGA from v1 to v2 (caused PGB to be updated to v2 since PGA v2 points to PGB v2) This all seemed to work correctly for me. Is there something different about my steps from what you have done? What versions of NiFi and NiFi Registry are you using? > CLI - Handling of embedded versioned process groups during export/import > > > Key: NIFI-5029 > URL: https://issues.apache.org/jira/browse/NIFI-5029 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC, Tools and Build >Reporter: Pierre Villard >Priority: Major > Labels: SDLC > > I'm in a situation where, in my dev environment, I have a versioned process > group A in bucket bA containing an embedded process group B from bucket bB. > Both A and B are versioned in the NiFi Registry of my dev environment. > I'm using the CLI to export both A and B from my dev Registry, and then > importing A and B in my prod Registry. The issue is that in the json > representing A, there is a reference to B containing the url of the dev > Registry, the bucket ID and the workflow ID representing B. > To import B in my prod registry, I first created a bucket bB and a workflow > B. New UUIDs have been generated for both in the prod registry. Then I did > the import of the JSON representing B. > Now I manually update the JSON representing A to set the UUIDs related to B > with the new values of my prod registry. Then I created a bucket and a > workflow for A in my prod registry, and did the import. > Problem occurs when I try to import the process group A in my prod NiFi. > It'll fail with an error looking like this: > {code:java} > #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080 > > Using a positional back-reference for 'A' > ERROR: Error executing command 'pg-change-version' : Error updating version > control information: The Flow Registry with ID > 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket > 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow > 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code} > I guess there are multiple ways of handling it. It could be on the CLI side > or on the Registry side (could be related to NIFIREG-148). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[ https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846115#comment-16846115 ] Doug Kalemba commented on NIFI-5029: I would like to offer support for the request made by [~jorjnagz] above. Nested versioned flows is not possible at the moment. My scenario is similar to his... I have PG-A v1 containing PG-B v1 in dev. I modify PG-B and commit my changes to the registry. It is now v2. Then I update PG-A to use v2 of PG-B and commit that. PG-A is now v2. I then move to the prod env and change PG-A from v1 to v2. When I look at PG-B i see that it is telling me that it is v2, but the v1 items that i changed have overwritten the v2 items and the pg is telling me that i now have local changes. Looking at the flow.xml.gz, i see it having 2 for PG-B but actually the v1 XML values. To me, this is a HUGE bug. It absolutely prevents modularizing the flows and cripples the power of the entire Registry feature. > CLI - Handling of embedded versioned process groups during export/import > > > Key: NIFI-5029 > URL: https://issues.apache.org/jira/browse/NIFI-5029 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC, Tools and Build >Reporter: Pierre Villard >Priority: Major > Labels: SDLC > > I'm in a situation where, in my dev environment, I have a versioned process > group A in bucket bA containing an embedded process group B from bucket bB. > Both A and B are versioned in the NiFi Registry of my dev environment. > I'm using the CLI to export both A and B from my dev Registry, and then > importing A and B in my prod Registry. The issue is that in the json > representing A, there is a reference to B containing the url of the dev > Registry, the bucket ID and the workflow ID representing B. > To import B in my prod registry, I first created a bucket bB and a workflow > B. New UUIDs have been generated for both in the prod registry. Then I did > the import of the JSON representing B. > Now I manually update the JSON representing A to set the UUIDs related to B > with the new values of my prod registry. Then I created a bucket and a > workflow for A in my prod registry, and did the import. > Problem occurs when I try to import the process group A in my prod NiFi. > It'll fail with an error looking like this: > {code:java} > #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080 > > Using a positional back-reference for 'A' > ERROR: Error executing command 'pg-change-version' : Error updating version > control information: The Flow Registry with ID > 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket > 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow > 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code} > I guess there are multiple ways of handling it. It could be on the CLI side > or on the Registry side (could be related to NIFIREG-148). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[ https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782225#comment-16782225 ] George Knaggs commented on NIFI-5029: - I am having similar issue, but in a simpler and perhaps more common use case. It happens every time I try to promote changes to next environment with NiFi Registry for a versioned flow that itself contains an embedded versioned group. It does not happen if I create a copy of a that same versioned flow on the same environment it was created (dev). I have a single instance of NiFi Registry that I use to promote changes between my environments. I am using GIT Repository for flow persistence with NiFi Registry. I am not using CLI, but the NiFi UI to import the versioned flow or update version. I am able to reproduce the issue if I import on test env the versioned flow with the embedded versioned group. I am also able to reproduce if I import on test env a versioned flow w/o any embedded versioned groups, but later add an embedded groups to version control on dev env, and then try to change the version on test to the latest version of the flow now with an embedded versioned group. It took me a while to understand what was causing the issue and traced back the problem only to a particular flow and only for versions after I added embedded groups to version control. My only choice is to prevent embedded version process groups. This is a completely broken feature and removes the advantage of version control for sub flows. I had hoped to have componitize my reusable sub flows under version control so when I fix a bug or expand functionality in these flows, I could easily update of the component flow in every full flows that uses it by simply changing the version. Please escalate the resolution of this bug in order to use this feature. > CLI - Handling of embedded versioned process groups during export/import > > > Key: NIFI-5029 > URL: https://issues.apache.org/jira/browse/NIFI-5029 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC, Tools and Build >Reporter: Pierre Villard >Priority: Major > > I'm in a situation where, in my dev environment, I have a versioned process > group A in bucket bA containing an embedded process group B from bucket bB. > Both A and B are versioned in the NiFi Registry of my dev environment. > I'm using the CLI to export both A and B from my dev Registry, and then > importing A and B in my prod Registry. The issue is that in the json > representing A, there is a reference to B containing the url of the dev > Registry, the bucket ID and the workflow ID representing B. > To import B in my prod registry, I first created a bucket bB and a workflow > B. New UUIDs have been generated for both in the prod registry. Then I did > the import of the JSON representing B. > Now I manually update the JSON representing A to set the UUIDs related to B > with the new values of my prod registry. Then I created a bucket and a > workflow for A in my prod registry, and did the import. > Problem occurs when I try to import the process group A in my prod NiFi. > It'll fail with an error looking like this: > {code:java} > #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080 > > Using a positional back-reference for 'A' > ERROR: Error executing command 'pg-change-version' : Error updating version > control information: The Flow Registry with ID > 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket > 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow > 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code} > I guess there are multiple ways of handling it. It could be on the CLI side > or on the Registry side (could be related to NIFIREG-148). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5029) CLI - Handling of embedded versioned process groups during export/import
[ https://issues.apache.org/jira/browse/NIFI-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655013#comment-16655013 ] Olivier Tarnus commented on NIFI-5029: -- I'm actually facing the same error message, but in slightly different context: copying flows between two registries. I've a DEV and PROD registries and wrote a simple script using NiPyApi to synchronize flow versions from DEV to PROD. This works perfectly until we add embedded process groups to version control: We have one main process group A, that embed multiple process groups B,C,D... all process groups versions being stored in the same bucket. Any NiFi instance connected to the DEV registry can successfully import the flows. Then we can successfully copy the flows to the PROD registry, but we hit the above error when importing the flow initially. I tried to first first add A to version control with all embedded process groups not versioned: we can then successfully import the process after a copy. Then I started version control on embedded process groups, successfully synced registries, but it does fail with the same error when trying to import the new version after the copy. This is not 100% the case described here (no copy and 2 buckets), but I think this is related to same area of code. Let me know if I should create a separate improvement requests. > CLI - Handling of embedded versioned process groups during export/import > > > Key: NIFI-5029 > URL: https://issues.apache.org/jira/browse/NIFI-5029 > Project: Apache NiFi > Issue Type: Improvement > Components: Flow Versioning, SDLC, Tools and Build >Reporter: Pierre Villard >Priority: Major > > I'm in a situation where, in my dev environment, I have a versioned process > group A in bucket bA containing an embedded process group B from bucket bB. > Both A and B are versioned in the NiFi Registry of my dev environment. > I'm using the CLI to export both A and B from my dev Registry, and then > importing A and B in my prod Registry. The issue is that in the json > representing A, there is a reference to B containing the url of the dev > Registry, the bucket ID and the workflow ID representing B. > To import B in my prod registry, I first created a bucket bB and a workflow > B. New UUIDs have been generated for both in the prod registry. Then I did > the import of the JSON representing B. > Now I manually update the JSON representing A to set the UUIDs related to B > with the new values of my prod registry. Then I created a bucket and a > workflow for A in my prod registry, and did the import. > Problem occurs when I try to import the process group A in my prod NiFi. > It'll fail with an error looking like this: > {code:java} > #> nifi pg-change-version -fv 2 -pgid &1 -u http://localhost:8080 > > Using a positional back-reference for 'A' > ERROR: Error executing command 'pg-change-version' : Error updating version > control information: The Flow Registry with ID > 56a49113-0162-1000-0a7e-4959d312d445 reports that no Flow exists with Bucket > 9d87ccc4-7084-4069-850a-7704135ea9e9, Flow > 7e47aeba-4908-41a6-88d7-e6deb1abef98, Version 2{code} > I guess there are multiple ways of handling it. It could be on the CLI side > or on the Registry side (could be related to NIFIREG-148). -- This message was sent by Atlassian JIRA (v7.6.3#76005)