Hi Bence, Joe Thank you guys for your fast response.
@Bence you are right, the whole issue seems to be related to nested flows which we try to sync. We are relying heavily on this as we are doing the integration on one NiFi system and then sync it to production via the NiFi Registry. Our parent flow contains multiple nested flows. Ok, as NIFI-10874 will be included in the next release we will skip the 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than a few days until we see the next release... Do you have an idea why the flow with the nested flow could lose the storageLocation? Because, we just change a small thing in one of the nested flows and suddenly one of the other nested flows have been losing it’s storageLocation in the parent flow and from this point the whole flow was broken. Now we are scared to change anything as it could brake anytime again. Cheers Josef From: Simon Bence <[email protected]> Reply to: "[email protected]" <[email protected]> Date: Tuesday, 29 November 2022 at 13:31 To: "[email protected]" <[email protected]> Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows Hi Josef, Thanks for your patience! I took a deeper look on what you were writing. In general, this is a sign for the case where the registry client cannot find the nested flow. As far as I understood in your case this happens when you are having a sync. Based on this you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 which is not part of NiFi 1.19.0. The next release should contain the fix that addresses your issue. Regards, Bence On 2022. Nov 28., at 17:48, Simon Bence <[email protected]<mailto:[email protected]>> wrote: Hi Josef, Thank you for raising the attention to this. The storageLocation is a new concept to generalise the information used to locate nested flows in a versioned flow. In general it can be there and does not cause issues, as of now NiFi falls back the previous way to locate nested flows. As Joe mentioned, there is a fix in 1.19, but I need to double check if it relates to this situation. Please give me some time, I will find you back. Regards, Bence On 2022. Nov 28., at 17:17, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Josef Sorry for the challenges you've hit there. I do think in 1.18 we had a bug/regression as we refactored our registry client. That issue should be resolved in 1.19 which just went live thanks to https://issues.apache.org/jira/browse/NIFI-10787. However, I am not positive if this will solve the scenario you've hit now but please if able try it out. Thanks On Mon, Nov 28, 2022 at 8:50 AM <[email protected]> wrote: Hi guys We had the following bug twice already and it broke the whole sync with the NiFi registry for the Flow/PG. First time was directly after we have upgraded from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it could be because of the upgrade, however it occurred again after a few NiFi Registry commits on NiFi 1.18.0… The error was the following when we tried to change the flow version or when we tried to start from scratch with that version from the NiFi Registry, so the version was broken in the NiFi Registry: <image001.png> We investigated the last NiFi Registry Commit and we saw in our GIT repo (we sync the “flow_storage” to GIT) on line 4847 that NiFi removed a “storageLocation” from another PG which makes no sense at all. We changed nothing there and especially why should NiFi remove only the storageLocation line… We have one specialty, as we have nested NiFi Registry flows, so one of the flows where the storageLocation has been removed was such a nested flow. <image002.png> Luckily we were able to resolve the error. We tried to add the line and commit it to GIT plus we dropped the the database to repopulate the DB, however it was broken again after a commit from NiFi. So we tried to manually create a new fake version on bucket.yml in the corresponding bucket folder and added as well the line in the snapshot again. We dropped then the DB and restarted NiFi Registry and voilà it was working again. However it was a nightmare to get it working again as the flow was completely broken, we couldn’t checkout the affected version at all. Any thoughts on this? Shall I fill a Jira Ticket? The problem is, we can’t really reproduce it, it looks like it happens randomly. As you could imagine, we can’t share our template as it contains a lot of confidential material. Cheers Josef
smime.p7s
Description: S/MIME Cryptographic Signature
