[
https://issues.apache.org/jira/browse/NIFI-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713610#comment-17713610
]
Patrick A. Mol commented on NIFI-11464:
---------------------------------------
I am sorry, but you are still missing the point.
The "reusables" flows are process groups that you would develop and test
separately (following FDLC) and use in other larger flows as basic building
blocks, comparable to a standard processor.
The other flows, like tenant_flows, are versioned as well at some point in
time, and then deployed to production.
When you first import a reusable process group, you might need to adjust the
controller services, but only if you are using different names. I get that.
That is not up for debate.
But the way nested versioned flows are handled, I would not be able to
develop/test reusables separately.
I would need to do this in tenant_flows and commit while inside tenant_flows
for deployment of tenant_flows to work properly.
If I then decide to develop "customer_flows", I cannot use any of the
"reusables" that were committed in "tenant_flows",
or I would need re-commit a new version of each in customer_flows.
My logs will then get flooded with messages that reusable_flow_Q is not the
latest version, because all uses in tenant_flows are now one version behind.
Alternatively I need to clone/branch the reusables for each development. That
is not really DRY and fun either.
While you could setup avro controller services at the root PG, this is not the
case with a DBCP controller service,
when each different tenant_flows deployment would need access to a different
database or use different credentials.
So I think the design needs tweaking, and a nested versioned flow should retain
an indicator that it is a versioned flow and where to find the original in the
registry,
but a copy of the nested flow with the settings it got as part of the larger
flow should be stored with the version of the larger flow.
If a larger versioned flow is checked out, and the original nested flow is no
longer available in the registry,
then it could mark the nested flow with a "?", as is already working currently.
The larger flow would still work, and you can proceed to fix the issue with the
missing version.
So while it looks like the last screenshot is a completely different issue,
there is overlap.
While you are asked for confirmation when you want to delete a flow version
from the registry,
there is no "referential integrity" telling you the version is in use.
So it is easy to topple the whole setup on purpose, but also when you have an
issue with the registry.
> importing a versioned flow with a nested versioned flow shows nested
> versioned flow with invalid controller services.
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-11464
> URL: https://issues.apache.org/jira/browse/NIFI-11464
> Project: Apache NiFi
> Issue Type: Bug
> Components: Flow Versioning
> Affects Versions: 1.21.0
> Environment: nifi 1.21.0 and nifi registry 1.21.0 (on ubuntu 20.04)
> Reporter: Patrick A. Mol
> Priority: Major
> Attachments: exported_flow_versions_pretty.zip,
> image-2023-04-17-17-34-08-898.png, image-2023-04-17-18-06-16-102.png,
> nested_versioned_flow_issue.xml, screenprints_reproduction_steps.zip
>
>
> When a flow (reusable_flow_Q) has controller services inherited from the
> hierarchy (process group reusables) and a version of the flow is stored, the
> flow version contains the references to these external controller services
> (as seen in an exported flow version [see below]).
> When this versioned flow is imported in another flow (tenant_flows) the
> controller services need to be reset to the controller services in the new
> hierarchy.
> When we have a working flow with the nested versioned flow ready in
> development we can check this flow into version control.
> When we then deploy the flow in production, the nested versioned flow shows
> up with invalid components. It shows the external controller service
> identifiers as stored in the flow version.
> When we then go back to development version of tenant_flows and make a minor
> change to the nested versioned flow reusable_flow_Q and commit this change to
> version control.
> Due to this version change, we need to also commit the changes for the
> tenant_flows process group.
> When we now go back to production, and import this new version of
> tenant_flows, the nested versioned flow reusable_flow_Q does not have invalid
> controller services.
> If you have several flows under development using the same reusable
> components,
> you will likely end up with invalid components after import.
> Depending on the amount of versioned flows used, it could be a lot of work.
> It could also lead to issues when using the ExecuteStateless processor.
> Please see attached template nested_version_flow_issue.xml for a starting
> point to reproduce the issue. It contains the steps.
> Screenprints are attached in a zip file show the process and diagnosis.
> Controller services identifiers in version 2.
> {code:java}
> $ fgrep -C 4 reusables_avro reusable_flow_Q.json.pretty
> "controllerServices": [
> {
> "identifier": "dc884171-4d75-3854-8604-afab91bd0e60",
> "instanceIdentifier": "8f647d06-0187-1000-4be9-14a61f55d904",
> "name": "reusables_avro_reader",
> "comments": "",
> "type": "org.apache.nifi.avro.AvroReader",
> "bundle": {
> "group": "org.apache.nifi",
> --
> },
> {
> "identifier": "b512b238-cdee-3642-b5cb-0c98d30dd133",
> "instanceIdentifier": "8f64f2c6-0187-1000-7557-ca63c88054dd",
> "name": "reusables_avro_writer",
> "comments": "",
> "type": "org.apache.nifi.avro.AvroRecordSetWriter",
> "bundle": {
> "group": "org.apache.nifi",
> $ head -15 reusable_flow_Q-version-2.json.pretty
> {
> "externalControllerServices": {
> "dc884171-4d75-3854-8604-afab91bd0e60": {
> "identifier": "dc884171-4d75-3854-8604-afab91bd0e60",
> "name": "reusables_avro_reader"
> },
> "b512b238-cdee-3642-b5cb-0c98d30dd133": {
> "identifier": "b512b238-cdee-3642-b5cb-0c98d30dd133",
> "name": "reusables_avro_writer"
> }
> },
> "flowContents": {
> "comments": "used to perform Q ...",
> "componentType": "PROCESS_GROUP",
> "connections": [
> {code}
> Controller services identifiers with version 3 committed in process group
> "tenant_flows".
> {code:java}
> pmo@hpmo:~/Documents.local/nested_versioned_flows_controller_issue$ fgrep -C
> 4 tenant_flow_avro tenant_flows-version-1.json.pretty
> ],
> "groupIdentifier": "a984831b-8587-3e17-bbbc-ef4b85c3898d",
> "identifier": "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22",
> "instanceIdentifier": "8f6cb319-0187-1000-b7fa-83340f7055f7",
> "name": "tenant_flow_avro_writer",
> "properties": {
> "compression-format": "NONE",
> "Schema Write Strategy": "avro-embedded",
> "schema-name": "${schema.name}",
> --
> ],
> "groupIdentifier": "a984831b-8587-3e17-bbbc-ef4b85c3898d",
> "identifier": "8ff96d88-3dc8-30ed-aeb8-757c26a7b807",
> "instanceIdentifier": "8f6c8a94-0187-1000-af54-2fee12838618",
> "name": "tenant_flow_avro_reader",
> "properties": {
> "schema-name": "${schema.name}",
> "cache-size": "1000",
> "schema-access-strategy": "embedded-avro-schema",
> pmo@hpmo:~/Documents.local/nested_versioned_flows_controller_issue$ head -15
> reusable_flow_Q-version-3.json.pretty
> {
> "externalControllerServices": {
> "8ff96d88-3dc8-30ed-aeb8-757c26a7b807": {
> "identifier": "8ff96d88-3dc8-30ed-aeb8-757c26a7b807",
> "name": "tenant_flow_avro_reader"
> },
> "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22": {
> "identifier": "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22",
> "name": "tenant_flow_avro_writer"
> }
> },
> "flowContents": {
> "comments": "used to perform Q ...",
> "componentType": "PROCESS_GROUP",
> "connections": [
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
