Good question Bence. I'm not sure it's worth the effort to have this
setting per registry. The reason is that, as of today, a flow can only be
versioned into a single registry. And a process group cannot be "attached"
to a versioned flow in the registry. So I'm not sure how frequent the
scenario you're describing is. One would commit new versions in dev
registry, and then when pushing to prod registry they'd have to stop
versioning and commit as a new version in "prod" registry, that seems very
unlikely and a very complicated process. A more common scenario is people
committing in the dev registry and then they have some scripts in place
(using the scripted event hook that you mentioned) to sync flows into a
prod registry. IMO we should just have something working for all configured
registries regardless of the implementations of the endpoint.
Le jeu. 18 janv. 2024 à 10:07, Simon Bence a
écrit :
> Thank you for the feedback! I look into this and check how it would be
> best to apply the rules engine here. Most probably the changes should be
> somewhere around StandardFlowRegistryClientNode so this can be used
> regardless of the actual plugged in client. However this comes with a
> further question: if it is part of the framework and there are more
> registry clients added to the NiFi instance, do we want to provide the
> option to differentiate between registries? As far as I know the rules
> engine, differentiation on the level of applied rules per registry would be
> challenging with the current setup but we can provide a toggle for the
> feature on a “per registry” basis. The motivation for this: in multiple
> occasions I saw cases where people used “in dev” registry for development
> before committing their flow into the final one. So from flow development
> lifecycle it might be too restricting to apply this for every registry (if
> the feature is turned on in the first place)
>
> Regards,
> Bence
>
> >
> > On 2024. Jan 17., at 17:59, Bryan Bende wrote:
> >
> > Good point, since registry client is an extension point, using the
> > rules on NiFi side allows it to work for any registry client.
> >
> > On Wed, Jan 17, 2024 at 11:49 AM Pierre Villard
> > wrote:
> >>
> >> Yeah so the more I think about it the more we should probably couple
> this
> >> with the new rules engine feature of NiFi 2.0. Users could define their
> >> rules there and then it could just be a property in NiFi saying whether
> or
> >> not it should prevent someone from committing a new version if one of
> the
> >> required rules is raising a violation in the process group being
> committed.
> >> This way this would work with any registry implementation, not just the
> >> NiFi Registry (I'm saying this in case we ever get to a pure Git-based
> >> implementation of the Registry endpoint).
> >>
> >> Le mer. 17 janv. 2024 à 20:33, Bryan Bende a écrit :
> >>
> >>> Michael,
> >>>
> >>> I think this validation idea came from the absence of a pull request
> >>> review model. Meaning, the rules being discussed here would check for
> >>> the things someone would most likely look for in a review of the flow
> >>> changes to decide if the flow is good. Obviously it wouldn't be able
> >>> to do everything a person would, but it could be better than nothing.
> >>> As far as I know, there isn't any active work around building a review
> >>> model.
> >>>
> >>> -Bryan
> >>>
> >>> On Wed, Jan 17, 2024 at 11:18 AM Michael Hogue
> >>> wrote:
>
> Could this be accomplished through git webhooks once changes are
> >>> committed
> through Registry and CI pipelines to perform whatever validation you
> >>> wish?
>
> The glaring weakness in this approach is that the changes have already
> >>> been
> committed. This makes me wonder if we can improve the "flow
> development
> lifecycle" to enable teams to review flow changes before they're
> committed, much like a pull request. Is this on the roadmap for
> Registry?
>
> Mike
>
> On Wed, Jan 17, 2024 at 4:14 PM Bryan Bende wrote:
>
> > Thanks guys, that makes sense in the context of the review model.
> >
> > Obviously some kind of general pre-event hook is the most flexible,
> > but I would say we should also consider whether we really want to be
> > calling out to arbitrary scripts during the main API requests, as
> well
> > as consider what someone would have to do to implement a scripted
> > rule. The current events are all metadata related, they don't have
> the
> > content of the snapshots, so in this case we'd have to pass the
> entire
> > json string of a snapshot as an arg to the script and then the script
> > has to parse through it looking for fields to validate/check. I
> wonder
> > if we should consider defining the types of rules ourselves and
> > allowing some kind of rules config file to be provided. This way the
> > rules are loaded during start up and applied in Java code
