RE: Nifi Registry git presistence
Hi Bryan, Thanks for the explanation. Unfortunately, we don't have possibility at all to integrate all the envs with one central NiFi registry nor to integrate dedicated NiFi registries between the envs, security reasons etc, no connectivity between them. The way we are having is to sync underlaying git repos when transferring the flows to higher logical env. We have also parameter context which needs to be env specific. Is there a way how to automate this, to have like some kind of tmeplate file which can be populated with specific env values? Just to avoid manually modifying parameter context file and then to import it to the higher env. Best regards, Tom -Original Message- From: Bryan Bende Sent: 23 June 2021 16:34 To: users@nifi.apache.org Subject: Re: Nifi Registry git presistence There is no automatic pulling from git. The way registry works is that there is a database (H2/MySql/Postgres) that stores all the metadata and knowledge of what items exist, and then a persistence provider for each time of versioned item (flows, extension bundles). Git is just an implementation of a flow persistence provider, so it was really meant to be treated like just another place to store a blob. There are also file system and database providers for flow persistence. When registry starts up, if the database is empty and git is configured, then it reads the git repo to rebuild the database. So the trick some people are using, is that in higher environments like prod where the registry is read-only, they periodically stop registry, delete the H2 database, git pull, and start again, and now everything is up to date. On Wed, Jun 23, 2021 at 10:21 AM Tomislav Novosel wrote: > > Hi Chris, Bryan, > > Thanks for the answers. > > How to configure "follower" Registries to pull from the "leader" > automatically? What if one of the Registries goes down, how to recover, is it > tricky? > > We have three separated envs, with separated nifi registries. Is it even > required to have git as a persistence layer? > What is the difference if we setup just file persistence layer in comparison > to git? > > I know Toolkit CLI can be used in our case for transfering the flows > between registries, but I'm curious and want to know pros and cons of having > one git repo for all the registries. > > Thanks, > Tom > -Original Message- > From: Bryan Bende > Sent: 23 June 2021 14:58 > To: users@nifi.apache.org > Subject: Re: Nifi Registry git presistence > > It wasn't really meant for that, but as others pointed out, if only one of > the registries is doing writes and the others are read-only for pulling into > nifi in another environment, then it can work. > > The best option is to have one central registry that all nifi's can access > (if you are willing to access the same git repo from all environments, then > why not the same nifi registry?). > > The second best option is the database persistence provider, with all > registries pointing at the same database. > > On Wed, Jun 23, 2021 at 7:56 AM Chris McKeever wrote: > > > > Yes -- however: > > - only one should be the authority > > - see the older thread of how you need to have the read-only-clients > > re-poll the remote git in order ot pick up the latest changes. > > > > On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel > > wrote: > >> > >> Hi to all, > >> > >> > >> > >> i sit possible to have one central Git repo for multiple NiFi > >> registries > >> > >> as persistance layer? > >> > >> Ii it possible to configure, for example, three NiFi registries > >> (through providers.xml file) > >> > >> to communicate with one git repo? > >> > >> > >> > >> Thanks, > >> > >> Tom
Re: Nifi Registry git presistence
There is no automatic pulling from git. The way registry works is that there is a database (H2/MySql/Postgres) that stores all the metadata and knowledge of what items exist, and then a persistence provider for each time of versioned item (flows, extension bundles). Git is just an implementation of a flow persistence provider, so it was really meant to be treated like just another place to store a blob. There are also file system and database providers for flow persistence. When registry starts up, if the database is empty and git is configured, then it reads the git repo to rebuild the database. So the trick some people are using, is that in higher environments like prod where the registry is read-only, they periodically stop registry, delete the H2 database, git pull, and start again, and now everything is up to date. On Wed, Jun 23, 2021 at 10:21 AM Tomislav Novosel wrote: > > Hi Chris, Bryan, > > Thanks for the answers. > > How to configure "follower" Registries to pull from the "leader" > automatically? What if one of the Registries goes down, how to recover, is it > tricky? > > We have three separated envs, with separated nifi registries. Is it even > required to have git as a persistence layer? > What is the difference if we setup just file persistence layer in comparison > to git? > > I know Toolkit CLI can be used in our case for transfering the flows between > registries, but I'm curious and want to know pros and cons of having > one git repo for all the registries. > > Thanks, > Tom > -Original Message- > From: Bryan Bende > Sent: 23 June 2021 14:58 > To: users@nifi.apache.org > Subject: Re: Nifi Registry git presistence > > It wasn't really meant for that, but as others pointed out, if only one of > the registries is doing writes and the others are read-only for pulling into > nifi in another environment, then it can work. > > The best option is to have one central registry that all nifi's can access > (if you are willing to access the same git repo from all environments, then > why not the same nifi registry?). > > The second best option is the database persistence provider, with all > registries pointing at the same database. > > On Wed, Jun 23, 2021 at 7:56 AM Chris McKeever wrote: > > > > Yes -- however: > > - only one should be the authority > > - see the older thread of how you need to have the read-only-clients > > re-poll the remote git in order ot pick up the latest changes. > > > > On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel > > wrote: > >> > >> Hi to all, > >> > >> > >> > >> i sit possible to have one central Git repo for multiple NiFi > >> registries > >> > >> as persistance layer? > >> > >> Ii it possible to configure, for example, three NiFi registries > >> (through providers.xml file) > >> > >> to communicate with one git repo? > >> > >> > >> > >> Thanks, > >> > >> Tom
RE: Nifi Registry git presistence
Hi Chris, Bryan, Thanks for the answers. How to configure "follower" Registries to pull from the "leader" automatically? What if one of the Registries goes down, how to recover, is it tricky? We have three separated envs, with separated nifi registries. Is it even required to have git as a persistence layer? What is the difference if we setup just file persistence layer in comparison to git? I know Toolkit CLI can be used in our case for transfering the flows between registries, but I'm curious and want to know pros and cons of having one git repo for all the registries. Thanks, Tom -Original Message- From: Bryan Bende Sent: 23 June 2021 14:58 To: users@nifi.apache.org Subject: Re: Nifi Registry git presistence It wasn't really meant for that, but as others pointed out, if only one of the registries is doing writes and the others are read-only for pulling into nifi in another environment, then it can work. The best option is to have one central registry that all nifi's can access (if you are willing to access the same git repo from all environments, then why not the same nifi registry?). The second best option is the database persistence provider, with all registries pointing at the same database. On Wed, Jun 23, 2021 at 7:56 AM Chris McKeever wrote: > > Yes -- however: > - only one should be the authority > - see the older thread of how you need to have the read-only-clients re-poll > the remote git in order ot pick up the latest changes. > > On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel > wrote: >> >> Hi to all, >> >> >> >> i sit possible to have one central Git repo for multiple NiFi >> registries >> >> as persistance layer? >> >> Ii it possible to configure, for example, three NiFi registries >> (through providers.xml file) >> >> to communicate with one git repo? >> >> >> >> Thanks, >> >> Tom
Re: Nifi Registry git presistence
It wasn't really meant for that, but as others pointed out, if only one of the registries is doing writes and the others are read-only for pulling into nifi in another environment, then it can work. The best option is to have one central registry that all nifi's can access (if you are willing to access the same git repo from all environments, then why not the same nifi registry?). The second best option is the database persistence provider, with all registries pointing at the same database. On Wed, Jun 23, 2021 at 7:56 AM Chris McKeever wrote: > > Yes -- however: > - only one should be the authority > - see the older thread of how you need to have the read-only-clients re-poll > the remote git in order ot pick up the latest changes. > > On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel > wrote: >> >> Hi to all, >> >> >> >> i sit possible to have one central Git repo for multiple NiFi registries >> >> as persistance layer? >> >> Ii it possible to configure, for example, three NiFi registries (through >> providers.xml file) >> >> to communicate with one git repo? >> >> >> >> Thanks, >> >> Tom
Re: Nifi Registry git presistence
Yes -- however: - only one should be the authority - see the older thread of how you need to have the read-only-clients re-poll the remote git in order ot pick up the latest changes. On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel < tomislav.novo...@clearpeaks.com> wrote: > Hi to all, > > > > i sit possible to have one central Git repo for multiple NiFi registries > > as persistance layer? > > Ii it possible to configure, for example, three NiFi registries (through > providers.xml file) > > to communicate with one git repo? > > > > Thanks, > > Tom >
