Re: Custom properties file for environmental promotion
Hello, Regarding Remote Process Groups, this is definitely something that needs to be improved. There is a JIRA to make the URL editable [1]. A significant amount of work has been done on the flow registry [2], and this will become the primary way to deploy flows across environments. The typical scenario would be to save your dev flow to the registry, and when importing it to QA or prod, you would then edit the RPG (based on the JIRA to make it editable) to be the URL for that environment. After that the URL would be set for that environment and would not be changed when upgrading to newer versions. Hope this helps. Thanks, Bryan [1] https://issues.apache.org/jira/browse/NIFI-4526 [2] https://github.com/apache/nifi-registry On Tue, Nov 7, 2017 at 11:58 PM, wildo wrote: > We have nearly wrapped up our testing with out Nifi scripts in dev, and are > now looking to push to QA. I found an article about creating a custom > properties file in order to specify each of your environmental specific > variables, and then specifying that file in nifi.properties at > nifi.variable.registry.properties. > > This will work fine omitting two cases I can think of. > > 1) We have a number of ListenTCP processors which require the "local network > interface" to be specified. I have read that Expression Language can access > system properties, but I haven't seen any example about how to use this. Can > anyone share how EL might be used to grab the local network interface for > each environment automatically? > > 2) We use Remote Process Groups with Site-to-Site for load balancing. In the > RPG, you have to specify an absolute url to the nodes in the remote site. > The RPG doesn't indicate that EL is acceptable in this field. Can anyone > chime in on the possibility of using EL to grab a property for the RPG url? > > Thanks! > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Great info Bryan- thanks! Regarding my first question, I talked to our admins and we only have one NIC anyway. So there is no need for me to limit it, and thus I don't have a need to use EL to discover the NIC. So that's good. Regarding the registry stuff, I found this [1] document which looks FANTASTIC. But I'm not able to find when/if this stuff will be released. My understanding is that it is not yet released, and therefore I'm assuming that specifying a custom.properties file via the nifi.variable.registry.properties is still the preferred method. Additionally, this will mean that: 1) Changes to this file require a service restart, correct? 2) Is it true that this needs to be specified for each node of a clustered environment? Thanks again! [1] https://nifi.apache.org/registry.html -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Currently, there is the variable properties file which would require a service restart and also would need to be on all nodes in a cluster. The last release (1.4.0) added a more user-friendly variable registry in the UI which you can access from the context palette for a given process group, near where the controller services for a PG are located. When editing variables in the UI, it will detect components that reference them and automatically restart them. This variable registry will be tightly integrated with the experience of using the flow registry. Given all of the above, there still isn't anything you can currently do for RPGs though... you will unfortunately have to recreate it in the target environment until they become editable. As far as when the registry will be released, there are no set timelines for apache projects so it will be based when the community believes it is mature enough to be released, and when someone volunteers to be the release manager. That being said, a lot of good work has been done already and it is maturing quickly. Thanks, Bryan On Wed, Nov 8, 2017 at 10:21 AM, wildo wrote: > Great info Bryan- thanks! > > Regarding my first question, I talked to our admins and we only have one NIC > anyway. So there is no need for me to limit it, and thus I don't have a need > to use EL to discover the NIC. So that's good. > > Regarding the registry stuff, I found this [1] document which looks > FANTASTIC. But I'm not able to find when/if this stuff will be released. My > understanding is that it is not yet released, and therefore I'm assuming > that specifying a custom.properties file via the > nifi.variable.registry.properties is still the preferred method. > Additionally, this will mean that: > 1) Changes to this file require a service restart, correct? > 2) Is it true that this needs to be specified for each node of a clustered > environment? > > Thanks again! > > [1] https://nifi.apache.org/registry.html > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Excellent info, as always Bryan. Very much appreciated! -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Oh no! I just realized that the Kerberos Principle and Keytab fields in PutHDFS do *not* support expression language. Oy. This is a nonstarter... Bryan- is there some way to submit an enhancement request for EL support on these fields? Same goes for the port number on ListenTCP- no EL support. Sorry- I'm not aware of the protocol for submitting such an enhancement. -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
It looks like the HDFS ones were added in NiFi 1.4.0 [1] [2], and the ListenTCP EL support for Port has an open Jira [3]. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-2923 [2] https://issues.apache.org/jira/browse/NIFI-4010 [3] https://issues.apache.org/jira/browse/NIFI-4458 On Mon, Nov 13, 2017 at 11:08 AM, wildo wrote: > Oh no! I just realized that the Kerberos Principle and Keytab fields in > PutHDFS do *not* support expression language. Oy. This is a nonstarter... > > Bryan- is there some way to submit an enhancement request for EL support on > these fields? Same goes for the port number on ListenTCP- no EL support. > Sorry- I'm not aware of the protocol for submitting such an enhancement. > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Thanks Matt! I was just coming back to update. I found the 1.4 docs that indicated that it was available there. We are currently using 1.2. I should have researched better... Thanks again! -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
If you guys would please entertain one more question: Unfortunately the site is currently limited to Nifi 1.2, and this effectively renders the custom properties file useless as it relates to the PutHDFS kerberos principal/keytab. I was thinking about this as I exported the template in order to put into source control. Is there any reason that I couldn't modify the xml template to replace these values before uploading the template into the new environment? Kerberos Principal d...@mydomain.com Kerberos Keytab /my/path/to/keytabs/dev.keytab Similarly, when it comes to the remote process group, is there any reason that I can't just modify these uris before uploading the template to the new environment? https://dev001.mydomain.com:1234/nifi https://dev001.mydomain.com:1234/nifi,https://dev002.mydomain.com:1234/nifi It would be quite simple to write a script that does a simple substitution on these values. But is there any reason that that wouldn't work? I imagine I'm hardly the first to think about editing the template xml itself... -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
In general that approach should work, there were a few community efforts that did something like this in the past [1][2]. For the RPG, you may need to substitute another value as well, because I believe the template also contains the UUID of the ports it is connected to, which will be different depending on the target URI, but I don't remember exactly how this part works, maybe others can chime in. Thanks, Bryan [1] https://github.com/aperepel/nifi-api-deploy [2] https://github.com/hermannpencole/nifi-config On Mon, Nov 13, 2017 at 1:47 PM, wildo wrote: > If you guys would please entertain one more question: > > Unfortunately the site is currently limited to Nifi 1.2, and this > effectively renders the custom properties file useless as it relates to the > PutHDFS kerberos principal/keytab. I was thinking about this as I exported > the template in order to put into source control. > > Is there any reason that I couldn't modify the xml template to replace these > values before uploading the template into the new environment? > > > Kerberos Principal > d...@mydomain.com > > > > Kerberos Keytab > /my/path/to/keytabs/dev.keytab > > > > Similarly, when it comes to the remote process group, is there any reason > that I can't just modify these uris before uploading the template to the new > environment? > > https://dev001.mydomain.com:1234/nifi > https://dev001.mydomain.com:1234/nifi,https://dev002.mydomain.com:1234/nifi > > > It would be quite simple to write a script that does a simple substitution > on these values. But is there any reason that that wouldn't work? I imagine > I'm hardly the first to think about editing the template xml itself... > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Thanks Bryan. I just wrapped this up and wanted to document for anyone in the future. -Yes, updating the kerberos principal/keytab values via the xml template worked just fine. -Yes, updating the target url fields on the RPG worked just fine. -Yes, Bryan was absolutely correct that you DO need to update the port ID in the xml template to point to the proper ID in the *new* environment. The big gotchya is that this is not only specified in the itself, but the ID is also specified as a on the parent process group that it lives in. If you miss this, then Nifi will throw an error when you try to add the template to the canvas stating something like "An ID to a connection was specified that does not exist." Sorry I don't have explicit error message in front of me. In summary: since we are limited to Nifi 1.2, this manual updating of the XML worked fine. Just be mindful of the gotchya mentioned above. -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/