RE: Nifi toolkit encrypt single properties
Hi David, Thanks for your answer. I had considered Hashicorp Vault for exactly the reason you state, but at this point it's too much overhead and not all of our NiFi clusters are at 1.14+ yet. You make a good point about the likelihood of changes "under the hood" causing problems. I'll focus instead on invoking encrypt-config.sh in the usual way and extracting the results from the output file somehow. Perhaps I can generate an input with a single property in it and get a single output. Kind regards, Isha -Oorspronkelijk bericht- Van: David Handermann Verzonden: dinsdag 25 januari 2022 15:25 Aan: dev@nifi.apache.org Onderwerp: Re: Nifi toolkit encrypt single properties Hi Isha, Thanks for contacting the developers list and describing what you are attempting to accomplish. Although there are several Java interfaces and implementation classes that could be leveraged to implement a custom approach to property encryption, that may not be the most reliable method. NiFi does not consider the Sensitive Property Provider interfaces and classes part of the public API, which means they could be subject to change in minor version updates. That being said, the input and output format for encrypted property values will remain supported across versions, so you could use the NiFi classes as a starting point. NiFi stores the serialized encrypted values in a custom format containing the initialization vector, followed by a delimiter, and then the encrypted content. It is possible to use standard AES-GCM support in other languages to read the value if you have the root key available. Recent versions of NiFi introduced support for additional nifi.properties protection schemes using external services such as HashiCorp Vault and Amazon Secrets Manager, as well as other service providers. The effective result is similar in that nifi.properties contains encrypted or placeholder values, but that might provide some different options. For example, the HashiCorp Vault Key Value store implementation uses a placeholder value for the property, instead of the encrypted string. This approach yields the same placeholder value, since HashiCorp Vault stores the actual property. That kind of strategy requires additional infrastructure, so that may not be a good fit for your environment, but it might be worth considering. The most stable integration from the perspective of an external tool is the encrypt-config.sh command and associated arguments. Regards, David Handermann On Tue, Jan 25, 2022 at 2:42 AM Isha Lamboo wrote: > Hi all, > > I hope this question is appropriate for the developers list, if not, > I’ll move it to users. > > I have an Ansible role for NiFi that includes generating the NiFi > properties files from templates and variables set per cluster/host as > well as updating keystore etc. > This works very well, except when it comes to protected properties as > set by the toolkit. The toolkit wants to operate on the actual > properties files, which causes Ansible to then see differences and want to > reset. > I tried with an intermediate processing dir, but then I lose the > ability to use Ansible’s –check and –diff options to see if any changes were > made. > In the end, I added the encrypted values by hand to the Ansible > variables files. > > As a next step, I’m looking at whether I can use the toolkit’s java > classes in my own wrapper to allow me to pass in a master key, > protection scheme and raw value and get out the encrypted one, so that > I can easily update my encrypted values in the Ansible inventory. > However, my Java skills are somewhat limited, so I would like to ask > first if this is even a good idea. > > Is this a sensible idea or does it conflict with NiFi or encryption > design principles I’m not aware of? > > Kind regards, > > Isha Lamboo > >
Re: Nifi toolkit encrypt single properties
Hi Isha, Thanks for contacting the developers list and describing what you are attempting to accomplish. Although there are several Java interfaces and implementation classes that could be leveraged to implement a custom approach to property encryption, that may not be the most reliable method. NiFi does not consider the Sensitive Property Provider interfaces and classes part of the public API, which means they could be subject to change in minor version updates. That being said, the input and output format for encrypted property values will remain supported across versions, so you could use the NiFi classes as a starting point. NiFi stores the serialized encrypted values in a custom format containing the initialization vector, followed by a delimiter, and then the encrypted content. It is possible to use standard AES-GCM support in other languages to read the value if you have the root key available. Recent versions of NiFi introduced support for additional nifi.properties protection schemes using external services such as HashiCorp Vault and Amazon Secrets Manager, as well as other service providers. The effective result is similar in that nifi.properties contains encrypted or placeholder values, but that might provide some different options. For example, the HashiCorp Vault Key Value store implementation uses a placeholder value for the property, instead of the encrypted string. This approach yields the same placeholder value, since HashiCorp Vault stores the actual property. That kind of strategy requires additional infrastructure, so that may not be a good fit for your environment, but it might be worth considering. The most stable integration from the perspective of an external tool is the encrypt-config.sh command and associated arguments. Regards, David Handermann On Tue, Jan 25, 2022 at 2:42 AM Isha Lamboo wrote: > Hi all, > > I hope this question is appropriate for the developers list, if not, I’ll > move it to users. > > I have an Ansible role for NiFi that includes generating the NiFi > properties files from templates and variables set per cluster/host as well > as updating keystore etc. > This works very well, except when it comes to protected properties as set > by the toolkit. The toolkit wants to operate on the actual properties > files, which causes Ansible to then see differences and want to reset. > I tried with an intermediate processing dir, but then I lose the ability > to use Ansible’s –check and –diff options to see if any changes were made. > In the end, I added the encrypted values by hand to the Ansible variables > files. > > As a next step, I’m looking at whether I can use the toolkit’s java > classes in my own wrapper to allow me to pass in a master key, protection > scheme and raw value and get out the encrypted one, so that I can easily > update my encrypted values in the Ansible inventory. > However, my Java skills are somewhat limited, so I would like to ask first > if this is even a good idea. > > Is this a sensible idea or does it conflict with NiFi or encryption design > principles I’m not aware of? > > Kind regards, > > Isha Lamboo > >
Nifi toolkit encrypt single properties
Hi all, I hope this question is appropriate for the developers list, if not, I’ll move it to users. I have an Ansible role for NiFi that includes generating the NiFi properties files from templates and variables set per cluster/host as well as updating keystore etc. This works very well, except when it comes to protected properties as set by the toolkit. The toolkit wants to operate on the actual properties files, which causes Ansible to then see differences and want to reset. I tried with an intermediate processing dir, but then I lose the ability to use Ansible’s –check and –diff options to see if any changes were made. In the end, I added the encrypted values by hand to the Ansible variables files. As a next step, I’m looking at whether I can use the toolkit’s java classes in my own wrapper to allow me to pass in a master key, protection scheme and raw value and get out the encrypted one, so that I can easily update my encrypted values in the Ansible inventory. However, my Java skills are somewhat limited, so I would like to ask first if this is even a good idea. Is this a sensible idea or does it conflict with NiFi or encryption design principles I’m not aware of? Kind regards, Isha Lamboo
