Re: Hashicorp vault transit engine for sensitive properties of processors
Cannon, There is also an ongoing effort to add a new framework-level component called a Parameter Provider [1], which will allow parameter values to be fetched into a NiFi Parameter Context from an external source. A HashiCorp Vault Key/Value Parameter Provider [2] has been proposed and is being worked. In this feature, sensitive parameter values are still encrypted as usual in the actual flow.xml.gz once fetched from the external source, so it doesn't exactly match your original question, but it seems to be one step closer. We hope to include this in a release sometime soon. [1] https://issues.apache.org/jira/browse/NIFI-8998 [2] https://issues.apache.org/jira/browse/NIFI-9401 Joe On Thu, Feb 3, 2022 at 9:06 AM David Handermann wrote: > Hi Cannon, > > The following NiFi Jira issue outlines a previous attempt to integrate > flow property storage and retrieval using HashiCorp Vault: > > https://issues.apache.org/jira/browse/NIFI-6825 > > I updated and closed the issue since the description itself outlines an > approach that no longer fits well in light of recent improvements to the > framework. > > Feel free to create a new issue and we can use that going forward. > > Regards, > David Handermann > > On Thu, Feb 3, 2022 at 12:23 AM Cannon Palms > wrote: > >> Thanks Joe! Do you know if there is an existing JIRA issue to track such >> a feature proposal (vault integration for flow secrets)? I’d be happy to >> create one if not. >> >> Cannon >> >> On Tue, Feb 1, 2022 at 3:46 PM Joe Gresock wrote: >> >>> Hi Cannon, >>> >>> Both the HashiCorp Vault Transit and Key/Value Sensitive Property >>> Providers are able to protect NiFi's configuration files (e.g., >>> nifi.properties, login-identity-providers.xml, and authorizers.xml). In >>> the case of the Transit implementation, you would use the encrypt-config.sh >>> tool from the NiFi Toolkit to encrypt properties in these files using the >>> Vault Transit Engine, and these will be decrypted using Vault as NiFi >>> starts up. The process is similar for the Key/Value implementation, but >>> the values of the properties are stored inside the Vault server instead of >>> being encrypted at rest in the configuration files. >>> >>> The properties in the flow.xml.gz file (e.g., your ConsumeMQTT processor >>> password) are protected by a different mechanism, and there is not >>> currently a Vault implementation that protects these. >>> >>> Hope this helps, >>> Joe >>> >>> On Tue, Feb 1, 2022 at 11:05 AM Cannon Palms >>> wrote: >>> Hello, From what I understand from the documentation, the transit engine of Hashicorp Vault is definitely supported for system properties. It is also clear that the standard key/value engine of Hashicorp vault is supported for sensitive processor properties (e.g. the password used to connect to an MQTT broker in a ConsumeMQTT processor). What I cannot tell is if NiFi supports using the transit engine for these sensitive properties of processors. I'd like to ensure that these properties are encrypted at rest inside of the registry, but decrypted using the transit engine and a provided vault encryption key at runtime. Is this currently supported? Or is the only the standard key/value engine supported for such properties? Thanks, Cannon >>>
Re: Hashicorp vault transit engine for sensitive properties of processors
Hi Cannon, The following NiFi Jira issue outlines a previous attempt to integrate flow property storage and retrieval using HashiCorp Vault: https://issues.apache.org/jira/browse/NIFI-6825 I updated and closed the issue since the description itself outlines an approach that no longer fits well in light of recent improvements to the framework. Feel free to create a new issue and we can use that going forward. Regards, David Handermann On Thu, Feb 3, 2022 at 12:23 AM Cannon Palms wrote: > Thanks Joe! Do you know if there is an existing JIRA issue to track such a > feature proposal (vault integration for flow secrets)? I’d be happy to > create one if not. > > Cannon > > On Tue, Feb 1, 2022 at 3:46 PM Joe Gresock wrote: > >> Hi Cannon, >> >> Both the HashiCorp Vault Transit and Key/Value Sensitive Property >> Providers are able to protect NiFi's configuration files (e.g., >> nifi.properties, login-identity-providers.xml, and authorizers.xml). In >> the case of the Transit implementation, you would use the encrypt-config.sh >> tool from the NiFi Toolkit to encrypt properties in these files using the >> Vault Transit Engine, and these will be decrypted using Vault as NiFi >> starts up. The process is similar for the Key/Value implementation, but >> the values of the properties are stored inside the Vault server instead of >> being encrypted at rest in the configuration files. >> >> The properties in the flow.xml.gz file (e.g., your ConsumeMQTT processor >> password) are protected by a different mechanism, and there is not >> currently a Vault implementation that protects these. >> >> Hope this helps, >> Joe >> >> On Tue, Feb 1, 2022 at 11:05 AM Cannon Palms >> wrote: >> >>> Hello, >>> >>> From what I understand from the documentation, the transit engine of >>> Hashicorp Vault is definitely supported for system properties. It is also >>> clear that the standard key/value engine of Hashicorp vault is supported >>> for sensitive processor properties (e.g. the password used to connect to an >>> MQTT broker in a ConsumeMQTT processor). >>> >>> What I cannot tell is if NiFi supports using the transit engine for >>> these sensitive properties of processors. >>> >>> I'd like to ensure that these properties are encrypted at rest inside of >>> the registry, but decrypted using the transit engine and a provided vault >>> encryption key at runtime. >>> >>> Is this currently supported? Or is the only the standard key/value >>> engine supported for such properties? >>> >>> Thanks, >>> Cannon >>> >>
Re: Hashicorp vault transit engine for sensitive properties of processors
Thanks Joe! Do you know if there is an existing JIRA issue to track such a feature proposal (vault integration for flow secrets)? I’d be happy to create one if not. Cannon On Tue, Feb 1, 2022 at 3:46 PM Joe Gresock wrote: > Hi Cannon, > > Both the HashiCorp Vault Transit and Key/Value Sensitive Property > Providers are able to protect NiFi's configuration files (e.g., > nifi.properties, login-identity-providers.xml, and authorizers.xml). In > the case of the Transit implementation, you would use the encrypt-config.sh > tool from the NiFi Toolkit to encrypt properties in these files using the > Vault Transit Engine, and these will be decrypted using Vault as NiFi > starts up. The process is similar for the Key/Value implementation, but > the values of the properties are stored inside the Vault server instead of > being encrypted at rest in the configuration files. > > The properties in the flow.xml.gz file (e.g., your ConsumeMQTT processor > password) are protected by a different mechanism, and there is not > currently a Vault implementation that protects these. > > Hope this helps, > Joe > > On Tue, Feb 1, 2022 at 11:05 AM Cannon Palms > wrote: > >> Hello, >> >> From what I understand from the documentation, the transit engine of >> Hashicorp Vault is definitely supported for system properties. It is also >> clear that the standard key/value engine of Hashicorp vault is supported >> for sensitive processor properties (e.g. the password used to connect to an >> MQTT broker in a ConsumeMQTT processor). >> >> What I cannot tell is if NiFi supports using the transit engine for these >> sensitive properties of processors. >> >> I'd like to ensure that these properties are encrypted at rest inside of >> the registry, but decrypted using the transit engine and a provided vault >> encryption key at runtime. >> >> Is this currently supported? Or is the only the standard key/value engine >> supported for such properties? >> >> Thanks, >> Cannon >> >
Re: Hashicorp vault transit engine for sensitive properties of processors
Hi Cannon, Both the HashiCorp Vault Transit and Key/Value Sensitive Property Providers are able to protect NiFi's configuration files (e.g., nifi.properties, login-identity-providers.xml, and authorizers.xml). In the case of the Transit implementation, you would use the encrypt-config.sh tool from the NiFi Toolkit to encrypt properties in these files using the Vault Transit Engine, and these will be decrypted using Vault as NiFi starts up. The process is similar for the Key/Value implementation, but the values of the properties are stored inside the Vault server instead of being encrypted at rest in the configuration files. The properties in the flow.xml.gz file (e.g., your ConsumeMQTT processor password) are protected by a different mechanism, and there is not currently a Vault implementation that protects these. Hope this helps, Joe On Tue, Feb 1, 2022 at 11:05 AM Cannon Palms wrote: > Hello, > > From what I understand from the documentation, the transit engine of > Hashicorp Vault is definitely supported for system properties. It is also > clear that the standard key/value engine of Hashicorp vault is supported > for sensitive processor properties (e.g. the password used to connect to an > MQTT broker in a ConsumeMQTT processor). > > What I cannot tell is if NiFi supports using the transit engine for these > sensitive properties of processors. > > I'd like to ensure that these properties are encrypted at rest inside of > the registry, but decrypted using the transit engine and a provided vault > encryption key at runtime. > > Is this currently supported? Or is the only the standard key/value engine > supported for such properties? > > Thanks, > Cannon >
Fwd: Hashicorp vault transit engine for sensitive properties of processors
Hello, >From what I understand from the documentation, the transit engine of Hashicorp Vault is definitely supported for system properties. It is also clear that the standard key/value engine of Hashicorp vault is supported for sensitive processor properties (e.g. the password used to connect to an MQTT broker in a ConsumeMQTT processor). What I cannot tell is if NiFi supports using the transit engine for these sensitive properties of processors. I'd like to ensure that these properties are encrypted at rest inside of the registry, but decrypted using the transit engine and a provided vault encryption key at runtime. Is this currently supported? Or is the only the standard key/value engine supported for such properties? Thanks, Cannon
