Hi Andy, I verified what you suggested: * Can you look for any other entries of the form “nifi.xyz.protected=“? -> Verified, no extra protected properties. * Are you sure that is being removed? -> I am sure.
When you say: *check the value of your master key to ensure it is the same key that encrypted that value. *How I can check that? Thanks On Mon, 9 Dec 2019 at 11:44, Andy LoPresto <alopre...@apache.org> wrote: > Thanks Juan. A couple notes: > > Using the same plaintext value for multiple keys will not cause a > technical problem, but it is bad security practice and is strongly > discouraged. It would not be the source of the issue here (however, you > need to use a fully-formed AES key for the provenance encryption key, and > it’s unlikely that would be the same value or format as a password for the > sensitive properties. That can cause other problems later on). > > As you are using the plain WriteAheadProvenanceRepository and not the > EncryptedWriteAheadProvenanceRepository, you do not need to provide (and in > fact, they are currently ignored) any properties for > nifi.provenance.encryption.*. So you can remove those lines entirely (and > probably should just for clarity and not to confuse anyone else who looks > at these properties). If you want to use the encrypted repository, you’ll > need to change the repository implementation (see step-by-step details in > the link I provided earlier). > > The nested exception was that one of the encrypted properties did not > contain the “||” delimiter. From visual inspection, it appears that all > properties you have listed here do contain the delimiter. That exception is > only thrown in one condition, and that is a simple string contains check > for the delimiter. Are you sure these are the only encrypted values in your > nifi.properties file, and that you are referencing the correct file? Can > you look for any other entries of the form “nifi.xyz.protected=“? > > You mentioned that it generates two unique entries for > “nifi.provenance.repository.encryption.key” and you remove the plaintext > one. Are you sure that is being removed? If the system believes that > property is encrypted (as indicated by the > nifi.provenance.repository.encryption.key.protected=aes/gcm/256” line > following it) and tries to decrypt the plaintext value, that would cause > the exception to be thrown. > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Dec 9, 2019, at 2:22 PM, Juan Pablo Gardella < > gardellajuanpa...@gmail.com> wrote: > > Thanks for answering my questions Andy, > > Below are the sensitive properties: > > # Provenance Repository Properties > > nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository > nifi.provenance.repository.debug.frequency=1_000_000 > > *nifi.provenance.repository.encryption.key=fbRg/ZgK7U8qJcrU||4nI1n1aRD0Tooq7TLSTyVDhkmX8* > nifi.provenance.repository.encryption.key.protected=aes/gcm/256 > nifi.provenance.repository.encryption.key.provider.location= > nifi.provenance.repository.encryption.key.id= > # security properties # > *nifi.sensitive.props.key=jtZiGY+mZyHPQIc1||/IJnMQBBXKN7VNkwMf6Oo7vZmAs* > nifi.sensitive.props.key.protected=aes/gcm/256 > nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL > nifi.sensitive.props.provider=BC > nifi.sensitive.props.additional.keys= > > nifi.security.keystore=/opt/certs/keystore.jks > nifi.security.keystoreType=JKS > > *nifi.security.keystorePasswd=GuuOm4fyK6yvo76H||av/NQmH7Hw8qK9k0NOMRSjp08tw+walt4D5JLpYPiCHG/Z7DDq5QZ+ui/dKOXxtapH76Gjpt3hMwmP0* > nifi.security.keystorePasswd.protected=aes/gcm/256 > > *nifi.security.keyPasswd=y4spsJvsy5Fzc3Uq||Q1vMntNgfLLMMSJuyPNn8+9aHlH+banQy82Ly0qrLWf6hNUTNgA+akyh86rlf2J5XZCONL3JCLX6mY0* > nifi.security.keyPasswd.protected=aes/gcm/256 > nifi.security.truststore=/opt/certs/truststore.jks > nifi.security.truststoreType=JKS > > *nifi.security.truststorePasswd=9r+fyOSjRUXQLcZG||YwAtPYorADqHSKFUmU4H3SbyqvYqqYNZiGidgCOUCibPdP2jiEAMGtLt5xyFsMcNPm5Pye2qXEioLR8* > nifi.security.truststorePasswd.protected=aes/gcm/256 > > These properties are generated by the toolkit. I using the same value for > nifi.sensitive.props.key value and the > nifi.provenance.repository.encryption.key, I was not aware they should be > different. Could be that the problem? > > Juan > > On Mon, 9 Dec 2019 at 08:20, Andy LoPresto <alopre...@apache.org> wrote: > >> Hi Juan, >> >> The error you are getting is saying that one of the protected properties >> is not of the expected format. While the Sensitive Property Provider >> mechanism is extensible (see NIFI-5481 [1] for additional options being >> added), the only natively supported one in 1.10.0 is AES/GCM encryption. >> This requires the sensitive properties to be in the format >> >> >> Wl9bXjSWX5DXs4Gm||EDnf18wwAAMJFckgNNfkRWiA4daSDWJCuRvSsbe99AaefQrkpmSqehJtyJGgEbhn402zSyztXi1EGPU >> >> Where the segment preceding the “||” delimiter is the Base64-encoded 16 >> byte initialization vector (IV), which is random and unique for each >> property, and the segment following the delimiter is the Base64-encoded >> cipher text. >> >> The error states that when NiFi tries to decrypt one of the five >> encrypted properties (it does not specify which in this case), it is not >> encoded in the proper form. Assuming you are using a strong key >> for nifi.bootstrap.sensitive.key in conf/bootstrap.conf, you can share the >> nifi.properties file with the encoded and encrypted values with this list >> to be verified for format, as no one will be able to decrypt them. However, >> if you do not wish to share them, please validate that they are all of the >> format specified above and encrypted with the same key that is present in >> bootstrap.conf. >> >> Another thing I noted is that you are replacing the >> nifi.sensitive.props.key value and the >> nifi.provenance.repository.encryption.key value with the same environment >> variable. These keys should not have the same value. The provenance >> repository key is designed to protect the provenance repository on disk and >> be rotated/migrated automatically. The formatting and provision of these >> keys is documented in the User Guide [2]. The key can be present in >> plaintext (raw hexadecimal encoding) or encrypted as any other sensitive >> configuration value in the nifi.properties file. >> >> The nifi.sensitive.props.key value is a password or other key derivation >> material used by NiFi to derive a strong key to encrypt the sensitive >> _property_ values - this means things like database passwords, FTP server >> passwords, keystore passwords, etc. that the NiFi flow uses and persists in >> an encrypted format in the flow.xml.gz file. >> >> If you believe the sensitive properties key you are injecting into the >> file is in the correct format (encoded as described above), check the value >> of your master key to ensure it is the same key that encrypted that value. >> If you are injecting a plaintext value like >> “my_bad_sensitive_props_password”, you must remove the master key from the >> bootstrap.conf file and ensure there is no sibling property present called >> NiFi.sensitive.props.key.protected which indicates that the value must be >> decrypted. >> >> I.e. the existing section like: >> >> >> nifi.sensitive.props.key=xPqEWK8a34r19J4z||UOFzOfZE/NQK4Xua8WWblf1/Ld+Pf7eQ1zg0U/qYW2sPwxyhhOXWwQmrUft6qA >> nifi.sensitive.props.key.protected=aes/gcm/128 >> >> Should change to look like: >> >> nifi.sensitive.props.key=my_bad_sensitive_props_password >> NiFi.sensitive.props.key.protected= # or remove this line entirely >> >> >> [1] https://github.com/apache/nifi/pull/3672 >> [2] >> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#encrypted-provenance >> >> >> Andy LoPresto >> alopre...@apache.org >> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Dec 8, 2019, at 8:01 PM, Juan Pablo Gardella < >> gardellajuanpa...@gmail.com> wrote: >> >> Hello all, >> >> I am trying to protect plain text passwords. I am using the latest docker >> image (1.10.0), and edited manually nifi.sensitive.props.key as below >> >> sed -i -e >> "s|^nifi.sensitive.props.key=.*$|nifi.sensitive.props.key=${NIFI_SENSITIVE_PROPS_KEY}|" >> /opt/nifi/nifi-current/conf/nifi.properties >> sed -i -e >> "s|^nifi.provenance.repository.encryption.key=.*$|nifi.provenance.repository.encryption.key=${NIFI_SENSITIVE_PROPS_KEY}|" >> /opt/nifi/nifi-current/conf/nifi.properties >> >> (this command for some reason does not update the file inside the >> Dockerfile, I have to do inside the container). >> >> After updated that property, I run following command inside the container: >> >> bash /opt/nifi/nifi-toolkit-current/bin/encrypt-config.sh -n >> /opt/nifi/nifi-current/conf/nifi.properties -b >> /opt/nifi/nifi-current/conf/bootstrap.conf -a >> /opt/nifi/nifi-current/conf/authorizers.xml -l >> /opt/nifi/nifi-current/conf/login-identity-providers.xml >> >> It prompts to put a master password and after that, I restart[1] the >> container but it failed to start with below error: >> >> nifi | 2019-12-08 18:57:31,777 INFO [main] >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 162 properties from >> /opt/nifi/nifi-current/./conf/nifi.properties >> *nifi | 2019-12-08 18:57:31,933 INFO [main] >> o.a.n.properties.ProtectedNiFiProperties There are 5 protected properties >> of 5 sensitive properties (100%)* >> nifi | 2019-12-08 18:57:31,935 ERROR [main] org.apache.nifi.NiFi >> Failure to launch NiFi due to java.lang.IllegalArgumentException: There was >> an issue decrypting protected properties >> nifi | java.lang.IllegalArgumentException: There was an issue >> decrypting protected properties >> nifi | at >> org.apache.nifi.NiFi.initializeProperties(NiFi.java:341) >> nifi | at >> org.apache.nifi.NiFi.convertArgumentsToValidatedNiFiProperties(NiFi.java:309) >> nifi | at org.apache.nifi.NiFi.main(NiFi.java:300) >> nifi | Caused by: java.lang.IllegalArgumentException: The cipher >> text does not contain the delimiter || -- it should be of the form >> Base64(IV) || Base64(cipherText) >> nifi | at >> org.apache.nifi.properties.AESSensitivePropertyProvider.unprotect(AESSensitivePropertyProvider.java:217) >> nifi | at >> org.apache.nifi.properties.ProtectedNiFiProperties.unprotectValue(ProtectedNiFiProperties.java:524) >> nifi | at >> org.apache.nifi.properties.ProtectedNiFiProperties.getUnprotectedProperties(ProtectedNiFiProperties.java:343) >> nifi | at >> org.apache.nifi.properties.NiFiPropertiesLoader.load(NiFiPropertiesLoader.java:209) >> nifi | at >> org.apache.nifi.properties.NiFiPropertiesLoader.load(NiFiPropertiesLoader.java:223) >> nifi | at >> org.apache.nifi.properties.NiFiPropertiesLoader.loadDefault(NiFiPropertiesLoader.java:130) >> nifi | at >> org.apache.nifi.properties.NiFiPropertiesLoader.get(NiFiPropertiesLoader.java:241) >> nifi | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> nifi | at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> nifi | at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> nifi | at java.lang.reflect.Method.invoke(Method.java:498) >> nifi | at >> org.apache.nifi.NiFi.initializeProperties(NiFi.java:336) >> nifi | ... 2 common frames omitted >> >> Any idea why it is failing? >> >> Thanks, >> Juan >> >> [1] Actually, after that command two entries are generated to >> nifi.provenance.repository.encryption.key= in the file, one with the plain >> text and the other encrypted. I have to remove manually the plain text one. >> >> >> >
