Hi Joe, Just to confirm here - was the nifi.security.keyPasswd not defined at all in your nifi.properties? Did you have to add the property and give it the correct value? Or was it in the nifi.properties file but blank? Or were the keyPasswd and keystorePasswd different values?
Thanks, Nathan On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock <jgres...@gmail.com> wrote: > Yep, setting the nifi.security.keyPasswd to the same as > nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre! > > On Tue, Mar 3, 2020 at 2:01 PM Joe Witt <joe.w...@gmail.com> wrote: > > > relevant change I believe is here: > > > > > https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775 > > and > > is from https://issues.apache.org/jira/browse/NIFI-6927 > > > > It *looks* to me like this was fixing an improper naming/usage issue that > > has been present but if so we probably should have addressed not in this > > bug fix line. Will defer to Troy/Andy for more context and next steps > > > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt <joe.w...@gmail.com> wrote: > > > > > If accurate....We need to look into whether this was a mistake and fix > it > > > if so. And we need to reflect this in the migration guide > > > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward <ryan.wa...@gmail.com> wrote: > > > > > >> Endre - thanks that was it > > >> > > >> On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs > > >> <andrewsmit...@protonmail.com.invalid> wrote: > > >> > > >> > Hi, > > >> > > > >> > One additional thing: > > >> > > > >> > we encountered something strange as well: > > >> > > > >> > on 1.11.2 clustered, kerberized: request replication worked well. > > >> > > > >> > on 1.11.3 clustered, kerberized: request replication did not work, > > >> unless > > >> > you specify, and set > > >> > nifi.security.keyPasswd > > >> > > > >> > to the very same password as the > > >> > > > >> > nifi.security.keystorePasswd > > >> > > > >> > For us this resolved the issue. > > >> > > > >> > Best regards, > > >> > Endre > > >> > > > >> > Sent with [ProtonMail](https://protonmail.com) Secure Email. > > >> > > > >> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > >> > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward <ryan.wa...@gmail.com > > > > >> > wrote: > > >> > > > >> > > Hi Joe - Did you resolve your issue? If so I am wondering what the > > fix > > >> > was as I'm seeing the same error on my cluster. > > >> > > > > >> > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs < > > >> > andrewsmit...@protonmail.com.invalid> wrote: > > >> > > > > >> > >> Hi Joe, > > >> > >> > > >> > >> 1. Have you tried connecting/debugging with openssl? From one > pod > > to > > >> > the other: > > >> > >> (openssl s_client -debug -CAfile > > >> > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt > > >> -connect > > >> > nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) > > >> > >> > > >> > >> 2. certs can also be verified by: > > >> > >> openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt > > >> > >> > > >> > >> 3. Can you check if no intermediary CAs are missing from the > nodes > > >> > truststore? > > >> > >> > > >> > >> 4. This exception is coming from inter-node communication > > >> (replication > > >> > of request from one node to the other). This means that it is > > unrelated > > >> to > > >> > external user's authentication by client certificate. The question > is: > > >> is > > >> > your inter node communication secured by the trusted root CA (that > you > > >> are > > >> > sure that the CA cert is present in the trust store) or is it > secured > > by > > >> > selfsigned CA (which's CA may be lacking from your truststore)? > > >> > >> > > >> > >> 5. `nifi.security.needClientAuth` is not part of NiFi properties > > any > > >> > more. If SSL is turned on, and no > > >> > `nifi.security.user.login.identity.provider` is set, then client > cert > > >> based > > >> > auth is the default. But supplying this property have no detrimental > > >> effect > > >> > anyhow. > > >> > >> > > >> > >> Best regards, > > >> > >> Endre > > >> > >> > > >> > >> Sent with ProtonMail Secure Email. > > >> > >> > > >> > >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > >> > >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock > > >> > jgres...@gmail.com wrote: > > >> > >> > > >> > >>> Were there any changes with how the trust store is used in > > 1.11.3? I > > >> > had a > > >> > >>> 1.11.0 deployment working with the following settings, but when > I > > >> > deployed > > >> > >>> 1.11.3, the cluster can't seem to replicate requests to itself: > > >> > >>> nifi.remote.input.host=<redacted> > > >> > >>> nifi.remote.input.secure=true > > >> > >>> nifi.remote.input.socket.port=32440 > > >> > >>> nifi.remote.input.http.enabled=true > > >> > >>> nifi.cluster.protocol.is.secure=true > > >> > >>> nifi.cluster.is.node=true > > >> > >>> > > >> > > > >> > > > nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > >> > >>> nifi.cluster.node.protocol.port=6007 > > >> > >>> > > >> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local > > >> > >>> nifi.web.https.port=8443 > > >> > >>> nifi.security.keystore=./conf/keystore.jks > > >> > >>> nifi.security.keystoreType=jks > > >> > >>> nifi.security.keystorePasswd=<password> > > >> > >>> nifi.security.keyPasswd= > > >> > >>> nifi.security.truststore=./conf/truststore.jks > > >> > >>> nifi.security.truststoreType=jks > > >> > >>> nifi.security.truststorePasswd=<password> > > >> > >>> nifi.security.needClientAuth=true > > >> > >>> A trusted client cert that worked against the old cluster is > > getting > > >> > the > > >> > >>> same trust error (PKIX path building failed). I've verified that > > the > > >> > >>> client cert was issued by an issuer that is definitely in the > > >> > >>> ./conf/truststore.jks as a trustedCertEntry. > > >> > >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] > > >> > >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator > > >> > >>> javax.net.ssl.SSLHandshakeException: > > >> > >>> sun.security.validator.ValidatorException: PKIX path building > > >> failed: > > >> > >>> sun.security.provider.certpath.SunCertPathBuilderException: > unable > > >> to > > >> > find > > >> > >>> valid certification path to r > > >> > >>> equested target > > >> > >>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > > >> > >>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) > > >> > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) > > >> > >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) > > >> > >>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) > > >> > >>> at > sun.security.ssl.Handshaker.process_record(Handshaker.java:965) > > >> > >>> at > > >> > >>> > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) > > >> > >>> at > > >> > >>> > > >> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) > > >> > >>> at > > >> > >>> > > >> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > > >> > >>> at > > >> > >>> > > okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) > > >> > >>> at okhttp3.RealCall.execute(RealCall.java:77) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839) > > >> > >>> at > > >> > >>> > > >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > >> > >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > >> > >>> at java.lang.Thread.run(Thread.java:748) > > >> > >>> Caused by: sun.security.validator.ValidatorException: PKIX path > > >> > building > > >> > >>> failed: > > sun.security.provider.certpath.SunCertPathBuilderException: > > >> > unable > > >> > >>> to find valid certification path to requested target > > >> > >>> at > > >> > >>> > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) > > >> > >>> at sun.security.validator.Validator.validate(Validator.java:262) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) > > >> > >>> ... 35 common frames omitted > > >> > >>> Caused by: > > >> sun.security.provider.certpath.SunCertPathBuilderException: > > >> > >>> unable to find valid certification path to requested target > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) > > >> > >>> at > > >> > >>> > > >> > > > >> > > > sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) > > >> > >>> at > > >> > >>> > java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) > > >> > >>> at > > >> > >>> > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) > > >> > >>> ... 41 common frames omitted > > >> > >>> Thanks, > > >> > >>> Joe > > >> > > > > > > > > -- > Be on your guard; stand firm in the faith; be courageous; be strong. Do > everything in love. -*1 Corinthians 16:13-14* >
