Hi, One additional thing:
we encountered something strange as well: on 1.11.2 clustered, kerberized: request replication worked well. on 1.11.3 clustered, kerberized: request replication did not work, unless you specify, and set nifi.security.keyPasswd to the very same password as the nifi.security.keystorePasswd For us this resolved the issue. Best regards, Endre Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 3, 2020 12:40 PM, Ryan Ward <ryan.wa...@gmail.com> wrote: > Hi Joe - Did you resolve your issue? If so I am wondering what the fix was as > I'm seeing the same error on my cluster. > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs > <andrewsmit...@protonmail.com.invalid> wrote: > >> Hi Joe, >> >> 1. Have you tried connecting/debugging with openssl? From one pod to the >> other: >> (openssl s_client -debug -CAfile ca-bundle-signing-node-certificates.crt >> -cert my-client-cert.crt -connect >> nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007) >> >> 2. certs can also be verified by: >> openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt >> >> 3. Can you check if no intermediary CAs are missing from the nodes >> truststore? >> >> 4. This exception is coming from inter-node communication (replication of >> request from one node to the other). This means that it is unrelated to >> external user's authentication by client certificate. The question is: is >> your inter node communication secured by the trusted root CA (that you are >> sure that the CA cert is present in the trust store) or is it secured by >> selfsigned CA (which's CA may be lacking from your truststore)? >> >> 5. `nifi.security.needClientAuth` is not part of NiFi properties any more. >> If SSL is turned on, and no `nifi.security.user.login.identity.provider` is >> set, then client cert based auth is the default. But supplying this property >> have no detrimental effect anyhow. >> >> Best regards, >> Endre >> >> Sent with ProtonMail Secure Email. >> >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock jgres...@gmail.com >> wrote: >> >>> Were there any changes with how the trust store is used in 1.11.3? I had a >>> 1.11.0 deployment working with the following settings, but when I deployed >>> 1.11.3, the cluster can't seem to replicate requests to itself: >>> nifi.remote.input.host=<redacted> >>> nifi.remote.input.secure=true >>> nifi.remote.input.socket.port=32440 >>> nifi.remote.input.http.enabled=true >>> nifi.cluster.protocol.is.secure=true >>> nifi.cluster.is.node=true >>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>> nifi.cluster.node.protocol.port=6007 >>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>> nifi.web.https.port=8443 >>> nifi.security.keystore=./conf/keystore.jks >>> nifi.security.keystoreType=jks >>> nifi.security.keystorePasswd=<password> >>> nifi.security.keyPasswd= >>> nifi.security.truststore=./conf/truststore.jks >>> nifi.security.truststoreType=jks >>> nifi.security.truststorePasswd=<password> >>> nifi.security.needClientAuth=true >>> A trusted client cert that worked against the old cluster is getting the >>> same trust error (PKIX path building failed). I've verified that the >>> client cert was issued by an issuer that is definitely in the >>> ./conf/truststore.jks as a trustedCertEntry. >>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator >>> javax.net.ssl.SSLHandshakeException: >>> sun.security.validator.ValidatorException: PKIX path building failed: >>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find >>> valid certification path to r >>> equested target >>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) >>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) >>> at >>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) >>> at >>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) >>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) >>> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) >>> at >>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) >>> at >>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) >>> at >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) >>> at >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) >>> at >>> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) >>> at >>> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) >>> at >>> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) >>> at >>> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) >>> at >>> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) >>> at >>> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) >>> at >>> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >>> at >>> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >>> at >>> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>> at >>> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>> at >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >>> at >>> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) >>> at okhttp3.RealCall.execute(RealCall.java:77) >>> at >>> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143) >>> at >>> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137) >>> at >>> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647) >>> at >>> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: sun.security.validator.ValidatorException: PKIX path building >>> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable >>> to find valid certification path to requested target >>> at >>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >>> at >>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) >>> at sun.security.validator.Validator.validate(Validator.java:262) >>> at >>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) >>> at >>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) >>> at >>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) >>> at >>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) >>> ... 35 common frames omitted >>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: >>> unable to find valid certification path to requested target >>> at >>> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) >>> at >>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) >>> at >>> java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) >>> at >>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) >>> ... 41 common frames omitted >>> Thanks, >>> Joe
