Joe, Can you share the keystores and truststores you are using? I understand the issue you’re encountering but we haven’t yet been able to reproduce it locally running with certs that work on 1.11.1. Please DO NOT share actual keystores if they contain real private keys, only if these are dev instances you are comfortable sharing.
If you cannot share these, can you please try generating a new set using the TLS Toolkit and verify that they also fail on 1.11.3 in your environment? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 26, 2020, at 11:07 AM, Joe Gresock <jgres...@gmail.com> wrote: > > Ok, I added all the server certs and my administrator's client cert to the > trust store, and they all still got PKIX path building failed. So I > redeployed nifi 1.11.1, and now it works again. > > Joe > > On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock <jgres...@gmail.com> wrote: > >> Yes, on Kubernetes. >> >> FWIW, I do see changes to >> nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java >> involving a new function createTrustSslContextWithTrustManagers(), among >> other related changes. >> >> I'm going to try directly adding the client certs to my trust store to see >> if that works (instead of just adding the client certs' issuer). >> >> On Wed, Feb 26, 2020 at 6:08 PM Joe Witt <joe.w...@gmail.com> wrote: >> >>> on kubernetes is a key detail here too... >>> >>> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock <jgres...@gmail.com> wrote: >>> >>>> Java 8 >>>> >>>> On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough <thena...@gmail.com> >>> wrote: >>>> >>>>> Hi Joe, >>>>> >>>>> I just set up a secure cluster with NiFi 1.11.3 and am not seeing any >>>>> issues like you describe. >>>>> >>>>> Are you running Java 8 or Java 11? >>>>> >>>>> Nathan >>>>> >>>>> On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock <jgres...@gmail.com> >>> wrote: >>>>> >>>>>> Were there any changes with how the trust store is used in 1.11.3? >>> I >>>>> had a >>>>>> 1.11.0 deployment working with the following settings, but when I >>>>> deployed >>>>>> 1.11.3, the cluster can't seem to replicate requests to itself: >>>>>> >>>>>> nifi.remote.input.host=<redacted> >>>>>> nifi.remote.input.secure=true >>>>>> nifi.remote.input.socket.port=32440 >>>>>> nifi.remote.input.http.enabled=true >>>>>> >>>>>> nifi.cluster.protocol.is.secure=true >>>>>> nifi.cluster.is.node=true >>>>>> >>>>>> >>>>> >>>> >>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>>>>> nifi.cluster.node.protocol.port=6007 >>>>>> >>>>>> >>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>>>>> nifi.web.https.port=8443 >>>>>> >>>>>> nifi.security.keystore=./conf/keystore.jks >>>>>> nifi.security.keystoreType=jks >>>>>> nifi.security.keystorePasswd=<password> >>>>>> nifi.security.keyPasswd= >>>>>> nifi.security.truststore=./conf/truststore.jks >>>>>> nifi.security.truststoreType=jks >>>>>> nifi.security.truststorePasswd=<password> >>>>>> nifi.security.needClientAuth=true >>>>>> >>>>>> A trusted client cert that worked against the old cluster is getting >>>> the >>>>>> same trust error (PKIX path building failed). I've verified that >>> the >>>>>> client cert was issued by an issuer that is definitely in the >>>>>> ./conf/truststore.jks as a trustedCertEntry. >>>>>> >>>>>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] >>>>>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator >>>>>> javax.net.ssl.SSLHandshakeException: >>>>>> sun.security.validator.ValidatorException: PKIX path building >>> failed: >>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable >>> to >>>>> find >>>>>> valid certification path to r >>>>>> equested target >>>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >>>>>> at >>>> sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) >>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) >>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) >>>>>> at >>>>>> >>>>> >>>> >>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) >>>>>> at >>>> sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) >>>>>> at >>>>> sun.security.ssl.Handshaker.process_record(Handshaker.java:965) >>>>>> at >>>>>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) >>>>>> at >>>>>> >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) >>>>>> at >>>>>> >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) >>>>>> at >>>>>> >>>>> >>>> >>> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >>>>>> at >>>>>> >>>>> >>>> >>> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >>>>>> at >>>>>> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) >>>>>> at okhttp3.RealCall.execute(RealCall.java:77) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839) >>>>>> at >>>>>> >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>> Caused by: sun.security.validator.ValidatorException: PKIX path >>>> building >>>>>> failed: sun.security.provider.certpath.SunCertPathBuilderException: >>>>> unable >>>>>> to find valid certification path to requested target >>>>>> at >>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >>>>>> at >>>>>> >>>>> >>>> >>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) >>>>>> at >>>> sun.security.validator.Validator.validate(Validator.java:262) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) >>>>>> ... 35 common frames omitted >>>>>> Caused by: >>> sun.security.provider.certpath.SunCertPathBuilderException: >>>>>> unable to find valid certification path to requested target >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) >>>>>> at >>>>>> >>>>>> >>>>> >>>> >>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) >>>>>> at >>>>>> java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) >>>>>> at >>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) >>>>>> ... 41 common frames omitted >>>>>> >>>>>> Thanks, >>>>>> Joe >>>>>> >>>>> >>>> >>> >>
