Good question -- I can't share these keystores and truststores, but I'll see if I can generate some test ones tomorrow. How should I send them to you?
On Wed, Feb 26, 2020 at 8:01 PM Andy LoPresto <alopre...@apache.org> wrote: > Joe, > > Can you share the keystores and truststores you are using? I understand > the issue you’re encountering but we haven’t yet been able to reproduce it > locally running with certs that work on 1.11.1. Please DO NOT share actual > keystores if they contain real private keys, only if these are dev > instances you are comfortable sharing. > > If you cannot share these, can you please try generating a new set using > the TLS Toolkit and verify that they also fail on 1.11.3 in your > environment? > > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Feb 26, 2020, at 11:07 AM, Joe Gresock <jgres...@gmail.com> wrote: > > > > Ok, I added all the server certs and my administrator's client cert to > the > > trust store, and they all still got PKIX path building failed. So I > > redeployed nifi 1.11.1, and now it works again. > > > > Joe > > > > On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock <jgres...@gmail.com> wrote: > > > >> Yes, on Kubernetes. > >> > >> FWIW, I do see changes to > >> > nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java > >> involving a new function createTrustSslContextWithTrustManagers(), among > >> other related changes. > >> > >> I'm going to try directly adding the client certs to my trust store to > see > >> if that works (instead of just adding the client certs' issuer). > >> > >> On Wed, Feb 26, 2020 at 6:08 PM Joe Witt <joe.w...@gmail.com> wrote: > >> > >>> on kubernetes is a key detail here too... > >>> > >>> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock <jgres...@gmail.com> > wrote: > >>> > >>>> Java 8 > >>>> > >>>> On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough <thena...@gmail.com> > >>> wrote: > >>>> > >>>>> Hi Joe, > >>>>> > >>>>> I just set up a secure cluster with NiFi 1.11.3 and am not seeing any > >>>>> issues like you describe. > >>>>> > >>>>> Are you running Java 8 or Java 11? > >>>>> > >>>>> Nathan > >>>>> > >>>>> On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock <jgres...@gmail.com> > >>> wrote: > >>>>> > >>>>>> Were there any changes with how the trust store is used in 1.11.3? > >>> I > >>>>> had a > >>>>>> 1.11.0 deployment working with the following settings, but when I > >>>>> deployed > >>>>>> 1.11.3, the cluster can't seem to replicate requests to itself: > >>>>>> > >>>>>> nifi.remote.input.host=<redacted> > >>>>>> nifi.remote.input.secure=true > >>>>>> nifi.remote.input.socket.port=32440 > >>>>>> nifi.remote.input.http.enabled=true > >>>>>> > >>>>>> nifi.cluster.protocol.is.secure=true > >>>>>> nifi.cluster.is.node=true > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local > >>>>>> nifi.cluster.node.protocol.port=6007 > >>>>>> > >>>>>> > >>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local > >>>>>> nifi.web.https.port=8443 > >>>>>> > >>>>>> nifi.security.keystore=./conf/keystore.jks > >>>>>> nifi.security.keystoreType=jks > >>>>>> nifi.security.keystorePasswd=<password> > >>>>>> nifi.security.keyPasswd= > >>>>>> nifi.security.truststore=./conf/truststore.jks > >>>>>> nifi.security.truststoreType=jks > >>>>>> nifi.security.truststorePasswd=<password> > >>>>>> nifi.security.needClientAuth=true > >>>>>> > >>>>>> A trusted client cert that worked against the old cluster is getting > >>>> the > >>>>>> same trust error (PKIX path building failed). I've verified that > >>> the > >>>>>> client cert was issued by an issuer that is definitely in the > >>>>>> ./conf/truststore.jks as a trustedCertEntry. > >>>>>> > >>>>>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] > >>>>>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator > >>>>>> javax.net.ssl.SSLHandshakeException: > >>>>>> sun.security.validator.ValidatorException: PKIX path building > >>> failed: > >>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable > >>> to > >>>>> find > >>>>>> valid certification path to r > >>>>>> equested target > >>>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > >>>>>> at > >>>> sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) > >>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) > >>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) > >>>>>> at > >>>>>> > >>>>> > >>>> > >>> > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) > >>>>>> at > >>>> sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) > >>>>>> at > >>>>> sun.security.ssl.Handshaker.process_record(Handshaker.java:965) > >>>>>> at > >>>>>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) > >>>>>> at > >>>>>> > >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) > >>>>>> at > >>>>>> > >>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) > >>>>>> at > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > >>>>>> at > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) > >>>>>> at > >>>>>> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) > >>>>>> at okhttp3.RealCall.execute(RealCall.java:77) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839) > >>>>>> at > >>>>>> > >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > >>>>>> at java.lang.Thread.run(Thread.java:748) > >>>>>> Caused by: sun.security.validator.ValidatorException: PKIX path > >>>> building > >>>>>> failed: sun.security.provider.certpath.SunCertPathBuilderException: > >>>>> unable > >>>>>> to find valid certification path to requested target > >>>>>> at > >>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) > >>>>>> at > >>>>>> > >>>>> > >>>> > >>> > sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) > >>>>>> at > >>>> sun.security.validator.Validator.validate(Validator.java:262) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) > >>>>>> ... 35 common frames omitted > >>>>>> Caused by: > >>> sun.security.provider.certpath.SunCertPathBuilderException: > >>>>>> unable to find valid certification path to requested target > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) > >>>>>> at > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) > >>>>>> at > >>>>>> java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) > >>>>>> at > >>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) > >>>>>> ... 41 common frames omitted > >>>>>> > >>>>>> Thanks, > >>>>>> Joe > >>>>>> > >>>>> > >>>> > >>> > >> > >
