You can post them on a temporary file sharing service, post them in the Apache NiFi Slack [1], or email them to me directly at alopre...@apache.org <mailto:alopre...@apache.org>. The mailing list software tends to strip attachments.
[1] https://apachenifi.slack.com <https://apachenifi.slack.com/> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 26, 2020, at 12:21 PM, Joe Gresock <jgres...@gmail.com> wrote: > > Good question -- I can't share these keystores and truststores, but I'll > see if I can generate some test ones tomorrow. How should I send them to > you? > > On Wed, Feb 26, 2020 at 8:01 PM Andy LoPresto <alopre...@apache.org> wrote: > >> Joe, >> >> Can you share the keystores and truststores you are using? I understand >> the issue you’re encountering but we haven’t yet been able to reproduce it >> locally running with certs that work on 1.11.1. Please DO NOT share actual >> keystores if they contain real private keys, only if these are dev >> instances you are comfortable sharing. >> >> If you cannot share these, can you please try generating a new set using >> the TLS Toolkit and verify that they also fail on 1.11.3 in your >> environment? >> >> >> Andy LoPresto >> alopre...@apache.org >> alopresto.apa...@gmail.com >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Feb 26, 2020, at 11:07 AM, Joe Gresock <jgres...@gmail.com> wrote: >>> >>> Ok, I added all the server certs and my administrator's client cert to >> the >>> trust store, and they all still got PKIX path building failed. So I >>> redeployed nifi 1.11.1, and now it works again. >>> >>> Joe >>> >>> On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock <jgres...@gmail.com> wrote: >>> >>>> Yes, on Kubernetes. >>>> >>>> FWIW, I do see changes to >>>> >> nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java >>>> involving a new function createTrustSslContextWithTrustManagers(), among >>>> other related changes. >>>> >>>> I'm going to try directly adding the client certs to my trust store to >> see >>>> if that works (instead of just adding the client certs' issuer). >>>> >>>> On Wed, Feb 26, 2020 at 6:08 PM Joe Witt <joe.w...@gmail.com> wrote: >>>> >>>>> on kubernetes is a key detail here too... >>>>> >>>>> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock <jgres...@gmail.com> >> wrote: >>>>> >>>>>> Java 8 >>>>>> >>>>>> On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough <thena...@gmail.com> >>>>> wrote: >>>>>> >>>>>>> Hi Joe, >>>>>>> >>>>>>> I just set up a secure cluster with NiFi 1.11.3 and am not seeing any >>>>>>> issues like you describe. >>>>>>> >>>>>>> Are you running Java 8 or Java 11? >>>>>>> >>>>>>> Nathan >>>>>>> >>>>>>> On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock <jgres...@gmail.com> >>>>> wrote: >>>>>>> >>>>>>>> Were there any changes with how the trust store is used in 1.11.3? >>>>> I >>>>>>> had a >>>>>>>> 1.11.0 deployment working with the following settings, but when I >>>>>>> deployed >>>>>>>> 1.11.3, the cluster can't seem to replicate requests to itself: >>>>>>>> >>>>>>>> nifi.remote.input.host=<redacted> >>>>>>>> nifi.remote.input.secure=true >>>>>>>> nifi.remote.input.socket.port=32440 >>>>>>>> nifi.remote.input.http.enabled=true >>>>>>>> >>>>>>>> nifi.cluster.protocol.is.secure=true >>>>>>>> nifi.cluster.is.node=true >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>>>>>>> nifi.cluster.node.protocol.port=6007 >>>>>>>> >>>>>>>> >>>>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>>>>>>> nifi.web.https.port=8443 >>>>>>>> >>>>>>>> nifi.security.keystore=./conf/keystore.jks >>>>>>>> nifi.security.keystoreType=jks >>>>>>>> nifi.security.keystorePasswd=<password> >>>>>>>> nifi.security.keyPasswd= >>>>>>>> nifi.security.truststore=./conf/truststore.jks >>>>>>>> nifi.security.truststoreType=jks >>>>>>>> nifi.security.truststorePasswd=<password> >>>>>>>> nifi.security.needClientAuth=true >>>>>>>> >>>>>>>> A trusted client cert that worked against the old cluster is getting >>>>>> the >>>>>>>> same trust error (PKIX path building failed). I've verified that >>>>> the >>>>>>>> client cert was issued by an issuer that is definitely in the >>>>>>>> ./conf/truststore.jks as a trustedCertEntry. >>>>>>>> >>>>>>>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] >>>>>>>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator >>>>>>>> javax.net.ssl.SSLHandshakeException: >>>>>>>> sun.security.validator.ValidatorException: PKIX path building >>>>> failed: >>>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable >>>>> to >>>>>>> find >>>>>>>> valid certification path to r >>>>>>>> equested target >>>>>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >>>>>>>> at >>>>>> sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) >>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) >>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) >>>>>>>> at >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) >>>>>>>> at >>>>>> sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) >>>>>>>> at >>>>>>> sun.security.ssl.Handshaker.process_record(Handshaker.java:965) >>>>>>>> at >>>>>>>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) >>>>>>>> at >>>>>>>> >>>>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) >>>>>>>> at >>>>>>>> >>>>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) >>>>>>>> at >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >>>>>>>> at >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) >>>>>>>> at >>>>>>>> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) >>>>>>>> at okhttp3.RealCall.execute(RealCall.java:77) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:143) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:137) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839) >>>>>>>> at >>>>>>>> >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>>>> Caused by: sun.security.validator.ValidatorException: PKIX path >>>>>> building >>>>>>>> failed: sun.security.provider.certpath.SunCertPathBuilderException: >>>>>>> unable >>>>>>>> to find valid certification path to requested target >>>>>>>> at >>>>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >>>>>>>> at >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) >>>>>>>> at >>>>>> sun.security.validator.Validator.validate(Validator.java:262) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) >>>>>>>> ... 35 common frames omitted >>>>>>>> Caused by: >>>>> sun.security.provider.certpath.SunCertPathBuilderException: >>>>>>>> unable to find valid certification path to requested target >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) >>>>>>>> at >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) >>>>>>>> at >>>>>>>> java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) >>>>>>>> at >>>>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) >>>>>>>> ... 41 common frames omitted >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Joe >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> >>
