Mike, thanks for putting together that PR. I have built everything successfully but I haven't been able to test this yet since I haven't built the new Docker image.
I assume you guys just use the ‘dockermaven’ folder to build the Dockerfile with src artifacts? My current dev machine is running out of disk space so I am spending some time trying to get it back to a runnable state. Hopefully I can build the PR and test it in Kubernetes this week sometime. Thanks, Peter On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen <mikerthom...@gmail.com>, wrote: > As a compromise, I upgraded to the latest 5.X client and manually incremented > Apache HttpClient to 4.5.10. PR is here: > > https://github.com/apache/nifi/pull/3828 > > There are integration tests for that package that automatically startup and > provision an ES node, and they all passed with this configuration. Hopefully > this PR will fix your issue since it appears to be an external dependency > causing it. > > > On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen <mikerthom...@gmail.com> wrote: > > > There's no hard and fast reason to stay with 5.X there, so you can build > > > your own copy of 1.9.2 with that dependency upgraded if you want to try > > > it out. I'll try to find time to test that change on 1.10.0-SNAPSHOT. > > > > > > > On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg <peter.mob...@gmail.com> > > > > wrote: > > > > > The certs in the TrustStore are marked as trusted. > > > > > > > > > > The host name specified in the ClientServiceImpl is: > > > > > https://quickstart-es-http.es-cluster:9200 > > > > > > > > > > The CN field of the server certificate is: > > > > > https://quickstart-es-http.es-cluster.es.local > > > > > > > > > > So at first glance it looks like the issue would be that the CN field > > > > > differs from the host name. However, Im not getting an error message > > > > > saying that the HostName doesn’t match the CN. According to the RFC > > > > > spec first mentioned by Andy the CN field should only be used if the > > > > > SubjectiveAlternativeName is empty. > > > > > > > > > > The Server Cert has the SAN set to: quickstart-es-http.es-cluster > > > > > > > > > > What is really strange to me is why I can connect using the > > > > > QueryElasticSearch processor with the same SSLContextService and > > > > > hostname but when I use the ClientServicesImpl it will not work. > > > > > > > > > > So I did some more investigation and it looks like the QueryES > > > > > processor uses the okHTTP library and ClientServicesImpl uses the > > > > > Elastic Search RestClient which in turn uses the HTTPClient from > > > > > Apache. > > > > > > > > > > There was an issue with the HTTPClient library in v4.5.2 > > > > > (https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it > > > > > didn’t do the right hostname verification. > > > > > > > > > > The version of Apache Nifi that Im using is 1.9.2 and it seems to use > > > > > v.5.6.8 of Elastic Search client libraries and that in turn uses > > > > > v4.5.2 of Apache HTTPClient. > > > > > > > > > > https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties > > > > > > > > > > But again if it was a HostName validation issue the error message > > > > > should read more like: “Certificate for <host> doesn’t match common > > > > > name of certificate subject…” > > > > > > > > > > On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen > > > > > <mikerthom...@gmail.com>, wrote: > > > > > > I'm far from a SSL/TLS expert, but let's get these out of the way: > > > > > > > > > > > > 1. Did you mark the server's cert as "trusted" when you created the > > > > > > trust store with keytool? > > > > > > 2. Are you sure that you're specifying the same hostname in the > > > > > > client service that is in the CN field in the server's cert? > > > > > > > > > > > > FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you > > > > > > can use it to generate valid certificates. It's meant for NiFi, but > > > > > > the certs it generates should be a solid step up from self-signed > > > > > > certs if you're not able to access a company CA. > > > > > > > > > > > > > On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg > > > > > > > <peter.mob...@gmail.com> wrote: > > > > > > > > Nope. No good. I even dump the network traffic and analyzed it > > > > > > > > in Wireshark. The ES server sends back two certificates (server > > > > > > > > + self-signed one) and both of them are present in my > > > > > > > > TrustStore. I am specifying both a TrustStore and a Keystore > > > > > > > > now but it still gives the error that it can’t find the > > > > > > > > certificate. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Peter > > > > > > > > On Oct 18, 2019, 4:44 PM -0500, Peter Moberg > > > > > > > > <peter.mob...@gmail.com>, wrote: > > > > > > > > > Think I might have found the issue. Will report tonight. > > > > > > > > > > > > > > > > > > Mike, please don’t spend any time debugging this because I > > > > > > > > > think it might be an issue on my side. Appreciate all the > > > > > > > > > help so far. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Peter > > > > > > > > > On Oct 18, 2019, 2:21 PM -0500, Peter Moberg > > > > > > > > > <peter.mob...@gmail.com>, wrote: > > > > > > > > > > Here it is: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process > > > > > > > > > > Thread-7] o.a.n.processors.standard.LookupRecord > > > > > > > > > > LookupRecord[id=df596687-016d-1000-0000-000065536eb2] > > > > > > > > > > Failed to process > > > > > > > > > > StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim > > > > > > > > > > [resourceClaim=StandardResourceClaim[id=1571410822200-1, > > > > > > > > > > container=default, section=1], offset=7180, > > > > > > > > > > length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]: > > > > > > > > > > org.apache.nifi.processor.exception.ProcessException: > > > > > > > > > > Failed to lookup coordinates {key=test} in Lookup Service > > > > > > > > > > org.apache.nifi.processor.exception.ProcessException: > > > > > > > > > > Failed to lookup coordinates {key=test} in Lookup Service > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > > > > > > > > > > at > > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > > > > > > > at > > > > > > > > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > > > > > > > > > at > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > > > > > > > > > at > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > > > > > > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > > > > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > > > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > > > > > > Caused by: org.apache.nifi.lookup.LookupFailureException: > > > > > > > > > > org.apache.nifi.lookup.LookupFailureException: > > > > > > > > > > javax.net.ssl.SSLHandshakeException: General SSLEngine > > > > > > > > > > problem > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175) > > > > > > > > > > at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown > > > > > > > > > > Source) > > > > > > > > > > at > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > > > > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87) > > > > > > > > > > at com.sun.proxy.$Proxy136.lookup(Unknown Source) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298) > > > > > > > > > > ... 17 common frames omitted > > > > > > > > > > Caused by: org.apache.nifi.lookup.LookupFailureException: > > > > > > > > > > javax.net.ssl.SSLHandshakeException: General SSLEngine > > > > > > > > > > problem > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288) > > > > > > > > > > at > > > > > > > > > > org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169) > > > > > > > > > > ... 23 common frames omitted > > > > > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General > > > > > > > > > > SSLEngine problem > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) > > > > > > > > > > at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) > > > > > > > > > > at > > > > > > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265) > > > > > > > > > > at > > > > > > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305) > > > > > > > > > > at > > > > > > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509) > > > > > > > > > > at > > > > > > > > > > org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120) > > > > > > > > > > at > > > > > > > > > > org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) > > > > > > > > > > at > > > > > > > > > > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) > > > > > > > > > > at > > > > > > > > > > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) > > > > > > > > > > at > > > > > > > > > > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) > > > > > > > > > > at > > > > > > > > > > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) > > > > > > > > > > at > > > > > > > > > > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) > > > > > > > > > > ... 1 common frames omitted > > > > > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General > > > > > > > > > > SSLEngine problem > > > > > > > > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) > > > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) > > > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) > > > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) > > > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) > > > > > > > > > > at java.security.AccessController.doPrivileged(Native > > > > > > > > > > Method) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) > > > > > > > > > > at > > > > > > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283) > > > > > > > > > > at > > > > > > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353) > > > > > > > > > > ... 9 common frames omitted > > > > > > > > > > Caused by: sun.security.validator.ValidatorException: PKIX > > > > > > > > > > path building failed: > > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilderException: > > > > > > > > > > unable to find valid certification path to requested target > > > > > > > > > > at > > > > > > > > > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) > > > > > > > > > > at > > > > > > > > > > sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) > > > > > > > > > > at > > > > > > > > > > sun.security.validator.Validator.validate(Validator.java:262) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) > > > > > > > > > > at > > > > > > > > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620) > > > > > > > > > > ... 17 common frames omitted > > > > > > > > > > Caused by: > > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilderException: > > > > > > > > > > unable to find valid certification path to requested target > > > > > > > > > > at > > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) > > > > > > > > > > at > > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) > > > > > > > > > > at > > > > > > > > > > java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) > > > > > > > > > > at > > > > > > > > > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) > > > > > > > > > > ... 23 common frames omitted > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Peter > > > > > > > > > > On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen > > > > > > > > > > <mikerthom...@gmail.com>, wrote: > > > > > > > > > > > Can you share the stacktrace from the logs? > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg > > > > > > > > > > > > <peter.mob...@gmail.com> wrote: > > > > > > > > > > > > > Mike, > > > > > > > > > > > > > > > > > > > > > > > > > > The SSLContextService only had the Trust store > > > > > > > > > > > > > configured. I think I seen that ticket before but > > > > > > > > > > > > > didn’t pay attention to the fact it wasn’t merged in > > > > > > > > > > > > > to the code I am running. > > > > > > > > > > > > > > > > > > > > > > > > > > However, I configured the service to have a KeyStore > > > > > > > > > > > > > now but I am getting the same errors… > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Peter > > > > > > > > > > > > > On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen > > > > > > > > > > > > > <mikerthom...@gmail.com>, wrote: > > > > > > > > > > > > > > Peter, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Are you configuring the service as a trust-only > > > > > > > > > > > > > > configuration? If so, that's been addressed in the > > > > > > > > > > > > > > 1.10 which is due for release in the near(ish) > > > > > > > > > > > > > > future. > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/NIFI-6228 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg > > > > > > > > > > > > > > > <peter.mob...@gmail.com> wrote: > > > > > > > > > > > > > > > > As a follow-up. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On the Nifi node I am able to do a GET to > > > > > > > > > > > > > > > > Elastic Search using curl. I specify the > > > > > > > > > > > > > > > > —cacert option giving it the self-signed root > > > > > > > > > > > > > > > > certificate. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Of course, this isn’t using the TrustStore but > > > > > > > > > > > > > > > > I am able to use the TrustStore if I use other > > > > > > > > > > > > > > > > ES processors… just not the > > > > > > > > > > > > > > > > ElasticSearchClientServicesImpl. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Oct 18, 2019, 12:48 AM -0500, Peter Moberg > > > > > > > > > > > > > > > > <peter.mob...@gmail.com>, wrote: > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks for your suggestions. Here is what I > > > > > > > > > > > > > > > > > have tried so far (still no luck). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Connecting with openssl and viewing the certs > > > > > > > > > > > > > > > > > it presents > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > openssl s_client -connect > > > > > > > > > > > > > > > > > quickstart-es-http.es-cluster -showcerts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If I then look inside the server cert I can > > > > > > > > > > > > > > > > > find this > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Server Cert: > > > > > > > > > > > > > > > > > Issuer: OU = quickstart, CN = quickstart-http > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > X509v3 Subject Alternative Name: > > > > > > > > > > > > > > > > > DNS:quickstart-es-http.es-cluster.es.local, > > > > > > > > > > > > > > > > > DNS:quickstart-es-http, > > > > > > > > > > > > > > > > > DNS:quickstart-es-http.es-cluster.svc, > > > > > > > > > > > > > > > > > DNS:quickstart-es-http.es-cluster > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If I look in to the self-signed root cert I > > > > > > > > > > > > > > > > > find this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Root Cert: > > > > > > > > > > > > > > > > > Subject: OU = quickstart, CN = quickstart-http > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I now double check my trust store to make > > > > > > > > > > > > > > > > > sure the Root Cert is there. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Trust store content > > > > > > > > > > > > > > > > > Your keystore contains 1 entry > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Alias name: ca_elastic > > > > > > > > > > > > > > > > > Creation date: Oct 16, 2019 > > > > > > > > > > > > > > > > > Entry type: trustedCertEntry > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Owner: CN=quickstart-http, OU=quickstart > > > > > > > > > > > > > > > > > Issuer: CN=quickstart-http, OU=quickstart > > > > > > > > > > > > > > > > > Serial number: > > > > > > > > > > > > > > > > > 5aa50b6806d2394fff6f98d2b7d4c287 > > > > > > > > > > > > > > > > > Valid from: Fri Oct 11 14:35:01 UTC 2019 > > > > > > > > > > > > > > > > > until: Sat Oct 10 14:36:01 UTC 2020 > > > > > > > > > > > > > > > > > Certificate fingerprints: > > > > > > > > > > > > > > > > > MD5: > > > > > > > > > > > > > > > > > 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 > > > > > > > > > > > > > > > > > SHA1: > > > > > > > > > > > > > > > > > 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 > > > > > > > > > > > > > > > > > SHA256: > > > > > > > > > > > > > > > > > B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF > > > > > > > > > > > > > > > > > Signature algorithm name: SHA256withRSA > > > > > > > > > > > > > > > > > Subject Public Key Algorithm: 2048-bit RSA key > > > > > > > > > > > > > > > > > Version: 3 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Extensions: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > #1: ObjectId: 2.5.29.19 Criticality=true > > > > > > > > > > > > > > > > > BasicConstraints:[ > > > > > > > > > > > > > > > > > CA:true > > > > > > > > > > > > > > > > > PathLen:2147483647 > > > > > > > > > > > > > > > > > ] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > #2: ObjectId: 2.5.29.37 Criticality=false > > > > > > > > > > > > > > > > > ExtendedKeyUsages [ > > > > > > > > > > > > > > > > > serverAuth > > > > > > > > > > > > > > > > > clientAuth > > > > > > > > > > > > > > > > > ] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > #3: ObjectId: 2.5.29.15 Criticality=true > > > > > > > > > > > > > > > > > KeyUsage [ > > > > > > > > > > > > > > > > > DigitalSignature > > > > > > > > > > > > > > > > > Key_CertSign > > > > > > > > > > > > > > > > > ] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So everything looks Ok. But when I run the > > > > > > > > > > > > > > > > > ElasticSearchClientServicesImpl with a > > > > > > > > > > > > > > > > > SSLContext pointing to my trust store I still > > > > > > > > > > > > > > > > > get the following output in the log. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Caused by: > > > > > > > > > > > > > > > > > javax.net.ssl.SSLHandshakeException: General > > > > > > > > > > > > > > > > > SSLEngine problem > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.Handshaker$1.run(Handshaker.java:992) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.Handshaker$1.run(Handshaker.java:989) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > java.security.AccessController.doPrivileged(Native > > > > > > > > > > > > > > > > > Method) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353) > > > > > > > > > > > > > > > > > ... 9 common frames omitted > > > > > > > > > > > > > > > > > Caused by: > > > > > > > > > > > > > > > > > sun.security.validator.ValidatorException: > > > > > > > > > > > > > > > > > PKIX path building failed: > > > > > > > > > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilderException: > > > > > > > > > > > > > > > > > unable to find valid certification path to > > > > > > > > > > > > > > > > > requested target > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.validator.Validator.validate(Validator.java:262) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) > > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620) > > > > > > > > > > > > > > > > > ... 17 common frames omitted > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Both the Nifi install and Elastic Search > > > > > > > > > > > > > > > > > install is running in Kubernetes. The address > > > > > > > > > > > > > > > > > I am using is a service address that is > > > > > > > > > > > > > > > > > backed by 3 ES instances. However, I double > > > > > > > > > > > > > > > > > checked all three of the ES nodes to make > > > > > > > > > > > > > > > > > sure that they returned back the same SSL > > > > > > > > > > > > > > > > > cert and they did. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The only thing I haven't been able to figure > > > > > > > > > > > > > > > > > out is how to check if Kubernetes/ES reacts > > > > > > > > > > > > > > > > > differently when you do a GET vs POST. Feels > > > > > > > > > > > > > > > > > strange that it would return different SSL > > > > > > > > > > > > > > > > > certs but stranger things have happened… > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto > > > > > > > > > > > > > > > > > <alopre...@apache.org>, wrote: > > > > > > > > > > > > > > > > > > Hi Peter, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If you can use openssl’s s_client command > > > > > > > > > > > > > > > > > > (example below) to connect to the endpoint > > > > > > > > > > > > > > > > > > and verify that the hostname matches the > > > > > > > > > > > > > > > > > > certificate and that the certificate > > > > > > > > > > > > > > > > > > contains a SubjectAlternativeName entry > > > > > > > > > > > > > > > > > > with that hostname (see RFC 6125 [1] for > > > > > > > > > > > > > > > > > > more details), this should help you debug > > > > > > > > > > > > > > > > > > the issue. The cause of the PKIX error is > > > > > > > > > > > > > > > > > > that the truststore doesn’t contain a > > > > > > > > > > > > > > > > > > certificate (or certificate chain) which > > > > > > > > > > > > > > > > > > matches the hostname presented by the > > > > > > > > > > > > > > > > > > remote endpoint. I think you understand > > > > > > > > > > > > > > > > > > that based on your message. The underlying > > > > > > > > > > > > > > > > > > reason for this is could be one of the > > > > > > > > > > > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * the server is behind an interface which > > > > > > > > > > > > > > > > > > responds differently to GET and POST/PUT > > > > > > > > > > > > > > > > > > requests > > > > > > > > > > > > > > > > > > * there is a load-balancer which is > > > > > > > > > > > > > > > > > > directing the requests coincidentally to > > > > > > > > > > > > > > > > > > different backend servers (one has the > > > > > > > > > > > > > > > > > > right cert; the other doesn’t) > > > > > > > > > > > > > > > > > > * I recall something around the addition of > > > > > > > > > > > > > > > > > > (some) Elastic Search components which > > > > > > > > > > > > > > > > > > handled TLS in an ES client-specific > > > > > > > > > > > > > > > > > > manner; I remember advocating for standard > > > > > > > > > > > > > > > > > > NiFi TLS interaction here but I am not sure > > > > > > > > > > > > > > > > > > what was ultimately contributed. If it’s > > > > > > > > > > > > > > > > > > not one of the above issues, I can > > > > > > > > > > > > > > > > > > investigate further. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hopefully this helps. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://tools.ietf.org/html/rfc6125#section-6.4.4 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > s_client example: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > $ openssl s_client -connect <host:port> > > > > > > > > > > > > > > > > > > -debug -state -cert <path_to_your_cert.pem> > > > > > > > > > > > > > > > > > > -key <path_to_your_key.pem> -CAfile > > > > > > > > > > > > > > > > > > <path_to_your_CA_cert.pem> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Andy LoPresto > > > > > > > > > > > > > > > > > > alopre...@apache.org > > > > > > > > > > > > > > > > > > alopresto.apa...@gmail.com > > > > > > > > > > > > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 > > > > > > > > > > > > > > > > > > BACE 3C6E F65B 2F7D EF69 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Oct 16, 2019, at 8:37 PM, Peter Moberg > > > > > > > > > > > > > > > > > > > <peter.mob...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have an Elastic Search cluster that is > > > > > > > > > > > > > > > > > > > setup with SSL. It uses a self-signed > > > > > > > > > > > > > > > > > > > cert for this. I am working with Apache > > > > > > > > > > > > > > > > > > > Nifi 1.9.2. I have a flow that has the > > > > > > > > > > > > > > > > > > > PutElasticSearchHttp component. I have > > > > > > > > > > > > > > > > > > > setup a SSLContextService for that > > > > > > > > > > > > > > > > > > > component where I have specified a trust > > > > > > > > > > > > > > > > > > > store that has the self-signed cert from > > > > > > > > > > > > > > > > > > > ES. I specify an https endpoint to access > > > > > > > > > > > > > > > > > > > Elastic Search and Im having no issues > > > > > > > > > > > > > > > > > > > populating my Elastic Search instance > > > > > > > > > > > > > > > > > > > using this flow. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have another flow where I want to do > > > > > > > > > > > > > > > > > > > some lookups. So I have been using the > > > > > > > > > > > > > > > > > > > LookupRecord processor. That one I have > > > > > > > > > > > > > > > > > > > associated with an > > > > > > > > > > > > > > > > > > > ElasticSearchClientServiceImpl which I > > > > > > > > > > > > > > > > > > > have setup to point to the same > > > > > > > > > > > > > > > > > > > SSLContextService as used above. I > > > > > > > > > > > > > > > > > > > specified the same HTTPS Url (triple > > > > > > > > > > > > > > > > > > > checked this). However, when I run this > > > > > > > > > > > > > > > > > > > second Flow I am not able to verify the > > > > > > > > > > > > > > > > > > > ES server's self-signed certificate. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I check the nifi-app.log and it says: > > > > > > > > > > > > > > > > > > > Caused by: > > > > > > > > > > > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilderException: > > > > > > > > > > > > > > > > > > > unable to find valid certification path > > > > > > > > > > > > > > > > > > > to requested target > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am a bit surprised that I am not able > > > > > > > > > > > > > > > > > > > to verify the same server certificate in > > > > > > > > > > > > > > > > > > > the two different flows. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Completely stuck on this so if anyone > > > > > > > > > > > > > > > > > > > have any pointers please let me know. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Peter > > > > > > > > > > > > > > > > > >
