Peter, Looks like this won't make it into 1.10, but it's a simple patch to apply if you download the source code for 1.10 and apply it manually.
Mike On Tue, Oct 22, 2019 at 2:46 PM Matt Burgess <mattyb...@gmail.com> wrote: > For saving space (if you want to run from the assembly/target directory) > you can add a -Ddir-only and it won’t build the tar.gz. > > > > > Sent from my iPhone > > On Oct 22, 2019, at 2:25 PM, Mike Thomsen <mikerthom...@gmail.com> wrote: > > > If you run `mvn clean install -DskipTests=true` from the root of the > source folder, you'll get a tar.gz build in $ROOT/nifi-assembly/target. I'd > recommend testing against that as it'll be faster. > > On Tue, Oct 22, 2019 at 1:56 PM Peter Moberg <peter.mob...@gmail.com> > wrote: > >> Mike, >> >> thanks for putting together that PR. I have built everything successfully >> but I haven't been able to test this yet since I haven't built the new >> Docker image. >> >> I assume you guys just use the ‘dockermaven’ folder to build the >> Dockerfile with src artifacts? >> >> My current dev machine is running out of disk space so I am spending some >> time trying to get it back to a runnable state. Hopefully I can build the >> PR and test it in Kubernetes this week sometime. >> >> Thanks, >> >> Peter >> On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen <mikerthom...@gmail.com>, >> wrote: >> >> As a compromise, I upgraded to the latest 5.X client and manually >> incremented Apache HttpClient to 4.5.10. PR is here: >> >> https://github.com/apache/nifi/pull/3828 >> >> There are integration tests for that package that automatically startup >> and provision an ES node, and they all passed with this configuration. >> Hopefully this PR will fix your issue since it appears to be an external >> dependency causing it. >> >> On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen <mikerthom...@gmail.com> >> wrote: >> >>> There's no hard and fast reason to stay with 5.X there, so you can build >>> your own copy of 1.9.2 with that dependency upgraded if you want to try it >>> out. I'll try to find time to test that change on 1.10.0-SNAPSHOT. >>> >>> On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg <peter.mob...@gmail.com> >>> wrote: >>> >>>> The certs in the TrustStore are marked as trusted. >>>> >>>> The host name specified in the ClientServiceImpl is: >>>> https://quickstart-es-http.es-cluster:9200 >>>> >>>> The CN field of the server certificate is: >>>> https://quickstart-es-http.es-cluster.es.local >>>> >>>> So at first glance it looks like the issue would be that the CN field >>>> differs from the host name. However, Im not getting an error message saying >>>> that the HostName doesn’t match the CN. According to the RFC spec first >>>> mentioned by Andy the CN field should only be used if the >>>> SubjectiveAlternativeName is empty. >>>> >>>> The Server Cert has the SAN set to: quickstart-es-http.es-cluster >>>> >>>> What is really strange to me is why I can connect using the >>>> QueryElasticSearch processor with the same SSLContextService and hostname >>>> but when I use the ClientServicesImpl it will not work. >>>> >>>> So I did some more investigation and it looks like the QueryES >>>> processor uses the okHTTP library and ClientServicesImpl uses the Elastic >>>> Search RestClient which in turn uses the HTTPClient from Apache. >>>> >>>> There was an issue with the HTTPClient library in v4.5.2 ( >>>> https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t >>>> do the right hostname verification. >>>> >>>> The version of Apache Nifi that Im using is 1.9.2 and it seems to use >>>> v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of >>>> Apache HTTPClient. >>>> >>>> >>>> https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties >>>> >>>> But again if it was a HostName validation issue the error message >>>> should read more like: “Certificate for <host> doesn’t match common name of >>>> certificate subject…” >>>> >>>> On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen <mikerthom...@gmail.com>, >>>> wrote: >>>> >>>> I'm far from a SSL/TLS expert, but let's get these out of the way: >>>> >>>> 1. Did you mark the server's cert as "trusted" when you created the >>>> trust store with keytool? >>>> 2. Are you sure that you're specifying the same hostname in the client >>>> service that is in the CN field in the server's cert? >>>> >>>> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can >>>> use it to generate valid certificates. It's meant for NiFi, but the certs >>>> it generates should be a solid step up from self-signed certs if you're not >>>> able to access a company CA. >>>> >>>> On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <peter.mob...@gmail.com> >>>> wrote: >>>> >>>>> Nope. No good. I even dump the network traffic and analyzed it in >>>>> Wireshark. The ES server sends back two certificates (server + self-signed >>>>> one) and both of them are present in my TrustStore. I am specifying both a >>>>> TrustStore and a Keystore now but it still gives the error that it can’t >>>>> find the certificate. >>>>> >>>>> Thanks, >>>>> >>>>> Peter >>>>> On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <peter.mob...@gmail.com>, >>>>> wrote: >>>>> >>>>> Think I might have found the issue. Will report tonight. >>>>> >>>>> Mike, please don’t spend any time debugging this because I think it >>>>> might be an issue on my side. Appreciate all the help so far. >>>>> >>>>> Thanks, >>>>> >>>>> Peter >>>>> On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <peter.mob...@gmail.com>, >>>>> wrote: >>>>> >>>>> Here it is: >>>>> >>>>> >>>>> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] >>>>> o.a.n.processors.standard.LookupRecord >>>>> LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process >>>>> StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim >>>>> [resourceClaim=StandardResourceClaim[id=1571410822200-1, >>>>> container=default, >>>>> section=1], offset=7180, >>>>> length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]: >>>>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup >>>>> coordinates {key=test} in Lookup Service >>>>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup >>>>> coordinates {key=test} in Lookup Service >>>>> at >>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300) >>>>> at >>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68) >>>>> at >>>>> org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134) >>>>> at >>>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212) >>>>> at >>>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180) >>>>> at >>>>> org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121) >>>>> at >>>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>>>> at >>>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) >>>>> at >>>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209) >>>>> at >>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) >>>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >>>>> at >>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >>>>> at >>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> Caused by: org.apache.nifi.lookup.LookupFailureException: >>>>> org.apache.nifi.lookup.LookupFailureException: >>>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem >>>>> at >>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175) >>>>> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> at >>>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87) >>>>> at com.sun.proxy.$Proxy136.lookup(Unknown Source) >>>>> at >>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298) >>>>> ... 17 common frames omitted >>>>> Caused by: org.apache.nifi.lookup.LookupFailureException: >>>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem >>>>> at >>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288) >>>>> at >>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169) >>>>> ... 23 common frames omitted >>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine >>>>> problem >>>>> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) >>>>> at >>>>> sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) >>>>> at >>>>> sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) >>>>> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) >>>>> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) >>>>> at >>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265) >>>>> at >>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305) >>>>> at >>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509) >>>>> at >>>>> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120) >>>>> at >>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) >>>>> at >>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) >>>>> at >>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) >>>>> at >>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) >>>>> at >>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) >>>>> at >>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) >>>>> ... 1 common frames omitted >>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine >>>>> problem >>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >>>>> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) >>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) >>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) >>>>> at >>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633) >>>>> at >>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) >>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) >>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) >>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) >>>>> at >>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283) >>>>> at >>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353) >>>>> ... 9 common frames omitted >>>>> Caused by: sun.security.validator.ValidatorException: PKIX path >>>>> building failed: >>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find >>>>> valid certification path to requested target >>>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >>>>> at >>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) >>>>> at sun.security.validator.Validator.validate(Validator.java:262) >>>>> at >>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) >>>>> at >>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) >>>>> at >>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) >>>>> at >>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620) >>>>> ... 17 common frames omitted >>>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: >>>>> unable to find valid certification path to requested target >>>>> at >>>>> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) >>>>> at >>>>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) >>>>> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) >>>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) >>>>> ... 23 common frames omitted >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Peter >>>>> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mikerthom...@gmail.com>, >>>>> wrote: >>>>> >>>>> Can you share the stacktrace from the logs? >>>>> >>>>> On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <peter.mob...@gmail.com> >>>>> wrote: >>>>> >>>>>> Mike, >>>>>> >>>>>> The SSLContextService only had the Trust store configured. I think I >>>>>> seen that ticket before but didn’t pay attention to the fact it wasn’t >>>>>> merged in to the code I am running. >>>>>> >>>>>> However, I configured the service to have a KeyStore now but I am >>>>>> getting the same errors… >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Peter >>>>>> On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mikerthom...@gmail.com>, >>>>>> wrote: >>>>>> >>>>>> Peter, >>>>>> >>>>>> Are you configuring the service as a trust-only configuration? If so, >>>>>> that's been addressed in the 1.10 which is due for release in the >>>>>> near(ish) >>>>>> future. >>>>>> >>>>>> https://issues.apache.org/jira/browse/NIFI-6228 >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Mike >>>>>> >>>>>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <peter.mob...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> As a follow-up. >>>>>>> >>>>>>> On the Nifi node I am able to do a GET to Elastic Search using curl. >>>>>>> I specify the —cacert option giving it the self-signed root certificate. >>>>>>> >>>>>>> Of course, this isn’t using the TrustStore but I am able to use the >>>>>>> TrustStore if I use other ES processors… just not the >>>>>>> ElasticSearchClientServicesImpl. >>>>>>> >>>>>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg < >>>>>>> peter.mob...@gmail.com>, wrote: >>>>>>> >>>>>>> Hi Andy, >>>>>>> >>>>>>> thanks for your suggestions. Here is what I have tried so far (still >>>>>>> no luck). >>>>>>> >>>>>>> Connecting with openssl and viewing the certs it presents >>>>>>> >>>>>>> *openssl s_client -connect quickstart-es-http.es >>>>>>> <http://quickstart-es-http.es>-cluster -showcerts* >>>>>>> >>>>>>> If I then look inside the server cert I can find this >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3 >>>>>>> Subject Alternative Name: DNS:quickstart-es-http.es >>>>>>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http, >>>>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc, >>>>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster* >>>>>>> >>>>>>> >>>>>>> If I look in to the self-signed root cert I find this: >>>>>>> >>>>>>> >>>>>>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http* >>>>>>> >>>>>>> >>>>>>> I now double check my trust store to make sure the Root Cert is >>>>>>> there. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Trust store content Your keystore contains 1 entry Alias name: >>>>>>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry >>>>>>> Owner: >>>>>>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, >>>>>>> OU=quickstart >>>>>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11 >>>>>>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate >>>>>>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1: >>>>>>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256: >>>>>>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF >>>>>>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: >>>>>>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19 >>>>>>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2: >>>>>>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth >>>>>>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [ >>>>>>> DigitalSignature Key_CertSign ]* >>>>>>> >>>>>>> So everything looks Ok. But when I run the >>>>>>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust >>>>>>> store I still get the following output in the log. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine >>>>>>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at >>>>>>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at >>>>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at >>>>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at >>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633) >>>>>>> at >>>>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) >>>>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at >>>>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at >>>>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at >>>>>>> java.security.AccessController.doPrivileged(Native Method) at >>>>>>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at >>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283) >>>>>>> at >>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353) >>>>>>> ... 9 common frames omitted Caused by: >>>>>>> sun.security.validator.ValidatorException: PKIX path building failed: >>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to >>>>>>> find >>>>>>> valid certification path to requested target at >>>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at >>>>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) >>>>>>> at sun.security.validator.Validator.validate(Validator.java:262) at >>>>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) >>>>>>> at >>>>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) >>>>>>> at >>>>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) >>>>>>> at >>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620) >>>>>>> ... 17 common frames omitted* >>>>>>> >>>>>>> Both the Nifi install and Elastic Search install is running in >>>>>>> Kubernetes. The address I am using is a service address that is backed >>>>>>> by 3 >>>>>>> ES instances. However, I double checked all three of the ES nodes to >>>>>>> make >>>>>>> sure that they returned back the same SSL cert and they did. >>>>>>> >>>>>>> The only thing I haven't been able to figure out is how to check if >>>>>>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels >>>>>>> strange >>>>>>> that it would return different SSL certs but stranger things have >>>>>>> happened… >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <alopre...@apache.org>, >>>>>>> wrote: >>>>>>> >>>>>>> Hi Peter, >>>>>>> >>>>>>> If you can use openssl’s s_client command (example below) to connect >>>>>>> to the endpoint and verify that the hostname matches the certificate and >>>>>>> that the certificate contains a SubjectAlternativeName entry with that >>>>>>> hostname (see RFC 6125 [1] for more details), this should help you debug >>>>>>> the issue. The cause of the PKIX error is that the truststore doesn’t >>>>>>> contain a certificate (or certificate chain) which matches the hostname >>>>>>> presented by the remote endpoint. I think you understand that based on >>>>>>> your >>>>>>> message. The underlying reason for this is could be one of the >>>>>>> following: >>>>>>> >>>>>>> * the server is behind an interface which responds differently to >>>>>>> GET and POST/PUT requests >>>>>>> * there is a load-balancer which is directing the requests >>>>>>> coincidentally to different backend servers (one has the right cert; the >>>>>>> other doesn’t) >>>>>>> * I recall something around the addition of (some) Elastic Search >>>>>>> components which handled TLS in an ES client-specific manner; I remember >>>>>>> advocating for standard NiFi TLS interaction here but I am not sure what >>>>>>> was ultimately contributed. If it’s not one of the above issues, I can >>>>>>> investigate further. >>>>>>> >>>>>>> Hopefully this helps. >>>>>>> >>>>>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4 >>>>>>> >>>>>>> s_client example: >>>>>>> >>>>>>> $ openssl s_client -connect <host:port> -debug -state -cert >>>>>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile >>>>>>> <path_to_your_CA_cert.pem> >>>>>>> >>>>>>> Andy LoPresto >>>>>>> alopre...@apache.org >>>>>>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* >>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>>>>> >>>>>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <peter.mob...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> I have an Elastic Search cluster that is setup with SSL. It uses a >>>>>>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a >>>>>>> flow that has the PutElasticSearchHttp component. I have setup a >>>>>>> SSLContextService for that component where I have specified a trust >>>>>>> store >>>>>>> that has the self-signed cert from ES. I specify an https endpoint to >>>>>>> access Elastic Search and Im having no issues populating my Elastic >>>>>>> Search >>>>>>> instance using this flow. >>>>>>> >>>>>>> I have another flow where I want to do some lookups. So I have been >>>>>>> using the LookupRecord processor. That one I have associated with an >>>>>>> ElasticSearchClientServiceImpl which I have setup to point to the same >>>>>>> SSLContextService as used above. I specified the same HTTPS Url (triple >>>>>>> checked this). However, when I run this second Flow I am not able to >>>>>>> verify >>>>>>> the ES server's self-signed certificate. >>>>>>> >>>>>>> I check the nifi-app.log and it says: >>>>>>> Caused by: >>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to >>>>>>> find >>>>>>> valid certification path to requested target >>>>>>> >>>>>>> I am a bit surprised that I am not able to verify the same server >>>>>>> certificate in the two different flows. >>>>>>> >>>>>>> Completely stuck on this so if anyone have any pointers please let >>>>>>> me know. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Peter >>>>>>> >>>>>>> >>>>>>>
