For saving space (if you want to run from the assembly/target directory) you can add a -Ddir-only and it won’t build the tar.gz.
Sent from my iPhone > On Oct 22, 2019, at 2:25 PM, Mike Thomsen <mikerthom...@gmail.com> wrote: > > > If you run `mvn clean install -DskipTests=true` from the root of the source > folder, you'll get a tar.gz build in $ROOT/nifi-assembly/target. I'd > recommend testing against that as it'll be faster. > >> On Tue, Oct 22, 2019 at 1:56 PM Peter Moberg <peter.mob...@gmail.com> wrote: >> Mike, >> >> thanks for putting together that PR. I have built everything successfully >> but I haven't been able to test this yet since I haven't built the new >> Docker image. >> >> I assume you guys just use the ‘dockermaven’ folder to build the Dockerfile >> with src artifacts? >> >> My current dev machine is running out of disk space so I am spending some >> time trying to get it back to a runnable state. Hopefully I can build the PR >> and test it in Kubernetes this week sometime. >> >> Thanks, >> >> Peter >>> On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen <mikerthom...@gmail.com>, >>> wrote: >>> As a compromise, I upgraded to the latest 5.X client and manually >>> incremented Apache HttpClient to 4.5.10. PR is here: >>> >>> https://github.com/apache/nifi/pull/3828 >>> >>> There are integration tests for that package that automatically startup and >>> provision an ES node, and they all passed with this configuration. >>> Hopefully this PR will fix your issue since it appears to be an external >>> dependency causing it. >>> >>>> On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen <mikerthom...@gmail.com> >>>> wrote: >>>> There's no hard and fast reason to stay with 5.X there, so you can build >>>> your own copy of 1.9.2 with that dependency upgraded if you want to try it >>>> out. I'll try to find time to test that change on 1.10.0-SNAPSHOT. >>>> >>>>> On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg <peter.mob...@gmail.com> >>>>> wrote: >>>>> The certs in the TrustStore are marked as trusted. >>>>> >>>>> The host name specified in the ClientServiceImpl is: >>>>> https://quickstart-es-http.es-cluster:9200 >>>>> >>>>> The CN field of the server certificate is: >>>>> https://quickstart-es-http.es-cluster.es.local >>>>> >>>>> So at first glance it looks like the issue would be that the CN field >>>>> differs from the host name. However, Im not getting an error message >>>>> saying that the HostName doesn’t match the CN. According to the RFC spec >>>>> first mentioned by Andy the CN field should only be used if the >>>>> SubjectiveAlternativeName is empty. >>>>> >>>>> The Server Cert has the SAN set to: quickstart-es-http.es-cluster >>>>> >>>>> What is really strange to me is why I can connect using the >>>>> QueryElasticSearch processor with the same SSLContextService and hostname >>>>> but when I use the ClientServicesImpl it will not work. >>>>> >>>>> So I did some more investigation and it looks like the QueryES processor >>>>> uses the okHTTP library and ClientServicesImpl uses the Elastic Search >>>>> RestClient which in turn uses the HTTPClient from Apache. >>>>> >>>>> There was an issue with the HTTPClient library in v4.5.2 >>>>> (https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t >>>>> do the right hostname verification. >>>>> >>>>> The version of Apache Nifi that Im using is 1.9.2 and it seems to use >>>>> v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 >>>>> of Apache HTTPClient. >>>>> >>>>> https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties >>>>> >>>>> But again if it was a HostName validation issue the error message should >>>>> read more like: “Certificate for <host> doesn’t match common name of >>>>> certificate subject…” >>>>> >>>>>> On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen <mikerthom...@gmail.com>, >>>>>> wrote: >>>>>> I'm far from a SSL/TLS expert, but let's get these out of the way: >>>>>> >>>>>> 1. Did you mark the server's cert as "trusted" when you created the >>>>>> trust store with keytool? >>>>>> 2. Are you sure that you're specifying the same hostname in the client >>>>>> service that is in the CN field in the server's cert? >>>>>> >>>>>> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can use >>>>>> it to generate valid certificates. It's meant for NiFi, but the certs it >>>>>> generates should be a solid step up from self-signed certs if you're not >>>>>> able to access a company CA. >>>>>> >>>>>>> On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <peter.mob...@gmail.com> >>>>>>> wrote: >>>>>>> Nope. No good. I even dump the network traffic and analyzed it in >>>>>>> Wireshark. The ES server sends back two certificates (server + >>>>>>> self-signed one) and both of them are present in my TrustStore. I am >>>>>>> specifying both a TrustStore and a Keystore now but it still gives the >>>>>>> error that it can’t find the certificate. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Peter >>>>>>>> On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <peter.mob...@gmail.com>, >>>>>>>> wrote: >>>>>>>> Think I might have found the issue. Will report tonight. >>>>>>>> >>>>>>>> Mike, please don’t spend any time debugging this because I think it >>>>>>>> might be an issue on my side. Appreciate all the help so far. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Peter >>>>>>>>> On Oct 18, 2019, 2:21 PM -0500, Peter Moberg >>>>>>>>> <peter.mob...@gmail.com>, wrote: >>>>>>>>> Here it is: >>>>>>>>> >>>>>>>>> >>>>>>>>> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] >>>>>>>>> o.a.n.processors.standard.LookupRecord >>>>>>>>> LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to >>>>>>>>> process >>>>>>>>> StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim >>>>>>>>> [resourceClaim=StandardResourceClaim[id=1571410822200-1, >>>>>>>>> container=default, section=1], offset=7180, >>>>>>>>> length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]: >>>>>>>>> org.apache.nifi.processor.exception.ProcessException: Failed to >>>>>>>>> lookup coordinates {key=test} in Lookup Service >>>>>>>>> org.apache.nifi.processor.exception.ProcessException: Failed to >>>>>>>>> lookup coordinates {key=test} in Lookup Service >>>>>>>>> at >>>>>>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300) >>>>>>>>> at >>>>>>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68) >>>>>>>>> at >>>>>>>>> org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134) >>>>>>>>> at >>>>>>>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212) >>>>>>>>> at >>>>>>>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180) >>>>>>>>> at >>>>>>>>> org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121) >>>>>>>>> at >>>>>>>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>>>>>>>> at >>>>>>>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) >>>>>>>>> at >>>>>>>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209) >>>>>>>>> at >>>>>>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) >>>>>>>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >>>>>>>>> at >>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>>>>> Caused by: org.apache.nifi.lookup.LookupFailureException: >>>>>>>>> org.apache.nifi.lookup.LookupFailureException: >>>>>>>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem >>>>>>>>> at >>>>>>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175) >>>>>>>>> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>>>>> at >>>>>>>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87) >>>>>>>>> at com.sun.proxy.$Proxy136.lookup(Unknown Source) >>>>>>>>> at >>>>>>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298) >>>>>>>>> ... 17 common frames omitted >>>>>>>>> Caused by: org.apache.nifi.lookup.LookupFailureException: >>>>>>>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem >>>>>>>>> at >>>>>>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288) >>>>>>>>> at >>>>>>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169) >>>>>>>>> ... 23 common frames omitted >>>>>>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine >>>>>>>>> problem >>>>>>>>> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) >>>>>>>>> at >>>>>>>>> sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) >>>>>>>>> at >>>>>>>>> sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) >>>>>>>>> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) >>>>>>>>> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) >>>>>>>>> at >>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265) >>>>>>>>> at >>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305) >>>>>>>>> at >>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509) >>>>>>>>> at >>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120) >>>>>>>>> at >>>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) >>>>>>>>> at >>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) >>>>>>>>> at >>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) >>>>>>>>> at >>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) >>>>>>>>> at >>>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) >>>>>>>>> at >>>>>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) >>>>>>>>> ... 1 common frames omitted >>>>>>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine >>>>>>>>> problem >>>>>>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >>>>>>>>> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) >>>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) >>>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) >>>>>>>>> at >>>>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633) >>>>>>>>> at >>>>>>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) >>>>>>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) >>>>>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) >>>>>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) >>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) >>>>>>>>> at >>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283) >>>>>>>>> at >>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353) >>>>>>>>> ... 9 common frames omitted >>>>>>>>> Caused by: sun.security.validator.ValidatorException: PKIX path >>>>>>>>> building failed: >>>>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to >>>>>>>>> find valid certification path to requested target >>>>>>>>> at >>>>>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >>>>>>>>> at >>>>>>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) >>>>>>>>> at sun.security.validator.Validator.validate(Validator.java:262) >>>>>>>>> at >>>>>>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) >>>>>>>>> at >>>>>>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) >>>>>>>>> at >>>>>>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) >>>>>>>>> at >>>>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620) >>>>>>>>> ... 17 common frames omitted >>>>>>>>> Caused by: >>>>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to >>>>>>>>> find valid certification path to requested target >>>>>>>>> at >>>>>>>>> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) >>>>>>>>> at >>>>>>>>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) >>>>>>>>> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) >>>>>>>>> at >>>>>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) >>>>>>>>> ... 23 common frames omitted >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Peter >>>>>>>>>> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen >>>>>>>>>> <mikerthom...@gmail.com>, wrote: >>>>>>>>>> Can you share the stacktrace from the logs? >>>>>>>>>> >>>>>>>>>>> On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg >>>>>>>>>>> <peter.mob...@gmail.com> wrote: >>>>>>>>>>> Mike, >>>>>>>>>>> >>>>>>>>>>> The SSLContextService only had the Trust store configured. I think >>>>>>>>>>> I seen that ticket before but didn’t pay attention to the fact it >>>>>>>>>>> wasn’t merged in to the code I am running. >>>>>>>>>>> >>>>>>>>>>> However, I configured the service to have a KeyStore now but I am >>>>>>>>>>> getting the same errors… >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Peter >>>>>>>>>>>> On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen >>>>>>>>>>>> <mikerthom...@gmail.com>, wrote: >>>>>>>>>>>> Peter, >>>>>>>>>>>> >>>>>>>>>>>> Are you configuring the service as a trust-only configuration? If >>>>>>>>>>>> so, that's been addressed in the 1.10 which is due for release in >>>>>>>>>>>> the near(ish) future. >>>>>>>>>>>> >>>>>>>>>>>> https://issues.apache.org/jira/browse/NIFI-6228 >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Mike >>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg >>>>>>>>>>>>> <peter.mob...@gmail.com> wrote: >>>>>>>>>>>>> As a follow-up. >>>>>>>>>>>>> >>>>>>>>>>>>> On the Nifi node I am able to do a GET to Elastic Search using >>>>>>>>>>>>> curl. I specify the —cacert option giving it the self-signed root >>>>>>>>>>>>> certificate. >>>>>>>>>>>>> >>>>>>>>>>>>> Of course, this isn’t using the TrustStore but I am able to use >>>>>>>>>>>>> the TrustStore if I use other ES processors… just not the >>>>>>>>>>>>> ElasticSearchClientServicesImpl. >>>>>>>>>>>>> >>>>>>>>>>>>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg >>>>>>>>>>>>>> <peter.mob...@gmail.com>, wrote: >>>>>>>>>>>>>> Hi Andy, >>>>>>>>>>>>>> >>>>>>>>>>>>>> thanks for your suggestions. Here is what I have tried so far >>>>>>>>>>>>>> (still no luck). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Connecting with openssl and viewing the certs it presents >>>>>>>>>>>>>> >>>>>>>>>>>>>> openssl s_client -connect quickstart-es-http.es-cluster >>>>>>>>>>>>>> -showcerts >>>>>>>>>>>>>> >>>>>>>>>>>>>> If I then look inside the server cert I can find this >>>>>>>>>>>>>> >>>>>>>>>>>>>> Server Cert: >>>>>>>>>>>>>> Issuer: OU = quickstart, CN = quickstart-http >>>>>>>>>>>>>> >>>>>>>>>>>>>> X509v3 Subject Alternative Name: >>>>>>>>>>>>>> DNS:quickstart-es-http.es-cluster.es.local, >>>>>>>>>>>>>> DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, >>>>>>>>>>>>>> DNS:quickstart-es-http.es-cluster >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> If I look in to the self-signed root cert I find this: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Root Cert: >>>>>>>>>>>>>> Subject: OU = quickstart, CN = quickstart-http >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I now double check my trust store to make sure the Root Cert is >>>>>>>>>>>>>> there. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Trust store content >>>>>>>>>>>>>> Your keystore contains 1 entry >>>>>>>>>>>>>> >>>>>>>>>>>>>> Alias name: ca_elastic >>>>>>>>>>>>>> Creation date: Oct 16, 2019 >>>>>>>>>>>>>> Entry type: trustedCertEntry >>>>>>>>>>>>>> >>>>>>>>>>>>>> Owner: CN=quickstart-http, OU=quickstart >>>>>>>>>>>>>> Issuer: CN=quickstart-http, OU=quickstart >>>>>>>>>>>>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 >>>>>>>>>>>>>> Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 >>>>>>>>>>>>>> 14:36:01 UTC 2020 >>>>>>>>>>>>>> Certificate fingerprints: >>>>>>>>>>>>>> MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 >>>>>>>>>>>>>> SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 >>>>>>>>>>>>>> SHA256: >>>>>>>>>>>>>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF >>>>>>>>>>>>>> Signature algorithm name: SHA256withRSA >>>>>>>>>>>>>> Subject Public Key Algorithm: 2048-bit RSA key >>>>>>>>>>>>>> Version: 3 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Extensions: >>>>>>>>>>>>>> >>>>>>>>>>>>>> #1: ObjectId: 2.5.29.19 Criticality=true >>>>>>>>>>>>>> BasicConstraints:[ >>>>>>>>>>>>>> CA:true >>>>>>>>>>>>>> PathLen:2147483647 >>>>>>>>>>>>>> ] >>>>>>>>>>>>>> >>>>>>>>>>>>>> #2: ObjectId: 2.5.29.37 Criticality=false >>>>>>>>>>>>>> ExtendedKeyUsages [ >>>>>>>>>>>>>> serverAuth >>>>>>>>>>>>>> clientAuth >>>>>>>>>>>>>> ] >>>>>>>>>>>>>> >>>>>>>>>>>>>> #3: ObjectId: 2.5.29.15 Criticality=true >>>>>>>>>>>>>> KeyUsage [ >>>>>>>>>>>>>> DigitalSignature >>>>>>>>>>>>>> Key_CertSign >>>>>>>>>>>>>> ] >>>>>>>>>>>>>> >>>>>>>>>>>>>> So everything looks Ok. But when I run the >>>>>>>>>>>>>> ElasticSearchClientServicesImpl with a SSLContext pointing to my >>>>>>>>>>>>>> trust store I still get the following output in the log. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Caused by: javax.net.ssl.SSLHandshakeException: General >>>>>>>>>>>>>> SSLEngine problem >>>>>>>>>>>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) >>>>>>>>>>>>>> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) >>>>>>>>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) >>>>>>>>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) >>>>>>>>>>>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) >>>>>>>>>>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) >>>>>>>>>>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) >>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353) >>>>>>>>>>>>>> ... 9 common frames omitted >>>>>>>>>>>>>> Caused by: sun.security.validator.ValidatorException: PKIX path >>>>>>>>>>>>>> building failed: >>>>>>>>>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: >>>>>>>>>>>>>> unable to find valid certification path to requested target >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) >>>>>>>>>>>>>> at sun.security.validator.Validator.validate(Validator.java:262) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620) >>>>>>>>>>>>>> ... 17 common frames omitted >>>>>>>>>>>>>> >>>>>>>>>>>>>> Both the Nifi install and Elastic Search install is running in >>>>>>>>>>>>>> Kubernetes. The address I am using is a service address that is >>>>>>>>>>>>>> backed by 3 ES instances. However, I double checked all three of >>>>>>>>>>>>>> the ES nodes to make sure that they returned back the same SSL >>>>>>>>>>>>>> cert and they did. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The only thing I haven't been able to figure out is how to check >>>>>>>>>>>>>> if Kubernetes/ES reacts differently when you do a GET vs POST. >>>>>>>>>>>>>> Feels strange that it would return different SSL certs but >>>>>>>>>>>>>> stranger things have happened… >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto >>>>>>>>>>>>>>> <alopre...@apache.org>, wrote: >>>>>>>>>>>>>>> Hi Peter, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If you can use openssl’s s_client command (example below) to >>>>>>>>>>>>>>> connect to the endpoint and verify that the hostname matches >>>>>>>>>>>>>>> the certificate and that the certificate contains a >>>>>>>>>>>>>>> SubjectAlternativeName entry with that hostname (see RFC 6125 >>>>>>>>>>>>>>> [1] for more details), this should help you debug the issue. >>>>>>>>>>>>>>> The cause of the PKIX error is that the truststore doesn’t >>>>>>>>>>>>>>> contain a certificate (or certificate chain) which matches the >>>>>>>>>>>>>>> hostname presented by the remote endpoint. I think you >>>>>>>>>>>>>>> understand that based on your message. The underlying reason >>>>>>>>>>>>>>> for this is could be one of the following: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> * the server is behind an interface which responds differently >>>>>>>>>>>>>>> to GET and POST/PUT requests >>>>>>>>>>>>>>> * there is a load-balancer which is directing the requests >>>>>>>>>>>>>>> coincidentally to different backend servers (one has the right >>>>>>>>>>>>>>> cert; the other doesn’t) >>>>>>>>>>>>>>> * I recall something around the addition of (some) Elastic >>>>>>>>>>>>>>> Search components which handled TLS in an ES client-specific >>>>>>>>>>>>>>> manner; I remember advocating for standard NiFi TLS interaction >>>>>>>>>>>>>>> here but I am not sure what was ultimately contributed. If it’s >>>>>>>>>>>>>>> not one of the above issues, I can investigate further. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hopefully this helps. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> s_client example: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> $ openssl s_client -connect <host:port> -debug -state -cert >>>>>>>>>>>>>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile >>>>>>>>>>>>>>> <path_to_your_CA_cert.pem> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Andy LoPresto >>>>>>>>>>>>>>> alopre...@apache.org >>>>>>>>>>>>>>> alopresto.apa...@gmail.com >>>>>>>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D >>>>>>>>>>>>>>> EF69 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg >>>>>>>>>>>>>>>> <peter.mob...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have an Elastic Search cluster that is setup with SSL. It >>>>>>>>>>>>>>>> uses a self-signed cert for this. I am working with Apache >>>>>>>>>>>>>>>> Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp >>>>>>>>>>>>>>>> component. I have setup a SSLContextService for that component >>>>>>>>>>>>>>>> where I have specified a trust store that has the self-signed >>>>>>>>>>>>>>>> cert from ES. I specify an https endpoint to access Elastic >>>>>>>>>>>>>>>> Search and Im having no issues populating my Elastic Search >>>>>>>>>>>>>>>> instance using this flow. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have another flow where I want to do some lookups. So I have >>>>>>>>>>>>>>>> been using the LookupRecord processor. That one I have >>>>>>>>>>>>>>>> associated with an ElasticSearchClientServiceImpl which I have >>>>>>>>>>>>>>>> setup to point to the same SSLContextService as used above. I >>>>>>>>>>>>>>>> specified the same HTTPS Url (triple checked this). However, >>>>>>>>>>>>>>>> when I run this second Flow I am not able to verify the ES >>>>>>>>>>>>>>>> server's self-signed certificate. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I check the nifi-app.log and it says: >>>>>>>>>>>>>>>> Caused by: >>>>>>>>>>>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: >>>>>>>>>>>>>>>> unable to find valid certification path to requested target >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am a bit surprised that I am not able to verify the same >>>>>>>>>>>>>>>> server certificate in the two different flows. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Completely stuck on this so if anyone have any pointers please >>>>>>>>>>>>>>>> let me know. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Peter >>>>>>>>>>>>>>>
