Think I might have found the issue. Will report tonight.
Mike, please don’t spend any time debugging this because I think it might be an
issue on my side. Appreciate all the help so far.
Thanks,
Peter
On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <peter.mob...@gmail.com>, wrote:
> Here it is:
>
>
> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7]
> o.a.n.processors.standard.LookupRecord
> LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process
> StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default,
> section=1], offset=7180,
> length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]:
> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
> coordinates {key=test} in Lookup Service
> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
> coordinates {key=test} in Lookup Service
> at
> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
> at
> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
> at
> org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
> at
> org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.nifi.lookup.LookupFailureException:
> org.apache.nifi.lookup.LookupFailureException:
> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at
> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
> at com.sun.proxy.$Proxy136.lookup(Unknown Source)
> at
> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
> ... 17 common frames omitted
> Caused by: org.apache.nifi.lookup.LookupFailureException:
> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at
> org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
> at
> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
> ... 23 common frames omitted
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
> at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
> at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> ... 1 common frames omitted
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> ... 9 common frames omitted
> Caused by: sun.security.validator.ValidatorException: PKIX path building
> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find valid certification path to requested target
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> at sun.security.validator.Validator.validate(Validator.java:262)
> at
> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> at
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> at
> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> ... 17 common frames omitted
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable
> to find valid certification path to requested target
> at
> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
> at
> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
> ... 23 common frames omitted
>
>
> Thanks,
>
> Peter
> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mikerthom...@gmail.com>, wrote:
> > Can you share the stacktrace from the logs?
> >
> > > On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <peter.mob...@gmail.com>
> > > wrote:
> > > > Mike,
> > > >
> > > > The SSLContextService only had the Trust store configured. I think I
> > > > seen that ticket before but didn’t pay attention to the fact it wasn’t
> > > > merged in to the code I am running.
> > > >
> > > > However, I configured the service to have a KeyStore now but I am
> > > > getting the same errors…
> > > >
> > > > Thanks,
> > > >
> > > > Peter
> > > > On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mikerthom...@gmail.com>,
> > > > wrote:
> > > > > Peter,
> > > > >
> > > > > Are you configuring the service as a trust-only configuration? If so,
> > > > > that's been addressed in the 1.10 which is due for release in the
> > > > > near(ish) future.
> > > > >
> > > > > https://issues.apache.org/jira/browse/NIFI-6228
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mike
> > > > >
> > > > > > On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg
> > > > > > <peter.mob...@gmail.com> wrote:
> > > > > > > As a follow-up.
> > > > > > >
> > > > > > > On the Nifi node I am able to do a GET to Elastic Search using
> > > > > > > curl. I specify the —cacert option giving it the self-signed root
> > > > > > > certificate.
> > > > > > >
> > > > > > > Of course, this isn’t using the TrustStore but I am able to use
> > > > > > > the TrustStore if I use other ES processors… just not the
> > > > > > > ElasticSearchClientServicesImpl.
> > > > > > >
> > > > > > > On Oct 18, 2019, 12:48 AM -0500, Peter Moberg
> > > > > > > <peter.mob...@gmail.com>, wrote:
> > > > > > > > Hi Andy,
> > > > > > > >
> > > > > > > > thanks for your suggestions. Here is what I have tried so far
> > > > > > > > (still no luck).
> > > > > > > >
> > > > > > > > Connecting with openssl and viewing the certs it presents
> > > > > > > >
> > > > > > > > openssl s_client -connect quickstart-es-http.es-cluster
> > > > > > > > -showcerts
> > > > > > > >
> > > > > > > > If I then look inside the server cert I can find this
> > > > > > > >
> > > > > > > > Server Cert:
> > > > > > > > Issuer: OU = quickstart, CN = quickstart-http
> > > > > > > >
> > > > > > > > X509v3 Subject Alternative Name:
> > > > > > > > DNS:quickstart-es-http.es-cluster.es.local,
> > > > > > > > DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc,
> > > > > > > > DNS:quickstart-es-http.es-cluster
> > > > > > > >
> > > > > > > >
> > > > > > > > If I look in to the self-signed root cert I find this:
> > > > > > > >
> > > > > > > > Root Cert:
> > > > > > > > Subject: OU = quickstart, CN = quickstart-http
> > > > > > > >
> > > > > > > >
> > > > > > > > I now double check my trust store to make sure the Root Cert
> > > > > > > > is there.
> > > > > > > >
> > > > > > > > Trust store content
> > > > > > > > Your keystore contains 1 entry
> > > > > > > >
> > > > > > > > Alias name: ca_elastic
> > > > > > > > Creation date: Oct 16, 2019
> > > > > > > > Entry type: trustedCertEntry
> > > > > > > >
> > > > > > > > Owner: CN=quickstart-http, OU=quickstart
> > > > > > > > Issuer: CN=quickstart-http, OU=quickstart
> > > > > > > > Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
> > > > > > > > Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10
> > > > > > > > 14:36:01 UTC 2020
> > > > > > > > Certificate fingerprints:
> > > > > > > > MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
> > > > > > > > SHA1:
> > > > > > > > 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
> > > > > > > > SHA256:
> > > > > > > > B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
> > > > > > > > Signature algorithm name: SHA256withRSA
> > > > > > > > Subject Public Key Algorithm: 2048-bit RSA key
> > > > > > > > Version: 3
> > > > > > > >
> > > > > > > > Extensions:
> > > > > > > >
> > > > > > > > #1: ObjectId: 2.5.29.19 Criticality=true
> > > > > > > > BasicConstraints:[
> > > > > > > > CA:true
> > > > > > > > PathLen:2147483647
> > > > > > > > ]
> > > > > > > >
> > > > > > > > #2: ObjectId: 2.5.29.37 Criticality=false
> > > > > > > > ExtendedKeyUsages [
> > > > > > > > serverAuth
> > > > > > > > clientAuth
> > > > > > > > ]
> > > > > > > >
> > > > > > > > #3: ObjectId: 2.5.29.15 Criticality=true
> > > > > > > > KeyUsage [
> > > > > > > > DigitalSignature
> > > > > > > > Key_CertSign
> > > > > > > > ]
> > > > > > > >
> > > > > > > > So everything looks Ok. But when I run the
> > > > > > > > ElasticSearchClientServicesImpl with a SSLContext pointing to
> > > > > > > > my trust store I still get the following output in the log.
> > > > > > > >
> > > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General
> > > > > > > > SSLEngine problem
> > > > > > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > > > > > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > > > > > > > at
> > > > > > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > > > > > > > at
> > > > > > > > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > > > > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > > > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > > > at
> > > > > > > > sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > > > > > > > at
> > > > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > > > > > > > at
> > > > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > > > > > > > ... 9 common frames omitted
> > > > > > > > Caused by: sun.security.validator.ValidatorException: PKIX path
> > > > > > > > building failed:
> > > > > > > > sun.security.provider.certpath.SunCertPathBuilderException:
> > > > > > > > unable to find valid certification path to requested target
> > > > > > > > at
> > > > > > > > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > > > > > at
> > > > > > > > sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > > > > > > at sun.security.validator.Validator.validate(Validator.java:262)
> > > > > > > > at
> > > > > > > > sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > > > > > > > at
> > > > > > > > sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > > > > > > > at
> > > > > > > > sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > > > > > > > at
> > > > > > > > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > > > > > > > ... 17 common frames omitted
> > > > > > > >
> > > > > > > > Both the Nifi install and Elastic Search install is running in
> > > > > > > > Kubernetes. The address I am using is a service address that is
> > > > > > > > backed by 3 ES instances. However, I double checked all three
> > > > > > > > of the ES nodes to make sure that they returned back the same
> > > > > > > > SSL cert and they did.
> > > > > > > >
> > > > > > > > The only thing I haven't been able to figure out is how to
> > > > > > > > check if Kubernetes/ES reacts differently when you do a GET vs
> > > > > > > > POST. Feels strange that it would return different SSL certs
> > > > > > > > but stranger things have happened…
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto
> > > > > > > > <alopre...@apache.org>, wrote:
> > > > > > > > > Hi Peter,
> > > > > > > > >
> > > > > > > > > If you can use openssl’s s_client command (example below) to
> > > > > > > > > connect to the endpoint and verify that the hostname matches
> > > > > > > > > the certificate and that the certificate contains a
> > > > > > > > > SubjectAlternativeName entry with that hostname (see RFC 6125
> > > > > > > > > [1] for more details), this should help you debug the issue.
> > > > > > > > > The cause of the PKIX error is that the truststore doesn’t
> > > > > > > > > contain a certificate (or certificate chain) which matches
> > > > > > > > > the hostname presented by the remote endpoint. I think you
> > > > > > > > > understand that based on your message. The underlying reason
> > > > > > > > > for this is could be one of the following:
> > > > > > > > >
> > > > > > > > > * the server is behind an interface which responds
> > > > > > > > > differently to GET and POST/PUT requests
> > > > > > > > > * there is a load-balancer which is directing the requests
> > > > > > > > > coincidentally to different backend servers (one has the
> > > > > > > > > right cert; the other doesn’t)
> > > > > > > > > * I recall something around the addition of (some) Elastic
> > > > > > > > > Search components which handled TLS in an ES client-specific
> > > > > > > > > manner; I remember advocating for standard NiFi TLS
> > > > > > > > > interaction here but I am not sure what was ultimately
> > > > > > > > > contributed. If it’s not one of the above issues, I can
> > > > > > > > > investigate further.
> > > > > > > > >
> > > > > > > > > Hopefully this helps.
> > > > > > > > >
> > > > > > > > > [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
> > > > > > > > >
> > > > > > > > > s_client example:
> > > > > > > > >
> > > > > > > > > $ openssl s_client -connect <host:port> -debug -state -cert
> > > > > > > > > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> > > > > > > > > <path_to_your_CA_cert.pem>
> > > > > > > > >
> > > > > > > > > Andy LoPresto
> > > > > > > > > alopre...@apache.org
> > > > > > > > > alopresto.apa...@gmail.com
> > > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B
> > > > > > > > > 2F7D EF69
> > > > > > > > >
> > > > > > > > > > On Oct 16, 2019, at 8:37 PM, Peter Moberg
> > > > > > > > > > <peter.mob...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > I have an Elastic Search cluster that is setup with SSL. It
> > > > > > > > > > uses a self-signed cert for this. I am working with Apache
> > > > > > > > > > Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp
> > > > > > > > > > component. I have setup a SSLContextService for that
> > > > > > > > > > component where I have specified a trust store that has the
> > > > > > > > > > self-signed cert from ES. I specify an https endpoint to
> > > > > > > > > > access Elastic Search and Im having no issues populating my
> > > > > > > > > > Elastic Search instance using this flow.
> > > > > > > > > >
> > > > > > > > > > I have another flow where I want to do some lookups. So I
> > > > > > > > > > have been using the LookupRecord processor. That one I have
> > > > > > > > > > associated with an ElasticSearchClientServiceImpl which I
> > > > > > > > > > have setup to point to the same SSLContextService as used
> > > > > > > > > > above. I specified the same HTTPS Url (triple checked
> > > > > > > > > > this). However, when I run this second Flow I am not able
> > > > > > > > > > to verify the ES server's self-signed certificate.
> > > > > > > > > >
> > > > > > > > > > I check the nifi-app.log and it says:
> > > > > > > > > > Caused by:
> > > > > > > > > > sun.security.provider.certpath.SunCertPathBuilderException:
> > > > > > > > > > unable to find valid certification path to requested target
> > > > > > > > > >
> > > > > > > > > > I am a bit surprised that I am not able to verify the same
> > > > > > > > > > server certificate in the two different flows.
> > > > > > > > > >
> > > > > > > > > > Completely stuck on this so if anyone have any pointers
> > > > > > > > > > please let me know.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Peter
> > > > > > > > >
