[jira] [Commented] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message
[ https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17345943#comment-17345943 ] Andreas Klingenstein commented on NIFI-8333: We don't have any custom processors installed. We only added one property to the invokeHTTP processor (see 5_config_invokeHttp_config_3.png) and a few other properties to other processors but that's it. I'm not sure if that's "custom code". And yes we have an SSL Context Service configured for our InvokeHTTP (-> 3_config_invokeHttp_config_1.png) and 7_sslcontextservice_properties.png for details. I think something is not properly cleaned up when the InvokeHTTP is terminated. That might lead to the fall back (using the JVM default trust store) the next time the processor is called. That would also explain why it's working fine after a restart of Nifi. Have you already tried to reproduce it? > Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long > running transactions) and misleading error-message > -- > > Key: NIFI-8333 > URL: https://issues.apache.org/jira/browse/NIFI-8333 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.12.1, 1.13.2 >Reporter: Andreas Klingenstein >Priority: Major > Labels: https, security > Attachments: 1_config_invokeHttp_settings.png, > 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png, > 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png, > 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png, > 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt > > > There seems to be an issue with InvokeHTTP 1.12.1 or > StandardRestrictedSSLContextService 1.12.1 or both. > We observed a lot of "SSLHandshakeException"s (...) "unable to find valid > certification path to requested target" in some situations. > At first we had a very close look into our keystore which is used in our > StandardRestrictedSSLContextService and made sure everything was fine. In the > end we figured out we had used an older, no longer valid oAuth token and the > webservice simlpy had rejected our request. > Then we got the "SSLHandshakeException"s out of the blue in situations of > very intense testing where we started and stopped tests runs over and over > again. The oAuth token was still valid and other processors of the same kind > and even exact copies of the now failing processor worked fine > We discovered the following steps to reproduce the issue in our environment > (Nifi 1.12.1 & 1.13.2): > - create input data for the InvokeHTTP > - make sure the contacted endpoint does not respond but have a high timeout > setting in InvokeHttp to be able to terminate the processor > - start InvokeHTTP > - let it run for some time > - stop the InvokeHTTP-processor > - "Terminate" should be available in the context menu if there are still > open requests waiting for an answer (or timeout) > - terminate it > - wait until all tasks are finally stopped > - start the InvokeHTTP again > "SSLHandshakeException" Errors should be visible now in the nifi-app.log on > all machines where the invokehttp-processor had to terminate some tasks. > They'll > stay until you restart those nifi instances > Our configuration > - Cluster: 3 server, one nifi instance each > - nifi in "http-mode" on port 8081 (nifi-properties) > more details in the screenshots > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message
[
https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17344877#comment-17344877
]
David Handermann commented on NIFI-8333:
[~aklingens] Thanks for following up and confirming the same issue on version
1.13.2. Do you have any custom processors installed or custom code in one of
the execute or invoke scripted processors? Related to that, do you have an SSL
Context Service configured for InvokeHTTP or not? If you do not have one
configured, then InvokeHTTP falls back to using the JVM default trust store.
If there is custom code setting the Java System property
{{javax.net.ssl.trustStore}}, that would explain why the issue only happens
after restarting InvokeHTTP.
> Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long
> running transactions) and misleading error-message
> --
>
> Key: NIFI-8333
> URL: https://issues.apache.org/jira/browse/NIFI-8333
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.1, 1.13.2
>Reporter: Andreas Klingenstein
>Priority: Major
> Labels: https, security
> Attachments: 1_config_invokeHttp_settings.png,
> 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png,
> 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png,
> 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png,
> 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt
>
>
> There seems to be an issue with InvokeHTTP 1.12.1 or
> StandardRestrictedSSLContextService 1.12.1 or both.
> We observed a lot of "SSLHandshakeException"s (...) "unable to find valid
> certification path to requested target" in some situations.
> At first we had a very close look into our keystore which is used in our
> StandardRestrictedSSLContextService and made sure everything was fine. In the
> end we figured out we had used an older, no longer valid oAuth token and the
> webservice simlpy had rejected our request.
> Then we got the "SSLHandshakeException"s out of the blue in situations of
> very intense testing where we started and stopped tests runs over and over
> again. The oAuth token was still valid and other processors of the same kind
> and even exact copies of the now failing processor worked fine
> We discovered the following steps to reproduce the issue in our environment
> (Nifi 1.12.1 & 1.13.2):
> - create input data for the InvokeHTTP
> - make sure the contacted endpoint does not respond but have a high timeout
> setting in InvokeHttp to be able to terminate the processor
> - start InvokeHTTP
> - let it run for some time
> - stop the InvokeHTTP-processor
> - "Terminate" should be available in the context menu if there are still
> open requests waiting for an answer (or timeout)
> - terminate it
> - wait until all tasks are finally stopped
> - start the InvokeHTTP again
> "SSLHandshakeException" Errors should be visible now in the nifi-app.log on
> all machines where the invokehttp-processor had to terminate some tasks.
> They'll
> stay until you restart those nifi instances
> Our configuration
> - Cluster: 3 server, one nifi instance each
> - nifi in "http-mode" on port 8081 (nifi-properties)
> more details in the screenshots
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message
[ https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343263#comment-17343263 ] Andreas Klingenstein commented on NIFI-8333: I was able to reproduce the issue in 1.13.2. Same steps as mentioned above. About your comment _"The stack traces in the attachment indicate that the truststore configured for use with InvokeHTTP does not include a certificate authority for the HTTPS server being accessed"_. We get that error only +after+ we canceled the invokeHTTP processor with "Terminate Task". Before that it's working fine and we get results from the webservice for many thousand requests on all our serves. There seems to happen something when the invokeHTTP is terminated - but only on the serves(s) Nifi must really terminate it. > Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long > running transactions) and misleading error-message > -- > > Key: NIFI-8333 > URL: https://issues.apache.org/jira/browse/NIFI-8333 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.12.1, 1.13.2 >Reporter: Andreas Klingenstein >Priority: Major > Labels: https, security > Attachments: 1_config_invokeHttp_settings.png, > 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png, > 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png, > 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png, > 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt > > > There seems to be an issue with InvokeHTTP 1.12.1 or > StandardRestrictedSSLContextService 1.12.1 or both. > We observed a lot of "SSLHandshakeException"s (...) "unable to find valid > certification path to requested target" in some situations. > At first we had a very close look into our keystore which is used in our > StandardRestrictedSSLContextService and made sure everything was fine. In the > end we figured out we had used an older, no longer valid oAuth token and the > webservice simlpy had rejected our request. > Then we got the "SSLHandshakeException"s out of the blue in situations of > very intense testing where we started and stopped tests runs over and over > again. The oAuth token was still valid and other processors of the same kind > and even exact copies of the now failing processor worked fine > We discovered the following steps to reproduce the issue in our environment > (Nifi 1.12.1 & 1.13.2): > - create input data for the InvokeHTTP > - make sure the contacted endpoint does not respond but have a high timeout > setting in InvokeHttp to be able to terminate the processor > - start InvokeHTTP > - let it run for some time > - stop the InvokeHTTP-processor > - "Terminate" should be available in the context menu if there are still > open requests waiting for an answer (or timeout) > - terminate it > - wait until all tasks are finally stopped > - start the InvokeHTTP again > "SSLHandshakeException" Errors should be visible now in the nifi-app.log on > all machines where the invokehttp-processor had to terminate some tasks. > They'll > stay until you restart those nifi instances > Our configuration > - Cluster: 3 server, one nifi instance each > - nifi in "http-mode" on port 8081 (nifi-properties) > more details in the screenshots > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message
[ https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17334460#comment-17334460 ] Andreas Klingenstein commented on NIFI-8333: Not yet, that upgrade is still on my list. I'll check as soon as possible. > Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long > running transactions) and misleading error-message > -- > > Key: NIFI-8333 > URL: https://issues.apache.org/jira/browse/NIFI-8333 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.12.1 >Reporter: Andreas Klingenstein >Priority: Major > Labels: https, security > Attachments: 1_config_invokeHttp_settings.png, > 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png, > 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png, > 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png, > 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt > > > There seems to be an issue with InvokeHTTP 1.12.1 or > StandardRestrictedSSLContextService 1.12.1 or both. > We observed a lot of "SSLHandshakeException"s (...) "unable to find valid > certification path to requested target" in some situations. > At first we had a very close look into our keystore which is used in our > StandardRestrictedSSLContextService and made sure everything was fine. In the > end we figured out we had used an older, no longer valid oAuth token and the > webservice simlpy had rejected our request. > Then we got the "SSLHandshakeException"s out of the blue in situations of > very intense testing where we started and stopped tests runs over and over > again. The oAuth token was still valid and other processors of the same kind > and even exact copies of the now failing processor worked fine > We discovered the following steps to reproduce the issue in our environment > (Nifi 1.12.1): > - create input data for the InvokeHTTP > - make sure the contacted endpoint does not respond but have a high timeout > setting in InvokeHttp to be able to terminate the processor > - start InvokeHTTP > - let it run for some time > - stop the InvokeHTTP-processor > - "Terminate" should be available in the context menu if there are still > open requests waiting for an answer (or timeout) > - terminate it > - wait until all tasks are finally stopped > - start the InvokeHTTP again > "SSLHandshakeException" Errors should be visible now in the nifi-app.log on > all machines where the invokehttp-processor had to terminate some tasks. > They'll > stay until you restart those nifi instances > Our configuration > - Cluster: 3 server, one nifi instance each > - nifi in "http-mode" on port 8081 (nifi-properties) > more details in the screenshots > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8333) Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long running transactions) and misleading error-message
[ https://issues.apache.org/jira/browse/NIFI-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333244#comment-17333244 ] David Handermann commented on NIFI-8333: [~aklingens] Have you tried reproducing the problem on NiFi 1.13.2? Recent updates to InvokeHTTP include a much more recent version of OkHttp, which contains a number of improvements. The stack traces in the attachment indicate that the truststore configured for use with InvokeHTTP does not include a certificate authority for the HTTPS server being accessed. That error should occur with every request, not just sporadically. However, if there is some type of round-robin load balancing occurring on the server side, it could be possible that InvokeHTTP is being routed to a different server that has an invalid certificate. If you are able to test the scenario using NiFi 1.13.2, that would be helpful. > Issues with invokehttp/SSL Service after "Terminate Task" (e.g. due to long > running transactions) and misleading error-message > -- > > Key: NIFI-8333 > URL: https://issues.apache.org/jira/browse/NIFI-8333 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.12.1 >Reporter: Andreas Klingenstein >Priority: Major > Labels: https, security > Attachments: 1_config_invokeHttp_settings.png, > 2_config_invokeHttp_scheduling.png, 3_config_invokeHttp_config_1.png, > 4_config_invokeHttp_config_2.png, 5_config_invokeHttp_config_3.png, > 6_sslcontextservice_settings.png, 7_sslcontextservice_properties.png, > 8_config_invokeHttp_Terminate_Task.png, SSLHandshakeException.txt > > > There seems to be an issue with InvokeHTTP 1.12.1 or > StandardRestrictedSSLContextService 1.12.1 or both. > We observed a lot of "SSLHandshakeException"s (...) "unable to find valid > certification path to requested target" in some situations. > At first we had a very close look into our keystore which is used in our > StandardRestrictedSSLContextService and made sure everything was fine. In the > end we figured out we had used an older, no longer valid oAuth token and the > webservice simlpy had rejected our request. > Then we got the "SSLHandshakeException"s out of the blue in situations of > very intense testing where we started and stopped tests runs over and over > again. The oAuth token was still valid and other processors of the same kind > and even exact copies of the now failing processor worked fine > We discovered the following steps to reproduce the issue in our environment > (Nifi 1.12.1): > - create input data for the InvokeHTTP > - make sure the contacted endpoint does not respond but have a high timeout > setting in InvokeHttp to be able to terminate the processor > - start InvokeHTTP > - let it run for some time > - stop the InvokeHTTP-processor > - "Terminate" should be available in the context menu if there are still > open requests waiting for an answer (or timeout) > - terminate it > - wait until all tasks are finally stopped > - start the InvokeHTTP again > "SSLHandshakeException" Errors should be visible now in the nifi-app.log on > all machines where the invokehttp-processor had to terminate some tasks. > They'll > stay until you restart those nifi instances > Our configuration > - Cluster: 3 server, one nifi instance each > - nifi in "http-mode" on port 8081 (nifi-properties) > more details in the screenshots > -- This message was sent by Atlassian Jira (v8.3.4#803005)
