[
https://issues.apache.org/jira/browse/HTTPCORE-782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17980458#comment-17980458
]
Jason Mathison commented on HTTPCORE-782:
-----------------------------------------
We had another customer reoccurrence that allowed us to gather the debug
tracing you asked for. We could not gather the tracing from the start of
incident due to the overwhelming amount of logging that would require, but did
capture it for a couple of seconds while the issue was happening.
We got these exact messages 15,034 times each in about 7 seconds.
"TLS decrypt; encrypted remaining = 4380"
"TLS decrypt; status = OK; handshake = NEED_TASK; bytesConsumed = 0;
bytesProduced = 0; end of stream = false"
The endpoint is using TLS 1.2.
We have a deploy going out now that will short circuit based on no data being
decrypted, which does seem like it would stop this sort of runaway thread.
Would there be any use /way to try to trace out the delegated tasks that
"NEED_TASK" implies are being waited on?
> Occasional thread stuck in
> org.apache.hc.core5.reactor.ssl.SSLIOSession.decryptData
> -----------------------------------------------------------------------------------
>
> Key: HTTPCORE-782
> URL: https://issues.apache.org/jira/browse/HTTPCORE-782
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore
> Affects Versions: 5.3.4
> Reporter: Jason Mathison
> Priority: Blocker
> Attachments: Thread Stack Traces
>
>
> We have a service that is occasionally having a thread get stuck in a state
> where it consumes as much CPU as possible. The thread will create and release
> a huge number of objects without causing an overall memory leak.
> We gathered a series of stack traces from this happening on one of our
> production instances that indicates that the thread is stuck inside of the
> "while (inEncryptedBuf.hasRemaining()) { " in
> org.apache.hc.core5.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java)
> This issue has been happening a couple of times a week for several weeks. We
> believe that it is always happening against a specific endpoint and are
> attempting to gather more information about what this endpoint is doing.
> This issue started happening a few weeks ago when we were on a slightly older
> version of this library. We happened to update to the latest version of these
> libraries and this is continuing to happen. We have had no success in causing
> this issue to happen in our testing environments where we could gather a heap
> dump.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org
For additional commands, e-mail: dev-h...@hc.apache.org
