On 11/01/2019 21:07, Christopher Schultz wrote: > Mark, > > On 1/11/19 13:50, Mark Thomas wrote: >> On 11/01/2019 17:31, Christopher Schultz wrote: >>> Mark, >>> >>> On 1/11/19 12:23, Mark Thomas wrote: >>>> On 10/01/2019 19:55, Mark Thomas wrote: >>> >>>> <snip/> >>> >>>>> I've just tracked down one leak although it is a relatively >>>>> small one. Next steps are to fix that leak and then find the >>>>> next one. And then repeat. >>> >>>> Found it. >>> >>>> The leak impacted NIO and NIO2 when used with OpenSSL. >>> >>>> The bug is in Tomcat Native. I have a fix that I am currently >>>> testing. That fix should be in the next Tomcat Native release. >>> >>>> For those interested in the technical details, Tomcat Native >>>> allocates some Native memory to track various attributes of >>>> each connection. >>> >>>> In APR/Native this memory is allocated from a new APR memory >>>> pool that is associated with the connection. When the >>>> connection closes, the pool is destroyed and the memory freed. >>>> All is good. >>> >>>> In NIO[2]+OpenSSL the per connection memory was allocated from >>>> the APR memory pool associated with the Connector so the memory >>>> was not released until the Connector was stopped. The fix is to >>>> create a pool per connection and use that for the per >>>> connection allocations. With a fix to do that in place, memory >>>> use is now broadly static under extended TLS load. >>> >>> Nice work. >>> >>> I think you can file this under "stuff we knew might be a >>> problem" and was the reason we were considering >>> reference-counting, etc. to make sure we didn't leak >>> *connections*. Here, the connection was cleaned-up, but some of >>> the associated data wasn't being cleaned from a shared cache. Do >>> I have that right? > >> Yes. The cache in this case was a memory pool. Think of it as >> managed malloc. You can destroy pool and clean up all the memory >> allocated through it without having to call free on each of the >> individual allocations. It was a tcn_ssl_ctxt_t structure that >> wasn't being released. It is a collection of 4 pointers so it >> wasn't too big but one for every connection mounts up over time. > > Does this affect all versions of Tomcat where JSSE+OpenSSL are in use? > Sounds like it would.
Yes. The bug (and the fix) were entirely in Tomcat Native. On the plus side, it means only Tomcat Native needs to be updated to get the fix. Mark > >>> Is there a way, short of bouncing Tomcat, to flush that cache? >>> For example, bouncing or otherwise re-initializing the connector >>> or something like that? > >> Yes, but I'm not sure how much practical use it is. You need to >> unbind the server socket - the necessary clean-up happens >> AbstractEnpoint.unbind(). That will happen on Connector.stop() or >> Connector.destroy() depending on bindOnInit. The issue is that >> unbinding the socket is going to cause connections to be dropped. >> The end result will be similar to bouncing Tomcat although it >> should be quicker (no need to reload webapps). > > I was mostly asking on behalf of others. Some people may find it > easier to bounce their connectors periodically rather than upgrade > right away. > > -chris > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
