[
https://issues.apache.org/jira/browse/TS-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554377#comment-14554377
]
Susan Hinrichs edited comment on TS-3554 at 5/21/15 2:59 PM:
-------------------------------------------------------------
Summary of investigations from yesterday.
[~reveller] reports 6GB increase in memory usage on each reload of his 6K long
ssl_mulitcert.config file.
I experimented with a 1K long ssl_multicert.config file and 5.3.x with the
limit_session_cache_alloc.diff and the ts-3554-53-2.diff patchs. I also made
more changes to enable the gperf tools heapchecker
(http://gperftools.googlecode.com/svn/trunk/doc/heap_checker.html) around the
ssl_multicert.config reload logic. Ran most of my tests with session caching
turned off.
If I reloaded without any traffic passing through, the heapchecker reported no
leaks and the VmSize increased by about 20KB sometimes.
If I passed traffic through without any reloads, the VmSize stayed stable.
This wasn't a huge traffic load. I ran "ab" with 4-6 concurrency to a constant
URL.
However, if I reloaded and passed traffic, I saw a VmSize increase of 25-30 MB
per reload. Still no real memory leaks reported by the heap checker, but the
number of "live" objects reported increased with each reload. So I assume that
the "leak" is objects left in a table somewhere (perhaps in openssl?).
So if we both reload and have some traffic passing through, I see a memory size
increase. Not as big as [~reveller] reported, but substantial. He also claims
that there was no traffic going through. But perhaps there was a little bit of
traffic passing through but not the full production load?
Next step is to figure what theses "leaked" but live objects are.
was (Author: shinrich):
Summary of investigations from yesterday.
[~reveller] reports 6GB increase in memory usage on each reload of his 6K long
ssl_mulitcert.config file.
I experimented with a 1K long ssl_multicert.config file and 5.3.x with the
limit_session_cache_alloc.diff and the ts-3554-53-2.diff patchs. I also made
more changes to enable the gperf tools heapchecker
(http://gperftools.googlecode.com/svn/trunk/doc/heap_checker.html) around the
ssl_multicert.config reload logic.
If I reloaded without any traffic passing through, the heapchecker reported no
leaks and the VmSize increased by about 20KB sometimes.
If I passed traffic through without any reloads, the VmSize stayed stable.
This wasn't a huge traffic load. I ran "ab" with 4-6 concurrency to a constant
URL.
However, if I reloaded and passed traffic, I saw a VmSize increase of 25-30 MB
per reload. Still no real memory leaks reported by the heap checker, but the
number of "live" objects reported increased with each reload. So I assume that
the "leak" is objects left in a table somewhere (perhaps in openssl?).
So if we both reload and have some traffic passing through, I see a memory size
increase. Not as big as [~reveller] reported, but substantial. He also claims
that there was no traffic going through. But perhaps there was a little bit of
traffic passing through but not the full production load?
Next step is to figure what theses "leaked" but live objects are.
> ATS memory leak reloading ssl_multicert.config with many ssl cert configs
> -------------------------------------------------------------------------
>
> Key: TS-3554
> URL: https://issues.apache.org/jira/browse/TS-3554
> Project: Traffic Server
> Issue Type: Bug
> Components: Configuration, Core, SSL
> Reporter: Steven Feltner
> Assignee: Susan Hinrichs
> Fix For: 6.0.0
>
> Attachments: limit_session_cache_alloc.diff, ts-3554-53-2.diff,
> ts-3554-53.diff
>
>
> ATS will consume all available memory on a server with 128GB of RAM.
> @shinrich suspects it may be due to CertLookup table not being freed on a
> config reload.
> Our current process:
> - New cert comes in
> - ssl_multicert.config and remap.config updated
> - traffic_line -x
> This reload could occur as often as every 3 mins with 5000+ certs configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
