[
https://issues.apache.org/jira/browse/TS-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Sorber updated TS-2824:
----------------------------
Description:
During our production testing of ATS master, I noticed that ATS memory keeps
growing beyond ram cache size limits. Upon enabling the memory allocation dump,
it looks like the ioBufAllocator objects keep growing endlessly. Talking to
[~psudaemon] and [~amc] on irc, it looks like the issue might be caused due to
NOT in-use proxy allocator (local thread) objects accumulating until they reach
the per thread object count limit (250). This may be significant memory for
large sized objects (e.g 1MB or 2MB). [~psudaemon] recommended to try reverting
the changes in TS-2592 that force to use Proxy allocator instead of global pool
objects for ioBuffers. (Note that my production host has ram cache set to 6G
and that is completely used, but, the ioBuffers just keep growing)
Here's the dump of memory allocations:
{code}
-bash-4.1$ sudo grep "ioBufAllocator\[13\]" traffic.out
0 | 0 | 1048576 |
memory/ioBufAllocator[13]
1174405120 | 1151336448 | 1048576 |
memory/ioBufAllocator[13]
4194304000 | 4186963968 | 1048576 |
memory/ioBufAllocator[13]
1912602624 | 1908408320 | 1048576 |
memory/ioBufAllocator[13]
771751936 | 759169024 | 1048576 |
memory/ioBufAllocator[13]
1610612736 | 1608515584 | 1048576 |
memory/ioBufAllocator[13]
1509949440 | 1484783616 | 1048576 |
memory/ioBufAllocator[13]
3724541952 | 3697278976 | 1048576 |
memory/ioBufAllocator[13]
2046820352 | 2023751680 | 1048576 |
memory/ioBufAllocator[13]
2046820352 | 2036334592 | 1048576 |
memory/ioBufAllocator[13]
2147483648 | 2128609280 | 1048576 |
memory/ioBufAllocator[13]
2181038080 | 2172649472 | 1048576 |
memory/ioBufAllocator[13]
2214592512 | 2204106752 | 1048576 |
memory/ioBufAllocator[13]
2281701376 | 2253389824 | 1048576 |
memory/ioBufAllocator[13]
2315255808 | 2287992832 | 1048576 |
memory/ioBufAllocator[13]
2348810240 | 2316304384 | 1048576 |
memory/ioBufAllocator[13]
2348810240 | 2336227328 | 1048576 |
memory/ioBufAllocator[13]
2382364672 | 2380267520 | 1048576 |
memory/ioBufAllocator[13]
2449473536 | 2424307712 | 1048576 |
memory/ioBufAllocator[13]
2449473536 | 2447376384 | 1048576 |
memory/ioBufAllocator[13]
2483027968 | 2474639360 | 1048576 |
memory/ioBufAllocator[13]
2516582400 | 2507145216 | 1048576 |
memory/ioBufAllocator[13]
2550136832 | 2543845376 | 1048576 |
memory/ioBufAllocator[13]
2617245696 | 2599419904 | 1048576 |
memory/ioBufAllocator[13]
2650800128 | 2626682880 | 1048576 |
memory/ioBufAllocator[13]
2684354560 | 2659188736 | 1048576 |
memory/ioBufAllocator[13]
2684354560 | 2675965952 | 1048576 |
memory/ioBufAllocator[13]
2717908992 | 2690646016 | 1048576 |
memory/ioBufAllocator[13]
2717908992 | 2713714688 | 1048576 |
memory/ioBufAllocator[13]
2785017856 | 2758803456 | 1048576 |
memory/ioBufAllocator[13]
2818572288 | 2792357888 | 1048576 |
memory/ioBufAllocator[13]
2852126720 | 2823815168 | 1048576 |
memory/ioBufAllocator[13]
2885681152 | 2882535424 | 1048576 |
memory/ioBufAllocator[13]
2952790016 | 2921332736 | 1048576 |
memory/ioBufAllocator[13]
2986344448 | 2959081472 | 1048576 |
memory/ioBufAllocator[13]
2986344448 | 2985295872 | 1048576 |
memory/ioBufAllocator[13]
3019898880 | 3014656000 | 1048576 |
memory/ioBufAllocator[13]
3087007744 | 3067084800 | 1048576 |
memory/ioBufAllocator[13]
3120562176 | 3104833536 | 1048576 |
memory/ioBufAllocator[13]
3154116608 | 3146776576 | 1048576 |
memory/ioBufAllocator[13]
3187671040 | 3186622464 | 1048576 |
memory/ioBufAllocator[13]
3221225472 | 3220176896 | 1048576 |
memory/ioBufAllocator[13]
3288334336 | 3268411392 | 1048576 |
memory/ioBufAllocator[13]
3321888768 | 3292528640 | 1048576 |
memory/ioBufAllocator[13]
3355443200 | 3344957440 | 1048576 |
memory/ioBufAllocator[13]
3388997632 | 3364880384 | 1048576 |
memory/ioBufAllocator[13]
3388997632 | 3387949056 | 1048576 |
memory/ioBufAllocator[13]
3422552064 | 3404726272 | 1048576 |
memory/ioBufAllocator[13]
3456106496 | 3423600640 | 1048576 |
memory/ioBufAllocator[13]
Complete memory dump at some point:
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
201326592 | 174063616 | 2097152 |
memory/ioBufAllocator[14]
3623878656 | 3597664256 | 1048576 |
memory/ioBufAllocator[13]
1811939328 | 1811939328 | 524288 |
memory/ioBufAllocator[12]
1400897536 | 1400635392 | 262144 |
memory/ioBufAllocator[11]
1283457024 | 1281490944 | 131072 |
memory/ioBufAllocator[10]
1663041536 | 1542127616 | 65536 | memory/ioBufAllocator[9]
1693450240 | 1058897920 | 32768 | memory/ioBufAllocator[8]
725614592 | 635879424 | 16384 | memory/ioBufAllocator[7]
398196736 | 357244928 | 8192 | memory/ioBufAllocator[6]
856686592 | 833167360 | 4096 | memory/ioBufAllocator[5]
0 | 0 | 2048 | memory/ioBufAllocator[4]
131072 | 61440 | 1024 | memory/ioBufAllocator[3]
65536 | 24576 | 512 | memory/ioBufAllocator[2]
1638400 | 1135616 | 256 | memory/ioBufAllocator[1]
1081344 | 643072 | 128 | memory/ioBufAllocator[0]
4325376 | 250368 | 512 | memory/FetchSMAllocator
0 | 0 | 592 |
memory/ICPRequestCont_allocator
0 | 0 | 112 |
memory/ICPPeerReadContAllocator
0 | 0 | 432 |
memory/PeerReadDataAllocator
4096 | 0 | 32 |
memory/MIMEFieldSDKHandle
0 | 0 | 240 | memory/INKVConnAllocator
12288 | 1248 | 96 | memory/INKContAllocator
4096 | 32 | 32 | memory/apiHookAllocator
0 | 0 | 96 |
memory/prefetchLockHandlerAllocator
0 | 0 | 320 |
memory/PrefetchBlasterAllocator
0 | 0 | 208 |
memory/prefetchUrlBlaster
0 | 0 | 224 | memory/blasterUrlList
0 | 0 | 192 |
memory/prefetchUrlEntryAllocator
0 | 0 | 128 |
memory/socksProxyAllocator
0 | 0 | 144 | memory/ObjectReloadCont
16625664 | 13732048 | 656 |
memory/httpClientSessionAllocator
56150016 | 1593072 | 7696 | memory/httpSMAllocator
200704 | 62720 | 224 |
memory/httpServerSessionAllocator
0 | 0 | 48 |
memory/CacheLookupHttpConfigAllocator
0 | 0 | 7744 |
memory/httpUpdateSMAllocator
1892352 | 109312 | 224 |
memory/spdyRequestAllocator
1794048 | 1698432 | 192 |
memory/spdyClientSessionAllocator
0 | 0 | 128 | memory/RemapPluginsAlloc
0 | 0 | 48 |
memory/CongestRequestParamAllocator
0 | 0 | 144 |
memory/CongestionDBContAllocator
360448 | 1792 | 256 |
memory/httpCacheAltAllocator
41943040 | 20996096 | 2048 | memory/hdrStrHeap
58982400 | 23791616 | 2048 | memory/hdrHeap
0 | 0 | 112 |
memory/OneWayTunnelAllocator
589824 | 2304 | 2304 |
memory/hostDBContAllocator
203136 | 33856 | 33856 | memory/dnsBufAllocator
327680 | 0 | 1280 | memory/dnsEntryAllocator
0 | 0 | 16 |
memory/DNSRequestDataAllocator
0 | 0 | 48 |
memory/ClusterVConnectionCache::Entry
0 | 0 | 576 |
memory/cacheContAllocator
0 | 0 | 112 |
memory/inControlAllocator
0 | 0 | 112 |
memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 |
memory/clusterVCAllocator
0 | 0 | 48 | memory/evacuationKey
6144 | 0 | 48 | memory/cacheRemoveCont
208896 | 207552 | 96 | memory/evacuationBlock
9621504 | 6307616 | 928 | memory/cacheVConnection
1474560 | 941280 | 160 | memory/openDirEntry
0 | 0 | 96 |
memory/RamCacheCLFUSEntry
8986624 | 8785728 | 64 | memory/RamCacheLRUEntry
69396480 | 68240160 | 720 | memory/sslNetVCAllocator
0 | 0 | 128 |
memory/UDPIOEventAllocator
13473792 | 11788192 | 688 | memory/netVCAllocator
0 | 0 | 128 |
memory/udpReadContAllocator
0 | 0 | 160 |
memory/udpPacketAllocator
0 | 0 | 384 | memory/socksAllocator
14876672 | 12889408 | 64 | memory/ioBlockAllocator
17006592 | 15877632 | 48 | memory/ioDataAllocator
34836480 | 27895920 | 240 | memory/ioAllocator
11970560 | 11386880 | 80 | memory/mutexAllocator
3170304 | 1586976 | 96 | memory/eventAllocator
4456448 | 159744 | 1024 | memory/ArenaBlock
top command showing traffic_server taking 15g RES mem:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6274 nobody 20 0 19.9g 15g 8428 S 480.3 65.2 1798:44
/home/y/bin/traffic_server -M --httpport
80:fd=7,81:fd=8,82:fd=9,83:fd=10,84:fd=11,85:fd=12,90:fd=13,80:fd=14:ipv6,81:fd=15
{code}
was:
During our production testing of ATS master, I noticed that ATS memory keeps
growing beyond ram cache size limits. Upon enabling the memory allocation dump,
it looks like the ioBufAllocator objects keep growing endlessly. Talking to
[~psudaemon] and [~amc] on irc, it looks like the issue might be caused due to
NOT in-use proxy allocator (local thread) objects accumulating until they reach
the per thread object count limit (250). This may be significant memory for
large sized objects (e.g 1MB or 2MB). [~psudaemon] recommended to try reverting
the changes in TS2592 that force to use Proxy allocator instead of global pool
objects for ioBuffers. (Note that my production host has ram cache set to 6G
and that is completely used, but, the ioBuffers just keep growing)
Here's the dump of memory allocations:
{code}
-bash-4.1$ sudo grep "ioBufAllocator\[13\]" traffic.out
0 | 0 | 1048576 |
memory/ioBufAllocator[13]
1174405120 | 1151336448 | 1048576 |
memory/ioBufAllocator[13]
4194304000 | 4186963968 | 1048576 |
memory/ioBufAllocator[13]
1912602624 | 1908408320 | 1048576 |
memory/ioBufAllocator[13]
771751936 | 759169024 | 1048576 |
memory/ioBufAllocator[13]
1610612736 | 1608515584 | 1048576 |
memory/ioBufAllocator[13]
1509949440 | 1484783616 | 1048576 |
memory/ioBufAllocator[13]
3724541952 | 3697278976 | 1048576 |
memory/ioBufAllocator[13]
2046820352 | 2023751680 | 1048576 |
memory/ioBufAllocator[13]
2046820352 | 2036334592 | 1048576 |
memory/ioBufAllocator[13]
2147483648 | 2128609280 | 1048576 |
memory/ioBufAllocator[13]
2181038080 | 2172649472 | 1048576 |
memory/ioBufAllocator[13]
2214592512 | 2204106752 | 1048576 |
memory/ioBufAllocator[13]
2281701376 | 2253389824 | 1048576 |
memory/ioBufAllocator[13]
2315255808 | 2287992832 | 1048576 |
memory/ioBufAllocator[13]
2348810240 | 2316304384 | 1048576 |
memory/ioBufAllocator[13]
2348810240 | 2336227328 | 1048576 |
memory/ioBufAllocator[13]
2382364672 | 2380267520 | 1048576 |
memory/ioBufAllocator[13]
2449473536 | 2424307712 | 1048576 |
memory/ioBufAllocator[13]
2449473536 | 2447376384 | 1048576 |
memory/ioBufAllocator[13]
2483027968 | 2474639360 | 1048576 |
memory/ioBufAllocator[13]
2516582400 | 2507145216 | 1048576 |
memory/ioBufAllocator[13]
2550136832 | 2543845376 | 1048576 |
memory/ioBufAllocator[13]
2617245696 | 2599419904 | 1048576 |
memory/ioBufAllocator[13]
2650800128 | 2626682880 | 1048576 |
memory/ioBufAllocator[13]
2684354560 | 2659188736 | 1048576 |
memory/ioBufAllocator[13]
2684354560 | 2675965952 | 1048576 |
memory/ioBufAllocator[13]
2717908992 | 2690646016 | 1048576 |
memory/ioBufAllocator[13]
2717908992 | 2713714688 | 1048576 |
memory/ioBufAllocator[13]
2785017856 | 2758803456 | 1048576 |
memory/ioBufAllocator[13]
2818572288 | 2792357888 | 1048576 |
memory/ioBufAllocator[13]
2852126720 | 2823815168 | 1048576 |
memory/ioBufAllocator[13]
2885681152 | 2882535424 | 1048576 |
memory/ioBufAllocator[13]
2952790016 | 2921332736 | 1048576 |
memory/ioBufAllocator[13]
2986344448 | 2959081472 | 1048576 |
memory/ioBufAllocator[13]
2986344448 | 2985295872 | 1048576 |
memory/ioBufAllocator[13]
3019898880 | 3014656000 | 1048576 |
memory/ioBufAllocator[13]
3087007744 | 3067084800 | 1048576 |
memory/ioBufAllocator[13]
3120562176 | 3104833536 | 1048576 |
memory/ioBufAllocator[13]
3154116608 | 3146776576 | 1048576 |
memory/ioBufAllocator[13]
3187671040 | 3186622464 | 1048576 |
memory/ioBufAllocator[13]
3221225472 | 3220176896 | 1048576 |
memory/ioBufAllocator[13]
3288334336 | 3268411392 | 1048576 |
memory/ioBufAllocator[13]
3321888768 | 3292528640 | 1048576 |
memory/ioBufAllocator[13]
3355443200 | 3344957440 | 1048576 |
memory/ioBufAllocator[13]
3388997632 | 3364880384 | 1048576 |
memory/ioBufAllocator[13]
3388997632 | 3387949056 | 1048576 |
memory/ioBufAllocator[13]
3422552064 | 3404726272 | 1048576 |
memory/ioBufAllocator[13]
3456106496 | 3423600640 | 1048576 |
memory/ioBufAllocator[13]
Complete memory dump at some point:
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
201326592 | 174063616 | 2097152 |
memory/ioBufAllocator[14]
3623878656 | 3597664256 | 1048576 |
memory/ioBufAllocator[13]
1811939328 | 1811939328 | 524288 |
memory/ioBufAllocator[12]
1400897536 | 1400635392 | 262144 |
memory/ioBufAllocator[11]
1283457024 | 1281490944 | 131072 |
memory/ioBufAllocator[10]
1663041536 | 1542127616 | 65536 | memory/ioBufAllocator[9]
1693450240 | 1058897920 | 32768 | memory/ioBufAllocator[8]
725614592 | 635879424 | 16384 | memory/ioBufAllocator[7]
398196736 | 357244928 | 8192 | memory/ioBufAllocator[6]
856686592 | 833167360 | 4096 | memory/ioBufAllocator[5]
0 | 0 | 2048 | memory/ioBufAllocator[4]
131072 | 61440 | 1024 | memory/ioBufAllocator[3]
65536 | 24576 | 512 | memory/ioBufAllocator[2]
1638400 | 1135616 | 256 | memory/ioBufAllocator[1]
1081344 | 643072 | 128 | memory/ioBufAllocator[0]
4325376 | 250368 | 512 | memory/FetchSMAllocator
0 | 0 | 592 |
memory/ICPRequestCont_allocator
0 | 0 | 112 |
memory/ICPPeerReadContAllocator
0 | 0 | 432 |
memory/PeerReadDataAllocator
4096 | 0 | 32 |
memory/MIMEFieldSDKHandle
0 | 0 | 240 | memory/INKVConnAllocator
12288 | 1248 | 96 | memory/INKContAllocator
4096 | 32 | 32 | memory/apiHookAllocator
0 | 0 | 96 |
memory/prefetchLockHandlerAllocator
0 | 0 | 320 |
memory/PrefetchBlasterAllocator
0 | 0 | 208 |
memory/prefetchUrlBlaster
0 | 0 | 224 | memory/blasterUrlList
0 | 0 | 192 |
memory/prefetchUrlEntryAllocator
0 | 0 | 128 |
memory/socksProxyAllocator
0 | 0 | 144 | memory/ObjectReloadCont
16625664 | 13732048 | 656 |
memory/httpClientSessionAllocator
56150016 | 1593072 | 7696 | memory/httpSMAllocator
200704 | 62720 | 224 |
memory/httpServerSessionAllocator
0 | 0 | 48 |
memory/CacheLookupHttpConfigAllocator
0 | 0 | 7744 |
memory/httpUpdateSMAllocator
1892352 | 109312 | 224 |
memory/spdyRequestAllocator
1794048 | 1698432 | 192 |
memory/spdyClientSessionAllocator
0 | 0 | 128 | memory/RemapPluginsAlloc
0 | 0 | 48 |
memory/CongestRequestParamAllocator
0 | 0 | 144 |
memory/CongestionDBContAllocator
360448 | 1792 | 256 |
memory/httpCacheAltAllocator
41943040 | 20996096 | 2048 | memory/hdrStrHeap
58982400 | 23791616 | 2048 | memory/hdrHeap
0 | 0 | 112 |
memory/OneWayTunnelAllocator
589824 | 2304 | 2304 |
memory/hostDBContAllocator
203136 | 33856 | 33856 | memory/dnsBufAllocator
327680 | 0 | 1280 | memory/dnsEntryAllocator
0 | 0 | 16 |
memory/DNSRequestDataAllocator
0 | 0 | 48 |
memory/ClusterVConnectionCache::Entry
0 | 0 | 576 |
memory/cacheContAllocator
0 | 0 | 112 |
memory/inControlAllocator
0 | 0 | 112 |
memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 |
memory/clusterVCAllocator
0 | 0 | 48 | memory/evacuationKey
6144 | 0 | 48 | memory/cacheRemoveCont
208896 | 207552 | 96 | memory/evacuationBlock
9621504 | 6307616 | 928 | memory/cacheVConnection
1474560 | 941280 | 160 | memory/openDirEntry
0 | 0 | 96 |
memory/RamCacheCLFUSEntry
8986624 | 8785728 | 64 | memory/RamCacheLRUEntry
69396480 | 68240160 | 720 | memory/sslNetVCAllocator
0 | 0 | 128 |
memory/UDPIOEventAllocator
13473792 | 11788192 | 688 | memory/netVCAllocator
0 | 0 | 128 |
memory/udpReadContAllocator
0 | 0 | 160 |
memory/udpPacketAllocator
0 | 0 | 384 | memory/socksAllocator
14876672 | 12889408 | 64 | memory/ioBlockAllocator
17006592 | 15877632 | 48 | memory/ioDataAllocator
34836480 | 27895920 | 240 | memory/ioAllocator
11970560 | 11386880 | 80 | memory/mutexAllocator
3170304 | 1586976 | 96 | memory/eventAllocator
4456448 | 159744 | 1024 | memory/ArenaBlock
top command showing traffic_server taking 15g RES mem:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6274 nobody 20 0 19.9g 15g 8428 S 480.3 65.2 1798:44
/home/y/bin/traffic_server -M --httpport
80:fd=7,81:fd=8,82:fd=9,83:fd=10,84:fd=11,85:fd=12,90:fd=13,80:fd=14:ipv6,81:fd=15
{code}
> ioBufferAllocator objects in ATS master keep growing endlessly
> --------------------------------------------------------------
>
> Key: TS-2824
> URL: https://issues.apache.org/jira/browse/TS-2824
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
> Reporter: Sudheer Vinukonda
>
> During our production testing of ATS master, I noticed that ATS memory keeps
> growing beyond ram cache size limits. Upon enabling the memory allocation
> dump, it looks like the ioBufAllocator objects keep growing endlessly.
> Talking to [~psudaemon] and [~amc] on irc, it looks like the issue might be
> caused due to NOT in-use proxy allocator (local thread) objects accumulating
> until they reach the per thread object count limit (250). This may be
> significant memory for large sized objects (e.g 1MB or 2MB). [~psudaemon]
> recommended to try reverting the changes in TS-2592 that force to use Proxy
> allocator instead of global pool objects for ioBuffers. (Note that my
> production host has ram cache set to 6G and that is completely used, but, the
> ioBuffers just keep growing)
> Here's the dump of memory allocations:
> {code}
> -bash-4.1$ sudo grep "ioBufAllocator\[13\]" traffic.out
> 0 | 0 | 1048576 |
> memory/ioBufAllocator[13]
> 1174405120 | 1151336448 | 1048576 |
> memory/ioBufAllocator[13]
> 4194304000 | 4186963968 | 1048576 |
> memory/ioBufAllocator[13]
> 1912602624 | 1908408320 | 1048576 |
> memory/ioBufAllocator[13]
> 771751936 | 759169024 | 1048576 |
> memory/ioBufAllocator[13]
> 1610612736 | 1608515584 | 1048576 |
> memory/ioBufAllocator[13]
> 1509949440 | 1484783616 | 1048576 |
> memory/ioBufAllocator[13]
> 3724541952 | 3697278976 | 1048576 |
> memory/ioBufAllocator[13]
> 2046820352 | 2023751680 | 1048576 |
> memory/ioBufAllocator[13]
> 2046820352 | 2036334592 | 1048576 |
> memory/ioBufAllocator[13]
> 2147483648 | 2128609280 | 1048576 |
> memory/ioBufAllocator[13]
> 2181038080 | 2172649472 | 1048576 |
> memory/ioBufAllocator[13]
> 2214592512 | 2204106752 | 1048576 |
> memory/ioBufAllocator[13]
> 2281701376 | 2253389824 | 1048576 |
> memory/ioBufAllocator[13]
> 2315255808 | 2287992832 | 1048576 |
> memory/ioBufAllocator[13]
> 2348810240 | 2316304384 | 1048576 |
> memory/ioBufAllocator[13]
> 2348810240 | 2336227328 | 1048576 |
> memory/ioBufAllocator[13]
> 2382364672 | 2380267520 | 1048576 |
> memory/ioBufAllocator[13]
> 2449473536 | 2424307712 | 1048576 |
> memory/ioBufAllocator[13]
> 2449473536 | 2447376384 | 1048576 |
> memory/ioBufAllocator[13]
> 2483027968 | 2474639360 | 1048576 |
> memory/ioBufAllocator[13]
> 2516582400 | 2507145216 | 1048576 |
> memory/ioBufAllocator[13]
> 2550136832 | 2543845376 | 1048576 |
> memory/ioBufAllocator[13]
> 2617245696 | 2599419904 | 1048576 |
> memory/ioBufAllocator[13]
> 2650800128 | 2626682880 | 1048576 |
> memory/ioBufAllocator[13]
> 2684354560 | 2659188736 | 1048576 |
> memory/ioBufAllocator[13]
> 2684354560 | 2675965952 | 1048576 |
> memory/ioBufAllocator[13]
> 2717908992 | 2690646016 | 1048576 |
> memory/ioBufAllocator[13]
> 2717908992 | 2713714688 | 1048576 |
> memory/ioBufAllocator[13]
> 2785017856 | 2758803456 | 1048576 |
> memory/ioBufAllocator[13]
> 2818572288 | 2792357888 | 1048576 |
> memory/ioBufAllocator[13]
> 2852126720 | 2823815168 | 1048576 |
> memory/ioBufAllocator[13]
> 2885681152 | 2882535424 | 1048576 |
> memory/ioBufAllocator[13]
> 2952790016 | 2921332736 | 1048576 |
> memory/ioBufAllocator[13]
> 2986344448 | 2959081472 | 1048576 |
> memory/ioBufAllocator[13]
> 2986344448 | 2985295872 | 1048576 |
> memory/ioBufAllocator[13]
> 3019898880 | 3014656000 | 1048576 |
> memory/ioBufAllocator[13]
> 3087007744 | 3067084800 | 1048576 |
> memory/ioBufAllocator[13]
> 3120562176 | 3104833536 | 1048576 |
> memory/ioBufAllocator[13]
> 3154116608 | 3146776576 | 1048576 |
> memory/ioBufAllocator[13]
> 3187671040 | 3186622464 | 1048576 |
> memory/ioBufAllocator[13]
> 3221225472 | 3220176896 | 1048576 |
> memory/ioBufAllocator[13]
> 3288334336 | 3268411392 | 1048576 |
> memory/ioBufAllocator[13]
> 3321888768 | 3292528640 | 1048576 |
> memory/ioBufAllocator[13]
> 3355443200 | 3344957440 | 1048576 |
> memory/ioBufAllocator[13]
> 3388997632 | 3364880384 | 1048576 |
> memory/ioBufAllocator[13]
> 3388997632 | 3387949056 | 1048576 |
> memory/ioBufAllocator[13]
> 3422552064 | 3404726272 | 1048576 |
> memory/ioBufAllocator[13]
> 3456106496 | 3423600640 | 1048576 |
> memory/ioBufAllocator[13]
> Complete memory dump at some point:
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 201326592 | 174063616 | 2097152 |
> memory/ioBufAllocator[14]
> 3623878656 | 3597664256 | 1048576 |
> memory/ioBufAllocator[13]
> 1811939328 | 1811939328 | 524288 |
> memory/ioBufAllocator[12]
> 1400897536 | 1400635392 | 262144 |
> memory/ioBufAllocator[11]
> 1283457024 | 1281490944 | 131072 |
> memory/ioBufAllocator[10]
> 1663041536 | 1542127616 | 65536 |
> memory/ioBufAllocator[9]
> 1693450240 | 1058897920 | 32768 |
> memory/ioBufAllocator[8]
> 725614592 | 635879424 | 16384 |
> memory/ioBufAllocator[7]
> 398196736 | 357244928 | 8192 |
> memory/ioBufAllocator[6]
> 856686592 | 833167360 | 4096 |
> memory/ioBufAllocator[5]
> 0 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 131072 | 61440 | 1024 |
> memory/ioBufAllocator[3]
> 65536 | 24576 | 512 |
> memory/ioBufAllocator[2]
> 1638400 | 1135616 | 256 |
> memory/ioBufAllocator[1]
> 1081344 | 643072 | 128 |
> memory/ioBufAllocator[0]
> 4325376 | 250368 | 512 |
> memory/FetchSMAllocator
> 0 | 0 | 592 |
> memory/ICPRequestCont_allocator
> 0 | 0 | 112 |
> memory/ICPPeerReadContAllocator
> 0 | 0 | 432 |
> memory/PeerReadDataAllocator
> 4096 | 0 | 32 |
> memory/MIMEFieldSDKHandle
> 0 | 0 | 240 |
> memory/INKVConnAllocator
> 12288 | 1248 | 96 |
> memory/INKContAllocator
> 4096 | 32 | 32 |
> memory/apiHookAllocator
> 0 | 0 | 96 |
> memory/prefetchLockHandlerAllocator
> 0 | 0 | 320 |
> memory/PrefetchBlasterAllocator
> 0 | 0 | 208 |
> memory/prefetchUrlBlaster
> 0 | 0 | 224 | memory/blasterUrlList
> 0 | 0 | 192 |
> memory/prefetchUrlEntryAllocator
> 0 | 0 | 128 |
> memory/socksProxyAllocator
> 0 | 0 | 144 |
> memory/ObjectReloadCont
> 16625664 | 13732048 | 656 |
> memory/httpClientSessionAllocator
> 56150016 | 1593072 | 7696 | memory/httpSMAllocator
> 200704 | 62720 | 224 |
> memory/httpServerSessionAllocator
> 0 | 0 | 48 |
> memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 7744 |
> memory/httpUpdateSMAllocator
> 1892352 | 109312 | 224 |
> memory/spdyRequestAllocator
> 1794048 | 1698432 | 192 |
> memory/spdyClientSessionAllocator
> 0 | 0 | 128 |
> memory/RemapPluginsAlloc
> 0 | 0 | 48 |
> memory/CongestRequestParamAllocator
> 0 | 0 | 144 |
> memory/CongestionDBContAllocator
> 360448 | 1792 | 256 |
> memory/httpCacheAltAllocator
> 41943040 | 20996096 | 2048 | memory/hdrStrHeap
> 58982400 | 23791616 | 2048 | memory/hdrHeap
> 0 | 0 | 112 |
> memory/OneWayTunnelAllocator
> 589824 | 2304 | 2304 |
> memory/hostDBContAllocator
> 203136 | 33856 | 33856 | memory/dnsBufAllocator
> 327680 | 0 | 1280 |
> memory/dnsEntryAllocator
> 0 | 0 | 16 |
> memory/DNSRequestDataAllocator
> 0 | 0 | 48 |
> memory/ClusterVConnectionCache::Entry
> 0 | 0 | 576 |
> memory/cacheContAllocator
> 0 | 0 | 112 |
> memory/inControlAllocator
> 0 | 0 | 112 |
> memory/outControlAllocator
> 0 | 0 | 32 |
> memory/byteBankAllocator
> 0 | 0 | 592 |
> memory/clusterVCAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 6144 | 0 | 48 | memory/cacheRemoveCont
> 208896 | 207552 | 96 | memory/evacuationBlock
> 9621504 | 6307616 | 928 |
> memory/cacheVConnection
> 1474560 | 941280 | 160 | memory/openDirEntry
> 0 | 0 | 96 |
> memory/RamCacheCLFUSEntry
> 8986624 | 8785728 | 64 |
> memory/RamCacheLRUEntry
> 69396480 | 68240160 | 720 |
> memory/sslNetVCAllocator
> 0 | 0 | 128 |
> memory/UDPIOEventAllocator
> 13473792 | 11788192 | 688 | memory/netVCAllocator
> 0 | 0 | 128 |
> memory/udpReadContAllocator
> 0 | 0 | 160 |
> memory/udpPacketAllocator
> 0 | 0 | 384 | memory/socksAllocator
> 14876672 | 12889408 | 64 |
> memory/ioBlockAllocator
> 17006592 | 15877632 | 48 | memory/ioDataAllocator
> 34836480 | 27895920 | 240 | memory/ioAllocator
> 11970560 | 11386880 | 80 | memory/mutexAllocator
> 3170304 | 1586976 | 96 | memory/eventAllocator
> 4456448 | 159744 | 1024 | memory/ArenaBlock
> top command showing traffic_server taking 15g RES mem:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 6274 nobody 20 0 19.9g 15g 8428 S 480.3 65.2 1798:44
> /home/y/bin/traffic_server -M --httpport
> 80:fd=7,81:fd=8,82:fd=9,83:fd=10,84:fd=11,85:fd=12,90:fd=13,80:fd=14:ipv6,81:fd=15
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
