Sudheer Vinukonda created TS-2824:
-------------------------------------
Summary: ioBufferAllocator objects in ATS master keep growing
endlessly
Key: TS-2824
URL: https://issues.apache.org/jira/browse/TS-2824
Project: Traffic Server
Issue Type: Bug
Components: Core
Reporter: Sudheer Vinukonda
During our production testing of ATS master, I noticed that ATS memory keeps
growing beyond ram cache size limits. Upon enabling the memory allocation dump,
it looks like the ioBufAllocator objects keep growing endlessly. Talking to
PSUdaemon and amc on irc, it looks like the issue might be caused due to NOT
in-use proxy allocator (local thread) objects keep accumulating until they
reach the per thread object count limit (250). This may be significant memory
for large sized objects (e.g 1MB or 2MB). PSUdaemon recommended to try
reverting the changes in TS2592 that force to use Proxy allocator instead of
global pool objects for ioBuffers. (Note that my production host has ram cache
set to 6G and that is completely used, but, the ioBuffers just keep growing)
Here's the dump of memory allocations:
{code}
-bash-4.1$ sudo grep "ioBufAllocator\[13\]" traffic.out
0 | 0 | 1048576 |
memory/ioBufAllocator[13]
1174405120 | 1151336448 | 1048576 |
memory/ioBufAllocator[13]
4194304000 | 4186963968 | 1048576 |
memory/ioBufAllocator[13]
1912602624 | 1908408320 | 1048576 |
memory/ioBufAllocator[13]
771751936 | 759169024 | 1048576 |
memory/ioBufAllocator[13]
1610612736 | 1608515584 | 1048576 |
memory/ioBufAllocator[13]
1509949440 | 1484783616 | 1048576 |
memory/ioBufAllocator[13]
3724541952 | 3697278976 | 1048576 |
memory/ioBufAllocator[13]
2046820352 | 2023751680 | 1048576 |
memory/ioBufAllocator[13]
2046820352 | 2036334592 | 1048576 |
memory/ioBufAllocator[13]
2147483648 | 2128609280 | 1048576 |
memory/ioBufAllocator[13]
2181038080 | 2172649472 | 1048576 |
memory/ioBufAllocator[13]
2214592512 | 2204106752 | 1048576 |
memory/ioBufAllocator[13]
2281701376 | 2253389824 | 1048576 |
memory/ioBufAllocator[13]
2315255808 | 2287992832 | 1048576 |
memory/ioBufAllocator[13]
2348810240 | 2316304384 | 1048576 |
memory/ioBufAllocator[13]
2348810240 | 2336227328 | 1048576 |
memory/ioBufAllocator[13]
2382364672 | 2380267520 | 1048576 |
memory/ioBufAllocator[13]
2449473536 | 2424307712 | 1048576 |
memory/ioBufAllocator[13]
2449473536 | 2447376384 | 1048576 |
memory/ioBufAllocator[13]
2483027968 | 2474639360 | 1048576 |
memory/ioBufAllocator[13]
2516582400 | 2507145216 | 1048576 |
memory/ioBufAllocator[13]
2550136832 | 2543845376 | 1048576 |
memory/ioBufAllocator[13]
2617245696 | 2599419904 | 1048576 |
memory/ioBufAllocator[13]
2650800128 | 2626682880 | 1048576 |
memory/ioBufAllocator[13]
2684354560 | 2659188736 | 1048576 |
memory/ioBufAllocator[13]
2684354560 | 2675965952 | 1048576 |
memory/ioBufAllocator[13]
2717908992 | 2690646016 | 1048576 |
memory/ioBufAllocator[13]
2717908992 | 2713714688 | 1048576 |
memory/ioBufAllocator[13]
2785017856 | 2758803456 | 1048576 |
memory/ioBufAllocator[13]
2818572288 | 2792357888 | 1048576 |
memory/ioBufAllocator[13]
2852126720 | 2823815168 | 1048576 |
memory/ioBufAllocator[13]
2885681152 | 2882535424 | 1048576 |
memory/ioBufAllocator[13]
2952790016 | 2921332736 | 1048576 |
memory/ioBufAllocator[13]
2986344448 | 2959081472 | 1048576 |
memory/ioBufAllocator[13]
2986344448 | 2985295872 | 1048576 |
memory/ioBufAllocator[13]
3019898880 | 3014656000 | 1048576 |
memory/ioBufAllocator[13]
3087007744 | 3067084800 | 1048576 |
memory/ioBufAllocator[13]
3120562176 | 3104833536 | 1048576 |
memory/ioBufAllocator[13]
3154116608 | 3146776576 | 1048576 |
memory/ioBufAllocator[13]
3187671040 | 3186622464 | 1048576 |
memory/ioBufAllocator[13]
3221225472 | 3220176896 | 1048576 |
memory/ioBufAllocator[13]
3288334336 | 3268411392 | 1048576 |
memory/ioBufAllocator[13]
3321888768 | 3292528640 | 1048576 |
memory/ioBufAllocator[13]
3355443200 | 3344957440 | 1048576 |
memory/ioBufAllocator[13]
3388997632 | 3364880384 | 1048576 |
memory/ioBufAllocator[13]
3388997632 | 3387949056 | 1048576 |
memory/ioBufAllocator[13]
3422552064 | 3404726272 | 1048576 |
memory/ioBufAllocator[13]
3456106496 | 3423600640 | 1048576 |
memory/ioBufAllocator[13]
Complete memory dump at some point:
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
201326592 | 174063616 | 2097152 |
memory/ioBufAllocator[14]
3623878656 | 3597664256 | 1048576 |
memory/ioBufAllocator[13]
1811939328 | 1811939328 | 524288 |
memory/ioBufAllocator[12]
1400897536 | 1400635392 | 262144 |
memory/ioBufAllocator[11]
1283457024 | 1281490944 | 131072 |
memory/ioBufAllocator[10]
1663041536 | 1542127616 | 65536 | memory/ioBufAllocator[9]
1693450240 | 1058897920 | 32768 | memory/ioBufAllocator[8]
725614592 | 635879424 | 16384 | memory/ioBufAllocator[7]
398196736 | 357244928 | 8192 | memory/ioBufAllocator[6]
856686592 | 833167360 | 4096 | memory/ioBufAllocator[5]
0 | 0 | 2048 | memory/ioBufAllocator[4]
131072 | 61440 | 1024 | memory/ioBufAllocator[3]
65536 | 24576 | 512 | memory/ioBufAllocator[2]
1638400 | 1135616 | 256 | memory/ioBufAllocator[1]
1081344 | 643072 | 128 | memory/ioBufAllocator[0]
4325376 | 250368 | 512 | memory/FetchSMAllocator
0 | 0 | 592 |
memory/ICPRequestCont_allocator
0 | 0 | 112 |
memory/ICPPeerReadContAllocator
0 | 0 | 432 |
memory/PeerReadDataAllocator
4096 | 0 | 32 |
memory/MIMEFieldSDKHandle
0 | 0 | 240 | memory/INKVConnAllocator
12288 | 1248 | 96 | memory/INKContAllocator
4096 | 32 | 32 | memory/apiHookAllocator
0 | 0 | 96 |
memory/prefetchLockHandlerAllocator
0 | 0 | 320 |
memory/PrefetchBlasterAllocator
0 | 0 | 208 |
memory/prefetchUrlBlaster
0 | 0 | 224 | memory/blasterUrlList
0 | 0 | 192 |
memory/prefetchUrlEntryAllocator
0 | 0 | 128 |
memory/socksProxyAllocator
0 | 0 | 144 | memory/ObjectReloadCont
16625664 | 13732048 | 656 |
memory/httpClientSessionAllocator
56150016 | 1593072 | 7696 | memory/httpSMAllocator
200704 | 62720 | 224 |
memory/httpServerSessionAllocator
0 | 0 | 48 |
memory/CacheLookupHttpConfigAllocator
0 | 0 | 7744 |
memory/httpUpdateSMAllocator
1892352 | 109312 | 224 |
memory/spdyRequestAllocator
1794048 | 1698432 | 192 |
memory/spdyClientSessionAllocator
0 | 0 | 128 | memory/RemapPluginsAlloc
0 | 0 | 48 |
memory/CongestRequestParamAllocator
0 | 0 | 144 |
memory/CongestionDBContAllocator
360448 | 1792 | 256 |
memory/httpCacheAltAllocator
41943040 | 20996096 | 2048 | memory/hdrStrHeap
58982400 | 23791616 | 2048 | memory/hdrHeap
0 | 0 | 112 |
memory/OneWayTunnelAllocator
589824 | 2304 | 2304 |
memory/hostDBContAllocator
203136 | 33856 | 33856 | memory/dnsBufAllocator
327680 | 0 | 1280 | memory/dnsEntryAllocator
0 | 0 | 16 |
memory/DNSRequestDataAllocator
0 | 0 | 48 |
memory/ClusterVConnectionCache::Entry
0 | 0 | 576 |
memory/cacheContAllocator
0 | 0 | 112 |
memory/inControlAllocator
0 | 0 | 112 |
memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 |
memory/clusterVCAllocator
0 | 0 | 48 | memory/evacuationKey
6144 | 0 | 48 | memory/cacheRemoveCont
208896 | 207552 | 96 | memory/evacuationBlock
9621504 | 6307616 | 928 | memory/cacheVConnection
1474560 | 941280 | 160 | memory/openDirEntry
0 | 0 | 96 |
memory/RamCacheCLFUSEntry
8986624 | 8785728 | 64 | memory/RamCacheLRUEntry
69396480 | 68240160 | 720 | memory/sslNetVCAllocator
0 | 0 | 128 |
memory/UDPIOEventAllocator
13473792 | 11788192 | 688 | memory/netVCAllocator
0 | 0 | 128 |
memory/udpReadContAllocator
0 | 0 | 160 |
memory/udpPacketAllocator
0 | 0 | 384 | memory/socksAllocator
14876672 | 12889408 | 64 | memory/ioBlockAllocator
17006592 | 15877632 | 48 | memory/ioDataAllocator
34836480 | 27895920 | 240 | memory/ioAllocator
11970560 | 11386880 | 80 | memory/mutexAllocator
3170304 | 1586976 | 96 | memory/eventAllocator
4456448 | 159744 | 1024 | memory/ArenaBlock
top command showing traffic_server taking 15g RES mem:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6274 nobody 20 0 19.9g 15g 8428 S 480.3 65.2 1798:44
/home/y/bin/traffic_server -M --httpport
80:fd=7,81:fd=8,82:fd=9,83:fd=10,84:fd=11,85:fd=12,90:fd=13,80:fd=14:ipv6,81:fd=15
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
