[
https://issues.apache.org/jira/browse/TS-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540840#comment-13540840
]
Yunkai Zhang edited comment on TS-1006 at 12/29/12 10:30 AM:
-------------------------------------------------------------
[~genext]
I'm not sure if there are memory leak in ClassAllocator, if yes, I'll try to
fix it.
But we can confirm that the original version of InkFreeList won't free memory
back to OS. So, let's improve InkFreeList firstly. And we should known that,
there is no perfect method to completely avoid memory fragment as long as we
allocate a big chunk and split it into small type_size. What we could do is to
reduce, not to avoid it.
Of couse, memory fragment will consume some of memory, but should not be too
big. Now, we have found that RamCacheLRU may lead to memory fragment, that's
[~kuotai]'s patch to fix.
So, I have two suggestions:
1) set a reasonable *ram_cache.size* and *max_overage* (we should leave some
space for memory fragment). If there are only memory fragment and no memory
leak, the total size should not increase infinitely. But in my experience, the
total size may increase for several days before it reach the top of memory
fragment.
2) wait for [~kuotai]'s patch, I'll send it soon, which will reduce memory
fragment caused by RamCacheLRU(CLFUS has the same problem but we haven't fix it
now, so I suggest you select LRU to test it).
was (Author: yunkai):
[~genext]
I'm not sure if there are memory leak in ClassAllocator, if yes, I'll try too
fix it.
But we can confirm that the original version of InkFreeList won't free memory
back to OS. So, let's improve InkFreeList firstly. And we should known that,
there is no perfect method to completely avoid memory fragment as long as we
allocate a big chunk and split it into small type_size. What we could do is to
reduce, not to avoid it.
Of couse, memory fragment will consume some of memory, but should not be too
big. Now, we have found that RamCacheLRU may lead to memory fragment, that's
[~kuotai]'s patch to fix.
So, I have two suggestions:
1) set a reasonable *ram_cache.size* and *max_overage* (we should leave some
space for memory fragment). If there are only memory fragment and no memory
leak, the total size should not increase infinitely. But in my experience, the
total size may increase for several days before it reach the top of memory
fragment.
2) wait for [~kuotai]'s patch, I'll send it soon, which will reduce memory
fragment caused by RamCacheLRU(CLFUS has the same problem but we haven't fit it
now, so I suggest you select LRU to test it).
> memory management, cut down memory waste ?
> ------------------------------------------
>
> Key: TS-1006
> URL: https://issues.apache.org/jira/browse/TS-1006
> Project: Traffic Server
> Issue Type: Improvement
> Components: Core
> Affects Versions: 3.1.1
> Reporter: Zhao Yongming
> Assignee: Bin Chen
> Fix For: 3.3.2
>
> Attachments: 0001-Allocator-optimize-InkFreeList-memory-pool.patch,
> 0002-Allocator-make-InkFreeList-memory-pool-configurable.patch,
> 0003-Allocator-store-InkChunkInfo-into-Chunk.patch,
> 0004-Allocator-optimize-alignment-size-to-avoid-mmap-fail.patch,
> 0005-Allocator-adjust-reclaiming-strategy-of-InkFreeList.patch,
> Memory-Usage-After-Introduced-New-Allocator.png, memusage.ods, memusage.ods
>
>
> when we review the memory usage in the production, there is something
> abnormal, ie, looks like TS take much memory than index data + common system
> waste, and here is some memory dump result by set
> "proxy.config.dump_mem_info_frequency"
> 1, the one on a not so busy forwarding system:
> physics memory: 32G
> RAM cache: 22G
> DISK: 6140 GB
> average_object_size 64000
> {code}
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 671088640 | 37748736 | 2097152 |
> memory/ioBufAllocator[14]
> 2248146944 | 2135949312 | 1048576 |
> memory/ioBufAllocator[13]
> 1711276032 | 1705508864 | 524288 |
> memory/ioBufAllocator[12]
> 1669332992 | 1667760128 | 262144 |
> memory/ioBufAllocator[11]
> 2214592512 | 2211840000 | 131072 |
> memory/ioBufAllocator[10]
> 2325741568 | 2323775488 | 65536 |
> memory/ioBufAllocator[9]
> 2091909120 | 2089123840 | 32768 |
> memory/ioBufAllocator[8]
> 1956642816 | 1956478976 | 16384 |
> memory/ioBufAllocator[7]
> 2094530560 | 2094071808 | 8192 |
> memory/ioBufAllocator[6]
> 356515840 | 355540992 | 4096 |
> memory/ioBufAllocator[5]
> 1048576 | 14336 | 2048 |
> memory/ioBufAllocator[4]
> 131072 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 65536 | 0 | 512 |
> memory/ioBufAllocator[2]
> 32768 | 0 | 256 |
> memory/ioBufAllocator[1]
> 16384 | 0 | 128 |
> memory/ioBufAllocator[0]
> 0 | 0 | 576 |
> memory/ICPRequestCont_allocator
> 0 | 0 | 112 |
> memory/ICPPeerReadContAllocator
> 0 | 0 | 432 |
> memory/PeerReadDataAllocator
> 0 | 0 | 32 |
> memory/MIMEFieldSDKHandle
> 0 | 0 | 240 |
> memory/INKVConnAllocator
> 0 | 0 | 96 |
> memory/INKContAllocator
> 4096 | 0 | 32 |
> memory/apiHookAllocator
> 0 | 0 | 288 |
> memory/FetchSMAllocator
> 0 | 0 | 80 |
> memory/prefetchLockHandlerAllocator
> 0 | 0 | 176 |
> memory/PrefetchBlasterAllocator
> 0 | 0 | 80 |
> memory/prefetchUrlBlaster
> 0 | 0 | 96 | memory/blasterUrlList
> 0 | 0 | 96 |
> memory/prefetchUrlEntryAllocator
> 0 | 0 | 128 |
> memory/socksProxyAllocator
> 0 | 0 | 144 |
> memory/ObjectReloadCont
> 3258368 | 576016 | 592 |
> memory/httpClientSessionAllocator
> 825344 | 139568 | 208 |
> memory/httpServerSessionAllocator
> 22597632 | 1284848 | 9808 | memory/httpSMAllocator
> 0 | 0 | 32 |
> memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 9856 |
> memory/httpUpdateSMAllocator
> 0 | 0 | 128 |
> memory/RemapPluginsAlloc
> 0 | 0 | 48 |
> memory/CongestRequestParamAllocator
> 0 | 0 | 128 |
> memory/CongestionDBContAllocator
> 5767168 | 704512 | 2048 | memory/hdrStrHeap
> 18350080 | 1153024 | 2048 | memory/hdrHeap
> 53248 | 2912 | 208 |
> memory/httpCacheAltAllocator
> 0 | 0 | 112 |
> memory/OneWayTunnelAllocator
> 157696 | 1232 | 1232 |
> memory/hostDBContAllocator
> 102240 | 17040 | 17040 | memory/dnsBufAllocator
> 323584 | 0 | 1264 |
> memory/dnsEntryAllocator
> 0 | 0 | 16 |
> memory/DNSRequestDataAllocator
> 0 | 0 | 1072 | memory/SRVAllocator
> 0 | 0 | 48 |
> memory/ClusterVConnectionCache::Entry
> 0 | 0 | 560 |
> memory/cacheContAllocator
> 0 | 0 | 112 |
> memory/inControlAllocator
> 0 | 0 | 112 |
> memory/outControlAllocator
> 0 | 0 | 32 |
> memory/byteBankAllocator
> 0 | 0 | 576 |
> memory/clusterVCAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 6144 | 0 | 48 | memory/cacheRemoveCont
> 270336 | 262560 | 96 | memory/evacuationBlock
> 4997120 | 3968416 | 976 |
> memory/cacheVConnection
> 798720 | 522080 | 160 | memory/openDirEntry
> 0 | 0 | 64 |
> memory/RamCacheLRUEntry
> 56426496 | 56426304 | 96 |
> memory/RamCacheCLFUSEntry
> 9584640 | 6168000 | 960 | memory/netVCAllocator
> 0 | 0 | 128 |
> memory/udpReadContAllocator
> 0 | 0 | 128 |
> memory/udpWorkContinuationAllocator
> 0 | 0 | 160 |
> memory/udpPacketAllocator
> 0 | 0 | 304 | memory/socksAllocator
> 139264 | 68544 | 1088 |
> memory/sslNetVCAllocator
> 0 | 0 | 128 |
> memory/UDPIOEventAllocator
> 671744 | 115520 | 64 |
> memory/ioBlockAllocator
> 28305408 | 28301520 | 48 | memory/ioDataAllocator
> 2273280 | 406320 | 240 | memory/ioAllocator
> 1904640 | 1489920 | 80 | memory/mutexAllocator
> 1105920 | 188544 | 96 | memory/eventAllocator
> 2359296 | 129024 | 1024 | memory/ArenaBlock
> {code}
> this box will crash every 2days, so the memory waste may no that high
> 2, our production reverse system:
> physics memory: 16G
> RAM cache: 8G
> DISK: 1516 GB
> average_object_size 16384
> and it run for a much long time:
> {code}
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 805306368 | 0 | 2097152 |
> memory/ioBufAllocator[14]
> 738197504 | 8388608 | 1048576 |
> memory/ioBufAllocator[13]
> 1258291200 | 46661632 | 524288 |
> memory/ioBufAllocator[12]
> 1300234240 | 183762944 | 262144 |
> memory/ioBufAllocator[11]
> 1170210816 | 466223104 | 131072 |
> memory/ioBufAllocator[10]
> 1790967808 | 1223426048 | 65536 |
> memory/ioBufAllocator[9]
> 2970615808 | 2601418752 | 32768 |
> memory/ioBufAllocator[8]
> 2067791872 | 2044608512 | 16384 |
> memory/ioBufAllocator[7]
> 1169424384 | 1169121280 | 8192 |
> memory/ioBufAllocator[6]
> 711458816 | 710463488 | 4096 |
> memory/ioBufAllocator[5]
> 1572864 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 131072 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 65536 | 0 | 512 |
> memory/ioBufAllocator[2]
> 32768 | 0 | 256 |
> memory/ioBufAllocator[1]
> 16384 | 0 | 128 |
> memory/ioBufAllocator[0]
> 0 | 0 | 576 |
> memory/ICPRequestCont_allocator
> 0 | 0 | 112 |
> memory/ICPPeerReadContAllocator
> 0 | 0 | 432 |
> memory/PeerReadDataAllocator
> 0 | 0 | 32 |
> memory/MIMEFieldSDKHandle
> 0 | 0 | 240 |
> memory/INKVConnAllocator
> 0 | 0 | 96 |
> memory/INKContAllocator
> 4096 | 0 | 32 |
> memory/apiHookAllocator
> 0 | 0 | 288 |
> memory/FetchSMAllocator
> 0 | 0 | 80 |
> memory/prefetchLockHandlerAllocator
> 0 | 0 | 176 |
> memory/PrefetchBlasterAllocator
> 0 | 0 | 80 |
> memory/prefetchUrlBlaster
> 0 | 0 | 96 | memory/blasterUrlList
> 0 | 0 | 96 |
> memory/prefetchUrlEntryAllocator
> 0 | 0 | 128 |
> memory/socksProxyAllocator
> 0 | 0 | 144 |
> memory/ObjectReloadCont
> 1136640 | 125504 | 592 |
> memory/httpClientSessionAllocator
> 372736 | 27248 | 208 |
> memory/httpServerSessionAllocator
> 11317248 | 39296 | 9824 | memory/httpSMAllocator
> 0 | 0 | 32 |
> memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 9888 |
> memory/httpUpdateSMAllocator
> 0 | 0 | 128 |
> memory/RemapPluginsAlloc
> 0 | 0 | 512 | memory/HCSMAllocator
> 0 | 0 | 48 |
> memory/VCEntryAllocator
> 0 | 0 | 96 |
> memory/HCEntryAllocator
> 0 | 0 | 64 |
> memory/HCHandlerAllocator
> 0 | 0 | 48 |
> memory/CongestRequestParamAllocator
> 0 | 0 | 128 |
> memory/CongestionDBContAllocator
> 6029312 | 643072 | 2048 | memory/hdrStrHeap
> 7077888 | 657408 | 2048 | memory/hdrHeap
> 26624 | 208 | 208 |
> memory/httpCacheAltAllocator
> 0 | 0 | 112 |
> memory/OneWayTunnelAllocator
> 630784 | 1232 | 1232 |
> memory/hostDBContAllocator
> 238560 | 17040 | 17040 | memory/dnsBufAllocator
> 161792 | 0 | 1264 |
> memory/dnsEntryAllocator
> 0 | 0 | 16 |
> memory/DNSRequestDataAllocator
> 0 | 0 | 1072 | memory/SRVAllocator
> 0 | 0 | 48 |
> memory/ClusterVConnectionCache::Entry
> 0 | 0 | 560 |
> memory/cacheContAllocator
> 0 | 0 | 112 |
> memory/inControlAllocator
> 0 | 0 | 112 |
> memory/outControlAllocator
> 0 | 0 | 32 |
> memory/byteBankAllocator
> 0 | 0 | 576 |
> memory/clusterVCAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 6144 | 0 | 48 | memory/cacheRemoveCont
> 17006592 | 14972928 | 96 | memory/evacuationBlock
> 1777664 | 759872 | 992 |
> memory/cacheVConnection
> 307200 | 111520 | 160 | memory/openDirEntry
> 0 | 0 | 64 |
> memory/RamCacheLRUEntry
> 104275968 | 104274048 | 96 |
> memory/RamCacheCLFUSEntry
> 3440640 | 1819200 | 960 | memory/netVCAllocator
> 0 | 0 | 128 |
> memory/udpReadContAllocator
> 0 | 0 | 128 |
> memory/udpWorkContinuationAllocator
> 0 | 0 | 160 |
> memory/udpPacketAllocator
> 0 | 0 | 304 | memory/socksAllocator
> 0 | 0 | 1088 |
> memory/sslNetVCAllocator
> 0 | 0 | 128 |
> memory/UDPIOEventAllocator
> 237568 | 22528 | 64 |
> memory/ioBlockAllocator
> 26087424 | 26081904 | 48 | memory/ioDataAllocator
> 890880 | 84240 | 240 | memory/ioAllocator
> 1525760 | 1403440 | 80 | memory/mutexAllocator
> 565248 | 129696 | 96 | memory/eventAllocator
> 1179648 | 4096 | 1024 | memory/ArenaBlock
> {code}
> our team is working on the memory free issue, trying to improve the memory
> management. and this a big project, the more input|comment the better.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
