[
https://issues.apache.org/jira/browse/TS-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540739#comment-13540739
]
jaekyung oh commented on TS-1006:
---------------------------------
Hi Yunkai Zhang.
i'm afraid there is memory leak a little because the memory usage tends to
increase. Now traffic server uses 8.7G bytes. Maybe it will use 9G tomorrow.
we set ram_cache.size 8000000000, ram_cache_cutoff 6000000000
part of yesterday debug logs :
[2aac3c888700:40][ink_queue.cc:00616][F] 8599.70M t:12861 f:71 m:71
avg:70.9 M:12790 csbase:64 csize:64 tsize:4096 cbsize:266240
[2aac3c888700:40][ink_queue.cc:00623][-] 8599.70M t:12791 f:1 m:71
avg:70.9 M:12790 csbase:64 csize:64 tsize:4096 cbsize:266240
[2aac3c888700:41][ink_queue.cc:00631][M] 8599.70M t:15469 f:0 m:1
avg:6.3 M:15469 csbase:32 csize:32 tsize:8192 cbsize:266240
[2aac3c888700:41][ink_queue.cc:00634][+] 8599.70M t:15501 f:31 m:1
avg:6.3 M:15469 csbase:32 csize:32 tsize:8192 cbsize:266240
[2aac3c888700:01][ink_queue.cc:00631][M] 8599.70M t:117925 f:0 m:12
avg:45.7 M:117925 csbase:256 csize:278 tsize:88 cbsize:24576
[2aac3c888700:01][ink_queue.cc:00634][+] 8599.70M t:118203 f:277 m:12
avg:45.7 M:117925 csbase:256 csize:278 tsize:88 cbsize:24576
[2aac3c888700:40][ink_queue.cc:00631][M] 8599.70M t:12791 f:0 m:1
avg:9.1 M:12791 csbase:64 csize:64 tsize:4096 cbsize:266240
[2aac3c888700:40][ink_queue.cc:00634][+] 8599.70M t:12855 f:63 m:1
avg:9.1 M:12791 csbase:64 csize:64 tsize:4096 cbsize:266240
[2aac3c888700:42][ink_queue.cc:00616][F] 8599.70M t:22873 f:18 m:16
avg:16.1 M:22855 csbase:32 csize:32 tsize:16384 cbsize:528384
[2aac3c888700:42][ink_queue.cc:00623][-] 8599.70M t:22857 f:2 m:16
avg:16.1 M:22855 csbase:32 csize:32 tsize:16384 cbsize:528384
[2aac3c888700:43][ink_queue.cc:00616][F] 8599.70M t:4474 f:37 m:26
avg:31.2 M:4437 csbase:32 csize:31 tsize:32768 cbsize:1019904
[2aac3c888700:43][ink_queue.cc:00623][-] 8599.70M t:4443 f:6 m:26
avg:31.2 M:4437 csbase:32 csize:31 tsize:32768 cbsize:1019904
[2aac3c888700:47][ink_queue.cc:00616][F] 8599.70M t:264 f:2 m:1
avg:1.3 M:262 csbase:32 csize:1 tsize:524288 cbsize:528384
[2aac3c888700:47][ink_queue.cc:00623][-] 8599.19M t:263 f:1 m:1
avg:1.3 M:262 csbase:32 csize:1 tsize:524288 cbsize:528384
[2aac3c888700:46][ink_queue.cc:00631][M] 8599.19M t:733 f:0 m:0
avg:1.1 M:733 csbase:32 csize:3 tsize:262144 cbsize:790528
[2aac3c888700:46][ink_queue.cc:00634][+] 8599.19M t:736 f:2 m:0
avg:1.1 M:733 csbase:32 csize:3 tsize:262144 cbsize:790528
[2aac3c888700:27][ink_queue.cc:00616][F] 8599.19M t:157 f:157 m:152
avg:150.7 M:0 csbase:128 csize:129 tsize:2048 cbsize:266240
[2aac3c888700:27][ink_queue.cc:00623][-] 8598.94M t:7 f:7 m:152
avg:150.7 M:0 csbase:128 csize:129 tsize:2048 cbsize:266240
[2aac3c888700:41][ink_queue.cc:00616][F] 8598.94M t:15501 f:30 m:30
avg:29.8 M:15471 csbase:32 csize:32 tsize:8192 cbsize:266240
current last debug logs :
[2aac3c989700:44][ink_queue.cc:00616][F] 8710.25M t:1461 f:2 m:1
avg:1.3 M:1459 csbase:32 csize:15 tsize:65536 cbsize:987136
[2aac3c989700:44][ink_queue.cc:00623][-] 8710.25M t:1460 f:1 m:1
avg:1.3 M:1459 csbase:32 csize:15 tsize:65536 cbsize:987136
[2aac3c989700:45][ink_queue.cc:00616][F] 8710.25M t:1536 f:2 m:1
avg:0.8 M:1534 csbase:32 csize:7 tsize:131072 cbsize:921600
[2aac3c989700:45][ink_queue.cc:00623][-] 8710.25M t:1536 f:2 m:1
avg:0.8 M:1534 csbase:32 csize:7 tsize:131072 cbsize:921600
[2aac3c989700:47][ink_queue.cc:00616][F] 8710.25M t:93 f:3 m:2
avg:1.9 M:90 csbase:32 csize:1 tsize:524288 cbsize:528384
[2aac3c989700:47][ink_queue.cc:00623][-] 8709.75M t:92 f:2 m:2
avg:1.9 M:90 csbase:32 csize:1 tsize:524288 cbsize:528384
[2aac3c989700:41][ink_queue.cc:00631][M] 8709.75M t:18766 f:0 m:1
avg:0.3 M:18766 csbase:32 csize:32 tsize:8192 cbsize:266240
[2aac3c989700:41][ink_queue.cc:00634][+] 8710.00M t:18798 f:31 m:1
avg:0.3 M:18766 csbase:32 csize:32 tsize:8192 cbsize:266240
[2aac3c686700:42][ink_queue.cc:00616][F] 8710.00M t:26868 f:32 m:32
avg:30.6 M:26836 csbase:32 csize:32 tsize:16384 cbsize:528384
[2aac3c686700:42][ink_queue.cc:00623][-] 8710.00M t:26838 f:2 m:32
avg:30.6 M:26836 csbase:32 csize:32 tsize:16384 cbsize:528384
[2aac3c686700:01][ink_queue.cc:00631][M] 8710.00M t:141808 f:0 m:2
avg:6.2 M:141808 csbase:256 csize:278 tsize:88 cbsize:24576
[2aac3c686700:01][ink_queue.cc:00634][+] 8710.02M t:142086 f:277 m:2
avg:6.2 M:141808 csbase:256 csize:278 tsize:88 cbsize:24576
[2aac3ca8a700:42][ink_queue.cc:00631][M] 8710.02M t:27525 f:0 m:1
avg:0.4 M:27525 csbase:32 csize:32 tsize:16384 cbsize:528384
[2aac3ca8a700:42][ink_queue.cc:00634][+] 8710.02M t:27557 f:31 m:1
avg:0.4 M:27525 csbase:32 csize:32 tsize:16384 cbsize:528384
[2aac3c686700:41][ink_queue.cc:00616][F] 8710.02M t:18884 f:32 m:30
avg:30.1 M:18852 csbase:32 csize:32 tsize:8192 cbsize:266240
[2aac3c686700:41][ink_queue.cc:00623][-] 8710.02M t:18854 f:2 m:30
avg:30.1 M:18852 csbase:32 csize:32 tsize:8192 cbsize:266240
[2aac3c686700:42][ink_queue.cc:00631][M] 8710.02M t:26838 f:0 m:1
avg:1.8 M:26838 csbase:32 csize:32 tsize:16384 cbsize:528384
[2aac3c686700:42][ink_queue.cc:00634][+] 8710.02M t:26870 f:31 m:1
avg:1.8 M:26838 csbase:32 csize:32 tsize:16384 cbsize:528384
if traffic server re-use freed memory(including gather fragment into chunk) (i
think) it should keep memory usage around 8G. is there any factor affect memory
usage in config?
thanks and happy new year to everyone.
> memory management, cut down memory waste ?
> ------------------------------------------
>
> Key: TS-1006
> URL: https://issues.apache.org/jira/browse/TS-1006
> Project: Traffic Server
> Issue Type: Improvement
> Components: Core
> Affects Versions: 3.1.1
> Reporter: Zhao Yongming
> Assignee: Bin Chen
> Fix For: 3.3.2
>
> Attachments: 0001-Allocator-optimize-InkFreeList-memory-pool.patch,
> 0002-Allocator-make-InkFreeList-memory-pool-configurable.patch,
> 0003-Allocator-store-InkChunkInfo-into-Chunk.patch,
> 0004-Allocator-optimize-alignment-size-to-avoid-mmap-fail.patch,
> 0005-Allocator-adjust-reclaiming-strategy-of-InkFreeList.patch,
> Memory-Usage-After-Introduced-New-Allocator.png, memusage.ods, memusage.ods
>
>
> when we review the memory usage in the production, there is something
> abnormal, ie, looks like TS take much memory than index data + common system
> waste, and here is some memory dump result by set
> "proxy.config.dump_mem_info_frequency"
> 1, the one on a not so busy forwarding system:
> physics memory: 32G
> RAM cache: 22G
> DISK: 6140 GB
> average_object_size 64000
> {code}
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 671088640 | 37748736 | 2097152 |
> memory/ioBufAllocator[14]
> 2248146944 | 2135949312 | 1048576 |
> memory/ioBufAllocator[13]
> 1711276032 | 1705508864 | 524288 |
> memory/ioBufAllocator[12]
> 1669332992 | 1667760128 | 262144 |
> memory/ioBufAllocator[11]
> 2214592512 | 2211840000 | 131072 |
> memory/ioBufAllocator[10]
> 2325741568 | 2323775488 | 65536 |
> memory/ioBufAllocator[9]
> 2091909120 | 2089123840 | 32768 |
> memory/ioBufAllocator[8]
> 1956642816 | 1956478976 | 16384 |
> memory/ioBufAllocator[7]
> 2094530560 | 2094071808 | 8192 |
> memory/ioBufAllocator[6]
> 356515840 | 355540992 | 4096 |
> memory/ioBufAllocator[5]
> 1048576 | 14336 | 2048 |
> memory/ioBufAllocator[4]
> 131072 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 65536 | 0 | 512 |
> memory/ioBufAllocator[2]
> 32768 | 0 | 256 |
> memory/ioBufAllocator[1]
> 16384 | 0 | 128 |
> memory/ioBufAllocator[0]
> 0 | 0 | 576 |
> memory/ICPRequestCont_allocator
> 0 | 0 | 112 |
> memory/ICPPeerReadContAllocator
> 0 | 0 | 432 |
> memory/PeerReadDataAllocator
> 0 | 0 | 32 |
> memory/MIMEFieldSDKHandle
> 0 | 0 | 240 |
> memory/INKVConnAllocator
> 0 | 0 | 96 |
> memory/INKContAllocator
> 4096 | 0 | 32 |
> memory/apiHookAllocator
> 0 | 0 | 288 |
> memory/FetchSMAllocator
> 0 | 0 | 80 |
> memory/prefetchLockHandlerAllocator
> 0 | 0 | 176 |
> memory/PrefetchBlasterAllocator
> 0 | 0 | 80 |
> memory/prefetchUrlBlaster
> 0 | 0 | 96 | memory/blasterUrlList
> 0 | 0 | 96 |
> memory/prefetchUrlEntryAllocator
> 0 | 0 | 128 |
> memory/socksProxyAllocator
> 0 | 0 | 144 |
> memory/ObjectReloadCont
> 3258368 | 576016 | 592 |
> memory/httpClientSessionAllocator
> 825344 | 139568 | 208 |
> memory/httpServerSessionAllocator
> 22597632 | 1284848 | 9808 | memory/httpSMAllocator
> 0 | 0 | 32 |
> memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 9856 |
> memory/httpUpdateSMAllocator
> 0 | 0 | 128 |
> memory/RemapPluginsAlloc
> 0 | 0 | 48 |
> memory/CongestRequestParamAllocator
> 0 | 0 | 128 |
> memory/CongestionDBContAllocator
> 5767168 | 704512 | 2048 | memory/hdrStrHeap
> 18350080 | 1153024 | 2048 | memory/hdrHeap
> 53248 | 2912 | 208 |
> memory/httpCacheAltAllocator
> 0 | 0 | 112 |
> memory/OneWayTunnelAllocator
> 157696 | 1232 | 1232 |
> memory/hostDBContAllocator
> 102240 | 17040 | 17040 | memory/dnsBufAllocator
> 323584 | 0 | 1264 |
> memory/dnsEntryAllocator
> 0 | 0 | 16 |
> memory/DNSRequestDataAllocator
> 0 | 0 | 1072 | memory/SRVAllocator
> 0 | 0 | 48 |
> memory/ClusterVConnectionCache::Entry
> 0 | 0 | 560 |
> memory/cacheContAllocator
> 0 | 0 | 112 |
> memory/inControlAllocator
> 0 | 0 | 112 |
> memory/outControlAllocator
> 0 | 0 | 32 |
> memory/byteBankAllocator
> 0 | 0 | 576 |
> memory/clusterVCAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 6144 | 0 | 48 | memory/cacheRemoveCont
> 270336 | 262560 | 96 | memory/evacuationBlock
> 4997120 | 3968416 | 976 |
> memory/cacheVConnection
> 798720 | 522080 | 160 | memory/openDirEntry
> 0 | 0 | 64 |
> memory/RamCacheLRUEntry
> 56426496 | 56426304 | 96 |
> memory/RamCacheCLFUSEntry
> 9584640 | 6168000 | 960 | memory/netVCAllocator
> 0 | 0 | 128 |
> memory/udpReadContAllocator
> 0 | 0 | 128 |
> memory/udpWorkContinuationAllocator
> 0 | 0 | 160 |
> memory/udpPacketAllocator
> 0 | 0 | 304 | memory/socksAllocator
> 139264 | 68544 | 1088 |
> memory/sslNetVCAllocator
> 0 | 0 | 128 |
> memory/UDPIOEventAllocator
> 671744 | 115520 | 64 |
> memory/ioBlockAllocator
> 28305408 | 28301520 | 48 | memory/ioDataAllocator
> 2273280 | 406320 | 240 | memory/ioAllocator
> 1904640 | 1489920 | 80 | memory/mutexAllocator
> 1105920 | 188544 | 96 | memory/eventAllocator
> 2359296 | 129024 | 1024 | memory/ArenaBlock
> {code}
> this box will crash every 2days, so the memory waste may no that high
> 2, our production reverse system:
> physics memory: 16G
> RAM cache: 8G
> DISK: 1516 GB
> average_object_size 16384
> and it run for a much long time:
> {code}
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 805306368 | 0 | 2097152 |
> memory/ioBufAllocator[14]
> 738197504 | 8388608 | 1048576 |
> memory/ioBufAllocator[13]
> 1258291200 | 46661632 | 524288 |
> memory/ioBufAllocator[12]
> 1300234240 | 183762944 | 262144 |
> memory/ioBufAllocator[11]
> 1170210816 | 466223104 | 131072 |
> memory/ioBufAllocator[10]
> 1790967808 | 1223426048 | 65536 |
> memory/ioBufAllocator[9]
> 2970615808 | 2601418752 | 32768 |
> memory/ioBufAllocator[8]
> 2067791872 | 2044608512 | 16384 |
> memory/ioBufAllocator[7]
> 1169424384 | 1169121280 | 8192 |
> memory/ioBufAllocator[6]
> 711458816 | 710463488 | 4096 |
> memory/ioBufAllocator[5]
> 1572864 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 131072 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 65536 | 0 | 512 |
> memory/ioBufAllocator[2]
> 32768 | 0 | 256 |
> memory/ioBufAllocator[1]
> 16384 | 0 | 128 |
> memory/ioBufAllocator[0]
> 0 | 0 | 576 |
> memory/ICPRequestCont_allocator
> 0 | 0 | 112 |
> memory/ICPPeerReadContAllocator
> 0 | 0 | 432 |
> memory/PeerReadDataAllocator
> 0 | 0 | 32 |
> memory/MIMEFieldSDKHandle
> 0 | 0 | 240 |
> memory/INKVConnAllocator
> 0 | 0 | 96 |
> memory/INKContAllocator
> 4096 | 0 | 32 |
> memory/apiHookAllocator
> 0 | 0 | 288 |
> memory/FetchSMAllocator
> 0 | 0 | 80 |
> memory/prefetchLockHandlerAllocator
> 0 | 0 | 176 |
> memory/PrefetchBlasterAllocator
> 0 | 0 | 80 |
> memory/prefetchUrlBlaster
> 0 | 0 | 96 | memory/blasterUrlList
> 0 | 0 | 96 |
> memory/prefetchUrlEntryAllocator
> 0 | 0 | 128 |
> memory/socksProxyAllocator
> 0 | 0 | 144 |
> memory/ObjectReloadCont
> 1136640 | 125504 | 592 |
> memory/httpClientSessionAllocator
> 372736 | 27248 | 208 |
> memory/httpServerSessionAllocator
> 11317248 | 39296 | 9824 | memory/httpSMAllocator
> 0 | 0 | 32 |
> memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 9888 |
> memory/httpUpdateSMAllocator
> 0 | 0 | 128 |
> memory/RemapPluginsAlloc
> 0 | 0 | 512 | memory/HCSMAllocator
> 0 | 0 | 48 |
> memory/VCEntryAllocator
> 0 | 0 | 96 |
> memory/HCEntryAllocator
> 0 | 0 | 64 |
> memory/HCHandlerAllocator
> 0 | 0 | 48 |
> memory/CongestRequestParamAllocator
> 0 | 0 | 128 |
> memory/CongestionDBContAllocator
> 6029312 | 643072 | 2048 | memory/hdrStrHeap
> 7077888 | 657408 | 2048 | memory/hdrHeap
> 26624 | 208 | 208 |
> memory/httpCacheAltAllocator
> 0 | 0 | 112 |
> memory/OneWayTunnelAllocator
> 630784 | 1232 | 1232 |
> memory/hostDBContAllocator
> 238560 | 17040 | 17040 | memory/dnsBufAllocator
> 161792 | 0 | 1264 |
> memory/dnsEntryAllocator
> 0 | 0 | 16 |
> memory/DNSRequestDataAllocator
> 0 | 0 | 1072 | memory/SRVAllocator
> 0 | 0 | 48 |
> memory/ClusterVConnectionCache::Entry
> 0 | 0 | 560 |
> memory/cacheContAllocator
> 0 | 0 | 112 |
> memory/inControlAllocator
> 0 | 0 | 112 |
> memory/outControlAllocator
> 0 | 0 | 32 |
> memory/byteBankAllocator
> 0 | 0 | 576 |
> memory/clusterVCAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 6144 | 0 | 48 | memory/cacheRemoveCont
> 17006592 | 14972928 | 96 | memory/evacuationBlock
> 1777664 | 759872 | 992 |
> memory/cacheVConnection
> 307200 | 111520 | 160 | memory/openDirEntry
> 0 | 0 | 64 |
> memory/RamCacheLRUEntry
> 104275968 | 104274048 | 96 |
> memory/RamCacheCLFUSEntry
> 3440640 | 1819200 | 960 | memory/netVCAllocator
> 0 | 0 | 128 |
> memory/udpReadContAllocator
> 0 | 0 | 128 |
> memory/udpWorkContinuationAllocator
> 0 | 0 | 160 |
> memory/udpPacketAllocator
> 0 | 0 | 304 | memory/socksAllocator
> 0 | 0 | 1088 |
> memory/sslNetVCAllocator
> 0 | 0 | 128 |
> memory/UDPIOEventAllocator
> 237568 | 22528 | 64 |
> memory/ioBlockAllocator
> 26087424 | 26081904 | 48 | memory/ioDataAllocator
> 890880 | 84240 | 240 | memory/ioAllocator
> 1525760 | 1403440 | 80 | memory/mutexAllocator
> 565248 | 129696 | 96 | memory/eventAllocator
> 1179648 | 4096 | 1024 | memory/ArenaBlock
> {code}
> our team is working on the memory free issue, trying to improve the memory
> management. and this a big project, the more input|comment the better.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
