[ https://issues.apache.org/jira/browse/TS-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153365#comment-13153365 ]
Zhao Yongming commented on TS-1006: ----------------------------------- we have made some enhancement and the result looks like the following, and we are still tracking for bugs and this is the first try, we will submit then codes after it stable for our usage. {code} [Nov 19 11:25:42.351] Server {0x4257a940} WARNING: Document 644CE0FF truncated at 10485040 of 21289520, missing fragment 9DD38ED0 allocated | in-use | type size | free list name --------------------|--------------------|------------|---------------------------------- 67108864 | 0 | 2097152 | memory/cacheBufAllocator[14] 33554432 | 0 | 1048576 | memory/cacheBufAllocator[13] 33554432 | 1572864 | 524288 | memory/cacheBufAllocator[12] 33554432 | 6029312 | 262144 | memory/cacheBufAllocator[11] 37748736 | 17301504 | 131072 | memory/cacheBufAllocator[10] 35651584 | 32964608 | 65536 | memory/cacheBufAllocator[9] 70254592 | 64782336 | 32768 | memory/cacheBufAllocator[8] 55574528 | 46366720 | 16384 | memory/cacheBufAllocator[7] 28573696 | 22749184 | 8192 | memory/cacheBufAllocator[6] 11534336 | 8888320 | 4096 | memory/cacheBufAllocator[5] 0 | 0 | 2048 | memory/cacheBufAllocator[4] 0 | 0 | 1024 | memory/cacheBufAllocator[3] 0 | 0 | 512 | memory/cacheBufAllocator[2] 0 | 0 | 256 | memory/cacheBufAllocator[1] 0 | 0 | 128 | memory/cacheBufAllocator[0] 1572864 | 997376 | 2048 | memory/hdrStrHeap 2621440 | 1466368 | 2048 | memory/hdrHeap 262144 | 203776 | 1024 | memory/ArenaBlock dallocated | in-use | type size | free list name --------------------|--------------------|------------|---------------------------------- 0 | 0 | 2097152 | memory/ioBufAllocator[14] 0 | 0 | 1048576 | memory/ioBufAllocator[13] 0 | 0 | 524288 | memory/ioBufAllocator[12] 0 | 0 | 262144 | memory/ioBufAllocator[11] 0 | 0 | 131072 | memory/ioBufAllocator[10] 0 | 0 | 65536 | memory/ioBufAllocator[9] 7340032 | 6455296 | 32768 | memory/ioBufAllocator[8] 1048576 | 720896 | 16384 | memory/ioBufAllocator[7] 1310720 | 1048576 | 8192 | memory/ioBufAllocator[6] 1572864 | 929792 | 4096 | memory/ioBufAllocator[5] 0 | 0 | 2048 | memory/ioBufAllocator[4] 0 | 0 | 1024 | memory/ioBufAllocator[3] 0 | 0 | 512 | memory/ioBufAllocator[2] 0 | 0 | 256 | memory/ioBufAllocator[1] 0 | 0 | 128 | memory/ioBufAllocator[0] 0 | 0 | 592 | memory/ICPRequestCont_allocator 0 | 0 | 128 | memory/ICPPeerReadContAllocator 0 | 0 | 432 | memory/PeerReadDataAllocator 0 | 0 | 32 | memory/MIMEFieldSDKHandle 0 | 0 | 256 | memory/INKVConnAllocator 0 | 0 | 112 | memory/INKContAllocator 0 | 0 | 32 | memory/apiHookAllocator 0 | 0 | 304 | memory/FetchSMAllocator 0 | 0 | 80 | memory/prefetchLockHandlerAllocator 0 | 0 | 176 | memory/PrefetchBlasterAllocator 0 | 0 | 80 | memory/prefetchUrlBlaster 0 | 0 | 96 | memory/blasterUrlList 0 | 0 | 96 | memory/prefetchUrlEntryAllocator 0 | 0 | 128 | memory/socksProxyAllocator 0 | 0 | 160 | memory/ObjectReloadCont 155648 | 120992 | 608 | memory/httpClientSessionAllocator 26624 | 1456 | 208 | memory/httpServerSessionAllocator 2531328 | 1967712 | 9888 | memory/httpSMAllocator 0 | 0 | 32 | memory/CacheLookupHttpConfigAllocator 0 | 0 | 9936 | memory/httpUpdateSMAllocator 0 | 0 | 128 | memory/RemapPluginsAlloc 0 | 0 | 544 | memory/HCSMAllocator 0 | 0 | 48 | memory/VCEntryAllocator 0 | 0 | 96 | memory/HCEntryAllocator 0 | 0 | 80 | memory/HCHandlerAllocator 0 | 0 | 48 | memory/CongestRequestParamAllocator 0 | 0 | 128 | memory/CongestionDBContAllocator 53248 | 29328 | 208 | memory/httpCacheAltAllocator 0 | 0 | 128 | memory/OneWayTunnelAllocator 157696 | 1232 | 1232 | memory/hostDBContAllocator 0 | 0 | 17040 | memory/dnsBufAllocator 0 | 0 | 1264 | memory/dnsEntryAllocator 0 | 0 | 16 | memory/DNSRequestDataAllocator 0 | 0 | 1072 | memory/SRVAllocator 0 | 0 | 48 | memory/ClusterVConnectionCache::Entry 0 | 0 | 560 | memory/cacheContAllocator 0 | 0 | 112 | memory/inControlAllocator 0 | 0 | 128 | memory/outControlAllocator 0 | 0 | 32 | memory/byteBankAllocator 0 | 0 | 592 | memory/clusterVCAllocator 0 | 0 | 48 | memory/evacuationKey 0 | 0 | 64 | memory/cacheRemoveCont 258048 | 183168 | 96 | memory/evacuationBlock 3461120 | 2541760 | 1040 | memory/cacheVConnection 491520 | 337440 | 160 | memory/openDirEntry 0 | 0 | 64 | memory/RamCacheLRUEntry 2297856 | 2290272 | 96 | memory/RamCacheCLFUSEntry 507904 | 489056 | 992 | memory/netVCAllocator 0 | 0 | 128 | memory/udpReadContAllocator 0 | 0 | 144 | memory/udpWorkContinuationAllocator 0 | 0 | 160 | memory/udpPacketAllocator 0 | 0 | 320 | memory/socksAllocator 0 | 0 | 1120 | memory/sslNetVCAllocator 0 | 0 | 128 | memory/UDPIOEventAllocator 49152 | 36352 | 64 | memory/ioBlockAllocator 620544 | 528576 | 48 | memory/ioDataAllocator 92160 | 49440 | 240 | memory/ioAllocator 1992704 | 1973104 | 112 | memory/mutexAllocator 98304 | 75264 | 96 | memory/eventAllocator {code} > memory management, cut down memory waste ? > ------------------------------------------ > > Key: TS-1006 > URL: https://issues.apache.org/jira/browse/TS-1006 > Project: Traffic Server > Issue Type: Improvement > Components: Core > Affects Versions: 3.1.1 > Reporter: Zhao Yongming > Assignee: Zhao Yongming > Fix For: 3.1.3 > > > when we review the memory usage in the production, there is something > abnormal, ie, looks like TS take much memory than index data + common system > waste, and here is some memory dump result by set > "proxy.config.dump_mem_info_frequency" > 1, the one on a not so busy forwarding system: > physics memory: 32G > RAM cache: 22G > DISK: 6140 GB > average_object_size 64000 > {code} > allocated | in-use | type size | free list name > --------------------|--------------------|------------|---------------------------------- > 671088640 | 37748736 | 2097152 | > memory/ioBufAllocator[14] > 2248146944 | 2135949312 | 1048576 | > memory/ioBufAllocator[13] > 1711276032 | 1705508864 | 524288 | > memory/ioBufAllocator[12] > 1669332992 | 1667760128 | 262144 | > memory/ioBufAllocator[11] > 2214592512 | 2211840000 | 131072 | > memory/ioBufAllocator[10] > 2325741568 | 2323775488 | 65536 | > memory/ioBufAllocator[9] > 2091909120 | 2089123840 | 32768 | > memory/ioBufAllocator[8] > 1956642816 | 1956478976 | 16384 | > memory/ioBufAllocator[7] > 2094530560 | 2094071808 | 8192 | > memory/ioBufAllocator[6] > 356515840 | 355540992 | 4096 | > memory/ioBufAllocator[5] > 1048576 | 14336 | 2048 | > memory/ioBufAllocator[4] > 131072 | 0 | 1024 | > memory/ioBufAllocator[3] > 65536 | 0 | 512 | > memory/ioBufAllocator[2] > 32768 | 0 | 256 | > memory/ioBufAllocator[1] > 16384 | 0 | 128 | > memory/ioBufAllocator[0] > 0 | 0 | 576 | > memory/ICPRequestCont_allocator > 0 | 0 | 112 | > memory/ICPPeerReadContAllocator > 0 | 0 | 432 | > memory/PeerReadDataAllocator > 0 | 0 | 32 | > memory/MIMEFieldSDKHandle > 0 | 0 | 240 | > memory/INKVConnAllocator > 0 | 0 | 96 | > memory/INKContAllocator > 4096 | 0 | 32 | > memory/apiHookAllocator > 0 | 0 | 288 | > memory/FetchSMAllocator > 0 | 0 | 80 | > memory/prefetchLockHandlerAllocator > 0 | 0 | 176 | > memory/PrefetchBlasterAllocator > 0 | 0 | 80 | > memory/prefetchUrlBlaster > 0 | 0 | 96 | memory/blasterUrlList > 0 | 0 | 96 | > memory/prefetchUrlEntryAllocator > 0 | 0 | 128 | > memory/socksProxyAllocator > 0 | 0 | 144 | > memory/ObjectReloadCont > 3258368 | 576016 | 592 | > memory/httpClientSessionAllocator > 825344 | 139568 | 208 | > memory/httpServerSessionAllocator > 22597632 | 1284848 | 9808 | memory/httpSMAllocator > 0 | 0 | 32 | > memory/CacheLookupHttpConfigAllocator > 0 | 0 | 9856 | > memory/httpUpdateSMAllocator > 0 | 0 | 128 | > memory/RemapPluginsAlloc > 0 | 0 | 48 | > memory/CongestRequestParamAllocator > 0 | 0 | 128 | > memory/CongestionDBContAllocator > 5767168 | 704512 | 2048 | memory/hdrStrHeap > 18350080 | 1153024 | 2048 | memory/hdrHeap > 53248 | 2912 | 208 | > memory/httpCacheAltAllocator > 0 | 0 | 112 | > memory/OneWayTunnelAllocator > 157696 | 1232 | 1232 | > memory/hostDBContAllocator > 102240 | 17040 | 17040 | memory/dnsBufAllocator > 323584 | 0 | 1264 | > memory/dnsEntryAllocator > 0 | 0 | 16 | > memory/DNSRequestDataAllocator > 0 | 0 | 1072 | memory/SRVAllocator > 0 | 0 | 48 | > memory/ClusterVConnectionCache::Entry > 0 | 0 | 560 | > memory/cacheContAllocator > 0 | 0 | 112 | > memory/inControlAllocator > 0 | 0 | 112 | > memory/outControlAllocator > 0 | 0 | 32 | > memory/byteBankAllocator > 0 | 0 | 576 | > memory/clusterVCAllocator > 0 | 0 | 48 | memory/evacuationKey > 6144 | 0 | 48 | memory/cacheRemoveCont > 270336 | 262560 | 96 | memory/evacuationBlock > 4997120 | 3968416 | 976 | > memory/cacheVConnection > 798720 | 522080 | 160 | memory/openDirEntry > 0 | 0 | 64 | > memory/RamCacheLRUEntry > 56426496 | 56426304 | 96 | > memory/RamCacheCLFUSEntry > 9584640 | 6168000 | 960 | memory/netVCAllocator > 0 | 0 | 128 | > memory/udpReadContAllocator > 0 | 0 | 128 | > memory/udpWorkContinuationAllocator > 0 | 0 | 160 | > memory/udpPacketAllocator > 0 | 0 | 304 | memory/socksAllocator > 139264 | 68544 | 1088 | > memory/sslNetVCAllocator > 0 | 0 | 128 | > memory/UDPIOEventAllocator > 671744 | 115520 | 64 | > memory/ioBlockAllocator > 28305408 | 28301520 | 48 | memory/ioDataAllocator > 2273280 | 406320 | 240 | memory/ioAllocator > 1904640 | 1489920 | 80 | memory/mutexAllocator > 1105920 | 188544 | 96 | memory/eventAllocator > 2359296 | 129024 | 1024 | memory/ArenaBlock > {code} > this box will crash every 2days, so the memory waste may no that high > 2, our production reverse system: > physics memory: 16G > RAM cache: 8G > DISK: 1516 GB > average_object_size 16384 > and it run for a much long time: > {code} > allocated | in-use | type size | free list name > --------------------|--------------------|------------|---------------------------------- > 805306368 | 0 | 2097152 | > memory/ioBufAllocator[14] > 738197504 | 8388608 | 1048576 | > memory/ioBufAllocator[13] > 1258291200 | 46661632 | 524288 | > memory/ioBufAllocator[12] > 1300234240 | 183762944 | 262144 | > memory/ioBufAllocator[11] > 1170210816 | 466223104 | 131072 | > memory/ioBufAllocator[10] > 1790967808 | 1223426048 | 65536 | > memory/ioBufAllocator[9] > 2970615808 | 2601418752 | 32768 | > memory/ioBufAllocator[8] > 2067791872 | 2044608512 | 16384 | > memory/ioBufAllocator[7] > 1169424384 | 1169121280 | 8192 | > memory/ioBufAllocator[6] > 711458816 | 710463488 | 4096 | > memory/ioBufAllocator[5] > 1572864 | 0 | 2048 | > memory/ioBufAllocator[4] > 131072 | 0 | 1024 | > memory/ioBufAllocator[3] > 65536 | 0 | 512 | > memory/ioBufAllocator[2] > 32768 | 0 | 256 | > memory/ioBufAllocator[1] > 16384 | 0 | 128 | > memory/ioBufAllocator[0] > 0 | 0 | 576 | > memory/ICPRequestCont_allocator > 0 | 0 | 112 | > memory/ICPPeerReadContAllocator > 0 | 0 | 432 | > memory/PeerReadDataAllocator > 0 | 0 | 32 | > memory/MIMEFieldSDKHandle > 0 | 0 | 240 | > memory/INKVConnAllocator > 0 | 0 | 96 | > memory/INKContAllocator > 4096 | 0 | 32 | > memory/apiHookAllocator > 0 | 0 | 288 | > memory/FetchSMAllocator > 0 | 0 | 80 | > memory/prefetchLockHandlerAllocator > 0 | 0 | 176 | > memory/PrefetchBlasterAllocator > 0 | 0 | 80 | > memory/prefetchUrlBlaster > 0 | 0 | 96 | memory/blasterUrlList > 0 | 0 | 96 | > memory/prefetchUrlEntryAllocator > 0 | 0 | 128 | > memory/socksProxyAllocator > 0 | 0 | 144 | > memory/ObjectReloadCont > 1136640 | 125504 | 592 | > memory/httpClientSessionAllocator > 372736 | 27248 | 208 | > memory/httpServerSessionAllocator > 11317248 | 39296 | 9824 | memory/httpSMAllocator > 0 | 0 | 32 | > memory/CacheLookupHttpConfigAllocator > 0 | 0 | 9888 | > memory/httpUpdateSMAllocator > 0 | 0 | 128 | > memory/RemapPluginsAlloc > 0 | 0 | 512 | memory/HCSMAllocator > 0 | 0 | 48 | > memory/VCEntryAllocator > 0 | 0 | 96 | > memory/HCEntryAllocator > 0 | 0 | 64 | > memory/HCHandlerAllocator > 0 | 0 | 48 | > memory/CongestRequestParamAllocator > 0 | 0 | 128 | > memory/CongestionDBContAllocator > 6029312 | 643072 | 2048 | memory/hdrStrHeap > 7077888 | 657408 | 2048 | memory/hdrHeap > 26624 | 208 | 208 | > memory/httpCacheAltAllocator > 0 | 0 | 112 | > memory/OneWayTunnelAllocator > 630784 | 1232 | 1232 | > memory/hostDBContAllocator > 238560 | 17040 | 17040 | memory/dnsBufAllocator > 161792 | 0 | 1264 | > memory/dnsEntryAllocator > 0 | 0 | 16 | > memory/DNSRequestDataAllocator > 0 | 0 | 1072 | memory/SRVAllocator > 0 | 0 | 48 | > memory/ClusterVConnectionCache::Entry > 0 | 0 | 560 | > memory/cacheContAllocator > 0 | 0 | 112 | > memory/inControlAllocator > 0 | 0 | 112 | > memory/outControlAllocator > 0 | 0 | 32 | > memory/byteBankAllocator > 0 | 0 | 576 | > memory/clusterVCAllocator > 0 | 0 | 48 | memory/evacuationKey > 6144 | 0 | 48 | memory/cacheRemoveCont > 17006592 | 14972928 | 96 | memory/evacuationBlock > 1777664 | 759872 | 992 | > memory/cacheVConnection > 307200 | 111520 | 160 | memory/openDirEntry > 0 | 0 | 64 | > memory/RamCacheLRUEntry > 104275968 | 104274048 | 96 | > memory/RamCacheCLFUSEntry > 3440640 | 1819200 | 960 | memory/netVCAllocator > 0 | 0 | 128 | > memory/udpReadContAllocator > 0 | 0 | 128 | > memory/udpWorkContinuationAllocator > 0 | 0 | 160 | > memory/udpPacketAllocator > 0 | 0 | 304 | memory/socksAllocator > 0 | 0 | 1088 | > memory/sslNetVCAllocator > 0 | 0 | 128 | > memory/UDPIOEventAllocator > 237568 | 22528 | 64 | > memory/ioBlockAllocator > 26087424 | 26081904 | 48 | memory/ioDataAllocator > 890880 | 84240 | 240 | memory/ioAllocator > 1525760 | 1403440 | 80 | memory/mutexAllocator > 565248 | 129696 | 96 | memory/eventAllocator > 1179648 | 4096 | 1024 | memory/ArenaBlock > {code} > our team is working on the memory free issue, trying to improve the memory > management. and this a big project, the more input|comment the better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira