[ https://issues.apache.org/jira/browse/TS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
taoyunxing updated TS-4246: --------------------------- Fix Version/s: 6.1.2 > ATS possible memory leak in CacheVC > ----------------------------------- > > Key: TS-4246 > URL: https://issues.apache.org/jira/browse/TS-4246 > Project: Traffic Server > Issue Type: Bug > Components: Cache > Reporter: taoyunxing > Fix For: 6.1.2 > > > CPU:40Core, Mem:132GB, Disk:300 data + 11* 2T, Network Card: 4*GB bond, ATS: > 5.3.1 or 6.1.1 > Recently I found possible memory leak in production host with ATS version > from 5.3.1 to 6.1.1, the memory tracker log is as following: > -------------------------------------------------------------------------------------------- > allocated | in-use | type size | free list name > --------------------|--------------------|------------|---------------------------------- > 67108864 | 33554432 | 2097152 | > memory/ioBufAllocator[14] > 41271951360 | 40564162560 | 1048576 | > memory/ioBufAllocator[13] > 570425344 | 566755328 | 524288 | > memory/ioBufAllocator[12] > 276824064 | 270794752 | 262144 | > memory/ioBufAllocator[11] > 155189248 | 150994944 | 131072 | > memory/ioBufAllocator[10] > 113246208 | 111607808 | 65536 | > memory/ioBufAllocator[9] > 1319108608 | 1278607360 | 32768 | > memory/ioBufAllocator[8] > 39845888 | 39501824 | 16384 | > memory/ioBufAllocator[7] > 78381056 | 55869440 | 8192 | > memory/ioBufAllocator[6] > 228589568 | 212971520 | 4096 | > memory/ioBufAllocator[5] > 262144 | 0 | 2048 | > memory/ioBufAllocator[4] > 131072 | 0 | 1024 | > memory/ioBufAllocator[3] > 65536 | 0 | 512 | > memory/ioBufAllocator[2] > 163840 | 256 | 256 | > memory/ioBufAllocator[1] > 1785856 | 768 | 128 | > memory/ioBufAllocator[0] > 0 | 0 | 512 | > memory/FetchSMAllocator > 0 | 0 | 592 | > memory/ICPRequestCont_allocator > 0 | 0 | 128 | > memory/ICPPeerReadContAllocator > 0 | 0 | 432 | > memory/PeerReadDataAllocator > 0 | 0 | 32 | > memory/MIMEFieldSDKHandle > 0 | 0 | 240 | > memory/INKVConnAllocator > 0 | 0 | 96 | > memory/INKContAllocator > 503808 | 492864 | 32 | > memory/apiHookAllocator > 0 | 0 | 128 | > memory/socksProxyAllocator > 17211392 | 17169856 | 704 | > memory/httpClientSessionAllocator > 180633600 | 170085888 | 8064 | memory/httpSMAllocator > 2236416 | 2229024 | 224 | > memory/httpServerSessionAllocator > 0 | 0 | 48 | > memory/CacheLookupHttpConfigAllocator > 0 | 0 | 848 | > memory/http2ClientSessionAllocator > 0 | 0 | 128 | > memory/RemapPluginsAlloc > 0 | 0 | 48 | > memory/CongestRequestParamAllocator > 0 | 0 | 160 | > memory/CongestionDBContAllocator > 4325376 | 4245248 | 256 | > memory/httpCacheAltAllocator > 155451392 | 146763776 | 2048 | memory/hdrStrHeap > 269221888 | 253577216 | 2048 | memory/hdrHeap > 0 | 0 | 128 | > memory/OneWayTunnelAllocator > 0 | 0 | 96 | > memory/hostDBFileContAllocator > 296960 | 4640 | 2320 | > memory/hostDBContAllocator > 135424 | 33856 | 33856 | memory/dnsBufAllocator > 163840 | 1280 | 1280 | > memory/dnsEntryAllocator > 0 | 0 | 16 | > memory/DNSRequestDataAllocator > 0 | 0 | 112 | > memory/inControlAllocator > 0 | 0 | 128 | > memory/outControlAllocator > 0 | 0 | 32 | > memory/byteBankAllocator > 0 | 0 | 592 | > memory/clusterVCAllocator > 0 | 0 | 48 | > memory/ClusterVConnectionCache::Entry > 0 | 0 | 576 | > memory/cacheContAllocator > 0 | 0 | 48 | memory/evacuationKey > 0 | 0 | 64 | memory/cacheRemoveCont > 380928 | 380928 | 96 | memory/evacuationBlock > 20003360 | 19959936 | 944 | > memory/cacheVConnection > 163840 | 156960 | 160 | memory/openDirEntry > 2260992 | 2253408 | 96 | > memory/RamCacheCLFUSEntry > 0 | 0 | 64 | > memory/RamCacheLRUEntry > 0 | 0 | 880 | > memory/sslNetVCAllocator > 0 | 0 | 128 | > memory/UDPIOEventAllocator > 24399872 | 23407008 | 736 | memory/netVCAllocator > 0 | 0 | 128 | > memory/udpReadContAllocator > 0 | 0 | 160 | > memory/udpPacketAllocator > 0 | 0 | 416 | memory/socksAllocator > 7053312 | 7051072 | 64 | > memory/ioBlockAllocator > 7441920 | 7440432 | 48 | memory/ioDataAllocator > 19355520 | 19350000 | 240 | memory/ioAllocator > 3402720 | 3306880 | 80 | memory/mutexAllocator > 147456 | 103680 | 96 | memory/eventAllocator > 22937600 | 21599232 | 1024 | memory/ArenaBlock > 6206100608 | 5329728512 | | TOTAL > Allocs | Frees | Size In-use | Avg Size | Location > > -----------|------------|----------------------|------------|-------------------------------------------------------------------- > 39443091 | 39399441 | 178790400 | 4096 | > memory/IOBuffer/ProtocolProbeSessionAccept.cc:66 > 61 | 61 | 0 | 0 | > memory/IOBuffer/Transform.cc:791 > 4064 | 4064 | 0 | 0 | > memory/IOBuffer/HttpClientSession.cc:211 > 10271910 | 10270634 | 23633920 | 18521 | > memory/IOBuffer/HttpSM.cc:5831 > 261122 | 261122 | 0 | 0 | > memory/IOBuffer/HttpSM.cc:5880 > 14785 | 14785 | 0 | 0 | > memory/IOBuffer/HttpSM.cc:5401 > 524803 | 494019 | 1008730112 | 32768 | > memory/IOBuffer/HttpSM.cc:6199 > 7365218 | 7365218 | 0 | 0 | > memory/IOBuffer/HttpSM.cc:6021 > 11827313 | 11827056 | 1052672 | 4096 | > memory/IOBuffer/HttpSM.cc:5750 > 51960076 | 51954854 | 170999808 | 32746 | > memory/IOBuffer/HttpSM.cc:6302 > 1156 | 1156 | 0 | 0 | > memory/IOBuffer/HttpSM.cc:6269 > 8806600 | 8803759 | 23273472 | 8192 | > memory/IOBuffer/HttpServerSession.cc:86 > 708080 | 708074 | 768 | 128 | > memory/IOBuffer/HttpTunnel.cc:106 > 34649 | 34648 | 256 | 256 | > memory/IOBuffer/HttpTunnel.cc:111 > 8493 | 8101 | 108728320 | 277368 | > memory/IOBuffer/Cache.cc:2476 > 31232454 | 31172959 | 41769619456 | 702069 | > memory/IOBuffer/Cache.cc:2490 > 43284829184 | | TOTAL > the possible memory leak point is allocated | in-use | > type size | free list name > --------------------|--------------------|------------|---------------------------------- > 67108864 | 33554432 | 2097152 | > memory/ioBufAllocator[14] > 41271951360 | 40564162560 | 1048576 | > memory/ioBufAllocator[13] > 570425344 | 566755328 | 524288 | > memory/ioBufAllocator[12] > 276824064 | 270794752 | 262144 | > memory/ioBufAllocator[11] > 155189248 | 150994944 | 131072 | > memory/ioBufAllocator[10] > 113246208 | 111607808 | 65536 | > memory/ioBufAllocator[9] > 1319108608 | 1278607360 | 32768 | > memory/ioBufAllocator[8] > 39845888 | 39501824 | 16384 | > memory/ioBufAllocator[7] > 78381056 | 55869440 | 8192 | > memory/ioBufAllocator[6] > 228589568 | 212971520 | 4096 | > memory/ioBufAllocator[5] > 262144 | 0 | 2048 | > memory/ioBufAllocator[4] > 131072 | 0 | 1024 | > memory/ioBufAllocator[3] > 65536 | 0 | 512 | > memory/ioBufAllocator[2] > 163840 | 256 | 256 | > memory/ioBufAllocator[1] > 1785856 | 768 | 128 | > memory/ioBufAllocator[0] > 0 | 0 | 512 | > memory/FetchSMAllocator > 0 | 0 | 592 | > memory/ICPRequestCont_allocator > 0 | 0 | 128 | > memory/ICPPeerReadContAllocator > 0 | 0 | 432 | > memory/PeerReadDataAllocator > 0 | 0 | 32 | > memory/MIMEFieldSDKHandle > 0 | 0 | 240 | > memory/INKVConnAllocator > 0 | 0 | 96 | > memory/INKContAllocator > 503808 | 492864 | 32 | > memory/apiHookAllocator > 0 | 0 | 128 | > memory/socksProxyAllocator > 17211392 | 17169856 | 704 | > memory/httpClientSessionAllocator > 180633600 | 170085888 | 8064 | memory/httpSMAllocator > 2236416 | 2229024 | 224 | > memory/httpServerSessionAllocator > 0 | 0 | 48 | > memory/CacheLookupHttpConfigAllocator > 0 | 0 | 848 | > memory/http2ClientSessionAllocator > 0 | 0 | 128 | > memory/RemapPluginsAlloc > 0 | 0 | 48 | > memory/CongestRequestParamAllocator > 0 | 0 | 160 | > memory/CongestionDBContAllocator > 4325376 | 4245248 | 256 | > memory/httpCacheAltAllocator > 155451392 | 146763776 | 2048 | memory/hdrStrHeap > 269221888 | 253577216 | 2048 | memory/hdrHeap > 0 | 0 | 128 | > memory/OneWayTunnelAllocator > 0 | 0 | 96 | > memory/hostDBFileContAllocator > 296960 | 4640 | 2320 | > memory/hostDBContAllocator > 135424 | 33856 | 33856 | memory/dnsBufAllocator > 163840 | 1280 | 1280 | > memory/dnsEntryAllocator > 0 | 0 | 16 | > memory/DNSRequestDataAllocator > 0 | 0 | 112 | > memory/inControlAllocator > 0 | 0 | 128 | > memory/outControlAllocator > 0 | 0 | 32 | > memory/byteBankAllocator > 0 | 0 | 592 | > memory/clusterVCAllocator > 0 | 0 | 48 | > memory/ClusterVConnectionCache::Entry > 0 | 0 | 576 | > memory/cacheContAllocator > 0 | 0 | 48 | memory/evacuationKey > 0 | 0 | 64 | memory/cacheRemoveCont > 380928 | 380928 | 96 | memory/evacuationBlock > 20003360 | 19959936 | 944 | > memory/cacheVConnection > 163840 | 156960 | 160 | memory/openDirEntry > 2260992 | 2253408 | 96 | > memory/RamCacheCLFUSEntry > 0 | 0 | 64 | > memory/RamCacheLRUEntry > 0 | 0 | 880 | > memory/sslNetVCAllocator > 0 | 0 | 128 | > memory/UDPIOEventAllocator > 24399872 | 23407008 | 736 | memory/netVCAllocator > 0 | 0 | 128 | > memory/udpReadContAllocator > 0 | 0 | 160 | > memory/udpPacketAllocator > 0 | 0 | 416 | memory/socksAllocator > 7053312 | 7051072 | 64 | > memory/ioBlockAllocator > 7441920 | 7440432 | 48 | memory/ioDataAllocator > 19355520 | 19350000 | 240 | memory/ioAllocator > 3402720 | 3306880 | 80 | memory/mutexAllocator > 147456 | 103680 | 96 | memory/eventAllocator > 22937600 | 21599232 | 1024 | memory/ArenaBlock > 6206100608 | 5329728512 | | TOTAL > Allocs | Frees | Size In-use | Avg Size | Location > > -----------|------------|----------------------|------------|-------------------------------------------------------------------- > 39443091 | 39399441 | 178790400 | 4096 | > memory/IOBuffer/ProtocolProbeSessionAccept.cc:66 > 61 | 61 | 0 | 0 | > memory/IOBuffer/Transform.cc:791 > 4064 | 4064 | 0 | 0 | > memory/IOBuffer/HttpClientSession.cc:211 > 10271910 | 10270634 | 23633920 | 18521 | > memory/IOBuffer/HttpSM.cc:5831 > 261122 | 261122 | 0 | 0 | > memory/IOBuffer/HttpSM.cc:5880 > 14785 | 14785 | 0 | 0 | > memory/IOBuffer/HttpSM.cc:5401 > 524803 | 494019 | 1008730112 | 32768 | > memory/IOBuffer/HttpSM.cc:6199 > 7365218 | 7365218 | 0 | 0 | > memory/IOBuffer/HttpSM.cc:6021 > 11827313 | 11827056 | 1052672 | 4096 | > memory/IOBuffer/HttpSM.cc:5750 > 51960076 | 51954854 | 170999808 | 32746 | > memory/IOBuffer/HttpSM.cc:6302 > 1156 | 1156 | 0 | 0 | > memory/IOBuffer/HttpSM.cc:6269 > 8806600 | 8803759 | 23273472 | 8192 | > memory/IOBuffer/HttpServerSession.cc:86 > 708080 | 708074 | 768 | 128 | > memory/IOBuffer/HttpTunnel.cc:106 > 34649 | 34648 | 256 | 256 | > memory/IOBuffer/HttpTunnel.cc:111 > 8493 | 8101 | 108728320 | 277368 | > memory/IOBuffer/Cache.cc:2476 > 31232454 | 31172959 | 41769619456 | 702069 | > memory/IOBuffer/Cache.cc:2490 > 43284829184 | | TOTAL > --------------------------------------------------------------------------------------------- > the possilbe memory leak position is in Cache.cc:2490, including new_CacheVC > and free_CacheVC, and memory leak has 41769619456 > byte. I wonder some places forget to call free_CacheVC, but not sure. > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)