[jira] [Commented] (TS-2796) Leaking CacheVConnections
[
https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995439#comment-13995439
]
Bryan Call commented on TS-2796:
Memory diff of a LinkedIn vs Yahoo production host:
{code}
name allocated in_use type_size
allocated(MB)
memory/ArenaBlock 786432 -54272 1024
0
memory/FetchSMAllocator 36864 0288
0
memory/RamCacheCLFUSEntry 364953600 364946400 96
348
memory/RamCacheLRUEntry -9601024 -9126016 64
-9
memory/cacheRemoveCont -6144 0 48
0
memory/cacheVConnection -9977856 -4804256928
-9
memory/dnsBufAllocator-135424 0 33856
0
memory/evacuationBlock-737280-737856 96
0
memory/evacuationKey -6144 0 48
0
memory/eventAllocator -73728 584832 96
0
memory/hdrStrHeap -26476544 -5216256 4096
-25
memory/httpClientSessionAlloca -48039936 -45016704656
-45
memory/httpSMAllocator -135473152 -105433456 9728
-129
memory/httpServerSessionAlloca 143360 382816224
0
memory/ioAllocator -18800640 -17246160240
-17
memory/ioBlockAllocator -9543680 -8705600 64
-9
memory/ioBufAllocator[0]-131072 0128
0
memory/ioBufAllocator[10] 264241152 -626130944 131072
252
memory/ioBufAllocator[11] 6526337024 -537133056 262144
6224
memory/ioBufAllocator[12] 4093640704 -545783808 524288
3904
memory/ioBufAllocator[13] -805306368 -6427770881048576
-768
memory/ioBufAllocator[2] -65536 0512
0
memory/ioBufAllocator[3]-131072 0 1024
0
memory/ioBufAllocator[4] -2359296 -8192 2048
-2
memory/ioBufAllocator[5] -585105408 -542396416 4096
-558
memory/ioBufAllocator[6] 6397624320 6433579008 8192
6101
memory/ioBufAllocator[7]1431358668814380695552 16384
13650
memory/ioBufAllocator[8] 9016705024 4105961472 32768
8599
memory/ioBufAllocator[9] 3965714432 -621150208 65536
3782
memory/ioDataAllocator 77807616 78438960 48
74
memory/mutexAllocator -6225920 -5812000 80
-5
memory/netVCAllocator -17719296 -15882048672
-16
memory/openDirEntry -1822720 -1109280160
-1
memory/sslNetVCAllocator -43683840 -37772640720
-41
{code}
> Leaking CacheVConnections
> -
>
> Key: TS-2796
> URL: https://issues.apache.org/jira/browse/TS-2796
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
>Affects Versions: 4.0.2, 4.2.1, 5.0.0
>Reporter: Brian Geffon
> Labels: yahoo
> Fix For: 5.0.0
>
>
> It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking
> CacheVConnections resulting in IOBufAllocator leaking also, here is an
> example:
> allocated |in-use | type size | free list name
>67108864 | 0 |2097152 |
> memory/ioBufAllocator[14]
>67108864 | 19922944 |1048576 |
> memory/ioBufAllocator[13]
> 4798283776 | 14155776 | 524288 |
> memory/ioBufAllocator[12]
> 7281311744 | 98304000 | 262144 |
> memory/ioBufAllocator[11]
> 1115684864 | 148242432 | 131072 |
> memory/ioBufAllocator[10]
> 497544 | 379977728 | 65536 |
> memory/ioBufAllocator[9]
> 9902751744 | 5223546880 | 32768 |
> memory/ioBufAllocator[8]
> 14762901504 |14762311680 | 16384 |
> memory/ioBufAllocator[7]
> 6558056448 | 6557859840 | 8192 |
> memory/ioBufAllocator[6]
>41418752 | 30502912 | 4096 |
> memory
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[
https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995894#comment-13995894
]
Alan M. Carroll commented on TS-2796:
-
Maybe check {{HTTPCacheAlt::copy_frag_offsets_from}} to verify that
{{m_frag_offsets}} is {{NULL}} before the copy. I think that should always be
true but perhaps not.
> Leaking CacheVConnections
> -
>
> Key: TS-2796
> URL: https://issues.apache.org/jira/browse/TS-2796
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
>Affects Versions: 4.0.2, 4.2.1, 5.0.0
>Reporter: Brian Geffon
> Labels: yahoo
> Fix For: 5.0.0
>
>
> It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking
> CacheVConnections resulting in IOBufAllocator leaking also, here is an
> example:
> allocated |in-use | type size | free list name
>67108864 | 0 |2097152 |
> memory/ioBufAllocator[14]
>67108864 | 19922944 |1048576 |
> memory/ioBufAllocator[13]
> 4798283776 | 14155776 | 524288 |
> memory/ioBufAllocator[12]
> 7281311744 | 98304000 | 262144 |
> memory/ioBufAllocator[11]
> 1115684864 | 148242432 | 131072 |
> memory/ioBufAllocator[10]
> 497544 | 379977728 | 65536 |
> memory/ioBufAllocator[9]
> 9902751744 | 5223546880 | 32768 |
> memory/ioBufAllocator[8]
> 14762901504 |14762311680 | 16384 |
> memory/ioBufAllocator[7]
> 6558056448 | 6557859840 | 8192 |
> memory/ioBufAllocator[6]
>41418752 | 30502912 | 4096 |
> memory/ioBufAllocator[5]
> 524288 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 0 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 0 | 0 |512 |
> memory/ioBufAllocator[2]
> 32768 | 0 |256 |
> memory/ioBufAllocator[1]
> 0 | 0 |128 |
> memory/ioBufAllocator[0]
> 2138112 |2124192 |928 |
> memory/cacheVConnection
> [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x.
> The code path in CacheVC that is allocating the IoBuffers is
> memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom
> the real issue here is the leaking CacheVC.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995956#comment-13995956 ] Zhao Yongming commented on TS-2796: --- I don't know what is this issue looking for, if we focus on the last line of memory dump, the memory/cacheVConnection, please ignore my comment. the most of the memory leaking in your memory dump result is memory/ioBufAllocator size > 32K. and from what I can guess, you are using the defualt CLFUS ram cache algorithm, which will produce this effect when the system running a long time, and the big objects in memory is replaced by the smaller ones, but memory used by big objects is not released to the system yet. and that issue is already adressed in TS-1006, and result in the reclaimable freelist memory management codes, already shiped in >4.0 releases, with a configure options to enable. so, if this is the cause, please help verify that your problem is still there with reclaimable freelist enabled, and you may test the simple LRU algorithm in the ram cache too. thanks > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995961#comment-13995961 ] Zhao Yongming commented on TS-2796: --- and if the recleamable freelist will help you, please help me promote it to be enabled by default > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[
https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995890#comment-13995890
]
Alan M. Carroll commented on TS-2796:
-
It's possible that we're leaking fragment tables, which were moved to the
alternates in 4.0. In general they should be cleared in {{free_CacheVC}} but
it's possible that during a write the table gets created but not freed by
{{CacheVC::destroy}}. The ones loaded from disk should be cleaned up by the
call to {{vector::clear()}} in {{free_CacheVC}}. One way to validate this would
be to do a check in {{free_CacheVC}} just before the call
{{cont->alternate.clear()}} and see if it still has an allocated fragment table
{code}m_frag_offsets && m_frag_offsets != m_integral_frag_offsets{code} This is
in iocore/cache/P_CacheInternal.h.
> Leaking CacheVConnections
> -
>
> Key: TS-2796
> URL: https://issues.apache.org/jira/browse/TS-2796
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
>Affects Versions: 4.0.2, 4.2.1, 5.0.0
>Reporter: Brian Geffon
> Labels: yahoo
> Fix For: 5.0.0
>
>
> It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking
> CacheVConnections resulting in IOBufAllocator leaking also, here is an
> example:
> allocated |in-use | type size | free list name
>67108864 | 0 |2097152 |
> memory/ioBufAllocator[14]
>67108864 | 19922944 |1048576 |
> memory/ioBufAllocator[13]
> 4798283776 | 14155776 | 524288 |
> memory/ioBufAllocator[12]
> 7281311744 | 98304000 | 262144 |
> memory/ioBufAllocator[11]
> 1115684864 | 148242432 | 131072 |
> memory/ioBufAllocator[10]
> 497544 | 379977728 | 65536 |
> memory/ioBufAllocator[9]
> 9902751744 | 5223546880 | 32768 |
> memory/ioBufAllocator[8]
> 14762901504 |14762311680 | 16384 |
> memory/ioBufAllocator[7]
> 6558056448 | 6557859840 | 8192 |
> memory/ioBufAllocator[6]
>41418752 | 30502912 | 4096 |
> memory/ioBufAllocator[5]
> 524288 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 0 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 0 | 0 |512 |
> memory/ioBufAllocator[2]
> 32768 | 0 |256 |
> memory/ioBufAllocator[1]
> 0 | 0 |128 |
> memory/ioBufAllocator[0]
> 2138112 |2124192 |928 |
> memory/cacheVConnection
> [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x.
> The code path in CacheVC that is allocating the IoBuffers is
> memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom
> the real issue here is the leaking CacheVC.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995389#comment-13995389 ] Bryan Call commented on TS-2796: This is from a 4.0.2 server in our CDN: -bash-4.1$ sudo grep cacheV traffic.out 12115968 |9591808 |928 | memory/cacheVConnection > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996050#comment-13996050 ] Leif Hedstrom commented on TS-2796: --- Good explanation, this actually now makes some sense as to the reclaimable freelist "issues"; the change in payload characteristics changing RAM cache objects. This explains why most of us don't see this issue ever, since we have very consistent payload characteristics. Does the CLFUS have different behavior here compared to the LRU RAM cache? It'd be surprising if it is, but if that is the case, we should fix CLFUS. > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996661#comment-13996661 ] Thomas Jackson commented on TS-2796: Update from the test-- looks like LRU leaks just as much, so definitely not that :( > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996652#comment-13996652 ] Thomas Jackson commented on TS-2796: If the CLFUS is the problem, it was a regression introduced between 3.2.x and 4.0.x since we use the default (CFLUS) on our older cache hosts and don't see the same memory leak. I have just changed the memory cache over to LRU, so I'll check on it in ~20m to see if its still leaking. > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996461#comment-13996461 ] Leif Hedstrom commented on TS-2796: --- I'm not keen on making this enabled by default. It's difficult to configure "correctly" for an experienced user, and worse, difficult to configure automatically. I'm +1 on making it such that it's compiled in at all time, with an on/off switch (which IMO should be off by default). > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996671#comment-13996671 ] Brian Geffon commented on TS-2796: -- Alan's explanation seems plausible, if RamCacheLRUEntry/RamCacheCLFUSEntry entries become leaked that would cause it. > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996658#comment-13996658 ] Brian Geffon commented on TS-2796: -- I do not believe that is the issue. Firstly, reclaimable freelist cannot solve our issue because (from the memory output) the freelist items are currently in use and there is nothing that can be recalimed. Also, I've tracked down the source of the IOBufferBlock allocations and they are coming from Cache.cc:2603 which is: buf = new_IOBufferData(iobuffer_size_to_index(io.aiocb.aio_nbytes, MAX_BUFFER_SIZE_INDEX), MEMALIGNED); I believe this issue is a valid leak as these buffers are not being returned to the freelist properly. > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996664#comment-13996664 ] Alan M. Carroll commented on TS-2796: - We know that the ram cache was significantly modified for 4.0 so it's plausible an issue was introduced. A key point here is that the IOBufferBlocks in the ram cache originally came from the CacheVC. When a CacheVC needs to read a fragment from disk, it gets an IOBufferBlock in the line Brian noted and sets the I/O to read to that memory. If at a later point it decides to put the fragment in the ram cache, it simply moves the IOBufferBlock there, it doesn't copy or re-read from disk. Therefore if the ram cache is not releasing the IOBufferBlocks as expected (or they are getting dropped somewhere in the transfer process) this is what we would see - growing memory usage from putatively in use IOBufferBlocks. In that case reclaimable free lists won't help. > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997221#comment-13997221 ] Zhao Yongming commented on TS-2796: --- yeah, I know you may think that the reclaim freelist is hard to manage and evil in coding, but if we can confirm it may help in this case, I'd like you think of enable it by default, we really should not waste so many time here, and pull back some not so experiencd users when they may think that we do have big memory problem in core. I'd push on the other enhancement you like to make it enable by default. :D > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997180#comment-13997180 ] Zhao Yongming commented on TS-2796: --- hmm, from what I know, many people have the memory issue, and it is a malloc and gc issue, but they just don't realized it. that is why I am pushing the reclaim freelist enabled by default. why not test it if you can vevify the result in hours. and please take a look at the 'allocated' - 'in-use' where memory/ioBufAllocator size > 32K, if you sum up them, that is the memory you leak. the same as TS-1006 > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[
https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995394#comment-13995394
]
Thomas Jackson commented on TS-2796:
We're running ATS 4.2.1 and we are seeing this leaking pretty fast (~1gb of an
hour). Memory
{code}
2138112 |2124192 |928 | memory/cacheVConnection
{code
> Leaking CacheVConnections
> -
>
> Key: TS-2796
> URL: https://issues.apache.org/jira/browse/TS-2796
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
>Affects Versions: 4.0.2, 4.2.1, 5.0.0
>Reporter: Brian Geffon
> Labels: yahoo
> Fix For: 5.0.0
>
>
> It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking
> CacheVConnections resulting in IOBufAllocator leaking also, here is an
> example:
> allocated |in-use | type size | free list name
>67108864 | 0 |2097152 |
> memory/ioBufAllocator[14]
>67108864 | 19922944 |1048576 |
> memory/ioBufAllocator[13]
> 4798283776 | 14155776 | 524288 |
> memory/ioBufAllocator[12]
> 7281311744 | 98304000 | 262144 |
> memory/ioBufAllocator[11]
> 1115684864 | 148242432 | 131072 |
> memory/ioBufAllocator[10]
> 497544 | 379977728 | 65536 |
> memory/ioBufAllocator[9]
> 9902751744 | 5223546880 | 32768 |
> memory/ioBufAllocator[8]
> 14762901504 |14762311680 | 16384 |
> memory/ioBufAllocator[7]
> 6558056448 | 6557859840 | 8192 |
> memory/ioBufAllocator[6]
>41418752 | 30502912 | 4096 |
> memory/ioBufAllocator[5]
> 524288 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 0 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 0 | 0 |512 |
> memory/ioBufAllocator[2]
> 32768 | 0 |256 |
> memory/ioBufAllocator[1]
> 0 | 0 |128 |
> memory/ioBufAllocator[0]
> 2138112 |2124192 |928 |
> memory/cacheVConnection
> [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x.
> The code path in CacheVC that is allocating the IoBuffers is
> memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom
> the real issue here is the leaking CacheVC.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003650#comment-14003650 ] Bryan Call commented on TS-2796: [~briang] Do you think there is a leak here? > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006172#comment-14006172 ] Zhao Yongming commented on TS-2796: --- any update on this issue? do you need me push on the code diffing in taobao's side? > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon >Assignee: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006184#comment-14006184 ] Brian Geffon commented on TS-2796: -- I believe it's actually related to proxy allocators, since fixing the ram cache size seems to help the situation. Brian > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon >Assignee: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007218#comment-14007218 ] Thomas Jackson commented on TS-2796: Not only fixing the ram cache size, but it has to be fixed small (somewhere between 1-5gb is the magic size). We have some more tests running to get a more exact number, but that is quite interesting. > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon >Assignee: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008259#comment-14008259 ] Tommy Lee commented on TS-2796: --- I'm seeing this leaking too. In my case i have freelist enabled but it doesn't helped too much. My scenario is: RAM: 98G RAM_CACHE_FIXED_SIZE: 35G STORAGE: 26TB OBJECT SIZE: 16KB I'll attach my mem_dump later. For now, i'm trying to decrease more and more RAM_CACHE size until find a magical number to stabilize. > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.0.2, 4.2.1, 5.0.0 >Reporter: Brian Geffon >Assignee: Brian Geffon > Labels: yahoo > Fix For: 5.0.0 > > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[ https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010393#comment-14010393 ] Thomas Jackson commented on TS-2796: FTR, the magic number is 2gb. If the ram_cache is 2gb or less it doesn't leak, anything over does. > Leaking CacheVConnections > - > > Key: TS-2796 > URL: https://issues.apache.org/jira/browse/TS-2796 > Project: Traffic Server > Issue Type: Bug > Components: Cache >Affects Versions: 4.2.1, 5.0.0 >Reporter: Brian Geffon >Assignee: Brian Geffon > Labels: yahoo > > It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking > CacheVConnections resulting in IOBufAllocator leaking also, here is an > example: > allocated |in-use | type size | free list name >67108864 | 0 |2097152 | > memory/ioBufAllocator[14] >67108864 | 19922944 |1048576 | > memory/ioBufAllocator[13] > 4798283776 | 14155776 | 524288 | > memory/ioBufAllocator[12] > 7281311744 | 98304000 | 262144 | > memory/ioBufAllocator[11] > 1115684864 | 148242432 | 131072 | > memory/ioBufAllocator[10] > 497544 | 379977728 | 65536 | > memory/ioBufAllocator[9] > 9902751744 | 5223546880 | 32768 | > memory/ioBufAllocator[8] > 14762901504 |14762311680 | 16384 | > memory/ioBufAllocator[7] > 6558056448 | 6557859840 | 8192 | > memory/ioBufAllocator[6] >41418752 | 30502912 | 4096 | > memory/ioBufAllocator[5] > 524288 | 0 | 2048 | > memory/ioBufAllocator[4] > 0 | 0 | 1024 | > memory/ioBufAllocator[3] > 0 | 0 |512 | > memory/ioBufAllocator[2] > 32768 | 0 |256 | > memory/ioBufAllocator[1] > 0 | 0 |128 | > memory/ioBufAllocator[0] > 2138112 |2124192 |928 | > memory/cacheVConnection > [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x. > The code path in CacheVC that is allocating the IoBuffers is > memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom > the real issue here is the leaking CacheVC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-2796) Leaking CacheVConnections
[
https://issues.apache.org/jira/browse/TS-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010401#comment-14010401
]
Brian Geffon commented on TS-2796:
--
This can't be related to TS-2824 because in 4.2.x ioBufferAllocator
(ProxyAllocator) wasn't used, and I've verified this on a core:
(gdb) p eventProcessor->all_ethreads[0]->ioBufAllocator
$37 = {{allocated = 0, freelist = 0x0} }
(gdb) p eventProcessor->all_ethreads[1]->ioBufAllocator
$38 = {{allocated = 0, freelist = 0x0} }
(gdb) p eventProcessor->all_ethreads[2]->ioBufAllocator
$39 = {{allocated = 0, freelist = 0x0} }
and so on, they are all empty, meaning that the memory above in the
ioBufAllocator aren't sitting on a class allocator, they aren't being returned
to a freelist somehow.
> Leaking CacheVConnections
> -
>
> Key: TS-2796
> URL: https://issues.apache.org/jira/browse/TS-2796
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
>Affects Versions: 4.2.1, 5.0.0
>Reporter: Brian Geffon
>Assignee: Brian Geffon
> Labels: yahoo
>
> It appears there is a memory leak in 4.0.x, 4.2.x, and master leaking
> CacheVConnections resulting in IOBufAllocator leaking also, here is an
> example:
> allocated |in-use | type size | free list name
>67108864 | 0 |2097152 |
> memory/ioBufAllocator[14]
>67108864 | 19922944 |1048576 |
> memory/ioBufAllocator[13]
> 4798283776 | 14155776 | 524288 |
> memory/ioBufAllocator[12]
> 7281311744 | 98304000 | 262144 |
> memory/ioBufAllocator[11]
> 1115684864 | 148242432 | 131072 |
> memory/ioBufAllocator[10]
> 497544 | 379977728 | 65536 |
> memory/ioBufAllocator[9]
> 9902751744 | 5223546880 | 32768 |
> memory/ioBufAllocator[8]
> 14762901504 |14762311680 | 16384 |
> memory/ioBufAllocator[7]
> 6558056448 | 6557859840 | 8192 |
> memory/ioBufAllocator[6]
>41418752 | 30502912 | 4096 |
> memory/ioBufAllocator[5]
> 524288 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 0 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 0 | 0 |512 |
> memory/ioBufAllocator[2]
> 32768 | 0 |256 |
> memory/ioBufAllocator[1]
> 0 | 0 |128 |
> memory/ioBufAllocator[0]
> 2138112 |2124192 |928 |
> memory/cacheVConnection
> [~bcall] has observed this issue on 4.0.x, and we have observed this on 4.2.x.
> The code path in CacheVC that is allocating the IoBuffers is
> memory/IOBuffer/Cache.cc:2603; however, that's just the observable symptom
> the real issue here is the leaking CacheVC.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
