[jira] [Comment Edited] (TS-1006) memory management, cut down memory waste ?
[
https://issues.apache.org/jira/browse/TS-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540840#comment-13540840
]
Yunkai Zhang edited comment on TS-1006 at 12/29/12 10:30 AM:
-
[~genext]
I'm not sure if there are memory leak in ClassAllocator, if yes, I'll try to
fix it.
But we can confirm that the original version of InkFreeList won't free memory
back to OS. So, let's improve InkFreeList firstly. And we should known that,
there is no perfect method to completely avoid memory fragment as long as we
allocate a big chunk and split it into small type_size. What we could do is to
reduce, not to avoid it.
Of couse, memory fragment will consume some of memory, but should not be too
big. Now, we have found that RamCacheLRU may lead to memory fragment, that's
[~kuotai]'s patch to fix.
So, I have two suggestions:
1) set a reasonable *ram_cache.size* and *max_overage* (we should leave some
space for memory fragment). If there are only memory fragment and no memory
leak, the total size should not increase infinitely. But in my experience, the
total size may increase for several days before it reach the top of memory
fragment.
2) wait for [~kuotai]'s patch, I'll send it soon, which will reduce memory
fragment caused by RamCacheLRU(CLFUS has the same problem but we haven't fix it
now, so I suggest you select LRU to test it).
was (Author: yunkai):
[~genext]
I'm not sure if there are memory leak in ClassAllocator, if yes, I'll try too
fix it.
But we can confirm that the original version of InkFreeList won't free memory
back to OS. So, let's improve InkFreeList firstly. And we should known that,
there is no perfect method to completely avoid memory fragment as long as we
allocate a big chunk and split it into small type_size. What we could do is to
reduce, not to avoid it.
Of couse, memory fragment will consume some of memory, but should not be too
big. Now, we have found that RamCacheLRU may lead to memory fragment, that's
[~kuotai]'s patch to fix.
So, I have two suggestions:
1) set a reasonable *ram_cache.size* and *max_overage* (we should leave some
space for memory fragment). If there are only memory fragment and no memory
leak, the total size should not increase infinitely. But in my experience, the
total size may increase for several days before it reach the top of memory
fragment.
2) wait for [~kuotai]'s patch, I'll send it soon, which will reduce memory
fragment caused by RamCacheLRU(CLFUS has the same problem but we haven't fit it
now, so I suggest you select LRU to test it).
memory management, cut down memory waste ?
--
Key: TS-1006
URL: https://issues.apache.org/jira/browse/TS-1006
Project: Traffic Server
Issue Type: Improvement
Components: Core
Affects Versions: 3.1.1
Reporter: Zhao Yongming
Assignee: Bin Chen
Fix For: 3.3.2
Attachments: 0001-Allocator-optimize-InkFreeList-memory-pool.patch,
0002-Allocator-make-InkFreeList-memory-pool-configurable.patch,
0003-Allocator-store-InkChunkInfo-into-Chunk.patch,
0004-Allocator-optimize-alignment-size-to-avoid-mmap-fail.patch,
0005-Allocator-adjust-reclaiming-strategy-of-InkFreeList.patch,
Memory-Usage-After-Introduced-New-Allocator.png, memusage.ods, memusage.ods
when we review the memory usage in the production, there is something
abnormal, ie, looks like TS take much memory than index data + common system
waste, and here is some memory dump result by set
proxy.config.dump_mem_info_frequency
1, the one on a not so busy forwarding system:
physics memory: 32G
RAM cache: 22G
DISK: 6140 GB
average_object_size 64000
{code}
allocated |in-use | type size | free list name
|||--
671088640 | 37748736 |2097152 |
memory/ioBufAllocator[14]
2248146944 | 2135949312 |1048576 |
memory/ioBufAllocator[13]
1711276032 | 1705508864 | 524288 |
memory/ioBufAllocator[12]
1669332992 | 1667760128 | 262144 |
memory/ioBufAllocator[11]
2214592512 | 221184 | 131072 |
memory/ioBufAllocator[10]
2325741568 | 2323775488 | 65536 |
memory/ioBufAllocator[9]
2091909120 | 2089123840 | 32768 |
memory/ioBufAllocator[8]
1956642816 | 1956478976 | 16384 |
memory/ioBufAllocator[7]
2094530560 | 2094071808 | 8192 |
memory/ioBufAllocator[6]
356515840 | 355540992 | 4096 |
memory/ioBufAllocator[5]
1048576 | 14336 | 2048 |
[jira] [Commented] (TS-1006) memory management, cut down memory waste ?
[
https://issues.apache.org/jira/browse/TS-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540843#comment-13540843
]
jaekyung oh commented on TS-1006:
-
got it. i'll talk about ram_cache.size with op team. thank you for your
immediate and easily understandable explanations.
memory management, cut down memory waste ?
--
Key: TS-1006
URL: https://issues.apache.org/jira/browse/TS-1006
Project: Traffic Server
Issue Type: Improvement
Components: Core
Affects Versions: 3.1.1
Reporter: Zhao Yongming
Assignee: Bin Chen
Fix For: 3.3.2
Attachments: 0001-Allocator-optimize-InkFreeList-memory-pool.patch,
0002-Allocator-make-InkFreeList-memory-pool-configurable.patch,
0003-Allocator-store-InkChunkInfo-into-Chunk.patch,
0004-Allocator-optimize-alignment-size-to-avoid-mmap-fail.patch,
0005-Allocator-adjust-reclaiming-strategy-of-InkFreeList.patch,
Memory-Usage-After-Introduced-New-Allocator.png, memusage.ods, memusage.ods
when we review the memory usage in the production, there is something
abnormal, ie, looks like TS take much memory than index data + common system
waste, and here is some memory dump result by set
proxy.config.dump_mem_info_frequency
1, the one on a not so busy forwarding system:
physics memory: 32G
RAM cache: 22G
DISK: 6140 GB
average_object_size 64000
{code}
allocated |in-use | type size | free list name
|||--
671088640 | 37748736 |2097152 |
memory/ioBufAllocator[14]
2248146944 | 2135949312 |1048576 |
memory/ioBufAllocator[13]
1711276032 | 1705508864 | 524288 |
memory/ioBufAllocator[12]
1669332992 | 1667760128 | 262144 |
memory/ioBufAllocator[11]
2214592512 | 221184 | 131072 |
memory/ioBufAllocator[10]
2325741568 | 2323775488 | 65536 |
memory/ioBufAllocator[9]
2091909120 | 2089123840 | 32768 |
memory/ioBufAllocator[8]
1956642816 | 1956478976 | 16384 |
memory/ioBufAllocator[7]
2094530560 | 2094071808 | 8192 |
memory/ioBufAllocator[6]
356515840 | 355540992 | 4096 |
memory/ioBufAllocator[5]
1048576 | 14336 | 2048 |
memory/ioBufAllocator[4]
131072 | 0 | 1024 |
memory/ioBufAllocator[3]
65536 | 0 |512 |
memory/ioBufAllocator[2]
32768 | 0 |256 |
memory/ioBufAllocator[1]
16384 | 0 |128 |
memory/ioBufAllocator[0]
0 | 0 |576 |
memory/ICPRequestCont_allocator
0 | 0 |112 |
memory/ICPPeerReadContAllocator
0 | 0 |432 |
memory/PeerReadDataAllocator
0 | 0 | 32 |
memory/MIMEFieldSDKHandle
0 | 0 |240 |
memory/INKVConnAllocator
0 | 0 | 96 |
memory/INKContAllocator
4096 | 0 | 32 |
memory/apiHookAllocator
0 | 0 |288 |
memory/FetchSMAllocator
0 | 0 | 80 |
memory/prefetchLockHandlerAllocator
0 | 0 |176 |
memory/PrefetchBlasterAllocator
0 | 0 | 80 |
memory/prefetchUrlBlaster
0 | 0 | 96 | memory/blasterUrlList
0 | 0 | 96 |
memory/prefetchUrlEntryAllocator
0 | 0 |128 |
memory/socksProxyAllocator
0 | 0 |144 |
memory/ObjectReloadCont
3258368 | 576016 |592 |
memory/httpClientSessionAllocator
825344 | 139568 |208 |
memory/httpServerSessionAllocator
22597632 |1284848 | 9808 | memory/httpSMAllocator
0 | 0 | 32 |
memory/CacheLookupHttpConfigAllocator
0 | 0 | 9856 |
memory/httpUpdateSMAllocator
0 | 0 |128 |
memory/RemapPluginsAlloc
0 | 0 | 48 |
memory/CongestRequestParamAllocator
0 | 0 |128 |
[jira] [Commented] (TS-1639) Unable to build on Mac OS X 10.7.5
[
https://issues.apache.org/jira/browse/TS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540932#comment-13540932
]
James Peach commented on TS-1639:
-
yep, that's it :)
Unable to build on Mac OS X 10.7.5
--
Key: TS-1639
URL: https://issues.apache.org/jira/browse/TS-1639
Project: Traffic Server
Issue Type: Bug
Components: Build
Reporter: Alan Cabrera
Assignee: James Peach
Filed Apple bug 12938480.
make all-am
CXX Diags.lo
Diags.cc:547: internal compiler error: Segmentation fault: 11
Please submit a full bug report,
with preprocessed source if appropriate.
See URL:http://developer.apple.com/bugreporter for instructions.
{standard input}:0:End-of-File not at end of a line
{standard input}:1239:End-of-File not at end of a line
{standard input}:unknown:Partial line at end of file ignored
{standard input}:unknown:Undefined local symbol L_.str1
{standard input}:unknown:Undefined local symbol L_.str2
{standard input}:unknown:Undefined local symbol L_.str3
{standard input}:unknown:Undefined local symbol L_.str4
{standard input}:unknown:Undefined local symbol L_.str5
{standard input}:unknown:Undefined local symbol L_.str6
{standard input}:unknown:Undefined local symbol L_.str7
{standard input}:unknown:Undefined local symbol L_.str8
{standard input}:unknown:Undefined local symbol L_.str
{standard input}:unknown:Undefined local symbol L_.str9
{standard input}:unknown:Undefined local symbol L_.str10
{standard input}:unknown:Undefined local symbol L_.str11
{standard input}:unknown:Undefined local symbol L_.str12
{standard input}:unknown:Undefined local symbol L_.str13
{standard input}:unknown:Undefined local symbol L_.str14
{standard input}:unknown:Undefined local symbol L_.str15
{standard input}:unknown:Undefined local symbol L_.str16
{standard input}:unknown:Undefined local symbol L_.str17
{standard input}:unknown:Undefined local symbol L_.str18
{standard input}:unknown:Undefined local symbol L_.str19
{standard input}:unknown:Undefined local symbol L_.str20
{standard input}:unknown:Undefined local symbol L_.str21
{standard input}:unknown:Undefined local symbol LBB8_5
make[3]: *** [Diags.lo] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1
Steps to reproduce:
git clone git://git.apache.org/trafficserver.git
cd trafficserver
autoreconf -i
./configure
make
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
