Re: [ceph-users] Fwd: MDS memory usage is very high
Hi, I forgot to say that maybe the Diff is lower than real (8Mb), because the memory usage was still high and i've prepared a new configuration with lower limit (5Mb). I've not reloaded the daemons for now, but maybe the configuration was loaded again today and that's the reason why is using less than 1Gb of RAM just now. Of course I've not rebooted the machine, but maybe if the daemon was killed for high memory usage then the new configuration is loaded now. Greetings! 2018-07-19 11:35 GMT+02:00 Daniel Carrasco : > Hello again, > > It is still early to say that is working fine now, but looks like the MDS > memory is now under 20% of RAM and the most of time between 6-9%. Maybe was > a mistake on configuration. > > As appointment, I've changed this client config: > [global] > ... > bluestore_cache_size_ssd = 805306360 > bluestore_cache_size_hdd = 805306360 > mds_cache_memory_limit = 536870910 > > [client] > client_reconnect_stale = true > client_cache_size = 32768 > client_mount_timeout = 30 > client_oc_max_objects = 2000 > client_oc_size = 629145600 > rbd_cache = true > rbd_cache_size = 671088640 > > > for this (just client cache sizes / 10): > [global] > ... > bluestore_cache_size_ssd = 80530636 > bluestore_cache_size_hdd = 80530636 > mds_cache_memory_limit = 53687091 > > [client] > client_cache_size = 32768 > client_mount_timeout = 30 > client_oc_max_objects = 2000 > client_oc_size = 62914560 > rbd_cache = true > rbd_cache_size = 67108864 > > > > Now the heap stats are: > mds.kavehome-mgto-pro-fs01 tcmalloc heap stats: > > MALLOC: 714063568 ( 681.0 MiB) Bytes in use by application > MALLOC: +0 (0.0 MiB) Bytes in page heap freelist > MALLOC: +132992224 ( 126.8 MiB) Bytes in central cache freelist > MALLOC: + 21929920 ( 20.9 MiB) Bytes in transfer cache freelist > MALLOC: + 31806608 ( 30.3 MiB) Bytes in thread cache freelists > MALLOC: + 30666912 ( 29.2 MiB) Bytes in malloc metadata > MALLOC: > MALLOC: =931459232 ( 888.3 MiB) Actual memory used (physical + swap) > MALLOC: + 21886803968 (20872.9 MiB) Bytes released to OS (aka unmapped) > MALLOC: > MALLOC: = 22818263200 (21761.2 MiB) Virtual address space used > MALLOC: > MALLOC: 21311 Spans in use > MALLOC: 18 Thread heaps in use > MALLOC: 8192 Tcmalloc page size > > Call ReleaseFreeMemory() to release freelist memory to the OS (via > madvise()). > Bytes released to the OS take up virtual address space but no physical > memory. > > And sometimes even better (taken later than above): > mds.kavehome-mgto-pro-fs01 tcmalloc heap stats: > > MALLOC: 516434072 ( 492.5 MiB) Bytes in use by application > MALLOC: +0 (0.0 MiB) Bytes in page heap freelist > MALLOC: + 7564936 (7.2 MiB) Bytes in central cache freelist > MALLOC: + 2751072 (2.6 MiB) Bytes in transfer cache freelist > MALLOC: + 2707072 (2.6 MiB) Bytes in thread cache freelists > MALLOC: + 2715808 (2.6 MiB) Bytes in malloc metadata > MALLOC: > MALLOC: =532172960 ( 507.5 MiB) Actual memory used (physical + swap) > MALLOC: + 573440 (0.5 MiB) Bytes released to OS (aka unmapped) > MALLOC: > MALLOC: =532746400 ( 508.1 MiB) Virtual address space used > MALLOC: > MALLOC: 21990 Spans in use > MALLOC: 16 Thread heaps in use > MALLOC: 8192 Tcmalloc page size > > Call ReleaseFreeMemory() to release freelist memory to the OS (via > madvise()). > Bytes released to the OS take up virtual address space but no physical > memory. > > > Greetings!! > > 2018-07-19 10:24 GMT+02:00 Daniel Carrasco : > >> Hello, >> >> Finally I've to remove CephFS and use a simple NFS, because the MDS >> daemon starts to use a lot of memory and is unstable. After reboot one node >> because it started to swap (the cluster will be able to survive without a >> node), the cluster goes down because one of the other MDS starts to use >> about 15Gb of RAM and crash all the time, so the cluster is unable to come >> back. The only solution is to reboot all nodes and is not good for HA. >> >> If somebody knows something about this, I'll be pleased to test it on a >> test environment to see if we can find a solution. >> >> Greetings! >> >> 2018-07-19 1:07 GMT+02:00 Daniel Carrasco : >> >>> Thanks again, >>> >>> I was trying to use fuse client instead Ubuntu 16.04 kernel module to >>> see if maybe is a client side problem, but CPU usage on fuse client is very >>> high (a 100% and even more in a two cores machine), so I'd to rever to >>> kernel client that uses much less CPU. >>> >>> Is a web server, so maybe
Re: [ceph-users] Fwd: MDS memory usage is very high
Hello again, It is still early to say that is working fine now, but looks like the MDS memory is now under 20% of RAM and the most of time between 6-9%. Maybe was a mistake on configuration. As appointment, I've changed this client config: [global] ... bluestore_cache_size_ssd = 805306360 bluestore_cache_size_hdd = 805306360 mds_cache_memory_limit = 536870910 [client] client_reconnect_stale = true client_cache_size = 32768 client_mount_timeout = 30 client_oc_max_objects = 2000 client_oc_size = 629145600 rbd_cache = true rbd_cache_size = 671088640 for this (just client cache sizes / 10): [global] ... bluestore_cache_size_ssd = 80530636 bluestore_cache_size_hdd = 80530636 mds_cache_memory_limit = 53687091 [client] client_cache_size = 32768 client_mount_timeout = 30 client_oc_max_objects = 2000 client_oc_size = 62914560 rbd_cache = true rbd_cache_size = 67108864 Now the heap stats are: mds.kavehome-mgto-pro-fs01 tcmalloc heap stats: MALLOC: 714063568 ( 681.0 MiB) Bytes in use by application MALLOC: +0 (0.0 MiB) Bytes in page heap freelist MALLOC: +132992224 ( 126.8 MiB) Bytes in central cache freelist MALLOC: + 21929920 ( 20.9 MiB) Bytes in transfer cache freelist MALLOC: + 31806608 ( 30.3 MiB) Bytes in thread cache freelists MALLOC: + 30666912 ( 29.2 MiB) Bytes in malloc metadata MALLOC: MALLOC: =931459232 ( 888.3 MiB) Actual memory used (physical + swap) MALLOC: + 21886803968 (20872.9 MiB) Bytes released to OS (aka unmapped) MALLOC: MALLOC: = 22818263200 (21761.2 MiB) Virtual address space used MALLOC: MALLOC: 21311 Spans in use MALLOC: 18 Thread heaps in use MALLOC: 8192 Tcmalloc page size Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. And sometimes even better (taken later than above): mds.kavehome-mgto-pro-fs01 tcmalloc heap stats: MALLOC: 516434072 ( 492.5 MiB) Bytes in use by application MALLOC: +0 (0.0 MiB) Bytes in page heap freelist MALLOC: + 7564936 (7.2 MiB) Bytes in central cache freelist MALLOC: + 2751072 (2.6 MiB) Bytes in transfer cache freelist MALLOC: + 2707072 (2.6 MiB) Bytes in thread cache freelists MALLOC: + 2715808 (2.6 MiB) Bytes in malloc metadata MALLOC: MALLOC: =532172960 ( 507.5 MiB) Actual memory used (physical + swap) MALLOC: + 573440 (0.5 MiB) Bytes released to OS (aka unmapped) MALLOC: MALLOC: =532746400 ( 508.1 MiB) Virtual address space used MALLOC: MALLOC: 21990 Spans in use MALLOC: 16 Thread heaps in use MALLOC: 8192 Tcmalloc page size Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. Greetings!! 2018-07-19 10:24 GMT+02:00 Daniel Carrasco : > Hello, > > Finally I've to remove CephFS and use a simple NFS, because the MDS daemon > starts to use a lot of memory and is unstable. After reboot one node > because it started to swap (the cluster will be able to survive without a > node), the cluster goes down because one of the other MDS starts to use > about 15Gb of RAM and crash all the time, so the cluster is unable to come > back. The only solution is to reboot all nodes and is not good for HA. > > If somebody knows something about this, I'll be pleased to test it on a > test environment to see if we can find a solution. > > Greetings! > > 2018-07-19 1:07 GMT+02:00 Daniel Carrasco : > >> Thanks again, >> >> I was trying to use fuse client instead Ubuntu 16.04 kernel module to see >> if maybe is a client side problem, but CPU usage on fuse client is very >> high (a 100% and even more in a two cores machine), so I'd to rever to >> kernel client that uses much less CPU. >> >> Is a web server, so maybe the problem is that. PHP and Nginx should open >> a lot of files and maybe that uses a lot of RAM. >> >> For now I've rebooted the machine because is the only way to free the >> memory, but I cannot restart the machine every few hours... >> >> Greetings!! >> >> 2018-07-19 1:00 GMT+02:00 Gregory Farnum : >> >>> Wow, yep, apparently the MDS has another 9GB of allocated RAM outside of >>> the cache! Hopefully one of the current FS users or devs has some idea. All >>> I can suggest is looking to see if there are a bunch of stuck requests or >>> something that are taking up memory which isn’t properly counted. >>> >>> On Wed, Jul 18, 2018 at 3:48 PM Daniel Carrasco >>> wrote: >>> Hello, thanks for your response.
Re: [ceph-users] Fwd: MDS memory usage is very high
Hello,
Finally I've to remove CephFS and use a simple NFS, because the MDS daemon
starts to use a lot of memory and is unstable. After reboot one node
because it started to swap (the cluster will be able to survive without a
node), the cluster goes down because one of the other MDS starts to use
about 15Gb of RAM and crash all the time, so the cluster is unable to come
back. The only solution is to reboot all nodes and is not good for HA.
If somebody knows something about this, I'll be pleased to test it on a
test environment to see if we can find a solution.
Greetings!
2018-07-19 1:07 GMT+02:00 Daniel Carrasco :
> Thanks again,
>
> I was trying to use fuse client instead Ubuntu 16.04 kernel module to see
> if maybe is a client side problem, but CPU usage on fuse client is very
> high (a 100% and even more in a two cores machine), so I'd to rever to
> kernel client that uses much less CPU.
>
> Is a web server, so maybe the problem is that. PHP and Nginx should open a
> lot of files and maybe that uses a lot of RAM.
>
> For now I've rebooted the machine because is the only way to free the
> memory, but I cannot restart the machine every few hours...
>
> Greetings!!
>
> 2018-07-19 1:00 GMT+02:00 Gregory Farnum :
>
>> Wow, yep, apparently the MDS has another 9GB of allocated RAM outside of
>> the cache! Hopefully one of the current FS users or devs has some idea. All
>> I can suggest is looking to see if there are a bunch of stuck requests or
>> something that are taking up memory which isn’t properly counted.
>>
>> On Wed, Jul 18, 2018 at 3:48 PM Daniel Carrasco
>> wrote:
>>
>>> Hello, thanks for your response.
>>>
>>> This is what I get:
>>>
>>> # ceph tell mds.kavehome-mgto-pro-fs01 heap stats
>>> 2018-07-19 00:43:46.142560 7f5a7a7fc700 0 client.1318388
>>> ms_handle_reset on 10.22.0.168:6800/1129848128
>>> 2018-07-19 00:43:46.181133 7f5a7b7fe700 0 client.1318391
>>> ms_handle_reset on 10.22.0.168:6800/1129848128
>>> mds.kavehome-mgto-pro-fs01 tcmalloc heap stats:
>>>
>>> MALLOC: 9982980144 ( 9520.5 MiB) Bytes in use by application
>>> MALLOC: +0 (0.0 MiB) Bytes in page heap freelist
>>> MALLOC: +172148208 ( 164.2 MiB) Bytes in central cache freelist
>>> MALLOC: + 19031168 ( 18.1 MiB) Bytes in transfer cache freelist
>>> MALLOC: + 23987552 ( 22.9 MiB) Bytes in thread cache freelists
>>> MALLOC: + 20869280 ( 19.9 MiB) Bytes in malloc metadata
>>> MALLOC:
>>> MALLOC: = 10219016352 ( 9745.6 MiB) Actual memory used (physical + swap)
>>> MALLOC: + 3913687040 ( 3732.4 MiB) Bytes released to OS (aka unmapped)
>>> MALLOC:
>>> MALLOC: = 14132703392 (13478.0 MiB) Virtual address space used
>>> MALLOC:
>>> MALLOC: 63875 Spans in use
>>> MALLOC: 16 Thread heaps in use
>>> MALLOC: 8192 Tcmalloc page size
>>>
>>> Call ReleaseFreeMemory() to release freelist memory to the OS (via
>>> madvise()).
>>> Bytes released to the OS take up virtual address space but no physical
>>> memory.
>>>
>>>
>>> I've tried the release command but it keeps using the same memory.
>>>
>>> greetings!
>>>
>>>
>>> 2018-07-19 0:25 GMT+02:00 Gregory Farnum :
>>>
The MDS think it's using 486MB of cache right now, and while that's
not a complete accounting (I believe you should generally multiply by
1.5 the configured cache limit to get a realistic memory consumption
model) it's obviously a long way from 12.5GB. You might try going in
with the "ceph daemon" command and looking at the heap stats (I forget
the exact command, but it will tell you if you run "help" against it)
and seeing what those say — you may have one of the slightly-broken
base systems and find that running the "heap release" (or similar
wording) command will free up a lot of RAM back to the OS!
-Greg
On Wed, Jul 18, 2018 at 1:53 PM, Daniel Carrasco
wrote:
> Hello,
>
> I've created a 3 nodes cluster with MON, MGR, OSD and MDS on all (2
MDS
> actives), and I've noticed that MDS is using a lot of memory (just
now is
> using 12.5GB of RAM):
> # ceph daemon mds.kavehome-mgto-pro-fs01 dump_mempools | jq -c
'.mds_co';
> ceph daemon mds.kavehome-mgto-pro-fs01 perf dump | jq '.mds_mem.rss'
> {"items":9272259,"bytes":510032260}
> 12466648
>
> I've configured the limit:
> mds_cache_memory_limit = 536870912
>
> But looks like is ignored, because is about 512Mb and is using a lot
more.
>
> Is there any way to limit the memory usage of MDS, because is giving
a lot
> of troubles because start to swap.
> Maybe I've to limit the cached inodes?
>
> The other active MDS is using a lot less memory (2.5Gb). but also is
using
> more than 512Mb. The standby
Re: [ceph-users] Fwd: MDS memory usage is very high
Thanks again,
I was trying to use fuse client instead Ubuntu 16.04 kernel module to see
if maybe is a client side problem, but CPU usage on fuse client is very
high (a 100% and even more in a two cores machine), so I'd to rever to
kernel client that uses much less CPU.
Is a web server, so maybe the problem is that. PHP and Nginx should open a
lot of files and maybe that uses a lot of RAM.
For now I've rebooted the machine because is the only way to free the
memory, but I cannot restart the machine every few hours...
Greetings!!
2018-07-19 1:00 GMT+02:00 Gregory Farnum :
> Wow, yep, apparently the MDS has another 9GB of allocated RAM outside of
> the cache! Hopefully one of the current FS users or devs has some idea. All
> I can suggest is looking to see if there are a bunch of stuck requests or
> something that are taking up memory which isn’t properly counted.
>
> On Wed, Jul 18, 2018 at 3:48 PM Daniel Carrasco
> wrote:
>
>> Hello, thanks for your response.
>>
>> This is what I get:
>>
>> # ceph tell mds.kavehome-mgto-pro-fs01 heap stats
>> 2018-07-19 00:43:46.142560 7f5a7a7fc700 0 client.1318388 ms_handle_reset
>> on 10.22.0.168:6800/1129848128
>> 2018-07-19 00:43:46.181133 7f5a7b7fe700 0 client.1318391 ms_handle_reset
>> on 10.22.0.168:6800/1129848128
>> mds.kavehome-mgto-pro-fs01 tcmalloc heap stats:
>>
>> MALLOC: 9982980144 ( 9520.5 MiB) Bytes in use by application
>> MALLOC: +0 (0.0 MiB) Bytes in page heap freelist
>> MALLOC: +172148208 ( 164.2 MiB) Bytes in central cache freelist
>> MALLOC: + 19031168 ( 18.1 MiB) Bytes in transfer cache freelist
>> MALLOC: + 23987552 ( 22.9 MiB) Bytes in thread cache freelists
>> MALLOC: + 20869280 ( 19.9 MiB) Bytes in malloc metadata
>> MALLOC:
>> MALLOC: = 10219016352 ( 9745.6 MiB) Actual memory used (physical + swap)
>> MALLOC: + 3913687040 ( 3732.4 MiB) Bytes released to OS (aka unmapped)
>> MALLOC:
>> MALLOC: = 14132703392 (13478.0 MiB) Virtual address space used
>> MALLOC:
>> MALLOC: 63875 Spans in use
>> MALLOC: 16 Thread heaps in use
>> MALLOC: 8192 Tcmalloc page size
>>
>> Call ReleaseFreeMemory() to release freelist memory to the OS (via
>> madvise()).
>> Bytes released to the OS take up virtual address space but no physical
>> memory.
>>
>>
>> I've tried the release command but it keeps using the same memory.
>>
>> greetings!
>>
>>
>> 2018-07-19 0:25 GMT+02:00 Gregory Farnum :
>>
>>> The MDS think it's using 486MB of cache right now, and while that's
>>> not a complete accounting (I believe you should generally multiply by
>>> 1.5 the configured cache limit to get a realistic memory consumption
>>> model) it's obviously a long way from 12.5GB. You might try going in
>>> with the "ceph daemon" command and looking at the heap stats (I forget
>>> the exact command, but it will tell you if you run "help" against it)
>>> and seeing what those say — you may have one of the slightly-broken
>>> base systems and find that running the "heap release" (or similar
>>> wording) command will free up a lot of RAM back to the OS!
>>> -Greg
>>>
>>> On Wed, Jul 18, 2018 at 1:53 PM, Daniel Carrasco
>>> wrote:
>>> > Hello,
>>> >
>>> > I've created a 3 nodes cluster with MON, MGR, OSD and MDS on all (2 MDS
>>> > actives), and I've noticed that MDS is using a lot of memory (just now
>>> is
>>> > using 12.5GB of RAM):
>>> > # ceph daemon mds.kavehome-mgto-pro-fs01 dump_mempools | jq -c
>>> '.mds_co';
>>> > ceph daemon mds.kavehome-mgto-pro-fs01 perf dump | jq '.mds_mem.rss'
>>> > {"items":9272259,"bytes":510032260}
>>> > 12466648
>>> >
>>> > I've configured the limit:
>>> > mds_cache_memory_limit = 536870912
>>> >
>>> > But looks like is ignored, because is about 512Mb and is using a lot
>>> more.
>>> >
>>> > Is there any way to limit the memory usage of MDS, because is giving a
>>> lot
>>> > of troubles because start to swap.
>>> > Maybe I've to limit the cached inodes?
>>> >
>>> > The other active MDS is using a lot less memory (2.5Gb). but also is
>>> using
>>> > more than 512Mb. The standby MDS is not using memory it all.
>>> >
>>> > I'm using the version:
>>> > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5)
>>> luminous
>>> > (stable).
>>> >
>>> > Thanks!!
>>> > --
>>> > _
>>> >
>>> > Daniel Carrasco Marín
>>> > Ingeniería para la Innovación i2TIC, S.L.
>>> > Tlf: +34 911 12 32 84 Ext: 223
>>> > www.i2tic.com
>>> > _
>>> >
>>> >
>>> >
>>> > --
>>> > _
>>> >
>>> > Daniel Carrasco Marín
>>> > Ingeniería para la Innovación i2TIC, S.L.
>>> > Tlf: +34 911 12 32 84 Ext: 223
>>> > www.i2tic.com
>>> > _
>>>
Re: [ceph-users] Fwd: MDS memory usage is very high
Wow, yep, apparently the MDS has another 9GB of allocated RAM outside of
the cache! Hopefully one of the current FS users or devs has some idea. All
I can suggest is looking to see if there are a bunch of stuck requests or
something that are taking up memory which isn’t properly counted.
On Wed, Jul 18, 2018 at 3:48 PM Daniel Carrasco
wrote:
> Hello, thanks for your response.
>
> This is what I get:
>
> # ceph tell mds.kavehome-mgto-pro-fs01 heap stats
> 2018-07-19 00:43:46.142560 7f5a7a7fc700 0 client.1318388 ms_handle_reset
> on 10.22.0.168:6800/1129848128
> 2018-07-19 00:43:46.181133 7f5a7b7fe700 0 client.1318391 ms_handle_reset
> on 10.22.0.168:6800/1129848128
> mds.kavehome-mgto-pro-fs01 tcmalloc heap
> stats:
> MALLOC: 9982980144 ( 9520.5 MiB) Bytes in use by application
> MALLOC: +0 (0.0 MiB) Bytes in page heap freelist
> MALLOC: +172148208 ( 164.2 MiB) Bytes in central cache freelist
> MALLOC: + 19031168 ( 18.1 MiB) Bytes in transfer cache freelist
> MALLOC: + 23987552 ( 22.9 MiB) Bytes in thread cache freelists
> MALLOC: + 20869280 ( 19.9 MiB) Bytes in malloc metadata
> MALLOC:
> MALLOC: = 10219016352 ( 9745.6 MiB) Actual memory used (physical + swap)
> MALLOC: + 3913687040 ( 3732.4 MiB) Bytes released to OS (aka unmapped)
> MALLOC:
> MALLOC: = 14132703392 (13478.0 MiB) Virtual address space used
> MALLOC:
> MALLOC: 63875 Spans in use
> MALLOC: 16 Thread heaps in use
> MALLOC: 8192 Tcmalloc page size
>
> Call ReleaseFreeMemory() to release freelist memory to the OS (via
> madvise()).
> Bytes released to the OS take up virtual address space but no physical
> memory.
>
>
> I've tried the release command but it keeps using the same memory.
>
> greetings!
>
>
> 2018-07-19 0:25 GMT+02:00 Gregory Farnum :
>
>> The MDS think it's using 486MB of cache right now, and while that's
>> not a complete accounting (I believe you should generally multiply by
>> 1.5 the configured cache limit to get a realistic memory consumption
>> model) it's obviously a long way from 12.5GB. You might try going in
>> with the "ceph daemon" command and looking at the heap stats (I forget
>> the exact command, but it will tell you if you run "help" against it)
>> and seeing what those say — you may have one of the slightly-broken
>> base systems and find that running the "heap release" (or similar
>> wording) command will free up a lot of RAM back to the OS!
>> -Greg
>>
>> On Wed, Jul 18, 2018 at 1:53 PM, Daniel Carrasco
>> wrote:
>> > Hello,
>> >
>> > I've created a 3 nodes cluster with MON, MGR, OSD and MDS on all (2 MDS
>> > actives), and I've noticed that MDS is using a lot of memory (just now
>> is
>> > using 12.5GB of RAM):
>> > # ceph daemon mds.kavehome-mgto-pro-fs01 dump_mempools | jq -c
>> '.mds_co';
>> > ceph daemon mds.kavehome-mgto-pro-fs01 perf dump | jq '.mds_mem.rss'
>> > {"items":9272259,"bytes":510032260}
>> > 12466648
>> >
>> > I've configured the limit:
>> > mds_cache_memory_limit = 536870912
>> >
>> > But looks like is ignored, because is about 512Mb and is using a lot
>> more.
>> >
>> > Is there any way to limit the memory usage of MDS, because is giving a
>> lot
>> > of troubles because start to swap.
>> > Maybe I've to limit the cached inodes?
>> >
>> > The other active MDS is using a lot less memory (2.5Gb). but also is
>> using
>> > more than 512Mb. The standby MDS is not using memory it all.
>> >
>> > I'm using the version:
>> > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous
>> > (stable).
>> >
>> > Thanks!!
>> > --
>> > _
>> >
>> > Daniel Carrasco Marín
>> > Ingeniería para la Innovación i2TIC, S.L.
>> > Tlf: +34 911 12 32 84 Ext: 223
>> > www.i2tic.com
>> > _
>> >
>> >
>> >
>> > --
>> > _
>> >
>> > Daniel Carrasco Marín
>> > Ingeniería para la Innovación i2TIC, S.L.
>> > Tlf: +34 911 12 32 84 Ext: 223
>> > www.i2tic.com
>> > _
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
>
>
>
> --
> _
>
> Daniel Carrasco Marín
> Ingeniería para la Innovación i2TIC, S.L.
> Tlf: +34 911 12 32 84 Ext: 223
> www.i2tic.com
> _
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Fwd: MDS memory usage is very high
Hello, thanks for your response.
This is what I get:
# ceph tell mds.kavehome-mgto-pro-fs01 heap stats
2018-07-19 00:43:46.142560 7f5a7a7fc700 0 client.1318388 ms_handle_reset
on 10.22.0.168:6800/1129848128
2018-07-19 00:43:46.181133 7f5a7b7fe700 0 client.1318391 ms_handle_reset
on 10.22.0.168:6800/1129848128
mds.kavehome-mgto-pro-fs01 tcmalloc heap
stats:
MALLOC: 9982980144 ( 9520.5 MiB) Bytes in use by application
MALLOC: +0 (0.0 MiB) Bytes in page heap freelist
MALLOC: +172148208 ( 164.2 MiB) Bytes in central cache freelist
MALLOC: + 19031168 ( 18.1 MiB) Bytes in transfer cache freelist
MALLOC: + 23987552 ( 22.9 MiB) Bytes in thread cache freelists
MALLOC: + 20869280 ( 19.9 MiB) Bytes in malloc metadata
MALLOC:
MALLOC: = 10219016352 ( 9745.6 MiB) Actual memory used (physical + swap)
MALLOC: + 3913687040 ( 3732.4 MiB) Bytes released to OS (aka unmapped)
MALLOC:
MALLOC: = 14132703392 (13478.0 MiB) Virtual address space used
MALLOC:
MALLOC: 63875 Spans in use
MALLOC: 16 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
Call ReleaseFreeMemory() to release freelist memory to the OS (via
madvise()).
Bytes released to the OS take up virtual address space but no physical
memory.
I've tried the release command but it keeps using the same memory.
greetings!
2018-07-19 0:25 GMT+02:00 Gregory Farnum :
> The MDS think it's using 486MB of cache right now, and while that's
> not a complete accounting (I believe you should generally multiply by
> 1.5 the configured cache limit to get a realistic memory consumption
> model) it's obviously a long way from 12.5GB. You might try going in
> with the "ceph daemon" command and looking at the heap stats (I forget
> the exact command, but it will tell you if you run "help" against it)
> and seeing what those say — you may have one of the slightly-broken
> base systems and find that running the "heap release" (or similar
> wording) command will free up a lot of RAM back to the OS!
> -Greg
>
> On Wed, Jul 18, 2018 at 1:53 PM, Daniel Carrasco
> wrote:
> > Hello,
> >
> > I've created a 3 nodes cluster with MON, MGR, OSD and MDS on all (2 MDS
> > actives), and I've noticed that MDS is using a lot of memory (just now is
> > using 12.5GB of RAM):
> > # ceph daemon mds.kavehome-mgto-pro-fs01 dump_mempools | jq -c '.mds_co';
> > ceph daemon mds.kavehome-mgto-pro-fs01 perf dump | jq '.mds_mem.rss'
> > {"items":9272259,"bytes":510032260}
> > 12466648
> >
> > I've configured the limit:
> > mds_cache_memory_limit = 536870912
> >
> > But looks like is ignored, because is about 512Mb and is using a lot
> more.
> >
> > Is there any way to limit the memory usage of MDS, because is giving a
> lot
> > of troubles because start to swap.
> > Maybe I've to limit the cached inodes?
> >
> > The other active MDS is using a lot less memory (2.5Gb). but also is
> using
> > more than 512Mb. The standby MDS is not using memory it all.
> >
> > I'm using the version:
> > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous
> > (stable).
> >
> > Thanks!!
> > --
> > _
> >
> > Daniel Carrasco Marín
> > Ingeniería para la Innovación i2TIC, S.L.
> > Tlf: +34 911 12 32 84 Ext: 223
> > www.i2tic.com
> > _
> >
> >
> >
> > --
> > _
> >
> > Daniel Carrasco Marín
> > Ingeniería para la Innovación i2TIC, S.L.
> > Tlf: +34 911 12 32 84 Ext: 223
> > www.i2tic.com
> > _
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
--
_
Daniel Carrasco Marín
Ingeniería para la Innovación i2TIC, S.L.
Tlf: +34 911 12 32 84 Ext: 223
www.i2tic.com
_
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Fwd: MDS memory usage is very high
The MDS think it's using 486MB of cache right now, and while that's
not a complete accounting (I believe you should generally multiply by
1.5 the configured cache limit to get a realistic memory consumption
model) it's obviously a long way from 12.5GB. You might try going in
with the "ceph daemon" command and looking at the heap stats (I forget
the exact command, but it will tell you if you run "help" against it)
and seeing what those say — you may have one of the slightly-broken
base systems and find that running the "heap release" (or similar
wording) command will free up a lot of RAM back to the OS!
-Greg
On Wed, Jul 18, 2018 at 1:53 PM, Daniel Carrasco wrote:
> Hello,
>
> I've created a 3 nodes cluster with MON, MGR, OSD and MDS on all (2 MDS
> actives), and I've noticed that MDS is using a lot of memory (just now is
> using 12.5GB of RAM):
> # ceph daemon mds.kavehome-mgto-pro-fs01 dump_mempools | jq -c '.mds_co';
> ceph daemon mds.kavehome-mgto-pro-fs01 perf dump | jq '.mds_mem.rss'
> {"items":9272259,"bytes":510032260}
> 12466648
>
> I've configured the limit:
> mds_cache_memory_limit = 536870912
>
> But looks like is ignored, because is about 512Mb and is using a lot more.
>
> Is there any way to limit the memory usage of MDS, because is giving a lot
> of troubles because start to swap.
> Maybe I've to limit the cached inodes?
>
> The other active MDS is using a lot less memory (2.5Gb). but also is using
> more than 512Mb. The standby MDS is not using memory it all.
>
> I'm using the version:
> ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous
> (stable).
>
> Thanks!!
> --
> _
>
> Daniel Carrasco Marín
> Ingeniería para la Innovación i2TIC, S.L.
> Tlf: +34 911 12 32 84 Ext: 223
> www.i2tic.com
> _
>
>
>
> --
> _
>
> Daniel Carrasco Marín
> Ingeniería para la Innovación i2TIC, S.L.
> Tlf: +34 911 12 32 84 Ext: 223
> www.i2tic.com
> _
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Fwd: MDS memory usage is very high
Hello,
I've created a 3 nodes cluster with MON, MGR, OSD and MDS on all (2 MDS
actives), and I've noticed that MDS is using a lot of memory (just now is
using 12.5GB of RAM):
# ceph daemon mds.kavehome-mgto-pro-fs01 dump_mempools | jq -c '.mds_co';
ceph daemon mds.kavehome-mgto-pro-fs01 perf dump | jq '.mds_mem.rss'
{"items":9272259,"bytes":510032260}
12466648
I've configured the limit:
mds_cache_memory_limit = 536870912
But looks like is ignored, because is about 512Mb and is using a lot more.
Is there any way to limit the memory usage of MDS, because is giving a lot
of troubles because start to swap.
Maybe I've to limit the cached inodes?
The other active MDS is using a lot less memory (2.5Gb). but also is using
more than 512Mb. The standby MDS is not using memory it all.
I'm using the version:
ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous
(stable).
Thanks!!
--
_
Daniel Carrasco Marín
Ingeniería para la Innovación i2TIC, S.L.
Tlf: +34 911 12 32 84 Ext: 223
www.i2tic.com
_
--
_
Daniel Carrasco Marín
Ingeniería para la Innovación i2TIC, S.L.
Tlf: +34 911 12 32 84 Ext: 223
www.i2tic.com
_
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
