Hi,
I'm running cephfs since 2 months now,
and my active msd memory usage is around 20G now (still growing).
ceph 1521539 10.8 31.2 20929836 20534868 ? Ssl janv.26 8573:34
/usr/bin/ceph-mds -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
USER PID %CPU %MEMVSZ RSS TT
Did the fs have lots of mount/umount? We recently found a memory leak
bug in that area https://github.com/ceph/ceph/pull/20148
Regards
Yan, Zheng
On Thu, Mar 22, 2018 at 5:29 PM, Alexandre DERUMIER wrote:
> Hi,
>
> I'm running cephfs since 2 months now,
>
> and my active msd memory usage is aro
amp; 12.2.4
I'll try to upgrade this buggy mds to 12.2.4 to see if it's helping.
- Mail original -
De: "Zheng Yan"
À: "aderumier"
Cc: "ceph-users"
Envoyé: Vendredi 23 Mars 2018 01:08:46
Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it
Hello Alexandre,
On Thu, Mar 22, 2018 at 2:29 AM, Alexandre DERUMIER wrote:
> Hi,
>
> I'm running cephfs since 2 months now,
>
> and my active msd memory usage is around 20G now (still growing).
>
> ceph 1521539 10.8 31.2 20929836 20534868 ? Ssl janv.26 8573:34
> /usr/bin/ceph-mds -f --cl
umier"
Cc: "ceph-users"
Envoyé: Mardi 27 Mars 2018 20:35:08
Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?
Hello Alexandre,
On Thu, Mar 22, 2018 at 2:29 AM, Alexandre DERUMIER
wrote:
> Hi,
>
> I'm running cephfs since 2 months now
},
"throttle-objecter_bytes": {
"val": 0,
"max": 104857600,
"get_started": 0,
"get": 0,
"get_sum": 0,
"get_or_fail_fail": 0,
"get_or_fail_success": 0,
"val": 0,
> "max": 104857600,
> "get_started": 0,
> "get": 1884071270,
> "get_sum": 12697353890803,
> "get_or_fail_fail": 0,
> "get_or_fail_success": 1884071270,
&
65197068,
"omap_del": 48058
},
"purge_queue": {
"pq_executing_ops": 0,
"pq_executing": 0,
"pq_executed": 619458
},
"throttle-msgr_dispatch_throttler-mds": {
"val": 0,
"
0,
> "req_lookupsnap": 0,
> "req_lssnap": 0,
> "req_mkdir": 12204,
> "req_mknod": 0,
> "req_mksnap": 0,
> "req_open": 106156167,
> "req_readdir": 20293077
Hello Brady,
On Thu, May 10, 2018 at 7:35 AM, Brady Deetz wrote:
> I am now seeing the exact same issues you are reporting. A heap release did
> nothing for me.
I'm not sure it's the same issue...
> [root@mds0 ~]# ceph daemon mds.mds0 config get mds_cache_memory_limit
> {
> "mds_cache_memor
[ceph-admin@mds0 ~]$ ps aux | grep ceph-mds
ceph1841 3.5 94.3 133703308 124425384 ? Ssl Apr04 1808:32
/usr/bin/ceph-mds -f --cluster ceph --id mds0 --setuser ceph --setgroup ceph
[ceph-admin@mds0 ~]$ sudo ceph daemon mds.mds0 cache status
{
"pool": {
"items": 173261056,
On Thu, May 10, 2018 at 12:00 PM, Brady Deetz wrote:
> [ceph-admin@mds0 ~]$ ps aux | grep ceph-mds
> ceph1841 3.5 94.3 133703308 124425384 ? Ssl Apr04 1808:32
> /usr/bin/ceph-mds -f --cluster ceph --id mds0 --setuser ceph --setgroup ceph
>
>
> [ceph-admin@mds0 ~]$ sudo ceph daemon mds.md
nvoyé: Jeudi 10 Mai 2018 21:11:19
Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?
On Thu, May 10, 2018 at 12:00 PM, Brady Deetz wrote:
> [ceph-admin@mds0 ~]$ ps aux | grep ceph-mds
> ceph 1841 3.5 94.3 133703308 124425384 ? Ssl Apr04 1808:32
> /usr/bin/ceph-mds
quot;imported": 0,
"imported_inodes": 0
}
}
Regards,
Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*
On Fri, May 11, 2018 at 3:13 PM Alexandre DERUMIER
wrote:
> Hi,
>
> I'm still seeing memory leak with 12
quot;
À: "ceph-users"
Envoyé: Vendredi 11 Mai 2018 20:18:27
Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?
You could use "mds_cache_size" to limit number of CAPS untill you have this
fixed, but I'd say for your number of caps and inodes, 20GB is norma
quot;:43683053,"bytes":5236668693}
8821728
{"items":44248833,"bytes":5310420155}
8821728
{"items":45013698,"bytes":5381693077}
8821728
{"items":44928825,"bytes":5313048602}
8821728
{"items":43828630,"bytes":5146482155}
On Sat, May 12, 2018 at 3:11 AM Alexandre DERUMIER
wrote:
> The documentation (luminous) say:
>
> >mds cache size
> >
> >Description:The number of inodes to cache. A value of 0 indicates an
> unlimited number. It is recommended to use mds_cache_memory_limit to limit
> the amount of memory t
y": 0,
"omap_wr": 3595801,
"omap_rd": 232070972,
"omap_del": 272598
},
"purge_queue": {
"pq_executing_ops": 0,
"pq_executing": 0,
"pq_executed": 1659514
},
"t
ive": 0,
> "command_send": 0,
> "command_resend": 0,
> "map_epoch": 3907,
> "map_full": 0,
> "map_inc": 601,
> "osd_sessions": 18,
> "osd_session_open": 20,
Envoyé: Jeudi 24 Mai 2018 11:34:20
Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?
On Tue, May 22, 2018 at 3:11 PM, Alexandre DERUMIER
wrote:
> Hi,some new stats, mds memory is not 16G,
>
> I have almost same number of items and bytes in cache vs some week
te"
ceph tell mds.xx heap stats
ceph daemon mds.xx config set mds_cache_size 0
>
>
> - Mail original -
> De: "Zheng Yan"
> À: "aderumier"
> Cc: "ceph-users"
> Envoyé: Jeudi 24 Mai 2018 11:34:20
> Objet: Re: [ceph-users] ceph mds mem
mds_cache_size 1 ?
- Mail original -
De: "Zheng Yan"
À: "aderumier"
Cc: "ceph-users"
Envoyé: Jeudi 24 Mai 2018 16:27:21
Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?
On Thu, May 24, 2018 at 7:22 PM, Alexandre DERUMIER
wr
ceph4-2.odiso.net config set mds_cache_size 0
{
"success": "mds_cache_size = '0' (not observed, change may require restart)
"
}
- Mail original -----
De: "Zheng Yan"
À: "aderumier"
Envoyé: Vendredi 25 Mai 2018 05:56:31
Objet: Re: [ceph-us
pace but no physical memory.
>
>
> root@ceph4-2:~# ceph daemon mds.ceph4-2.odiso.net config set mds_cache_size 0
> {
> "success": "mds_cache_size = '0' (not observed, change may require
> restart) "
> }
>
> - Mail original -
> De
tes released to the OS take up virtual address space but no physical
>> memory.
>>
>>
>> root@ceph4-2:~# ceph daemon mds.ceph4-2.odiso.net config set mds_cache_size 0
>> {
>> "success": "mds_cache_size = '0' (not observed, change may
I'm not sure this is a cache issue. To me, this feels like a memory leak.
I'm now at 129GB (haven't had a window to upgrade yet) on a configured 80GB
cache.
[root@mds0 ceph-admin]# ceph daemon mds.mds0 cache status
{
"pool": {
"items": 166753076,
"bytes": 71766944952
}
}
serve the rss memory increase.
>
>
>
>
> - Mail original -
> De: "Zheng Yan"
> À: "aderumier"
> Cc: "ceph-users"
> Envoyé: Jeudi 24 Mai 2018 11:34:20
> Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?
>
> On
ot;sum": 0.00000,
"avgtime": 0.0
}
},
"throttle-objecter_bytes": {
"val": 0,
"max": 104857600,
"get_started": 0,
"get": 0,
"get_sum": 0,
": 605991
> },
> "purge_queue": {
> "pq_executing_ops": 0,
> "pq_executing": 0,
> "pq_executed": 3118819
> },
> "throttle-msgr_dispatch_throttler-mds": {
> "val": 0
ease to test it.
Thanks you very much again !
I'll keep you in touch in this thread.
Regards,
Alexandre
- Mail original -
De: "Zheng Yan"
À: "aderumier"
Cc: "ceph-users"
Envoyé: Mardi 29 Mai 2018 04:40:27
Objet: Re: [ceph-users] ceph mds memory usa
"ceph-users"
Envoyé: Mardi 29 Mai 2018 04:40:27
Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?
Could you try path https://github.com/ceph/ceph/pull/22240/files.
The leakage of MMDSBeacon messages can explain your issue.
Regards
Yan, Zheng
On Mon, May 28, 2018
31 matches
Mail list logo
