Hello, How many time is neccesary?, because is a production environment and memory profiler + low cache size because the problem, gives a lot of CPU usage from OSD and MDS that makes it fails while profiler is running. Is there any problem if is done in a low traffic time? (less usage and maybe it don't fails, but maybe less info about usage).
Greetings! 2018-07-24 10:21 GMT+02:00 Yan, Zheng <uker...@gmail.com>: > I mean: > > ceph tell mds.x heap start_profiler > > ... wait for some time > > ceph tell mds.x heap stop_profiler > > pprof --text /usr/bin/ceph-mds > /var/log/ceph/ceph-mds.x.profile.<largest number>.heap > > > > > On Tue, Jul 24, 2018 at 3:18 PM Daniel Carrasco <d.carra...@i2tic.com> > wrote: > > > > This is what i get: > > > > -------------------------------------------------------- > > -------------------------------------------------------- > > -------------------------------------------------------- > > :/# ceph tell mds.kavehome-mgto-pro-fs01 heap dump > > 2018-07-24 09:05:19.350720 7fc562ffd700 0 client.1452545 > ms_handle_reset on 10.22.0.168:6800/1685786126 > > 2018-07-24 09:05:29.103903 7fc563fff700 0 client.1452548 > ms_handle_reset on 10.22.0.168:6800/1685786126 > > mds.kavehome-mgto-pro-fs01 dumping heap profile now. > > ------------------------------------------------ > > MALLOC: 760199640 ( 725.0 MiB) Bytes in use by application > > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist > > MALLOC: + 246962320 ( 235.5 MiB) Bytes in central cache freelist > > MALLOC: + 43933664 ( 41.9 MiB) Bytes in transfer cache freelist > > MALLOC: + 41012664 ( 39.1 MiB) Bytes in thread cache freelists > > MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata > > MALLOC: ------------ > > MALLOC: = 1102295200 ( 1051.2 MiB) Actual memory used (physical + swap) > > MALLOC: + 4268335104 ( 4070.6 MiB) Bytes released to OS (aka unmapped) > > MALLOC: ------------ > > MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used > > MALLOC: > > MALLOC: 33027 Spans in use > > MALLOC: 19 Thread heaps in use > > MALLOC: 8192 Tcmalloc page size > > ------------------------------------------------ > > Call ReleaseFreeMemory() to release freelist memory to the OS (via > madvise()). > > Bytes released to the OS take up virtual address space but no physical > memory. > > > > > > -------------------------------------------------------- > > -------------------------------------------------------- > > -------------------------------------------------------- > > :/# ceph tell mds.kavehome-mgto-pro-fs01 heap stats > > 2018-07-24 09:14:25.747706 7f94fffff700 0 client.1452578 > ms_handle_reset on 10.22.0.168:6800/1685786126 > > 2018-07-24 09:14:25.754034 7f95057fa700 0 client.1452581 > ms_handle_reset on 10.22.0.168:6800/1685786126 > > mds.kavehome-mgto-pro-fs01 tcmalloc heap stats:------------------------ > ------------------------ > > MALLOC: 960649328 ( 916.1 MiB) Bytes in use by application > > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist > > MALLOC: + 108867288 ( 103.8 MiB) Bytes in central cache freelist > > MALLOC: + 37179424 ( 35.5 MiB) Bytes in transfer cache freelist > > MALLOC: + 40143000 ( 38.3 MiB) Bytes in thread cache freelists > > MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata > > MALLOC: ------------ > > MALLOC: = 1157025952 ( 1103.4 MiB) Actual memory used (physical + swap) > > MALLOC: + 4213604352 ( 4018.4 MiB) Bytes released to OS (aka unmapped) > > MALLOC: ------------ > > MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used > > MALLOC: > > MALLOC: 33028 Spans in use > > MALLOC: 19 Thread heaps in use > > MALLOC: 8192 Tcmalloc page size > > ------------------------------------------------ > > Call ReleaseFreeMemory() to release freelist memory to the OS (via > madvise()). > > Bytes released to the OS take up virtual address space but no physical > memory. > > > > -------------------------------------------------------- > > -------------------------------------------------------- > > -------------------------------------------------------- > > After heap release: > > :/# ceph tell mds.kavehome-mgto-pro-fs01 heap stats > > 2018-07-24 09:15:28.540203 7f2f7affd700 0 client.1443339 > ms_handle_reset on 10.22.0.168:6800/1685786126 > > 2018-07-24 09:15:28.547153 7f2f7bfff700 0 client.1443342 > ms_handle_reset on 10.22.0.168:6800/1685786126 > > mds.kavehome-mgto-pro-fs01 tcmalloc heap stats:------------------------ > ------------------------ > > MALLOC: 710315776 ( 677.4 MiB) Bytes in use by application > > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist > > MALLOC: + 246471880 ( 235.1 MiB) Bytes in central cache freelist > > MALLOC: + 40802848 ( 38.9 MiB) Bytes in transfer cache freelist > > MALLOC: + 38689304 ( 36.9 MiB) Bytes in thread cache freelists > > MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata > > MALLOC: ------------ > > MALLOC: = 1046466720 ( 998.0 MiB) Actual memory used (physical + swap) > > MALLOC: + 4324163584 ( 4123.8 MiB) Bytes released to OS (aka unmapped) > > MALLOC: ------------ > > MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used > > MALLOC: > > MALLOC: 33177 Spans in use > > MALLOC: 19 Thread heaps in use > > MALLOC: 8192 Tcmalloc page size > > ------------------------------------------------ > > Call ReleaseFreeMemory() to release freelist memory to the OS (via > madvise()). > > Bytes released to the OS take up virtual address space but no physical > memory. > > > > > > The other commands fails with a curl error: > > Failed to get profile: curl 'http:///pprof/profile?seconds=30' > > /root/pprof/.tmp.ceph-mds.1532416424.: > > > > > > Greetings!! > > > > 2018-07-24 5:35 GMT+02:00 Yan, Zheng <uker...@gmail.com>: > >> > >> could you profile memory allocation of mds > >> > >> http://docs.ceph.com/docs/mimic/rados/troubleshooting/memory-profiling/ > >> On Tue, Jul 24, 2018 at 7:54 AM Daniel Carrasco <d.carra...@i2tic.com> > wrote: > >> > > >> > Yeah, is also my thread. This thread was created before lower the > cache size from 512Mb to 8Mb. I thought that maybe was my fault and I did a > misconfiguration, so I've ignored the problem until now. > >> > > >> > Greetings! > >> > > >> > El mar., 24 jul. 2018 1:00, Gregory Farnum <gfar...@redhat.com> > escribió: > >> >> > >> >> On Mon, Jul 23, 2018 at 11:08 AM Patrick Donnelly < > pdonn...@redhat.com> wrote: > >> >>> > >> >>> On Mon, Jul 23, 2018 at 5:48 AM, Daniel Carrasco < > d.carra...@i2tic.com> wrote: > >> >>> > Hi, thanks for your response. > >> >>> > > >> >>> > Clients are about 6, and 4 of them are the most of time on > standby. Only two > >> >>> > are active servers that are serving the webpage. Also we've a > varnish on > >> >>> > front, so are not getting all the load (below 30% in PHP is not > much). > >> >>> > About the MDS cache, now I've the mds_cache_memory_limit at 8Mb. > >> >>> > >> >>> What! Please post `ceph daemon mds.<name> config diff`, `... perf > >> >>> dump`, and `... dump_mempools ` from the server the active MDS is > on. > >> >>> > >> >>> > I've tested > >> >>> > also 512Mb, but the CPU usage is the same and the MDS RAM usage > grows up to > >> >>> > 15GB (on a 16Gb server it starts to swap and all fails). With > 8Mb, at least > >> >>> > the memory usage is stable on less than 6Gb (now is using about > 1GB of RAM). > >> >>> > >> >>> We've seen reports of possible memory leaks before and the potential > >> >>> fixes for those were in 12.2.6. How fast does your MDS reach 15GB? > >> >>> Your MDS cache size should be configured to 1-8GB (depending on your > >> >>> preference) so it's disturbing to see you set it so low. > >> >> > >> >> > >> >> See also the thread "[ceph-users] Fwd: MDS memory usage is very > high", which had more discussion of that. The MDS daemon seemingly had > 9.5GB of allocated RSS but only believed 489MB was in use for the cache... > >> >> -Greg > >> >> > >> >>> > >> >>> > >> >>> -- > >> >>> Patrick Donnelly > >> >>> _______________________________________________ > >> >>> ceph-users mailing list > >> >>> ceph-users@lists.ceph.com > >> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > >> > _______________________________________________ > >> > ceph-users mailing list > >> > ceph-users@lists.ceph.com > >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > -- > > _________________________________________ > > > > Daniel Carrasco Marín > > Ingeniería para la Innovación i2TIC, S.L. > > Tlf: +34 911 12 32 84 Ext: 223 > > www.i2tic.com > > _________________________________________ > -- _________________________________________ Daniel Carrasco Marín Ingeniería para la Innovación i2TIC, S.L. Tlf: +34 911 12 32 84 Ext: 223 www.i2tic.com _________________________________________
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
