James Page james.p...@ubuntu.com writes:
Any chance you could enable the profiler following:
http://ceph.com/docs/master/rados/troubleshooting/memory-profiling/
I have a hunch that this might actually be related to the version of
google-perftools with have in precise - but this at least will give us
some more information about what's chewing memory.
So, given:
57274 root 20 0 1427m 713m 9932 S1 4.5 250:20.11 /usr/bin/ceph-osd
--cluster=ceph -i 34 -f
58731 root 20 0 1001m 470m 10m S 80 2.9 229:51.78 /usr/bin/ceph-osd
--cluster=ceph -i 36 -f
60881 root 20 0 998m 461m 11m S0 2.9 181:30.65 /usr/bin/ceph-osd
--cluster=ceph -i 37 -f
64272 root 20 0 910m 351m 11m S1 2.2 197:16.14 /usr/bin/ceph-osd
--cluster=ceph -i 35 -f
64584 root 20 0 855m 321m 10m S1 2.0 101:52.52 /usr/bin/ceph-osd
--cluster=ceph -i 2 -f
60613 root 20 0 824m 307m 9744 S0 1.9 164:26.79 /usr/bin/ceph-osd
--cluster=ceph -i 3 -f
58493 root 20 0 785m 296m 8852 S0 1.9 147:28.98 /usr/bin/ceph-osd
--cluster=ceph -i 4 -f
64948 root 20 0 773m 273m 9.9m S0 1.7 150:23.20 /usr/bin/ceph-osd
--cluster=ceph -i 33 -f
I choose OSD 34.
2013-08-29 13:09:57.574853 7fcdbcdeb700 0 log [INF] : osd.34 started profiler
2013-08-29 13:09:57.576890 7fcdbcdeb700 0 osd.34 6002 do_command r=0
2013-08-29 13:10:06.076818 7fcdbcdeb700 0 log [INF] : osd.34tcmalloc heap
stats:
2013-08-29 13:10:06.138029 7fcdbcdeb700 0 log [INF] : MALLOC: 666882640 (
636.0 MB) Bytes in use by application
2013-08-29 13:10:06.138057 7fcdbcdeb700 0 log [INF] : MALLOC: + 62881792 (
60.0 MB) Bytes in page heap freelist
2013-08-29 13:10:06.138080 7fcdbcdeb700 0 log [INF] : MALLOC: + 3149808 (
3.0 MB) Bytes in central cache freelist
2013-08-29 13:10:06.138097 7fcdbcdeb700 0 log [INF] : MALLOC: +83456 (
0.1 MB) Bytes in transfer cache freelist
2013-08-29 13:10:06.208103 7fcdbcdeb700 0 log [INF] : MALLOC: + 5633984 (
5.4 MB) Bytes in thread cache freelists
2013-08-29 13:10:06.209350 7fcdbcdeb700 0 log [INF] : MALLOC: + 6160384 (
5.9 MB) Bytes in malloc metadata
2013-08-29 13:10:06.209386 7fcdbcdeb700 0 log [INF] : MALLOC:
2013-08-29 13:10:06.209404 7fcdbcdeb700 0 log [INF] : MALLOC: =744792064 (
710.3 MB) Actual memory used (physical + swap)
2013-08-29 13:10:06.210670 7fcdbcdeb700 0 log [INF] : MALLOC: +143736832 (
137.1 MB) Bytes released to OS (aka unmapped)
2013-08-29 13:10:06.210685 7fcdbcdeb700 0 log [INF] : MALLOC:
2013-08-29 13:10:06.210903 7fcdbcdeb700 0 log [INF] : MALLOC: =888528896 (
847.4 MB) Virtual address space used
2013-08-29 13:10:06.210919 7fcdbcdeb700 0 log [INF] : MALLOC:
2013-08-29 13:10:06.211010 7fcdbcdeb700 0 log [INF] : MALLOC: 75893
Spans in use
2013-08-29 13:10:06.211338 7fcdbcdeb700 0 log [INF] : MALLOC:193
Thread heaps in use
2013-08-29 13:10:06.211350 7fcdbcdeb700 0 log [INF] : MALLOC: 4096
Tcmalloc page size
2013-08-29 13:10:06.211361 7fcdbcdeb700 0 log [INF] :
2013-08-29 13:10:06.211373 7fcdbcdeb700 0 log [INF] : Call ReleaseFreeMemory()
to release freelist memory to the OS (via madvise()).
2013-08-29 13:10:06.211396 7fcdbcdeb700 0 log [INF] : Bytes released to the OS
take
I've uploaded the heap dump to
https://chinstrap.canonical.com/~james/nx/osd.34.profile.0001.heap.gz
(This is only available to Canonical employees as it's not clear to me
if there's any potential leak of data in the heap dump.)
Please let me know if you need any more information.
--
James
** Changed in: ceph (Ubuntu)
Status: Incomplete = New
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1215014
Title:
ceph-osd from Folsom leaks memory constantly
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1215014/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
