I try to dump perf counter via admin socket, but I don't know what does these numbers actual mean or does these numbers have any thing to do with the different memory usage between arm and amd processors, so I attach the dump log as attachment(mon.a runs on AMD processor, mon.c runs on ARM processor).
PS: after days of running(mon.b is 6 day, mon.c is 3 day), the memory consumption of both monitor running on arm board become stable, some what about 600MB, here is the heap stats: mon.btcmalloc heap stats:------------------------------------------------ MALLOC: 594258992 ( 566.7 MiB) Bytes in use by application MALLOC: + 19529728 ( 18.6 MiB) Bytes in page heap freelist MALLOC: + 3885120 ( 3.7 MiB) Bytes in central cache freelist MALLOC: + 6486528 ( 6.2 MiB) Bytes in transfer cache freelist MALLOC: + 12202384 ( 11.6 MiB) Bytes in thread cache freelists MALLOC: + 2889952 ( 2.8 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 639252704 ( 609.6 MiB) Actual memory used (physical + swap) MALLOC: + 122880 ( 0.1 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 639375584 ( 609.8 MiB) Virtual address space used MALLOC: MALLOC: 10231 Spans in use MALLOC: 24 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the mon.ctcmalloc heap stats:------------------------------------------------ MALLOC: 593987584 ( 566.5 MiB) Bytes in use by application MALLOC: + 23969792 ( 22.9 MiB) Bytes in page heap freelist MALLOC: + 2172640 ( 2.1 MiB) Bytes in central cache freelist MALLOC: + 5874688 ( 5.6 MiB) Bytes in transfer cache freelist MALLOC: + 9268512 ( 8.8 MiB) Bytes in thread cache freelists MALLOC: + 2889952 ( 2.8 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 638163168 ( 608.6 MiB) Actual memory used (physical + swap) MALLOC: + 163840 ( 0.2 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 638327008 ( 608.8 MiB) Virtual address space used MALLOC: MALLOC: 9796 Spans in use MALLOC: 14 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the On Fri, Nov 8, 2013 at 12:03 PM, Gregory Farnum <g...@inktank.com> wrote: > I don't think this is anything we've observed before. Normally when a > Ceph node is using more memory than its peers it's a consequence of > something in that node getting backed up. You might try looking at the > perf counters via the admin socket and seeing if something about them > is different between your ARM and AMD processors. > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > On Tue, Nov 5, 2013 at 7:21 AM, Yu Changyuan <rei...@gmail.com> wrote: > > Finally, my tiny ceph cluster get 3 monitors, newly added mon.b and mon.c > > both running on cubieboard2, which is cheap but still with enough cpu > > power(dual-core arm A7 cpu, 1.2G) and memory(1G). > > > > But compare to mon.a which running on an amd64 cpu, both mon.b and mon.c > > easily consume too much memory, so I want to know whether this is caused > by > > memory leak. Below is the output of 'ceph tell mon.a heap stats' and > 'ceph > > tell mon.c heap stats'(mon.c only start 12hr ago, while mon.a already > > running for more than 10 days) > > > > mon.atcmalloc heap stats:------------------------------------------------ > > MALLOC: 5480160 ( 5.2 MiB) Bytes in use by application > > MALLOC: + 28065792 ( 26.8 MiB) Bytes in page heap freelist > > MALLOC: + 15242312 ( 14.5 MiB) Bytes in central cache freelist > > MALLOC: + 10116608 ( 9.6 MiB) Bytes in transfer cache freelist > > MALLOC: + 10432216 ( 9.9 MiB) Bytes in thread cache freelists > > MALLOC: + 1667224 ( 1.6 MiB) Bytes in malloc metadata > > MALLOC: ------------ > > MALLOC: = 71004312 ( 67.7 MiB) Actual memory used (physical + swap) > > MALLOC: + 57540608 ( 54.9 MiB) Bytes released to OS (aka unmapped) > > MALLOC: ------------ > > MALLOC: = 128544920 ( 122.6 MiB) Virtual address space used > > MALLOC: > > MALLOC: 4655 Spans in use > > MALLOC: 34 Thread heaps in use > > MALLOC: 8192 Tcmalloc page size > > ------------------------------------------------ > > Call ReleaseFreeMemory() to release freelist memory to the OS (via > > madvise()). > > Bytes released to the > > > > > > mon.ctcmalloc heap stats:------------------------------------------------ > > MALLOC: 175861640 ( 167.7 MiB) Bytes in use by application > > MALLOC: + 2220032 ( 2.1 MiB) Bytes in page heap freelist > > MALLOC: + 1007560 ( 1.0 MiB) Bytes in central cache freelist > > MALLOC: + 2871296 ( 2.7 MiB) Bytes in transfer cache freelist > > MALLOC: + 4686000 ( 4.5 MiB) Bytes in thread cache freelists > > MALLOC: + 2758880 ( 2.6 MiB) Bytes in malloc metadata > > MALLOC: ------------ > > MALLOC: = 189405408 ( 180.6 MiB) Actual memory used (physical + swap) > > MALLOC: + 0 ( 0.0 MiB) Bytes released to OS (aka unmapped) > > MALLOC: ------------ > > MALLOC: = 189405408 ( 180.6 MiB) Virtual address space used > > MALLOC: > > MALLOC: 3445 Spans in use > > MALLOC: 14 Thread heaps in use > > MALLOC: 8192 Tcmalloc page size > > ------------------------------------------------ > > Call ReleaseFreeMemory() to release freelist memory to the OS (via > > madvise()). > > Bytes released to the > > > > The ceph versin is 0.67.4, compiled with tcmalloc enabled, > > gcc(armv7a-hardfloat-linux-gnueabi-gcc) version 4.7.3 and I also try to > dump > > heap, but I can not find anything useful, below is a recent dump, output > by > > command "pprof --text /usr/bin/ceph-mon mon.c.profile.0021.heap". What > extra > > step should I take to make the dump more meaningful? > > > > Using local file /usr/bin/ceph-mon. > > Using local file mon.c.profile.0021.heap. > > Total: 149.3 MB > > 146.2 97.9% 97.9% 146.2 97.9% 00000000b6a7ce7c > > 1.4 0.9% 98.9% 1.4 0.9% std::basic_string::_Rep::_S_create > > ??:0 > > 1.4 0.9% 99.8% 1.4 0.9% 00000000002dd794 > > 0.1 0.1% 99.9% 0.1 0.1% 00000000b6a81170 > > 0.1 0.1% 99.9% 0.1 0.1% 00000000b6a80894 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e2ac > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a81410 > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000367450 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d4474 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000028847c > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e8d8 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000020c80c > > 0.0 0.0% 100.0% 0.0 0.0% 000000000028bd20 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a63248 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a83478 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a806f0 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000002eb8b8 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000024efb4 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000027e550 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a77104 > > 0.0 0.0% 100.0% 0.0 0.0% _dl_mcount ??:0 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000003673ec > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7a91c > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000295e44 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7ee38 > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000283948 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000002a53c4 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7665c > > 0.0 0.0% 100.0% 0.0 0.0% 00000000002c4590 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e88c > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a8456c > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a76ed4 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a842f0 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a72bd0 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a73cf8 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7100c > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7dec4 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000035e6e8 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a78f68 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7de9c > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000220528 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000035e7c0 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a6b2f8 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a80a04 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a62e7c > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a66f50 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e958 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a6cfb8 > > 0.0 0.0% 100.0% 0.0 0.0% leveldb::DBImpl::MakeRoomForWrite > > (inline) ??:0 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000020797c > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a69de0 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d0af0 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d0ebc > > 0.0 0.0% 100.0% 0.0 0.0% 00000000002a0cd4 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000036909c > > 0.0 0.0% 100.0% 0.0 0.0% 000000000040b02c > > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d0b68 > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000392fa0 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a64404 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a791b4 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d9824 > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000213928 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000002a0cb8 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000002a4fcc > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a725ac > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a66308 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a79068 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000000013b2 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000040b000 > > 0.0 0.0% 100.0% 0.1 0.1% 00000000004d839b > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f29887 > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f3eb6b > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f6e1cb > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f6edab > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f873ab > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f8a26b > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f8b0cb > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f92dab > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f9c96b > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fa24bf > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fadd8b > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fb06ab > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fb0d0b > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fb494b > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fbad6b > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fbb2cb > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fbea6b > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fed0eb > > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fed69b > > 0.0 0.0% 100.0% 0.0 0.0% 000000000129920b > > 0.0 0.0% 100.0% 0.0 0.0% 00000000014250eb > > 0.0 0.0% 100.0% 0.0 0.0% 000000000166cfc5 > > 0.0 0.0% 100.0% 0.1 0.1% 000000000166d711 > > 0.0 0.0% 100.0% 0.0 0.0% 0000000003531d2b > > 0.0 0.0% 100.0% 0.0 0.0% 000000000379adbb > > 0.0 0.0% 100.0% 0.0 0.0% 0000000004e888fb > > 0.0 0.0% 100.0% 0.0 0.0% 0000000004e894ab > > 0.0 0.0% 100.0% 0.0 0.0% 0000000004e8951b > > 0.0 0.0% 100.0% 0.0 0.0% 00000000060146d3 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000601482f > > 0.0 0.0% 100.0% 0.0 0.0% 00000000060fcd2b > > 0.0 0.0% 100.0% 0.0 0.0% 00000000060fd33b > > 0.0 0.0% 100.0% 0.0 0.0% 00000000060fdfbb > > 0.0 0.0% 100.0% 0.0 0.0% 000000000a820749 > > 0.0 0.0% 100.0% 0.0 0.0% 000000000bfb1950 > > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a43f23 > > 0.0 0.0% 100.0% 0.0 0.0% __clone ??:0 > > 0.0 0.0% 100.0% 0.0 0.0% leveldb::DBImpl::MakeRoomForWrite > > ??:0 > > 0.0 0.0% 100.0% 0.2 0.1% std::num_put::do_put@806e4 ??:0 > > 0.0 0.0% 100.0% 0.4 0.2% std::num_put::do_put@80b44 ??:0 > > 0.0 0.0% 100.0% 0.1 0.1% std::num_put::do_put@80e00 ??:0 > > > > > > PS: there's a cubietruck > > board(http://docs.cubieboard.org/products/start#cubietruck_cubieboard3) > > released recently, which features a dual-core arm A7 cpu, 2G RAM, 1Gbit > eth > > port, and a sata 2.0 port, for $89, maybe suitable for cheap dedicate osd > > server with single disk. > > > > -- > > Best regards, > > Changyuan > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- Best regards, Changyuan
mon_a.dump.json
Description: application/json
mon_a.schema.json
Description: application/json
mon_c.dump.json
Description: application/json
mon_c.schema.json
Description: application/json
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
