On Wed, Oct 3, 2012 at 4:56 PM, Gregory Farnum <g...@inktank.com> wrote: > On Wed, Oct 3, 2012 at 4:23 PM, Tren Blackburn <t...@eotnetworks.com> wrote: >> On Wed, Oct 3, 2012 at 4:15 PM, Gregory Farnum <g...@inktank.com> wrote: >>> On Wed, Oct 3, 2012 at 3:22 PM, Tren Blackburn <t...@eotnetworks.com> wrote: >>>> Hi List; >>>> >>>> I was advised to use the "mds cache size" option to limit the memory >>>> that the mds process will take. I have it set to "32768". However it >>>> the ceph-mds process is now at 50GB and still growing. >>>> >>>> fern ceph # ps wwaux | grep ceph-mds >>>> root 895 4.3 26.6 53269304 52725820 ? Ssl Sep28 312:29 >>>> /usr/bin/ceph-mds -i fern --pid-file /var/run/ceph/mds.fern.pid -c >>>> /etc/ceph/ceph.conf >>>> >>>> Have I specified the limit incorrectly? How far will it go? >>> >>> Oof. That looks correct; it sounds like we have a leak or some other >>> kind of bug. I believe you're on Gentoo; did you build with tcmalloc? >>> If so, can you run "ceph -w" in one window and then "ceph mds tell 0 >>> heap stats" and send back the output? >>> If you didn't build with tcmalloc, can you do so and try again? We >>> have noticed fragmentation issues with the default memory allocator, >>> which is why we switched (though I can't imagine it'd balloon that far >>> — but tcmalloc will give us some better options to diagnose it). Sorry >>> I didn't mention this before! >> >> Hey Greg! Good recall, I am on Gentoo, and I did build with tcmalloc. > > Search is a wonderful thing. ;) > >> Here is the information you requested: >> >> 2012-10-03 16:20:43.979673 mds.0 [INF] mds.ferntcmalloc heap >> stats:------------------------------------------------ >> 2012-10-03 16:20:43.979676 mds.0 [INF] MALLOC: 53796808560 (51304.6 >> MiB) Bytes in use by application >> 2012-10-03 16:20:43.979679 mds.0 [INF] MALLOC: + 753664 ( 0.7 >> MiB) Bytes in page heap freelist >> 2012-10-03 16:20:43.979681 mds.0 [INF] MALLOC: + 93299048 ( 89.0 >> MiB) Bytes in central cache freelist >> 2012-10-03 16:20:43.979683 mds.0 [INF] MALLOC: + 6110720 ( 5.8 >> MiB) Bytes in transfer cache freelist >> 2012-10-03 16:20:43.979685 mds.0 [INF] MALLOC: + 84547880 ( 80.6 >> MiB) Bytes in thread cache freelists >> 2012-10-03 16:20:43.979686 mds.0 [INF] MALLOC: + 84606976 ( 80.7 >> MiB) Bytes in malloc metadata >> 2012-10-03 16:20:43.979688 mds.0 [INF] MALLOC: ------------ >> 2012-10-03 16:20:43.979690 mds.0 [INF] MALLOC: = 54066126848 (51561.5 >> MiB) Actual memory used (physical + swap) >> 2012-10-03 16:20:43.979691 mds.0 [INF] MALLOC: + 0 ( 0.0 >> MiB) Bytes released to OS (aka unmapped) >> 2012-10-03 16:20:43.979693 mds.0 [INF] MALLOC: ------------ >> 2012-10-03 16:20:43.979694 mds.0 [INF] MALLOC: = 54066126848 (51561.5 >> MiB) Virtual address space used >> 2012-10-03 16:20:43.979700 mds.0 [INF] MALLOC: >> 2012-10-03 16:20:43.979702 mds.0 [INF] MALLOC: 609757 >> Spans in use >> 2012-10-03 16:20:43.979703 mds.0 [INF] MALLOC: 395 >> Thread heaps in use >> 2012-10-03 16:20:43.979705 mds.0 [INF] MALLOC: 8192 >> Tcmalloc page size >> 2012-10-03 16:20:43.979710 mds.0 [INF] > > So tcmalloc thinks the MDS is actually using >50GB of RAM. ie, we have a leak. > > Sage suggests we check out the perfcounters (specifically, how many > log segments are open). "ceph --admin-daemon </path/to/socket> > perfcounters_dump" I believe the default path is > /var/run/ceph/ceph-mds.a.asok.
Got it... --- Start --- fern ceph # ceph --admin-daemon /var/run/ceph/ceph-mds.fern.asok perfcounters_dump {"mds":{"req":0,"reply":48446606,"replyl":{"avgcount":48446606,"sum":28781.3},"fw":0,"dir_f":1238738,"dir_c":1709578,"dir_sp":0,"dir_ffc":0,"imax":32768,"i":9236006,"itop":421,"ibot":2,"iptail":9235583,"ipin":9236004,"iex":20572348,"icap":9235995,"cap":9235995,"dis":0,"t":60401624,"thit":43843666,"tfw":0,"tdis":0,"tdirf":1235679,"trino":0,"tlock":0,"l":347,"q":0,"popanyd":0,"popnest":0,"sm":2,"ex":0,"iexp":0,"im":0,"iim":0},"mds_log":{"evadd":41768893,"evex":41734641,"evtrm":41734641,"ev":34252,"evexg":0,"evexd":1158,"segadd":44958,"segex":44928,"segtrm":44928,"seg":31,"segexg":0,"segexd":1,"expos":188437496802,"wrpos":188567160172,"rdpos":0,"jlat":0},"mds_mem":{"ino":9236008,"ino+":20540696,"ino-":11304688,"dir":1219715,"dir+":2806911,"dir-":1587196,"dn":9236006,"dn+":29809444,"dn-":20573438,"cap":9235995,"cap+":20077556,"cap-":10841561,"rss":52843824,"heap":10792,"malloc":-1925579,"buf":0},"mds_server":{"hcreq":48446606,"hsreq":0,"hcsess":0,"dcreq":51199273,"dsreq":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":6842412,"op_send_bytes":0,"op_resend":216654,"op_ack":1238738,"op_commit":5387021,"op":6625759,"op_r":1238738,"op_w":5387021,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":0,"osdop_read":0,"osdop_write":3542566,"osdop_writefull":43897,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":90980,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":3419156,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_tmap_up":231883,"osdop_tmap_put":1477695,"osdop_tmap_get":1238738,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"map_epoch":69,"map_full":0,"map_inc":67,"osd_sessions":2418,"osd_session_open":452224,"osd_session_close":452032,"osd_laggy":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get":76466053,"get_sum":17902766691,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":76466053,"put_sum":17902766691,"wait":{"avgcount":624620,"sum":1996.57}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":6625759,"take_sum":193714902197,"put":5296041,"put_sum":193714902197,"wait":{"avgcount":0,"sum":0}},"throttle-objecter_ops":{"val":0,"max":1024,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":6625759,"take_sum":6625759,"put":6625759,"put_sum":6625759,"wait":{"avgcount":0,"sum":0}}} --- End --- > > If this doesn't provide us a clue, I'm afraid we're going to have to > start keeping track of heap usage with tcmalloc or run the daemon > through massif... Hmm, well let me know if there's anything else I can provide. And thanks again for your help. t. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html