Re: [ceph-users] Useful visualizations / metrics

[Craig Lewis]

I've been graphing disk latency, osd latency, and RGW latency.  It's a bit tricky to pull out of ceph --admin-daemon ceph-osd.0.asok perf dump though.  perf dump gives you the total ops and total op time.  You have to track the delta of those two values, then divide the deltas to get the average latency over your sample interval. I had some alerting on those values, but it was too noisy.  The graphs are helpful though, especially the graphs that have all of a single node's disks (one graph) and OSDs (second graph) on it.  Viewing both graphs helped me identify several problems, including a failing disk and a bad write cache battery. I'm not getting much out of the RGW latency graph though. It's pretty much just the sum of all the OSD latency graphs during that sample interval. Craig Lewis Senior Systems Engineer Office +1.714.602.1309 Email cle...@centraldesktop.com Central Desktop. Work together in ways you never thought possible. Connect with us   Website  |  Twitter  |  Facebook  |  LinkedIn  |  Blog On 4/12/14 07:37 , Greg Poirier wrote: Curious as to how you define cluster latency. On Sat, Apr 12, 2014 at 7:21 AM, Jason Villalta <ja...@rubixnet.com> wrote: Hi, i have not don't anything with metrics yet but the only ones I personally would be interested in is total capacity utilization and cluster latency. Just my 2 cents. On Sat, Apr 12, 2014 at 10:02 AM, Greg Poirier <greg.poir...@opower.com> wrote: I'm in the process of building a dashboard for our Ceph nodes. I was wondering if anyone out there had instrumented their OSD / MON clusters and found particularly useful visualizations. At first, I was trying to do ridiculous things (like graphing % used for every disk in every OSD host), but I realized quickly that that is simply too many metrics and far too visually dense to be useful. I am attempting to put together a few simpler, more dense visualizations like... overcall cluster utilization, aggregate cpu and memory utilization per osd host, etc. Just looking for some suggestions.  Thanks! _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- --  Jason Villalta Co-founder 800.799.4407x1230 | www.RubixTechnology.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com