Re: [ceph-users] Monitoring ceph statistics using rados python module
Thank you, that should do the trick.
2014-05-14 6:41 GMT+02:00 Kai Zhang :
> Hi Adrian,
>
> You may be interested in "rados -p poo_name df --format json", although
> it's pool oriented, you could probably add the values together :)
>
> Regards,
> Kai
>
> 在 2014-05-13 08:33:11，"Adrian Banasiak" 写道：
>
> Thanks for sugestion with admin daemon but it looks like single osd
> oriented. I have used perf dump on mon socket and it output some
> interesting data in case of monitoring whole cluster:
> { "cluster": { "num_mon": 4,
> "num_mon_quorum": 4,
> "num_osd": 29,
> "num_osd_up": 29,
> "num_osd_in": 29,
> "osd_epoch": 1872,
> "osd_kb": 20218112516,
> "osd_kb_used": 5022202696,
> "osd_kb_avail": 15195909820,
> "num_pool": 4,
> "num_pg": 3500,
> "num_pg_active_clean": 3500,
> "num_pg_active": 3500,
> "num_pg_peering": 0,
> "num_object": 400746,
> "num_object_degraded": 0,
> "num_object_unfound": 0,
> "num_bytes": 1678788329609,
> "num_mds_up": 0,
> "num_mds_in": 0,
> "num_mds_failed": 0,
> "mds_epoch": 1},
>
> Unfortunately cluster wide IO statistics are still missing.
>
>
> 2014-05-13 17:17 GMT+02:00 Haomai Wang :
>
>> Not sure your demand.
>>
>> I use "ceph --admin-daemon /var/run/ceph/ceph-osd.x.asok perf dump" to
>> get the monitor infos. And the result can be parsed by simplejson
>> easily via python.
>>
>> On Tue, May 13, 2014 at 10:56 PM, Adrian Banasiak
>> wrote:
>> > Hi, i am working with test Ceph cluster and now I want to implement
>> Zabbix
>> > monitoring with items such as:
>> >
>> > - whoe cluster IO (for example ceph -s -> recovery io 143 MB/s, 35
>> > objects/s)
>> > - pg statistics
>> >
>> > I would like to create single script in python to retrive values using
>> rados
>> > python module, but there are only few informations in documentation
>> about
>> > module usage. I've created single function which calculates all pools
>> > current read/write statistics but i cant find out how to add recovery IO
>> > usage and pg statistics:
>> >
>> > read = 0
>> > write = 0
>> > for pool in conn.list_pools():
>> > io = conn.open_ioctx(pool)
>> > stats[pool] = io.get_stats()
>> > read+=int(stats[pool]['num_rd'])
>> > write+=int(stats[pool]['num_wr'])
>> >
>> > Could someone share his knowledge about rados module for retriving ceph
>> > statistics?
>> >
>> > BTW Ceph is awesome!
>> >
>> > --
>> > Best regards, Adrian Banasiak
>> > email: adr...@banasiak.it
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>>
>
>
>
> --
> Pozdrawiam, Adrian Banasiak
> email: adr...@banasiak.it
>
>
--
Pozdrawiam, Adrian Banasiak
email: adr...@banasiak.it
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Monitoring ceph statistics using rados python module
Hi Adrian,
You may be interested in "rados -p poo_name df --format json", although it's
pool oriented, you could probably add the values together :)
Regards,
Kai
在 2014-05-13 08:33:11，"Adrian Banasiak" 写道：
Thanks for sugestion with admin daemon but it looks like single osd oriented. I
have used perf dump on mon socket and it output some interesting data in case
of monitoring whole cluster:
{ "cluster": { "num_mon": 4,
"num_mon_quorum": 4,
"num_osd": 29,
"num_osd_up": 29,
"num_osd_in": 29,
"osd_epoch": 1872,
"osd_kb": 20218112516,
"osd_kb_used": 5022202696,
"osd_kb_avail": 15195909820,
"num_pool": 4,
"num_pg": 3500,
"num_pg_active_clean": 3500,
"num_pg_active": 3500,
"num_pg_peering": 0,
"num_object": 400746,
"num_object_degraded": 0,
"num_object_unfound": 0,
"num_bytes": 1678788329609,
"num_mds_up": 0,
"num_mds_in": 0,
"num_mds_failed": 0,
"mds_epoch": 1},
Unfortunately cluster wide IO statistics are still missing.
2014-05-13 17:17 GMT+02:00 Haomai Wang :
Not sure your demand.
I use "ceph --admin-daemon /var/run/ceph/ceph-osd.x.asok perf dump" to
get the monitor infos. And the result can be parsed by simplejson
easily via python.
On Tue, May 13, 2014 at 10:56 PM, Adrian Banasiak wrote:
> Hi, i am working with test Ceph cluster and now I want to implement Zabbix
> monitoring with items such as:
>
> - whoe cluster IO (for example ceph -s -> recovery io 143 MB/s, 35
> objects/s)
> - pg statistics
>
> I would like to create single script in python to retrive values using rados
> python module, but there are only few informations in documentation about
> module usage. I've created single function which calculates all pools
> current read/write statistics but i cant find out how to add recovery IO
> usage and pg statistics:
>
> read = 0
> write = 0
> for pool in conn.list_pools():
> io = conn.open_ioctx(pool)
> stats[pool] = io.get_stats()
> read+=int(stats[pool]['num_rd'])
> write+=int(stats[pool]['num_wr'])
>
> Could someone share his knowledge about rados module for retriving ceph
> statistics?
>
> BTW Ceph is awesome!
>
> --
> Best regards, Adrian Banasiak
> email: adr...@banasiak.it
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Best Regards,
Wheat
--
Pozdrawiam, Adrian Banasiak
email: adr...@banasiak.it___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Monitoring ceph statistics using rados python module
On 5/13/14 09:33 , Adrian Banasiak wrote:
Thanks for sugestion with admin daemon but it looks like single osd
oriented. I have used perf dump on mon socket and it output some
interesting data in case of monitoring whole cluster:
{ "cluster": { "num_mon": 4,
"num_mon_quorum": 4,
"num_osd": 29,
"num_osd_up": 29,
"num_osd_in": 29,
"osd_epoch": 1872,
"osd_kb": 20218112516,
"osd_kb_used": 5022202696,
"osd_kb_avail": 15195909820,
"num_pool": 4,
"num_pg": 3500,
"num_pg_active_clean": 3500,
"num_pg_active": 3500,
"num_pg_peering": 0,
"num_object": 400746,
"num_object_degraded": 0,
"num_object_unfound": 0,
"num_bytes": 1678788329609,
"num_mds_up": 0,
"num_mds_in": 0,
"num_mds_failed": 0,
"mds_epoch": 1},
Unfortunately cluster wide IO statistics are still missing.
I'm getting cluster wide OPs and Bandwidth from ceph pg stat -f json.
I'm using this section:
{
"pg_stats_delta": {
"stat_sum": {
"num_bytes": 0,
"num_objects": 31851793,
"num_object_clones": 0,
"num_object_copies": 100208267,
"num_objects_missing_on_primary": 0,
"num_objects_degraded": 4687903,
"num_objects_unfound": 0,
"num_read": 315072058,
"num_read_kb": 55549447422,
"num_write": 223701235,
"num_write_kb": 20457441876,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 74138172,
"num_bytes_recovered": 62776621391330,
"num_keys_recovered": 1129447173},
"stat_cat_sum": {},
"log_size": 7191821,
"ondisk_log_size": 7191821},
I'm tracking num_write, num_write_kb, num_read, and num_read_kb.
Although I see some other things that I should be tracking too
Those values appear to be counters, so you probably want to track the
change from the previous sample rather than the absolute value.
--
*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com
*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website | Facebook
| LinkedIn
| Blog
Re: [ceph-users] Monitoring ceph statistics using rados python module
python-cephclient may be of some use to you
https://github.com/dmsimard/python-cephclient
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Mike Dawson
> Sent: Tuesday, May 13, 2014 10:04 AM
> To: Adrian Banasiak; Haomai Wang
> Cc: ceph-us...@ceph.com
> Subject: Re: [ceph-users] Monitoring ceph statistics using rados python module
>
> Adrian,
>
> Yes, it is single OSD oriented.
>
> Like Haomai, we monitor perf dumps from individual OSD admin sockets. On
> new enough versions of ceph, you can do 'ceph daemon osd.x perf dump',
> which is a shorter way to ask for the same output as 'ceph --admin-daemon
> /var/run/ceph/ceph-osd.x.asok perf dump'. Keep in mind, either version has to
> be run locally on the host where osd.x is running.
>
> We use Sensu to take samples and push them to Graphite. We have the ability
> to then build dashboards showing the whole cluster, units in our CRUSH tree,
> hosts, or an individual OSDs.
>
> I have found that monitoring each OSD's admin daemon is critical. Often times
> a single OSD can affect performance of the entire cluster. Without individual
> data, these types of issues can be quite difficult to pinpoint.
>
> Also, note that Inktank has developed Calamari. There are rumors that it may
> be open sourced at some point in the future.
>
> Cheers,
> Mike Dawson
>
>
> On 5/13/2014 12:33 PM, Adrian Banasiak wrote:
> > Thanks for sugestion with admin daemon but it looks like single osd
> > oriented. I have used perf dump on mon socket and it output some
> > interesting data in case of monitoring whole cluster:
> > { "cluster": { "num_mon": 4,
> >"num_mon_quorum": 4,
> >"num_osd": 29,
> >"num_osd_up": 29,
> >"num_osd_in": 29,
> >"osd_epoch": 1872,
> >"osd_kb": 20218112516,
> >"osd_kb_used": 5022202696,
> >"osd_kb_avail": 15195909820,
> >"num_pool": 4,
> >"num_pg": 3500,
> >"num_pg_active_clean": 3500,
> >"num_pg_active": 3500,
> >"num_pg_peering": 0,
> >"num_object": 400746,
> >"num_object_degraded": 0,
> >"num_object_unfound": 0,
> >"num_bytes": 1678788329609,
> >"num_mds_up": 0,
> >"num_mds_in": 0,
> >"num_mds_failed": 0,
> >"mds_epoch": 1},
> >
> > Unfortunately cluster wide IO statistics are still missing.
> >
> >
> > 2014-05-13 17:17 GMT+02:00 Haomai Wang > <mailto:haomaiw...@gmail.com>>:
> >
> > Not sure your demand.
> >
> > I use "ceph --admin-daemon /var/run/ceph/ceph-osd.x.asok perf dump" to
> > get the monitor infos. And the result can be parsed by simplejson
> > easily via python.
> >
> > On Tue, May 13, 2014 at 10:56 PM, Adrian Banasiak
> > mailto:adr...@banasiak.it>> wrote:
> > > Hi, i am working with test Ceph cluster and now I want to
> > implement Zabbix
> > > monitoring with items such as:
> > >
> > > - whoe cluster IO (for example ceph -s -> recovery io 143 MB/s, 35
> > > objects/s)
> > > - pg statistics
> > >
> > > I would like to create single script in python to retrive values
> > using rados
> > > python module, but there are only few informations in
> > documentation about
> > > module usage. I've created single function which calculates all pools
> > > current read/write statistics but i cant find out how to add
> > recovery IO
> > > usage and pg statistics:
> > >
> > > read = 0
> > > write = 0
> > > for pool in conn.list_pools():
> > > io = conn.open_ioctx(pool)
> > > stats[pool] = io.get_stats()
> > > read+=int(stats[pool]['num_rd'])
> > > write+=int(stats[pool]['num_wr'])
> > >
> > > Could someone share his knowledge about rados module for
> > retriving ceph
> > > statistics?
> > >
> > > BTW Ceph is awesome!
> > >
> > > --
> > > Best regards, Adrian Banasiak
> >
Re: [ceph-users] Monitoring ceph statistics using rados python module
Adrian,
Yes, it is single OSD oriented.
Like Haomai, we monitor perf dumps from individual OSD admin sockets. On
new enough versions of ceph, you can do 'ceph daemon osd.x perf dump',
which is a shorter way to ask for the same output as 'ceph
--admin-daemon /var/run/ceph/ceph-osd.x.asok perf dump'. Keep in mind,
either version has to be run locally on the host where osd.x is running.
We use Sensu to take samples and push them to Graphite. We have the
ability to then build dashboards showing the whole cluster, units in our
CRUSH tree, hosts, or an individual OSDs.
I have found that monitoring each OSD's admin daemon is critical. Often
times a single OSD can affect performance of the entire cluster. Without
individual data, these types of issues can be quite difficult to pinpoint.
Also, note that Inktank has developed Calamari. There are rumors that it
may be open sourced at some point in the future.
Cheers,
Mike Dawson
On 5/13/2014 12:33 PM, Adrian Banasiak wrote:
Thanks for sugestion with admin daemon but it looks like single osd
oriented. I have used perf dump on mon socket and it output some
interesting data in case of monitoring whole cluster:
{ "cluster": { "num_mon": 4,
"num_mon_quorum": 4,
"num_osd": 29,
"num_osd_up": 29,
"num_osd_in": 29,
"osd_epoch": 1872,
"osd_kb": 20218112516,
"osd_kb_used": 5022202696,
"osd_kb_avail": 15195909820,
"num_pool": 4,
"num_pg": 3500,
"num_pg_active_clean": 3500,
"num_pg_active": 3500,
"num_pg_peering": 0,
"num_object": 400746,
"num_object_degraded": 0,
"num_object_unfound": 0,
"num_bytes": 1678788329609,
"num_mds_up": 0,
"num_mds_in": 0,
"num_mds_failed": 0,
"mds_epoch": 1},
Unfortunately cluster wide IO statistics are still missing.
2014-05-13 17:17 GMT+02:00 Haomai Wang mailto:haomaiw...@gmail.com>>:
Not sure your demand.
I use "ceph --admin-daemon /var/run/ceph/ceph-osd.x.asok perf dump" to
get the monitor infos. And the result can be parsed by simplejson
easily via python.
On Tue, May 13, 2014 at 10:56 PM, Adrian Banasiak
mailto:adr...@banasiak.it>> wrote:
> Hi, i am working with test Ceph cluster and now I want to
implement Zabbix
> monitoring with items such as:
>
> - whoe cluster IO (for example ceph -s -> recovery io 143 MB/s, 35
> objects/s)
> - pg statistics
>
> I would like to create single script in python to retrive values
using rados
> python module, but there are only few informations in
documentation about
> module usage. I've created single function which calculates all pools
> current read/write statistics but i cant find out how to add
recovery IO
> usage and pg statistics:
>
> read = 0
> write = 0
> for pool in conn.list_pools():
> io = conn.open_ioctx(pool)
> stats[pool] = io.get_stats()
> read+=int(stats[pool]['num_rd'])
> write+=int(stats[pool]['num_wr'])
>
> Could someone share his knowledge about rados module for
retriving ceph
> statistics?
>
> BTW Ceph is awesome!
>
> --
> Best regards, Adrian Banasiak
> email: adr...@banasiak.it
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Best Regards,
Wheat
--
Pozdrawiam, Adrian Banasiak
email: adr...@banasiak.it
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Monitoring ceph statistics using rados python module
Thanks for sugestion with admin daemon but it looks like single osd
oriented. I have used perf dump on mon socket and it output some
interesting data in case of monitoring whole cluster:
{ "cluster": { "num_mon": 4,
"num_mon_quorum": 4,
"num_osd": 29,
"num_osd_up": 29,
"num_osd_in": 29,
"osd_epoch": 1872,
"osd_kb": 20218112516,
"osd_kb_used": 5022202696,
"osd_kb_avail": 15195909820,
"num_pool": 4,
"num_pg": 3500,
"num_pg_active_clean": 3500,
"num_pg_active": 3500,
"num_pg_peering": 0,
"num_object": 400746,
"num_object_degraded": 0,
"num_object_unfound": 0,
"num_bytes": 1678788329609,
"num_mds_up": 0,
"num_mds_in": 0,
"num_mds_failed": 0,
"mds_epoch": 1},
Unfortunately cluster wide IO statistics are still missing.
2014-05-13 17:17 GMT+02:00 Haomai Wang :
> Not sure your demand.
>
> I use "ceph --admin-daemon /var/run/ceph/ceph-osd.x.asok perf dump" to
> get the monitor infos. And the result can be parsed by simplejson
> easily via python.
>
> On Tue, May 13, 2014 at 10:56 PM, Adrian Banasiak
> wrote:
> > Hi, i am working with test Ceph cluster and now I want to implement
> Zabbix
> > monitoring with items such as:
> >
> > - whoe cluster IO (for example ceph -s -> recovery io 143 MB/s, 35
> > objects/s)
> > - pg statistics
> >
> > I would like to create single script in python to retrive values using
> rados
> > python module, but there are only few informations in documentation about
> > module usage. I've created single function which calculates all pools
> > current read/write statistics but i cant find out how to add recovery IO
> > usage and pg statistics:
> >
> > read = 0
> > write = 0
> > for pool in conn.list_pools():
> > io = conn.open_ioctx(pool)
> > stats[pool] = io.get_stats()
> > read+=int(stats[pool]['num_rd'])
> > write+=int(stats[pool]['num_wr'])
> >
> > Could someone share his knowledge about rados module for retriving ceph
> > statistics?
> >
> > BTW Ceph is awesome!
> >
> > --
> > Best regards, Adrian Banasiak
> > email: adr...@banasiak.it
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
>
> --
> Best Regards,
>
> Wheat
>
--
Pozdrawiam, Adrian Banasiak
email: adr...@banasiak.it
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Monitoring ceph statistics using rados python module
Not sure your demand. I use "ceph --admin-daemon /var/run/ceph/ceph-osd.x.asok perf dump" to get the monitor infos. And the result can be parsed by simplejson easily via python. On Tue, May 13, 2014 at 10:56 PM, Adrian Banasiak wrote: > Hi, i am working with test Ceph cluster and now I want to implement Zabbix > monitoring with items such as: > > - whoe cluster IO (for example ceph -s -> recovery io 143 MB/s, 35 > objects/s) > - pg statistics > > I would like to create single script in python to retrive values using rados > python module, but there are only few informations in documentation about > module usage. I've created single function which calculates all pools > current read/write statistics but i cant find out how to add recovery IO > usage and pg statistics: > > read = 0 > write = 0 > for pool in conn.list_pools(): > io = conn.open_ioctx(pool) > stats[pool] = io.get_stats() > read+=int(stats[pool]['num_rd']) > write+=int(stats[pool]['num_wr']) > > Could someone share his knowledge about rados module for retriving ceph > statistics? > > BTW Ceph is awesome! > > -- > Best regards, Adrian Banasiak > email: adr...@banasiak.it > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Best Regards, Wheat ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
