[ceph-users] Re: Metrics for object sizes

[Anthony D'Atri]

Average size is good to know, but doesn’t give much information about 
distribution.  In a multitenant deployment, most likely there will be buckets 
with a variety of internal distributions too, all small, all large, mix etc.

A cluster with 1M 1kB objects and 1M 10MB objects will have an average 
effectively the same as a cluster with 2M 5MB objects.

> Do keep in mind that small objects are much more expensive then large 
> objects. The metadata overhead becomes costly and can even become problematic 
> if you have millions of tiny (few kb) objects.

Absolutely, which is one reason why it’s good to have more detailed 
information, to inform architecture to consider media types, multiple storage 
classes, Varnish caches etc. 

> Hope this is still relevant... but you may want to have a look at this
> example:
> 
> https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.lua
> https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.md
> 
> where we log RGW object sizes to Prometheus.
> would be easy to change it so it is per bucket and not per operation type.


That’s really terrific, SO wish I would have had that a couple of years ago!  
Lua scripting is shaping up to have all kinds of great applications, especially 
since it’s fast.
Makes me want to go write up a Grafana histogram pane, though it might be 
really slow over a long time region.

> Just got the chance to have a look, but I see lua scripting is new in version 
> pacific ☹
> I have octopus 15.2.14, will it be backported or no chance?


I can’t speak to the potential for backporting, but running a Pacific RGW as a 
client of an Octopus cluster *might* work fine, but there also might be subtle 
incompatibilities.


> 
> 
> Actually it's a good idea, metrics are already there so easy to make the 
> Grafana dash, thank you 😊
> 
> Istvan Szabo
> Senior Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
> 
> -Original Message-
> From: Christian Rohmann  
> Sent: Thursday, October 14, 2021 3:08 PM
> To: ceph-users@ceph.io
> Subject: [ceph-users] Re: Metrics for object sizes
> 
> Email received from the internet. If in doubt, don't click any link nor open 
> any attachment !
> 
> 
> On 23/04/2021 03:53, Szabo, Istvan (Agoda) wrote:
>> Objects inside RGW buckets like in couch base software they have their own 
>> metrics and has this information.
> 
> Not as detailed as you would like, but how about using the bucket stats on 
> bucket size and number of objects?
>  $ radosgw-admin bucket stats --bucket mybucket
> 
> 
> Doing a bucket_size / number_of_objects gives you an average object size per 
> bucket and that certainly is an indication on buckets with rather small 
> objects.
> 
> 
> 
> Regards
> 
> 
> Christian
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
> ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Metrics for object sizes

[Szabo, Istvan (Agoda)]

Actually it's a good idea, metrics are already there so easy to make the 
Grafana dash, thank you 😊

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Christian Rohmann  
Sent: Thursday, October 14, 2021 3:08 PM
To: ceph-users@ceph.io
Subject: [ceph-users] Re: Metrics for object sizes

Email received from the internet. If in doubt, don't click any link nor open 
any attachment !


On 23/04/2021 03:53, Szabo, Istvan (Agoda) wrote:
> Objects inside RGW buckets like in couch base software they have their own 
> metrics and has this information.

Not as detailed as you would like, but how about using the bucket stats on 
bucket size and number of objects?
  $ radosgw-admin bucket stats --bucket mybucket


Doing a bucket_size / number_of_objects gives you an average object size per 
bucket and that certainly is an indication on buckets with rather small objects.



Regards


Christian

___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Metrics for object sizes

[Christian Rohmann]

On 23/04/2021 03:53, Szabo, Istvan (Agoda) wrote:

Objects inside RGW buckets like in couch base software they have their own 
metrics and has this information.


Not as detailed as you would like, but how about using the bucket stats 
on bucket size and number of objects?

 $ radosgw-admin bucket stats --bucket mybucket


Doing a bucket_size / number_of_objects gives you an average object size 
per bucket and that certainly is an indication on

buckets with rather small objects.



Regards


Christian

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Metrics for object sizes

[Szabo, Istvan (Agoda)]

Hi,

Just got the chance to have a look, but I see lua scripting is new in version 
pacific ☹
I have octopus 15.2.14, will it be backported or no chance?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Yuval Lifshitz 
Sent: Tuesday, September 14, 2021 7:38 PM
To: Szabo, Istvan (Agoda) 
Cc: Wido den Hollander ; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: Metrics for object sizes

Email received from the internet. If in doubt, don't click any link nor open 
any attachment !

Hi Istvan,
Hope this is still relevant... but you may want to have a look at this example:

https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.lua
https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.md

where we log RGW object sizes to Prometheus.
would be easy to change it so it is per bucket and not per operation type.

Yuval

On Fri, Apr 23, 2021 at 4:53 AM Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:
Objects inside RGW buckets like in couch base software they have their own 
metrics and has this information.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: 
istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com><mailto:istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>>
---

On 2021. Apr 22., at 14:00, Wido den Hollander 
mailto:w...@42on.com>> wrote:



On 21/04/2021 11:46, Szabo, Istvan (Agoda) wrote:
Hi,
Is there any clusterwise metric regarding object sizes?
I'd like to collect some information about the users what is the object sizes 
in their buckets.

Are you talking about RADOS objects or objects inside RGW buckets?

I think you are talking about RGW, but I just wanted to check.

Afaik this information is not available for both RADOS and RGW.

Do keep in mind that small objects are much more expensive then large objects. 
The metadata overhead becomes costly and can even become problematic if you 
have millions of tiny (few kb) objects.

Wido


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to 
ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>
___
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to 
ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Metrics for object sizes

[Szabo, Istvan (Agoda)]

Definitely still pending from my side, thank you very much I’ll give a try.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

On 2021. Sep 14., at 14:38, Yuval Lifshitz  wrote:


Email received from the internet. If in doubt, don't click any link nor open 
any attachment !

Hi Istvan,
Hope this is still relevant... but you may want to have a look at this example:

https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.lua
https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.md

where we log RGW object sizes to Prometheus.
would be easy to change it so it is per bucket and not per operation type.

Yuval

On Fri, Apr 23, 2021 at 4:53 AM Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:
Objects inside RGW buckets like in couch base software they have their own 
metrics and has this information.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: 
istvan.sz...@agoda.com>
---

On 2021. Apr 22., at 14:00, Wido den Hollander 
mailto:w...@42on.com>> wrote:



On 21/04/2021 11:46, Szabo, Istvan (Agoda) wrote:
Hi,
Is there any clusterwise metric regarding object sizes?
I'd like to collect some information about the users what is the object sizes 
in their buckets.

Are you talking about RADOS objects or objects inside RGW buckets?

I think you are talking about RGW, but I just wanted to check.

Afaik this information is not available for both RADOS and RGW.

Do keep in mind that small objects are much more expensive then large objects. 
The metadata overhead becomes costly and can even become problematic if you 
have millions of tiny (few kb) objects.

Wido


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Metrics for object sizes

[Yuval Lifshitz]

Hi Istvan,
Hope this is still relevant... but you may want to have a look at this
example:

https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.lua
https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.md

where we log RGW object sizes to Prometheus.
would be easy to change it so it is per bucket and not per operation type.

Yuval

On Fri, Apr 23, 2021 at 4:53 AM Szabo, Istvan (Agoda) <
istvan.sz...@agoda.com> wrote:

> Objects inside RGW buckets like in couch base software they have their own
> metrics and has this information.
>
> Istvan Szabo
> Senior Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
>
> On 2021. Apr 22., at 14:00, Wido den Hollander  wrote:
>
> 
>
> On 21/04/2021 11:46, Szabo, Istvan (Agoda) wrote:
> Hi,
> Is there any clusterwise metric regarding object sizes?
> I'd like to collect some information about the users what is the object
> sizes in their buckets.
>
> Are you talking about RADOS objects or objects inside RGW buckets?
>
> I think you are talking about RGW, but I just wanted to check.
>
> Afaik this information is not available for both RADOS and RGW.
>
> Do keep in mind that small objects are much more expensive then large
> objects. The metadata overhead becomes costly and can even become
> problematic if you have millions of tiny (few kb) objects.
>
> Wido
>
> 
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright
> or other legal rules. If you have received it by mistake please let us know
> by reply email and delete it from your system. It is prohibited to copy
> this message or disclose its content to anyone. Any confidentiality or
> privilege is not waived or lost by any mistaken delivery or unauthorized
> disclosure of the message. All messages sent to and from Agoda may be
> monitored to ensure compliance with company policies, to protect the
> company's interests and to remove potential malware. Electronic messages
> may be intercepted, amended, lost or deleted, or contain viruses.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Metrics for object sizes

[Szabo, Istvan (Agoda)]

Objects inside RGW buckets like in couch base software they have their own 
metrics and has this information.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

On 2021. Apr 22., at 14:00, Wido den Hollander  wrote:



On 21/04/2021 11:46, Szabo, Istvan (Agoda) wrote:
Hi,
Is there any clusterwise metric regarding object sizes?
I'd like to collect some information about the users what is the object sizes 
in their buckets.

Are you talking about RADOS objects or objects inside RGW buckets?

I think you are talking about RGW, but I just wanted to check.

Afaik this information is not available for both RADOS and RGW.

Do keep in mind that small objects are much more expensive then large objects. 
The metadata overhead becomes costly and can even become problematic if you 
have millions of tiny (few kb) objects.

Wido


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Metrics for object sizes

[Wido den Hollander]

On 21/04/2021 11:46, Szabo, Istvan (Agoda) wrote:

Hi,

Is there any clusterwise metric regarding object sizes?

I'd like to collect some information about the users what is the object sizes 
in their buckets.


Are you talking about RADOS objects or objects inside RGW buckets?

I think you are talking about RGW, but I just wanted to check.

Afaik this information is not available for both RADOS and RGW.

Do keep in mind that small objects are much more expensive then large 
objects. The metadata overhead becomes costly and can even become 
problematic if you have millions of tiny (few kb) objects.


Wido






This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io