[ceph-users] Re: RGW access logs with bucket name

2023-11-02 Thread Dan van der Ster 
Using the ops log is a good option -- I had missed that it can now log
to a file. In Quincy:

# ceph config set global rgw_ops_log_rados false
# ceph config set global rgw_ops_log_file_path
'/var/log/ceph/ops-log-$cluster-$name.log'
# ceph config set global rgw_enable_ops_log true

Then restart all RGWs.

Thanks!

Dan

--
Dan van der Ster
CTO

Clyso GmbH
p: +49 89 215252722 | a: Vancouver, Canada
w: https://clyso.com | e: dan.vanders...@clyso.com

We are hiring: https://www.clyso.com/jobs/

On Mon, Oct 30, 2023 at 7:19 AM Casey Bodley  wrote:
>
> another option is to enable the rgw ops log, which includes the bucket
> name for each request
>
> the http access log line that's visible at log level 1 follows a known
> apache format that users can scrape, so i've resisted adding extra
> s3-specific stuff like bucket/object names there. there was some
> recent discussion around this in
> https://github.com/ceph/ceph/pull/50350, which had originally extended
> that access log line
>
> On Mon, Oct 30, 2023 at 6:03 AM Boris Behrens  wrote:
> >
> > Hi Dan,
> >
> > we are currently moving all the logging into lua scripts, so it is not an
> > issue anymore for us.
> >
> > Thanks
> >
> > ps: the ceph analyzer is really cool. plusplus
> >
> > Am Sa., 28. Okt. 2023 um 22:03 Uhr schrieb Dan van der Ster <
> > dan.vanders...@clyso.com>:
> >
> > > Hi Boris,
> > >
> > > I found that you need to use debug_rgw=10 to see the bucket name :-/
> > >
> > > e.g.
> > > 2023-10-28T19:55:42.288+ 7f34dde06700 10 req 3268931155513085118
> > > 0.0s s->object=... s->bucket=xyz-bucket-123
> > >
> > > Did you find a more convenient way in the meantime? I think we should
> > > log bucket name at level 1.
> > >
> > > Cheers, Dan
> > >
> > > --
> > > Dan van der Ster
> > > CTO
> > >
> > > Clyso GmbH
> > > p: +49 89 215252722 | a: Vancouver, Canada
> > > w: https://clyso.com | e: dan.vanders...@clyso.com
> > >
> > > Try our Ceph Analyzer: https://analyzer.clyso.com
> > >
> > > On Thu, Mar 30, 2023 at 4:15 AM Boris Behrens  wrote:
> > > >
> > > > Sadly not.
> > > > I only see the the path/query of a request, but not the hostname.
> > > > So when a bucket is accessed via hostname (
> > > https://bucket.TLD/object?query)
> > > > I only see the object and the query (GET /object?query).
> > > > When a bucket is accessed bia path (https://TLD/bucket/object?query) I
> > > can
> > > > see also the bucket in the log (GET bucket/object?query)
> > > >
> > > > Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) <
> > > > istvan.sz...@agoda.com>:
> > > >
> > > > > It has the full url begins with the bucket name in the beast logs http
> > > > > requests, hasn’t it?
> > > > >
> > > > > Istvan Szabo
> > > > > Staff Infrastructure Engineer
> > > > > ---
> > > > > Agoda Services Co., Ltd.
> > > > > e: istvan.sz...@agoda.com
> > > > > ---
> > > > >
> > > > > On 2023. Mar 30., at 17:44, Boris Behrens  wrote:
> > > > >
> > > > > Email received from the internet. If in doubt, don't click any link
> > > nor
> > > > > open any attachment !
> > > > > 
> > > > >
> > > > > Bringing up that topic again:
> > > > > is it possible to log the bucket name in the rgw client logs?
> > > > >
> > > > > currently I am only to know the bucket name when someone access the
> > > bucket
> > > > > via https://TLD/bucket/object instead of https://bucket.TLD/object.
> > > > >
> > > > > Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens 
> > > > >  > > >:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I am looking forward to move our logs from
> > > > >
> > > > > /var/log/ceph/ceph-client...log to our logaggregator.
> > > > >
> > > > >
> > > > > Is there a way to have the bucket name in the log file?
> > > > >
> > > > >
> > > > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work
> > > with
> > > > >
> > > > > this.
> > > > >
> > > > >
> > > > > Cheers and happy new year
> > > > >
> > > > > Boris
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
> > > im
> > > > > groüen Saal.
> > > > > ___
> > > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > > > >
> > > > >
> > > > > --
> > > > > This message is confidential and is for the sole use of the intended
> > > > > recipient(s). It may also be privileged or otherwise protected by
> > > copyright
> > > > > or other legal rules. If you have received it by mistake please let us
> > > know
> > > > > by reply email and delete it from your system. It is prohibited to 
> > > > > copy
> > > > > this message or disclose its content to anyone. Any confidentiality or
> > > > > privilege is not waived or lost by any mistaken delivery or
> > > unauthorized
> >

[ceph-users] Re: RGW access logs with bucket name

2023-10-30 Thread Casey Bodley 
another option is to enable the rgw ops log, which includes the bucket
name for each request

the http access log line that's visible at log level 1 follows a known
apache format that users can scrape, so i've resisted adding extra
s3-specific stuff like bucket/object names there. there was some
recent discussion around this in
https://github.com/ceph/ceph/pull/50350, which had originally extended
that access log line

On Mon, Oct 30, 2023 at 6:03 AM Boris Behrens  wrote:
>
> Hi Dan,
>
> we are currently moving all the logging into lua scripts, so it is not an
> issue anymore for us.
>
> Thanks
>
> ps: the ceph analyzer is really cool. plusplus
>
> Am Sa., 28. Okt. 2023 um 22:03 Uhr schrieb Dan van der Ster <
> dan.vanders...@clyso.com>:
>
> > Hi Boris,
> >
> > I found that you need to use debug_rgw=10 to see the bucket name :-/
> >
> > e.g.
> > 2023-10-28T19:55:42.288+ 7f34dde06700 10 req 3268931155513085118
> > 0.0s s->object=... s->bucket=xyz-bucket-123
> >
> > Did you find a more convenient way in the meantime? I think we should
> > log bucket name at level 1.
> >
> > Cheers, Dan
> >
> > --
> > Dan van der Ster
> > CTO
> >
> > Clyso GmbH
> > p: +49 89 215252722 | a: Vancouver, Canada
> > w: https://clyso.com | e: dan.vanders...@clyso.com
> >
> > Try our Ceph Analyzer: https://analyzer.clyso.com
> >
> > On Thu, Mar 30, 2023 at 4:15 AM Boris Behrens  wrote:
> > >
> > > Sadly not.
> > > I only see the the path/query of a request, but not the hostname.
> > > So when a bucket is accessed via hostname (
> > https://bucket.TLD/object?query)
> > > I only see the object and the query (GET /object?query).
> > > When a bucket is accessed bia path (https://TLD/bucket/object?query) I
> > can
> > > see also the bucket in the log (GET bucket/object?query)
> > >
> > > Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) <
> > > istvan.sz...@agoda.com>:
> > >
> > > > It has the full url begins with the bucket name in the beast logs http
> > > > requests, hasn’t it?
> > > >
> > > > Istvan Szabo
> > > > Staff Infrastructure Engineer
> > > > ---
> > > > Agoda Services Co., Ltd.
> > > > e: istvan.sz...@agoda.com
> > > > ---
> > > >
> > > > On 2023. Mar 30., at 17:44, Boris Behrens  wrote:
> > > >
> > > > Email received from the internet. If in doubt, don't click any link
> > nor
> > > > open any attachment !
> > > > 
> > > >
> > > > Bringing up that topic again:
> > > > is it possible to log the bucket name in the rgw client logs?
> > > >
> > > > currently I am only to know the bucket name when someone access the
> > bucket
> > > > via https://TLD/bucket/object instead of https://bucket.TLD/object.
> > > >
> > > > Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens  > >:
> > > >
> > > > Hi,
> > > >
> > > > I am looking forward to move our logs from
> > > >
> > > > /var/log/ceph/ceph-client...log to our logaggregator.
> > > >
> > > >
> > > > Is there a way to have the bucket name in the log file?
> > > >
> > > >
> > > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work
> > with
> > > >
> > > > this.
> > > >
> > > >
> > > > Cheers and happy new year
> > > >
> > > > Boris
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
> > im
> > > > groüen Saal.
> > > > ___
> > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > > >
> > > >
> > > > --
> > > > This message is confidential and is for the sole use of the intended
> > > > recipient(s). It may also be privileged or otherwise protected by
> > copyright
> > > > or other legal rules. If you have received it by mistake please let us
> > know
> > > > by reply email and delete it from your system. It is prohibited to copy
> > > > this message or disclose its content to anyone. Any confidentiality or
> > > > privilege is not waived or lost by any mistaken delivery or
> > unauthorized
> > > > disclosure of the message. All messages sent to and from Agoda may be
> > > > monitored to ensure compliance with company policies, to protect the
> > > > company's interests and to remove potential malware. Electronic
> > messages
> > > > may be intercepted, amended, lost or deleted, or contain viruses.
> > > >
> > >
> > >
> > > --
> > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> > > groüen Saal.
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe

[ceph-users] Re: RGW access logs with bucket name

2023-10-30 Thread Boris Behrens 
Hi Dan,

we are currently moving all the logging into lua scripts, so it is not an
issue anymore for us.

Thanks

ps: the ceph analyzer is really cool. plusplus

Am Sa., 28. Okt. 2023 um 22:03 Uhr schrieb Dan van der Ster <
dan.vanders...@clyso.com>:

> Hi Boris,
>
> I found that you need to use debug_rgw=10 to see the bucket name :-/
>
> e.g.
> 2023-10-28T19:55:42.288+ 7f34dde06700 10 req 3268931155513085118
> 0.0s s->object=... s->bucket=xyz-bucket-123
>
> Did you find a more convenient way in the meantime? I think we should
> log bucket name at level 1.
>
> Cheers, Dan
>
> --
> Dan van der Ster
> CTO
>
> Clyso GmbH
> p: +49 89 215252722 | a: Vancouver, Canada
> w: https://clyso.com | e: dan.vanders...@clyso.com
>
> Try our Ceph Analyzer: https://analyzer.clyso.com
>
> On Thu, Mar 30, 2023 at 4:15 AM Boris Behrens  wrote:
> >
> > Sadly not.
> > I only see the the path/query of a request, but not the hostname.
> > So when a bucket is accessed via hostname (
> https://bucket.TLD/object?query)
> > I only see the object and the query (GET /object?query).
> > When a bucket is accessed bia path (https://TLD/bucket/object?query) I
> can
> > see also the bucket in the log (GET bucket/object?query)
> >
> > Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) <
> > istvan.sz...@agoda.com>:
> >
> > > It has the full url begins with the bucket name in the beast logs http
> > > requests, hasn’t it?
> > >
> > > Istvan Szabo
> > > Staff Infrastructure Engineer
> > > ---
> > > Agoda Services Co., Ltd.
> > > e: istvan.sz...@agoda.com
> > > ---
> > >
> > > On 2023. Mar 30., at 17:44, Boris Behrens  wrote:
> > >
> > > Email received from the internet. If in doubt, don't click any link
> nor
> > > open any attachment !
> > > 
> > >
> > > Bringing up that topic again:
> > > is it possible to log the bucket name in the rgw client logs?
> > >
> > > currently I am only to know the bucket name when someone access the
> bucket
> > > via https://TLD/bucket/object instead of https://bucket.TLD/object.
> > >
> > > Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens  >:
> > >
> > > Hi,
> > >
> > > I am looking forward to move our logs from
> > >
> > > /var/log/ceph/ceph-client...log to our logaggregator.
> > >
> > >
> > > Is there a way to have the bucket name in the log file?
> > >
> > >
> > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work
> with
> > >
> > > this.
> > >
> > >
> > > Cheers and happy new year
> > >
> > > Boris
> > >
> > >
> > >
> > >
> > > --
> > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
> im
> > > groüen Saal.
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > >
> > >
> > > --
> > > This message is confidential and is for the sole use of the intended
> > > recipient(s). It may also be privileged or otherwise protected by
> copyright
> > > or other legal rules. If you have received it by mistake please let us
> know
> > > by reply email and delete it from your system. It is prohibited to copy
> > > this message or disclose its content to anyone. Any confidentiality or
> > > privilege is not waived or lost by any mistaken delivery or
> unauthorized
> > > disclosure of the message. All messages sent to and from Agoda may be
> > > monitored to ensure compliance with company policies, to protect the
> > > company's interests and to remove potential malware. Electronic
> messages
> > > may be intercepted, amended, lost or deleted, or contain viruses.
> > >
> >
> >
> > --
> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> > groüen Saal.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW access logs with bucket name

2023-10-28 Thread Dan van der Ster 
Hi Boris,

I found that you need to use debug_rgw=10 to see the bucket name :-/

e.g.
2023-10-28T19:55:42.288+ 7f34dde06700 10 req 3268931155513085118
0.0s s->object=... s->bucket=xyz-bucket-123

Did you find a more convenient way in the meantime? I think we should
log bucket name at level 1.

Cheers, Dan

--
Dan van der Ster
CTO

Clyso GmbH
p: +49 89 215252722 | a: Vancouver, Canada
w: https://clyso.com | e: dan.vanders...@clyso.com

Try our Ceph Analyzer: https://analyzer.clyso.com

On Thu, Mar 30, 2023 at 4:15 AM Boris Behrens  wrote:
>
> Sadly not.
> I only see the the path/query of a request, but not the hostname.
> So when a bucket is accessed via hostname (https://bucket.TLD/object?query)
> I only see the object and the query (GET /object?query).
> When a bucket is accessed bia path (https://TLD/bucket/object?query) I can
> see also the bucket in the log (GET bucket/object?query)
>
> Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) <
> istvan.sz...@agoda.com>:
>
> > It has the full url begins with the bucket name in the beast logs http
> > requests, hasn’t it?
> >
> > Istvan Szabo
> > Staff Infrastructure Engineer
> > ---
> > Agoda Services Co., Ltd.
> > e: istvan.sz...@agoda.com
> > ---
> >
> > On 2023. Mar 30., at 17:44, Boris Behrens  wrote:
> >
> > Email received from the internet. If in doubt, don't click any link nor
> > open any attachment !
> > 
> >
> > Bringing up that topic again:
> > is it possible to log the bucket name in the rgw client logs?
> >
> > currently I am only to know the bucket name when someone access the bucket
> > via https://TLD/bucket/object instead of https://bucket.TLD/object.
> >
> > Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens :
> >
> > Hi,
> >
> > I am looking forward to move our logs from
> >
> > /var/log/ceph/ceph-client...log to our logaggregator.
> >
> >
> > Is there a way to have the bucket name in the log file?
> >
> >
> > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with
> >
> > this.
> >
> >
> > Cheers and happy new year
> >
> > Boris
> >
> >
> >
> >
> > --
> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> > groüen Saal.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> >
> > --
> > This message is confidential and is for the sole use of the intended
> > recipient(s). It may also be privileged or otherwise protected by copyright
> > or other legal rules. If you have received it by mistake please let us know
> > by reply email and delete it from your system. It is prohibited to copy
> > this message or disclose its content to anyone. Any confidentiality or
> > privilege is not waived or lost by any mistaken delivery or unauthorized
> > disclosure of the message. All messages sent to and from Agoda may be
> > monitored to ensure compliance with company policies, to protect the
> > company's interests and to remove potential malware. Electronic messages
> > may be intercepted, amended, lost or deleted, or contain viruses.
> >
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW access logs with bucket name

2023-03-30 Thread Boris Behrens 
Sadly not.
I only see the the path/query of a request, but not the hostname.
So when a bucket is accessed via hostname (https://bucket.TLD/object?query)
I only see the object and the query (GET /object?query).
When a bucket is accessed bia path (https://TLD/bucket/object?query) I can
see also the bucket in the log (GET bucket/object?query)

Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) <
istvan.sz...@agoda.com>:

> It has the full url begins with the bucket name in the beast logs http
> requests, hasn’t it?
>
> Istvan Szabo
> Staff Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
>
> On 2023. Mar 30., at 17:44, Boris Behrens  wrote:
>
> Email received from the internet. If in doubt, don't click any link nor
> open any attachment !
> 
>
> Bringing up that topic again:
> is it possible to log the bucket name in the rgw client logs?
>
> currently I am only to know the bucket name when someone access the bucket
> via https://TLD/bucket/object instead of https://bucket.TLD/object.
>
> Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens :
>
> Hi,
>
> I am looking forward to move our logs from
>
> /var/log/ceph/ceph-client...log to our logaggregator.
>
>
> Is there a way to have the bucket name in the log file?
>
>
> Or can I write the rgw_enable_ops_log into a file? Maybe I could work with
>
> this.
>
>
> Cheers and happy new year
>
> Boris
>
>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> --
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright
> or other legal rules. If you have received it by mistake please let us know
> by reply email and delete it from your system. It is prohibited to copy
> this message or disclose its content to anyone. Any confidentiality or
> privilege is not waived or lost by any mistaken delivery or unauthorized
> disclosure of the message. All messages sent to and from Agoda may be
> monitored to ensure compliance with company policies, to protect the
> company's interests and to remove potential malware. Electronic messages
> may be intercepted, amended, lost or deleted, or contain viruses.
>


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW access logs with bucket name

2023-03-30 Thread Szabo, Istvan (Agoda) 
It has the full url begins with the bucket name in the beast logs http 
requests, hasn’t it?

Istvan Szabo
Staff Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

On 2023. Mar 30., at 17:44, Boris Behrens  wrote:

Email received from the internet. If in doubt, don't click any link nor open 
any attachment !


Bringing up that topic again:
is it possible to log the bucket name in the rgw client logs?

currently I am only to know the bucket name when someone access the bucket
via https://TLD/bucket/object instead of https://bucket.TLD/object.

Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens :

Hi,
I am looking forward to move our logs from
/var/log/ceph/ceph-client...log to our logaggregator.

Is there a way to have the bucket name in the log file?

Or can I write the rgw_enable_ops_log into a file? Maybe I could work with
this.

Cheers and happy new year
Boris



--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW access logs with bucket name

2023-03-30 Thread Boris Behrens 
Bringing up that topic again:
is it possible to log the bucket name in the rgw client logs?

currently I am only to know the bucket name when someone access the bucket
via https://TLD/bucket/object instead of https://bucket.TLD/object.

Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens :

> Hi,
> I am looking forward to move our logs from
> /var/log/ceph/ceph-client...log to our logaggregator.
>
> Is there a way to have the bucket name in the log file?
>
> Or can I write the rgw_enable_ops_log into a file? Maybe I could work with
> this.
>
> Cheers and happy new year
>  Boris
>


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io