[ceph-users] Re: x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?
That hasn't been glued up, but likely will be. Something similar for k8s has been discussed. You could certainly use lua to do something custom. I suspect lua to customize the canned ops-log will become an option, too. Matt On Fri, May 7, 2021 at 5:51 PM David Orman wrote: > > Has anyone figured out an elegant way to emit this from inside cephadm > managed/containerized ceph, so it can be handled via the host's > journald and processed/shipped? We had gone down that path before, but > decided to hold off on the suggestion that the LUA-based scripting > might be a better option. > > David > > On Fri, May 7, 2021 at 4:21 PM Matt Benjamin wrote: > > > > Hi David, > > > > I think the solution is most likely the ops log. It is called for > > every op, and has the transaction id. > > > > Matt > > > > On Fri, May 7, 2021 at 4:58 PM David Orman wrote: > > > > > > Hi Yuval, > > > > > > We've managed to get an upgrade done with the 16.2.3 release in a > > > testing cluster, and we've been able to implement some of the logging > > > I need via this mechanism, but the logs are emitted only when > > > debug_rgw is set to 20. I don't need to log any of that level of data > > > (we used centralized logging and the sheer volume of this output is > > > staggering); I'm just trying to get the full request log, to include > > > the transactionID, so I can match it up with the logging we do on our > > > load balancer solution. Is there another mechanism to emit these logs > > > at normal log levels? RGWDebugLog() doesn't appear to be what I'm > > > actually looking for. My intent is to emit JSON logs using this > > > mechanism, in the end, with all of the required fields for requests. > > > The current "beast: " log lines don't contain the information we need, > > > such as txid, which is what we're attempting to solve for - but can't > > > afford to have full debug logging enabled in production clusters. > > > > > > Thanks! > > > David > > > > > > On Thu, Apr 1, 2021 at 11:21 AM Yuval Lifshitz > > > wrote: > > > > > > > > Hi David, > > > > Don't have any good idea for "octopus" (other than ops log), but you > > > > can do that (and more) in "pacific" using lua scripting on the RGW: > > > > https://docs.ceph.com/en/pacific/radosgw/lua-scripting/ > > > > > > > > Yuval > > > > > > > > On Thu, Apr 1, 2021 at 7:11 PM David Orman wrote: > > > >> > > > >> Hi, > > > >> > > > >> Is there any way to log the x-amz-request-id along with the request in > > > >> the rgw logs? We're using beast and don't see an option in the > > > >> configuration documentation to add headers to the request lines. We > > > >> use centralized logging and would like to be able to search all layers > > > >> of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. > > > >> > > > >> Right now, all we see is this: > > > >> > > > >> debug 2021-04-01T15:55:31.105+ 7f54e599b700 1 beast: > > > >> 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+] "PUT > > > >> /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; > > > >> amd64)" - > > > >> > > > >> We've also tried this: > > > >> > > > >> ceph config set global rgw_enable_ops_log true > > > >> ceph config set global rgw_ops_log_socket_path /tmp/testlog > > > >> > > > >> After doing this, inside the rgw container, we can socat - > > > >> UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that > > > >> we want, but there has to be a better way to do this, where the logs > > > >> are emitted like the request logs above by beast, so that we can > > > >> handle it using journald. If there's an alternative that would > > > >> accomplish the same thing, we're very open to suggestions. > > > >> > > > >> Thank you, > > > >> David > > > >> ___ > > > >> ceph-users mailing list -- ceph-users@ceph.io > > > >> To unsubscribe send an email to ceph-users-le...@ceph.io > > > >> > > > ___ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > > > > > > > > -- > > > > Matt Benjamin > > Red Hat, Inc. > > 315 West Huron Street, Suite 140A > > Ann Arbor, Michigan 48103 > > > > http://www.redhat.com/en/technologies/storage > > > > tel. 734-821-5101 > > fax. 734-769-8938 > > cel. 734-216-5309 > > > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?
Has anyone figured out an elegant way to emit this from inside cephadm managed/containerized ceph, so it can be handled via the host's journald and processed/shipped? We had gone down that path before, but decided to hold off on the suggestion that the LUA-based scripting might be a better option. David On Fri, May 7, 2021 at 4:21 PM Matt Benjamin wrote: > > Hi David, > > I think the solution is most likely the ops log. It is called for > every op, and has the transaction id. > > Matt > > On Fri, May 7, 2021 at 4:58 PM David Orman wrote: > > > > Hi Yuval, > > > > We've managed to get an upgrade done with the 16.2.3 release in a > > testing cluster, and we've been able to implement some of the logging > > I need via this mechanism, but the logs are emitted only when > > debug_rgw is set to 20. I don't need to log any of that level of data > > (we used centralized logging and the sheer volume of this output is > > staggering); I'm just trying to get the full request log, to include > > the transactionID, so I can match it up with the logging we do on our > > load balancer solution. Is there another mechanism to emit these logs > > at normal log levels? RGWDebugLog() doesn't appear to be what I'm > > actually looking for. My intent is to emit JSON logs using this > > mechanism, in the end, with all of the required fields for requests. > > The current "beast: " log lines don't contain the information we need, > > such as txid, which is what we're attempting to solve for - but can't > > afford to have full debug logging enabled in production clusters. > > > > Thanks! > > David > > > > On Thu, Apr 1, 2021 at 11:21 AM Yuval Lifshitz wrote: > > > > > > Hi David, > > > Don't have any good idea for "octopus" (other than ops log), but you can > > > do that (and more) in "pacific" using lua scripting on the RGW: > > > https://docs.ceph.com/en/pacific/radosgw/lua-scripting/ > > > > > > Yuval > > > > > > On Thu, Apr 1, 2021 at 7:11 PM David Orman wrote: > > >> > > >> Hi, > > >> > > >> Is there any way to log the x-amz-request-id along with the request in > > >> the rgw logs? We're using beast and don't see an option in the > > >> configuration documentation to add headers to the request lines. We > > >> use centralized logging and would like to be able to search all layers > > >> of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. > > >> > > >> Right now, all we see is this: > > >> > > >> debug 2021-04-01T15:55:31.105+ 7f54e599b700 1 beast: > > >> 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+] "PUT > > >> /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; > > >> amd64)" - > > >> > > >> We've also tried this: > > >> > > >> ceph config set global rgw_enable_ops_log true > > >> ceph config set global rgw_ops_log_socket_path /tmp/testlog > > >> > > >> After doing this, inside the rgw container, we can socat - > > >> UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that > > >> we want, but there has to be a better way to do this, where the logs > > >> are emitted like the request logs above by beast, so that we can > > >> handle it using journald. If there's an alternative that would > > >> accomplish the same thing, we're very open to suggestions. > > >> > > >> Thank you, > > >> David > > >> ___ > > >> ceph-users mailing list -- ceph-users@ceph.io > > >> To unsubscribe send an email to ceph-users-le...@ceph.io > > >> > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309 > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?
Hi David, I think the solution is most likely the ops log. It is called for every op, and has the transaction id. Matt On Fri, May 7, 2021 at 4:58 PM David Orman wrote: > > Hi Yuval, > > We've managed to get an upgrade done with the 16.2.3 release in a > testing cluster, and we've been able to implement some of the logging > I need via this mechanism, but the logs are emitted only when > debug_rgw is set to 20. I don't need to log any of that level of data > (we used centralized logging and the sheer volume of this output is > staggering); I'm just trying to get the full request log, to include > the transactionID, so I can match it up with the logging we do on our > load balancer solution. Is there another mechanism to emit these logs > at normal log levels? RGWDebugLog() doesn't appear to be what I'm > actually looking for. My intent is to emit JSON logs using this > mechanism, in the end, with all of the required fields for requests. > The current "beast: " log lines don't contain the information we need, > such as txid, which is what we're attempting to solve for - but can't > afford to have full debug logging enabled in production clusters. > > Thanks! > David > > On Thu, Apr 1, 2021 at 11:21 AM Yuval Lifshitz wrote: > > > > Hi David, > > Don't have any good idea for "octopus" (other than ops log), but you can do > > that (and more) in "pacific" using lua scripting on the RGW: > > https://docs.ceph.com/en/pacific/radosgw/lua-scripting/ > > > > Yuval > > > > On Thu, Apr 1, 2021 at 7:11 PM David Orman wrote: > >> > >> Hi, > >> > >> Is there any way to log the x-amz-request-id along with the request in > >> the rgw logs? We're using beast and don't see an option in the > >> configuration documentation to add headers to the request lines. We > >> use centralized logging and would like to be able to search all layers > >> of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. > >> > >> Right now, all we see is this: > >> > >> debug 2021-04-01T15:55:31.105+ 7f54e599b700 1 beast: > >> 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+] "PUT > >> /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; > >> amd64)" - > >> > >> We've also tried this: > >> > >> ceph config set global rgw_enable_ops_log true > >> ceph config set global rgw_ops_log_socket_path /tmp/testlog > >> > >> After doing this, inside the rgw container, we can socat - > >> UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that > >> we want, but there has to be a better way to do this, where the logs > >> are emitted like the request logs above by beast, so that we can > >> handle it using journald. If there's an alternative that would > >> accomplish the same thing, we're very open to suggestions. > >> > >> Thank you, > >> David > >> ___ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-le...@ceph.io > >> > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?
Hi Yuval, We've managed to get an upgrade done with the 16.2.3 release in a testing cluster, and we've been able to implement some of the logging I need via this mechanism, but the logs are emitted only when debug_rgw is set to 20. I don't need to log any of that level of data (we used centralized logging and the sheer volume of this output is staggering); I'm just trying to get the full request log, to include the transactionID, so I can match it up with the logging we do on our load balancer solution. Is there another mechanism to emit these logs at normal log levels? RGWDebugLog() doesn't appear to be what I'm actually looking for. My intent is to emit JSON logs using this mechanism, in the end, with all of the required fields for requests. The current "beast: " log lines don't contain the information we need, such as txid, which is what we're attempting to solve for - but can't afford to have full debug logging enabled in production clusters. Thanks! David On Thu, Apr 1, 2021 at 11:21 AM Yuval Lifshitz wrote: > > Hi David, > Don't have any good idea for "octopus" (other than ops log), but you can do > that (and more) in "pacific" using lua scripting on the RGW: > https://docs.ceph.com/en/pacific/radosgw/lua-scripting/ > > Yuval > > On Thu, Apr 1, 2021 at 7:11 PM David Orman wrote: >> >> Hi, >> >> Is there any way to log the x-amz-request-id along with the request in >> the rgw logs? We're using beast and don't see an option in the >> configuration documentation to add headers to the request lines. We >> use centralized logging and would like to be able to search all layers >> of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. >> >> Right now, all we see is this: >> >> debug 2021-04-01T15:55:31.105+ 7f54e599b700 1 beast: >> 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+] "PUT >> /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; >> amd64)" - >> >> We've also tried this: >> >> ceph config set global rgw_enable_ops_log true >> ceph config set global rgw_ops_log_socket_path /tmp/testlog >> >> After doing this, inside the rgw container, we can socat - >> UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that >> we want, but there has to be a better way to do this, where the logs >> are emitted like the request logs above by beast, so that we can >> handle it using journald. If there's an alternative that would >> accomplish the same thing, we're very open to suggestions. >> >> Thank you, >> David >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?
Hi Folks, A Red Hat SA (Mustafa Aydin) suggested, some while back, a concise formula for relaying ops-log to syslog, basically a script executing socat unix-connect:/var/run/ceph/opslog,reuseaddr UNIX-CLIENT:/dev/log & I haven't experimented with it. Matt On Thu, Apr 1, 2021 at 12:22 PM Yuval Lifshitz wrote: > > Hi David, > Don't have any good idea for "octopus" (other than ops log), but you can do > that (and more) in "pacific" using lua scripting on the RGW: > https://docs.ceph.com/en/pacific/radosgw/lua-scripting/ > > Yuval > > On Thu, Apr 1, 2021 at 7:11 PM David Orman wrote: > > > Hi, > > > > Is there any way to log the x-amz-request-id along with the request in > > the rgw logs? We're using beast and don't see an option in the > > configuration documentation to add headers to the request lines. We > > use centralized logging and would like to be able to search all layers > > of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. > > > > Right now, all we see is this: > > > > debug 2021-04-01T15:55:31.105+ 7f54e599b700 1 beast: > > 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+] "PUT > > /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; > > amd64)" - > > > > We've also tried this: > > > > ceph config set global rgw_enable_ops_log true > > ceph config set global rgw_ops_log_socket_path /tmp/testlog > > > > After doing this, inside the rgw container, we can socat - > > UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that > > we want, but there has to be a better way to do this, where the logs > > are emitted like the request logs above by beast, so that we can > > handle it using journald. If there's an alternative that would > > accomplish the same thing, we're very open to suggestions. > > > > Thank you, > > David > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?
Hi David, Don't have any good idea for "octopus" (other than ops log), but you can do that (and more) in "pacific" using lua scripting on the RGW: https://docs.ceph.com/en/pacific/radosgw/lua-scripting/ Yuval On Thu, Apr 1, 2021 at 7:11 PM David Orman wrote: > Hi, > > Is there any way to log the x-amz-request-id along with the request in > the rgw logs? We're using beast and don't see an option in the > configuration documentation to add headers to the request lines. We > use centralized logging and would like to be able to search all layers > of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. > > Right now, all we see is this: > > debug 2021-04-01T15:55:31.105+ 7f54e599b700 1 beast: > 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+] "PUT > /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; > amd64)" - > > We've also tried this: > > ceph config set global rgw_enable_ops_log true > ceph config set global rgw_ops_log_socket_path /tmp/testlog > > After doing this, inside the rgw container, we can socat - > UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that > we want, but there has to be a better way to do this, where the logs > are emitted like the request logs above by beast, so that we can > handle it using journald. If there's an alternative that would > accomplish the same thing, we're very open to suggestions. > > Thank you, > David > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
