[ceph-users] Re: Bug in RGW header x-amz-date parsing
hi Subu, On Tue, Dec 7, 2021 at 12:10 PM Subu Sankara Subramanian wrote: > > Folks, > > Is there a bug in ceph RGW date parsing? > https://github.com/ceph/ceph/blob/master/src/rgw/rgw_auth_s3.cc#L223 - this > line parses the date in x-amz-date as RFC 2616. BUT the format specified by > Amazon S3 is ISO 8601 basic - MMDDTHHMMSSZ ( > https://docs.aws.amazon.com/ses/latest/APIReference-V2/CommonParameters.html). this is a link to the Amazon Simple Email Service v2 API, it doesn't refer to aws v2 signatures. v2 signatures for s3 are documented at https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html#RESTAuthenticationTimeStamp, which states that "The value of the x-amz-date header must be in one of the RFC 2616 formats (http://www.ietf.org/rfc/rfc2616.txt)." > I think this causes an error in my tests stating NOTICE: failed to parse > date for auth header. Am I reading the code wrong? > > s3Cmd specified for testing in rook docs uses SigV4 (as recommended) so am > assuming this code path isn't getting exercised. I am trying to run the > benchmark from https://github.com/wasabi-tech/s3-benchmark and this uses > sig v2. FWIW it works once I remove the header and add a HTTP date instead. > > Sorry I don't have access to the bug tracker (pending account creation), so > mailing here. > > Thanks. Subu > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Bug in RGW header x-amz-date parsing
On Tue, Dec 07, 2021 at 12:39:12PM -0500, Casey Bodley wrote: > hi Subu, > > On Tue, Dec 7, 2021 at 12:10 PM Subu Sankara Subramanian > wrote: > > > > Folks, > > > > Is there a bug in ceph RGW date parsing? > > https://github.com/ceph/ceph/blob/master/src/rgw/rgw_auth_s3.cc#L223 - this > > line parses the date in x-amz-date as RFC 2616. BUT the format specified by > > Amazon S3 is ISO 8601 basic - MMDDTHHMMSSZ ( > > https://docs.aws.amazon.com/ses/latest/APIReference-V2/CommonParameters.html). > > this is a link to the Amazon Simple Email Service v2 API, it doesn't > refer to aws v2 signatures. v2 signatures for s3 are documented at > https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html#RESTAuthenticationTimeStamp, > which states that "The value of the x-amz-date header must be in one > of the RFC 2616 formats (http://www.ietf.org/rfc/rfc2616.txt)." Yes, there is a bug, but it's not the one you're thinking of. The S3 specification says one thing (RFC2616 dates [3]), and AWS's S3 implementation does something slightly different: It accepts non-GMT time offsets, and correctly converts them to GMT/UTC. RGW on the other hand has a subtle failure: 1. RFC2616 section 3.3.1 [1] Specifies that HTTP timestamps must be in GMT. 2. Not all clients follow this, and may send it with other timezones! 3. RGW uses strptime(s, "%a, %d %b %Y %H:%M:%S %z", t) to try and parse the time. 4. glibc strptime accepts the %z specifier, but discards the value! [2] 5. If the server is set to UTC, and the client sends a non-GMT date, it will be parsed as if it's in GMT, and thus potentially be offset by some time (usually a multiple of hours due to the geo-political nature of timezones). 6. If making a SigV2 request, the timezone used 4th line of the string to sign (timestamp) is *not* GMT/ offset, e.g. "Sun, 12 Dec 2021 12:52:48 -0800", RGW will respond with RequestTimeTooSkewed response. Partial demo for this bug at [4]. [1] https://datatracker.ietf.org/doc/html/rfc2616#section-3.3.1 [2] https://man7.org/linux/man-pages/man3/strptime.3.html#NOTES [3] https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html#RESTAuthenticationTimeStamp [4] https://gist.github.com/robbat2/930974e21dcdf2b31944f954c46dd904 Credit to one of my co-workers, Max Kuznetsov, for finding the initial conditions that lead this bug discovery. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Bug in RGW header x-amz-date parsing
Sorry for the late reply - vacation :/ Seems there is some confusion here: the AWS GO SDK sets the format as ISO 8601 basic - https://github.com/aws/aws-sdk-go/blob/main/aws/signer/v4/v4.go#L84. So I don't know where the divergence is happening :(. FWIW the aws go sdk works fine against S3 and fails for ceph. A very straightforward repro can be had by running https://github.com/wasabi-tech/s3-benchmark/blob/master/s3-benchmark.go against ceph - Is it possible to see why this is? (FWIW, I have created an AWS support case for this internally - I THINK s3 processes both formats, I am not sure). Thanks. Subu On Sun, Dec 12, 2021 at 12:58 PM Robin H. Johnson wrote: > On Tue, Dec 07, 2021 at 12:39:12PM -0500, Casey Bodley wrote: > > hi Subu, > > > > On Tue, Dec 7, 2021 at 12:10 PM Subu Sankara Subramanian > > wrote: > > > > > > Folks, > > > > > > Is there a bug in ceph RGW date parsing? > > > https://github.com/ceph/ceph/blob/master/src/rgw/rgw_auth_s3.cc#L223 > - this > > > line parses the date in x-amz-date as RFC 2616. BUT the format > specified by > > > Amazon S3 is ISO 8601 basic - MMDDTHHMMSSZ ( > > > > https://docs.aws.amazon.com/ses/latest/APIReference-V2/CommonParameters.html > ). > > > > this is a link to the Amazon Simple Email Service v2 API, it doesn't > > refer to aws v2 signatures. v2 signatures for s3 are documented at > > > https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html#RESTAuthenticationTimeStamp > , > > which states that "The value of the x-amz-date header must be in one > > of the RFC 2616 formats (http://www.ietf.org/rfc/rfc2616.txt)." > Yes, there is a bug, but it's not the one you're thinking of. > > The S3 specification says one thing (RFC2616 dates [3]), and AWS's S3 > implementation does something slightly different: > It accepts non-GMT time offsets, and correctly converts them to GMT/UTC. > > RGW on the other hand has a subtle failure: > 1. RFC2616 section 3.3.1 [1] >Specifies that HTTP timestamps must be in GMT. > 2. Not all clients follow this, and may send it with other timezones! > 3. RGW uses strptime(s, "%a, %d %b %Y %H:%M:%S %z", t) to try and parse >the time. > 4. glibc strptime accepts the %z specifier, but discards the value! [2] > 5. If the server is set to UTC, and the client sends a non-GMT date, >it will be parsed as if it's in GMT, and thus potentially be offset >by some time (usually a multiple of hours due to the geo-political >nature of timezones). > 6. If making a SigV2 request, the timezone used 4th line of the string to >sign (timestamp) is *not* GMT/ offset, e.g. "Sun, 12 Dec 2021 >12:52:48 -0800", RGW will respond with RequestTimeTooSkewed response. > > Partial demo for this bug at [4]. > > [1] https://datatracker.ietf.org/doc/html/rfc2616#section-3.3.1 > [2] https://man7.org/linux/man-pages/man3/strptime.3.html#NOTES > [3] > https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html#RESTAuthenticationTimeStamp > [4] https://gist.github.com/robbat2/930974e21dcdf2b31944f954c46dd904 > > Credit to one of my co-workers, Max Kuznetsov, for finding the initial > conditions that lead this bug discovery. > > -- > Robin Hugh Johnson > Gentoo Linux: Dev, Infra Lead, Foundation Treasurer > E-Mail : robb...@gentoo.org > GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 > GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
