[ceph-users] Re: RGW accessing real source IP address of a client (e.g. in S3 bucket policies)
Hey Casey, all, On 16/06/2023 17:00, Casey Bodley wrote: But when applying a bucket policy with aws:SourceIp it seems to only work if I set the internal IP of the HAProxy instance, not the public IP of the client. So the actual remote address is NOT used in my case. Did I miss any config setting anywhere? your 'rgw remote addr param' config looks right. with that same config, i was able to set a bucket policy that denied access based on I found the issue. Embarrassingly it was simply a NAT-Hairpin which was applied to the traffic from the server I was testing with. In short: Even though I targeted the public IP from the HAProxy instance the internal IP address of my test server was maintained as source since both machines are on the same network segment. That is why I first thought the LB IP was applied to the policy, but not the actual public source IP of the client. In reality it was simply the private, RFC1918, IP of the test machine that came in as source. Sorry for the noise and thanks for your help. Christian P.S. With IPv6, this would not have happened. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW accessing real source IP address of a client (e.g. in S3 bucket policies)
On Fri, Jun 16, 2023 at 2:55 AM Christian Rohmann
wrote:
>
> On 15/06/2023 15:46, Casey Bodley wrote:
>
> * In case of HTTP via headers like "X-Forwarded-For". This is
> apparently supported only for logging the source in the "rgw ops log" ([1])?
> Or is this info used also when evaluating the source IP condition within
> a bucket policy?
>
> yes, the aws:SourceIp condition key does use the value from
> X-Forwarded-For when present
>
> I have an HAProxy in front of the RGWs which has
>
> "option forwardfor" set to add the "X-Forwarded-For" header.
>
> Then the RGWs have "rgw remote addr param = http_x_forwarded_for" set,
> according to
> https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr_param
>
> and I also see remote_addr properly logged within the rgw ops log.
>
>
>
> But when applying a bucket policy with aws:SourceIp it seems to only work if
> I set the internal IP of the HAProxy instance, not the public IP of the
> client.
> So the actual remote address is NOT used in my case.
>
>
> Did I miss any config setting anywhere?
>
>
>
>
> Regards and thanks for your help
>
>
> Christian
>
>
your 'rgw remote addr param' config looks right. with that same
config, i was able to set a bucket policy that denied access based on
that X-Forwarded-For header:
$ cat bucketpolicy.json
{
"Version": "2012-10-17",
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPAllow",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::testbucket",
"arn:aws:s3:::testbucket/*"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": "127.0.0.1"
}
}
}
]
}
$ s3cmd mb s3://testbucket
$ s3cmd setpolicy bucketpolicy.json s3://testbucket
$ s3cmd --add-header=X-Forwarded-For:127.0.0.2 put bucketpolicy.json
s3://testbucket
upload: 'bucketpolicy.json' -> 's3://testbucket/bucketpolicy.json' [1 of 1]
489 of 489 100% in0s42.95 KB/s done
$ s3cmd --add-header=X-Forwarded-For:127.0.0.1 put bucketpolicy.json
s3://testbucket
upload: 'bucketpolicy.json' -> 's3://testbucket/bucketpolicy.json' [1 of 1]
489 of 489 100% in0s11.08 KB/s done
ERROR: S3 error: 403 (AccessDenied)
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW accessing real source IP address of a client (e.g. in S3 bucket policies)
On 15/06/2023 15:46, Casey Bodley wrote: * In case of HTTP via headers like "X-Forwarded-For". This is apparently supported only for logging the source in the "rgw ops log" ([1])? Or is this info used also when evaluating the source IP condition within a bucket policy? yes, the aws:SourceIp condition key does use the value from X-Forwarded-For when present I have an HAProxy in front of the RGWs which has "option forwardfor" set to add the "X-Forwarded-For" header. Then the RGWs have "rgw remote addr param = http_x_forwarded_for" set, according to https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr_param and I also see remote_addr properly logged within the rgw ops log. But when applying a bucket policy with aws:SourceIp it seems to only work if I set the internal IP of the HAProxy instance, not the public IP of the client. So the actual remote address is NOT used in my case. Did I miss any config setting anywhere? Regards and thanks for your help Christian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW accessing real source IP address of a client (e.g. in S3 bucket policies)
On Thu, Jun 15, 2023 at 4:48 PM Casey Bodley wrote: > On Thu, Jun 15, 2023 at 7:23 AM Christian Rohmann > wrote: > > > > Hello Ceph-Users, > > > > context or motivation of my question is S3 bucket policies and other > > cases using the source IP address as condition. > > > > I was wondering if and how RadosGW is able to access the source IP > > address of clients if receiving their connections via a loadbalancer / > > reverse proxy like HAProxy. > > So naturally that is where the connection originates from in that case, > > rendering a policy based on IP addresses useless. > > > > Depending on whether the connection balanced as HTTP or TCP there are > > two ways to carry information about the actual source: > > > > * In case of HTTP via headers like "X-Forwarded-For". This is > > apparently supported only for logging the source in the "rgw ops log" > ([1])? > > Or is this info used also when evaluating the source IP condition within > > a bucket policy? > > yes, the aws:SourceIp condition key does use the value from > X-Forwarded-For when present > > maybe use lua scripting to implement your policy based on this HTTP header? (won't help in the TCP balancer case, as lua only see HTTP/S3 level stuff) > > > > * In case of TCP loadbalancing, there is the proxy protocol v2. This > > unfortunately seems not even supposed by the BEAST library which RGW > uses. > > I opened feature requests ... > > > > ** https://tracker.ceph.com/issues/59422 > > ** https://github.com/chriskohlhoff/asio/issues/1091 > > ** https://github.com/boostorg/beast/issues/2484 > > > > but there is no outcome yet. > > > > > > Regards > > > > > > Christian > > > > > > [1] > > > https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr_param > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW accessing real source IP address of a client (e.g. in S3 bucket policies)
On Thu, Jun 15, 2023 at 7:23 AM Christian Rohmann wrote: > > Hello Ceph-Users, > > context or motivation of my question is S3 bucket policies and other > cases using the source IP address as condition. > > I was wondering if and how RadosGW is able to access the source IP > address of clients if receiving their connections via a loadbalancer / > reverse proxy like HAProxy. > So naturally that is where the connection originates from in that case, > rendering a policy based on IP addresses useless. > > Depending on whether the connection balanced as HTTP or TCP there are > two ways to carry information about the actual source: > > * In case of HTTP via headers like "X-Forwarded-For". This is > apparently supported only for logging the source in the "rgw ops log" ([1])? > Or is this info used also when evaluating the source IP condition within > a bucket policy? yes, the aws:SourceIp condition key does use the value from X-Forwarded-For when present > > * In case of TCP loadbalancing, there is the proxy protocol v2. This > unfortunately seems not even supposed by the BEAST library which RGW uses. > I opened feature requests ... > > ** https://tracker.ceph.com/issues/59422 > ** https://github.com/chriskohlhoff/asio/issues/1091 > ** https://github.com/boostorg/beast/issues/2484 > > but there is no outcome yet. > > > Regards > > > Christian > > > [1] > https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_remote_addr_param > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
