[ceph-users] Re: AssumeRoleWithWebIdentity in RGW with Azure AD
This is very helpful, I'll take a look at it.
Thanks,
Pritha
On Thu, Jul 11, 2024 at 8:04 PM Ryan Rempel wrote:
> Thanks!
>
> I took a crack at it myself, and have some work-in-progress here:
>
> https://github.com/cmu-rgrempel/ceph/pull/1
>
> Feel free to use any of that if you like it. It's working for me, but I've
> only tested it with Azure AD – I haven't tested the cases that it used to
> work for. (I believe it doesn't break them, but haven't tested).
>
> --
>
> Ryan Rempel
>
>
> --
> *From:* Pritha Srivastava
> *Sent:* Monday, July 8, 2024 10:38 PM
>
> Hi Ryan,
>
> This appears to be a known issue and is tracked here:
> https://tracker.ceph.com/issues/54562. There is a workaround mentioned in
> the tracker that has worked and you can try that. Otherwise, I will be
> working on this 'invalid padding' problem very soon.
>
> Thanks,
> Pritha
>
> On Tue, Jul 9, 2024 at 1:16 AM Ryan Rempel wrote:
>
> I'm trying to setup the OIDC provider for RGW so that I can have roles
> that can be assumed by people logging into their regular Azure AD
> identities. The client I'm planning to use is Cyberduck – it seems like one
> of the few GUI S3 clients that manages the OIDC login process in a way that
> could work for relatively naive users.
>
> I've gotten a fair ways down the road. I've been able to configure
> Cyberduck so that it performs the login with Azure AD, gets an identity
> token, and then sends it to Ceph to engage with the
> AssumeRoleWithWebIdentity process. However, I then get an error, which
> shows up in the Ceph rgw logs like this:
>
> 2024-07-08T17:18:09.749+ 7fb2d7845700 0 req 15967124976712370684
> 1.284013867s sts:assume_role_web_identity Signature validation failed: evp
> verify final failed: 0 error:0407008A:rsa
> routines:RSA_padding_check_PKCS1_type_1:invalid padding
>
> I turned the logging for rgw up to 20 to see if I could follow along to
> see how much of the process succeeds and learn more about what fails. I can
> then see logging messages from this file in the source code:
>
>
> https://github.com/ceph/ceph/blob/08d7ff952d78d1bbda04d5ff7e3db1e733301072/src/rgw/rgw_rest_sts.cc
>
> We get to WebTokenEngine::get_from_jwt, and it logs the JWT payload in a
> way that seems to be as expected. The logs then indicate that a request is
> sent to the /.well-known/openid-configuration endpoint that appears to be
> appropriate for the issuer of the JWT. The logs eventually indicate what
> looks like a successful and appropriate response to that. The logs then
> show that a request is sent to the jwks_uri that is indicated in the
> openid-configuration document. The response to that is logged, and it
> appears to be appropriate.
>
> We then get some logging starting with "Certificate is", so it looks like
> we're getting as far as WebTokenEngine::validate_signature. So, several
> things appear to have happened successfully – we've loading the OIDC
> provider that corresponds to the iss, and we've found a client ID that
> corresponds to what I registered when I configured things. (This is why I
> say we appear to be a fair ways down the road – a lot of this is working).
>
> It looks as though what's happening in the code now is that it's iterating
> through the certificates given in the jwks_uri content. There are 6
> certificates listed, but the code only gets as far as the first one.
> Looking at the code, what appears to be happening is that, among the
> various certificates in the jwks_uri, it's finding the first one which
> matches a thumbprint registered with Ceph (that is, which I registered with
> Ceph). This must be succeeding (for the first certificate), because the
> "Signature validation failed" logging comes later. So, the code does verify
> that the thumbprint of the first certificate matches one of the thumbprints
> I registered with Ceph for this OIDC provider.
>
> We then get to a part of the code where it tries to verify the JWT using
> the certificate, with jwt::verify. Given what gets logged ("Signature
> validateion failed: ", this must be throwing an exception.
>
> The thing I find surprising about this is that there really isn't any
> reason to think that the first certificate listed in the jwks_uri content
> is going to be the certificate used to sign the JWT. If I understand JWT
> correctly, it's appropriate to sign the JWT with any of the certificates
> listed in the jwks_uri content. Furthermore, the JWT header includes a
> reference to the kid, so it's possible for Ceph to know exactly which
> certificate the JWT purports to be signed by. And, Ceph knows that there
> might be multiple thumbprints, because we can reg
[ceph-users] Re: AssumeRoleWithWebIdentity in RGW with Azure AD
Hi Ryan,
This appears to be a known issue and is tracked here:
https://tracker.ceph.com/issues/54562. There is a workaround mentioned in
the tracker that has worked and you can try that. Otherwise, I will be
working on this 'invalid padding' problem very soon.
Thanks,
Pritha
On Tue, Jul 9, 2024 at 1:16 AM Ryan Rempel wrote:
> I'm trying to setup the OIDC provider for RGW so that I can have roles
> that can be assumed by people logging into their regular Azure AD
> identities. The client I'm planning to use is Cyberduck – it seems like one
> of the few GUI S3 clients that manages the OIDC login process in a way that
> could work for relatively naive users.
>
> I've gotten a fair ways down the road. I've been able to configure
> Cyberduck so that it performs the login with Azure AD, gets an identity
> token, and then sends it to Ceph to engage with the
> AssumeRoleWithWebIdentity process. However, I then get an error, which
> shows up in the Ceph rgw logs like this:
>
> 2024-07-08T17:18:09.749+ 7fb2d7845700 0 req 15967124976712370684
> 1.284013867s sts:assume_role_web_identity Signature validation failed: evp
> verify final failed: 0 error:0407008A:rsa
> routines:RSA_padding_check_PKCS1_type_1:invalid padding
>
> I turned the logging for rgw up to 20 to see if I could follow along to
> see how much of the process succeeds and learn more about what fails. I can
> then see logging messages from this file in the source code:
>
>
> https://github.com/ceph/ceph/blob/08d7ff952d78d1bbda04d5ff7e3db1e733301072/src/rgw/rgw_rest_sts.cc
>
> We get to WebTokenEngine::get_from_jwt, and it logs the JWT payload in a
> way that seems to be as expected. The logs then indicate that a request is
> sent to the /.well-known/openid-configuration endpoint that appears to be
> appropriate for the issuer of the JWT. The logs eventually indicate what
> looks like a successful and appropriate response to that. The logs then
> show that a request is sent to the jwks_uri that is indicated in the
> openid-configuration document. The response to that is logged, and it
> appears to be appropriate.
>
> We then get some logging starting with "Certificate is", so it looks like
> we're getting as far as WebTokenEngine::validate_signature. So, several
> things appear to have happened successfully – we've loading the OIDC
> provider that corresponds to the iss, and we've found a client ID that
> corresponds to what I registered when I configured things. (This is why I
> say we appear to be a fair ways down the road – a lot of this is working).
>
> It looks as though what's happening in the code now is that it's iterating
> through the certificates given in the jwks_uri content. There are 6
> certificates listed, but the code only gets as far as the first one.
> Looking at the code, what appears to be happening is that, among the
> various certificates in the jwks_uri, it's finding the first one which
> matches a thumbprint registered with Ceph (that is, which I registered with
> Ceph). This must be succeeding (for the first certificate), because the
> "Signature validation failed" logging comes later. So, the code does verify
> that the thumbprint of the first certificate matches one of the thumbprints
> I registered with Ceph for this OIDC provider.
>
> We then get to a part of the code where it tries to verify the JWT using
> the certificate, with jwt::verify. Given what gets logged ("Signature
> validateion failed: ", this must be throwing an exception.
>
> The thing I find surprising about this is that there really isn't any
> reason to think that the first certificate listed in the jwks_uri content
> is going to be the certificate used to sign the JWT. If I understand JWT
> correctly, it's appropriate to sign the JWT with any of the certificates
> listed in the jwks_uri content. Furthermore, the JWT header includes a
> reference to the kid, so it's possible for Ceph to know exactly which
> certificate the JWT purports to be signed by. And, Ceph knows that there
> might be multiple thumbprints, because we can register 5. So, the logic of
> trying the first valid certificate in x5c and then stopping if it fails
> seems broken, actually.
>
> I suppose what I could do as a workaround is try to figure out whether
> Azure AD is consistently using the same kid to sign the JWTs for me, and
> then only register that thumbprint with Ceph. Then, Ceph would actually
> choose the correct certificate (as the others wouldn't match a thumbprint I
> registered). I may try this – in part, just to verify what I think is
> happening. But it would be awfully fragile – I don't believe there is any
> requirement in JWT to just use one of the certificates listed in x5c.
>
> An alternative would be to try rewriting the code to apply a different
> kind of logic. The way it ought to work (it seems to me) is something like
> this:
>
>
> *
> Get the openid_configuration, and get the jwks stuff from the jwks_uri
> (which Ceph does already).
> *
> Look at
[ceph-users] Re: ceph api rgw/role
There are REST APIs 1. UpdateRole to modify a role 2. UpdateAssumeRolePolicy to modify the trust policy of a role. What exactly are you trying to modify for a role? Thanks, Pritha On Tue, Apr 23, 2024 at 11:00 AM farhad kh wrote: > hi , i used ceph api for create rgw/role but ther is not api for delete or > edit rgw/role . > how can i delete them or edit ? > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW STS Token Forbidden error since upgrading to Quincy 17.2.6
Hi Austin,
Do you have rgw debug logs that can help debug this?
Can you provide more information, as to which user is trying to assume the
role - which tenants the user and role belong to?
Can you please open a tracker issue with all this information?
Thanks,
Pritha
On Wed, Jun 14, 2023 at 6:14 PM Austin Axworthy
wrote:
> Hi Pritha,
>
> I have added the bucket to the resource, but I am still running into the
> same Forbidden response.
>
> Thanks,
> Austin
>
>
> -Original Message-
> From: Pritha Srivastava
> Sent: June 14, 2023 4:59 AM
> To: Austin Axworthy
> Cc: ceph-users@ceph.io
> Subject: [ceph-users] Re: RGW STS Token Forbidden error since upgrading to
> Quincy 17.2.6
>
> Hi Austin,
>
> Can you try by adding the bucket arn to the Resource section of the
> policy, like the following:
>
> "Resource": [
>"arn:aws:s3:::bucket1",
> "arn:aws:s3:::bucket1/*",
> "arn:aws:s3:::bucket2/*"
>
> ]
>
> Thanks,
> Pritha
>
> On Tue, Jun 13, 2023 at 6:02 PM Austin Axworthy
> wrote:
>
> > Hi,
> >
> >
> >
> > We are using STS tokens to grant temporary access for users. I was
> > running ceph version 15.2.16 and this method worked as expected until
> > the cluster was upgraded to 17.2.6.
> >
> >
> >
> > Using boto3 I am able to assume the correct role, but when trying to
> > use the temporary credentials a forbidden error is reported. This was
> > working before the upgrade. I have also tested on a 15.2.17 cluster
> > and experience the same issue.
> >
> >
> >
> > I have pasted the created role, and boto3 code I am using as well as
> > the error I am running into. Any insight on this issue would be
> > greatly appreciated.
> >
> >
> >
> > Error: botocore.exceptions.ClientError: An error occurred (403) when
> > calling the HeadObject operation: Forbidden
> >
> > Role:
> >
> > {
> >
> > "RoleId": "499eba48-8431-48f7-9aca-76000b9c01cc",
> >
> > "RoleName": "DefaultNoS3Access",
> >
> > "Path": "/",
> >
> > "Arn": "arn:aws:iam:::role/DefaultNoS3Access",
> >
> > "CreateDate": "2023-05-11T12:04:44.367Z",
> >
> > "MaxSessionDuration": 3600,
> >
> > "AssumeRolePolicyDocument":
> >
> > "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Pr
> > incipa
> >
> > l\":{\"AWS\":[\"arn:aws:iam:::user/user\"]},\"Action\":[\"sts:AssumeRo
> > le\"]}
> > ]}"
> >
> > },
> >
> >
> >
> >
> >
> > Boto3 Code:
> >
> > response = sts_client.assume_role(
> >
> > RoleArn=role_arn,
> >
> > RoleSessionName="test",
> >
> > Policy=json.dumps(
> >
> > {
> >
> > "Version": "2012-10-17",
> >
> > "Statement": [
> >
> > {
> >
> > "Effect": "Allow",
> >
> > "Action": [
> >
> > "s3:GetObject",
> >
> > "s3:PutObjectAcl",
> >
> > "s3:PutObject",
> >
> > "s3:ListBucket"
> >
> > ],
> >
> > "Resource": [
> >
> > "arn:aws:s3:::bucket1/*",
> >
> > "arn:aws:s3:::bucket2/*"
> >
> > ]
> >
> > }
> >
> > ]
> >
> > }, separators=(',', ':')
> >
> > )
> >
> > )
> >
> >
> >
> > credentials = response['Credentials']
> >
> >
> >
> > s3_client = boto3.client(
> >
> > 's3',
> >
> > aws_access_key_id=credentials['AccessKeyId'],
> >
> > aws_secret_access_key=credentials['SecretAccessKey'],
> >
> > aws_session_token=credentials['SessionToken'],
> >
> > endpoint_url="http://IP:8080;,
> >
> > region_name=""
> >
> > )
> >
> >
> >
> > bucket_name = 'bucket1'
> >
> > file_key = 'test.txt'
> >
> > local_file_path = '/'
> >
> >
> >
> > s3_client.download_file(bucket_name, file_key, local_file_path)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> > email to ceph-users-le...@ceph.io
> >
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW STS Token Forbidden error since upgrading to Quincy 17.2.6
Hi Austin,
Can you try by adding the bucket arn to the Resource section of the policy,
like the following:
"Resource": [
"arn:aws:s3:::bucket1",
"arn:aws:s3:::bucket1/*",
"arn:aws:s3:::bucket2/*"
]
Thanks,
Pritha
On Tue, Jun 13, 2023 at 6:02 PM Austin Axworthy
wrote:
> Hi,
>
>
>
> We are using STS tokens to grant temporary access for users. I was running
> ceph version 15.2.16 and this method worked as expected until the cluster
> was upgraded to 17.2.6.
>
>
>
> Using boto3 I am able to assume the correct role, but when trying to use
> the
> temporary credentials a forbidden error is reported. This was working
> before
> the upgrade. I have also tested on a 15.2.17 cluster and experience the
> same
> issue.
>
>
>
> I have pasted the created role, and boto3 code I am using as well as the
> error I am running into. Any insight on this issue would be greatly
> appreciated.
>
>
>
> Error: botocore.exceptions.ClientError: An error occurred (403) when
> calling
> the HeadObject operation: Forbidden
>
> Role:
>
> {
>
> "RoleId": "499eba48-8431-48f7-9aca-76000b9c01cc",
>
> "RoleName": "DefaultNoS3Access",
>
> "Path": "/",
>
> "Arn": "arn:aws:iam:::role/DefaultNoS3Access",
>
> "CreateDate": "2023-05-11T12:04:44.367Z",
>
> "MaxSessionDuration": 3600,
>
> "AssumeRolePolicyDocument":
>
> "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principa
>
> l\":{\"AWS\":[\"arn:aws:iam:::user/user\"]},\"Action\":[\"sts:AssumeRole\"]}
> ]}"
>
> },
>
>
>
>
>
> Boto3 Code:
>
> response = sts_client.assume_role(
>
> RoleArn=role_arn,
>
> RoleSessionName="test",
>
> Policy=json.dumps(
>
> {
>
> "Version": "2012-10-17",
>
> "Statement": [
>
> {
>
> "Effect": "Allow",
>
> "Action": [
>
> "s3:GetObject",
>
> "s3:PutObjectAcl",
>
> "s3:PutObject",
>
> "s3:ListBucket"
>
> ],
>
> "Resource": [
>
> "arn:aws:s3:::bucket1/*",
>
> "arn:aws:s3:::bucket2/*"
>
> ]
>
> }
>
> ]
>
> }, separators=(',', ':')
>
> )
>
> )
>
>
>
> credentials = response['Credentials']
>
>
>
> s3_client = boto3.client(
>
> 's3',
>
> aws_access_key_id=credentials['AccessKeyId'],
>
> aws_secret_access_key=credentials['SecretAccessKey'],
>
> aws_session_token=credentials['SessionToken'],
>
> endpoint_url="http://IP:8080;,
>
> region_name=""
>
> )
>
>
>
> bucket_name = 'bucket1'
>
> file_key = 'test.txt'
>
> local_file_path = '/'
>
>
>
> s3_client.download_file(bucket_name, file_key, local_file_path)
>
>
>
>
>
>
>
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: External Auth (AssumeRoleWithWebIdentity) , STS by default, generic policies and isolation by ownership
Hi Christian,
Replies are inline.
On Wed, Mar 15, 2023 at 9:27 PM Christian Rohmann <
christian.rohm...@inovex.de> wrote:
> Hello ceph-users,
>
> unhappy with the capabilities in regards to bucket access policies when
> using the Keystone authentication module
> I posted to this ML a while back -
>
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/S2TV7GVFJTWPYA6NVRXDL2JXYUIQGMIN/
>
> In general I'd still like to hear how others are making use of external
> authentication and STS and what your
> experiences are in replacing e.g. Keystone authentication
>
>
>
> In the meantime we looked into OIDC authentication (via Keycloak) and
> the potentials there.
> While this works in general, AssumeRoleWithWebIdentity comes back with
> an STS token and that can be used to access S3 buckets,
> I am wondering about a few things:
>
>
> 1) How to enable STS for everyone (without user-individual policy to
> AssumeRole)
>
> In the documentation on STS
> (https://docs.ceph.com/en/quincy/radosgw/STS/#sts-in-ceph) and also
> STS-Lite (https://docs.ceph.com/en/quincy/radosgw/STSLite/#sts-lite)
> it's implied at one has to attach an dedicated policy to allow for STS
> to each user individually. This does not scale well with thousands of
> users. Also when using a federated / external authentication, there is no
> explicit user creation "A shadow user is created corresponding to every
> federated user. The user id is derived from the ‘sub’ field of the
> incoming web token."
>
> Is there a way to automatically have a role corresponding to each user
> that can be assumed via a OIDC token?
> So an implicit role that would allow for an externally authenticated
> user to have full access to S3 and all buckets owned?
> Looking at STS Lite documentation, it seems all the more natural to be
> able to allow keystone users to make use of STS.
>
> Is there any way to apply such an AssumeRole policy "globally" or for a
> whole set of users at the same time?
>
Attributes based access control has been added to STS, where tags can be
attached to users and resources and generic policies can be written using
them like the following (
https://docs.ceph.com/en/quincy/radosgw/session-tags/):
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":["s3:*"],
"Resource":["*"],
"Condition":{"StringEquals":{"s3:ResourceTag/Department":
"${aws:PrincipalTag/Department}"}}
}]
}
> I just found PR https://github.com/ceph/ceph/pull/44434 aiming to add
> policy variables such as ${aws:username} to allow for generic policies.
> But this is more about restricting bucket names or granting access to
> certain pattern of names.
>
>
>
> 2) Isolation in S3 Multi-Tenancy with external IdP
> (AssumeRoleWithWebIdentity), how does bucket ownership come into play?
>
> Following the question about generic policies for STS I am wondering
> about the role (no pun intended) that the bucket ownership or tenant
> play here?
> If one creates a role policy of e.g.
>
>
> {"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}}
>
> Would this allow someone assuming this role access to all, "*", buckets,
> or just those owned by the user that created this role policy?
>
> AssumeRole* APIs allow access to buckets owned by the user that creates
the role.
In case there is a generic role policy which allows s3:* actions, the
federated user/ local user assuming a 'role' will be allowed to create and
own a bucket in the tenant in which the role already exists.
Although a typical use case for AssumeRole API is to give access to
existing buckets of users in a tenant (cross account access).
>
> In case of Keystone auth the owner of a bucket is the project, not the
> individual (human) user. So this creates somewhat of a tenant which I'd
> want to isolate.
>
>
>
> 3) Allowing users to create their own roles and policies by default
>
> Is there a way to allow users to create their own roles and policies to
> use them by default?
> All the examples talk about the requirement for admin caps and
> individual setting of '--caps="user-policy=*'.
>
> A user needs to have perms to create a role or a user policy.
> If there was a default role + policy (question #1) that could be applied
> to externally authenticated users, I'd like for them to be able to
> create new roles and policies to grant access to their buckets to other
> users.
>
> Externally authenticated users can be allowed to create new roles by
allowing iam actions like 'iam:CreateRole', 'iam:PutRolePolicy' in the role
policy that the external user assumes. But there is no default role +
policy.
>
>
>
>
> Regards
>
>
> Christian
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To
[ceph-users] Re: How do I troubleshoot radosgw errors STS?
I will look into the bug that you submitted.
Thanks,
Pritha
On Thu, Mar 2, 2023 at 3:46 AM wrote:
> Hello,
>
> I just submitted: https://tracker.ceph.com/issues/58890
>
> Here are more details about the configuration. Note that I've tried a URL
> with and without a trailing `/` slash like what appears in the ISS.
>
> STS OpenIDConnectProvider
>
>
> {
> "ClientIDList": [
> "radosgw"
> ],
> "CreateDate": "2023-03-01T04:05:45.93+00:00",
> "ThumbprintList": [
> "16A1FBBEE0DC3F78C2013326B2EBA2B9F6D59575"
> ],
> "Url": "https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c
> "
> }
>
>
> Role document with the ARN used in the AssumeRoleWithIdentity call. The
> token returns a "sub" claim with the value of "mathew.utter", e.g. me.
>
>
> {
> "RoleId": "53186307-cc98-4904-b867-aa6c2fb10291",
> "RoleName": "AssumeRoleWithWebIdentityForOIDC",
> "Path": "/",
> "Arn": "arn:aws:iam:::role/AssumeRoleWithWebIdentityForOIDC",
> "CreateDate": "2023-03-01T04:05:46.417Z",
> "MaxSessionDuration": 3600,
> "AssumeRolePolicyDocument":
> "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":[\"arn:aws:iam:::oidc-provider/login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c\"]},\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c:sub\":\"mathew.utter\"}}}]}"
> }
>
>
> Policy attached to the role:
>
>
> {
> "Permission policy":
> "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"s3:*\"],\"Resource\":[\"arn:aws:s3:::*\"]}]}"
> }
>
>
>
> There would be a role and policy created for each OIDC user, which is why
> I'm user the "sub" in the Role.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: How do I troubleshoot radosgw errors STS?
Hi,
What version of ceph are you using? Can you share the trust policy that is
attached to the role being assumed?
Thanks,
Pritha
On Wed, Mar 1, 2023 at 9:07 PM wrote:
> I've setup RadosGW with STS ontop of my ceph cluster. It works great and
> fine but I'm also trying to setup authentication with an OpenIDConnect
> provider. I'm have a hard time troubleshooting issues because the radosgw
> log file doesn't have much information in it. For example when I try to use
> the `sts:AssumeRoleWithWebIdentity` API it fails with `{'Code':
> 'AccessDenied', ...}` and all I see is the beat log showing an HTTP 403.
>
> Is there a way to enable more verbose logging so I can see what is failing
> and why I'm getting certain errors with STS, S3, or IAM apis?
>
> My ceph.conf looks like this for each node (mildly redacted):
>
> ```
> [client.radosgw.pve4]
> host = pve4
> keyring = /etc/pve/priv/ceph.client.radosgw.keyring
> log file = /var/log/ceph/client.radosgw.$host.log
> rgw_dns_name = s3.lab
> rgw_frontends = beast endpoint=0.0.0.0:7480 ssl_endpoint=0.0.0.0:443
> ssl_certificate=/etc/pve/priv/ceph/s3.lab.crt
> ssl_private_key=/etc/pve/priv/ceph/s3.lab.key
> rgw_sts_key =
> rgw_s3_auth_use_sts = true
> rgw_enable_apis = s3, s3website, admin, sts, iam
> ```
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW cannot list or create openidconnect providers
Hi,
Have you added oidc-provider caps to the user that is trying to create the
openid connect provider/ list openid connect providers, in your case the
user which has the access key as 'L70QT3LN71SQXWHS97Y4'. (
https://docs.ceph.com/en/quincy/radosgw/oidc/)
Thanks,
Pritha
On Fri, Feb 17, 2023 at 4:54 AM wrote:
> Hello,
>
> I'm attempting to setup an OpenIDConnect provider with RGW. I'm doing this
> using the boto3 API & Python. However it seems that the APIs are failing in
> some unexpected ways because radosgw was not setup correctly. There is
> sample code below, and yes, I know there are "secrets" in it - but this is
> an offline test lab so I am fine with this.
>
> The first error shows this in the logs.
>
> 2023-02-16T00:45:26.860-0500 7fe19fef7700 1 == starting new request
> req=0x7fe2ccb54680 =
> 2023-02-16T00:45:26.904-0500 7fe19def3700 0 req 17562030806519127926
> 0.044000439s ERROR: listing filtered objects failed: OIDC pool:
> default.rgw.meta: oidc_url.: (2) No such file or directory
> 2023-02-16T00:45:26.904-0500 7fe19aeed700 1 == req done
> req=0x7fe2ccb54680 op status=-2 http_status=404 latency=0.044000439s ==
> 2023-02-16T00:45:26.904-0500 7fe19aeed700 1 beast: 0x7fe2ccb54680:
> 10.20.104.178 - authentik [16/Feb/2023:00:45:26.860 -0500] "POST /
> HTTP/1.1" 404 189 - "Boto3/1.26.71 Python/3.11.1
> Linux/6.0.6-76060006-generic Botocore/1.29.72" - latency=0.044000439s
>
> So the object "oidc_url" is missing from the "default.rgw.meta" pool?
>
> rados --pool default.rgw.meta ls --all
> users.uid root.buckets
> users.uid authentik.buckets
> roottest4
> root.bucket.meta.test2:3866fac0-854b-48b5-b3b7-bf84a166a404.1165645.1
> users.keys ZVBTLTYRRPY7JU39WOR9
> users.uid authentik
> users.uid cephadmin
> users.keys NIVIV0JSKD9D2LDC3IH4
> users.uid root
> users.email tes...@lab.dev
> users.keys L70QT3LN71SQXWHS97Y4
> root.bucket.meta.test:3866fac0-854b-48b5-b3b7-bf84a166a404.1204730.1
> root.bucket.meta.test4:3866fac0-854b-48b5-b3b7-bf84a166a404.1204730.2
> roottest
> roottest2
>
> Well the object is clearly not there and I do not know how to fix this.
>
> The second error produces this error in the log:
>
> 2023-02-16T01:11:29.304-0500 7fe1976e6700 1 == starting new request
> req=0x7fe2ccb54680 =
> 2023-02-16T01:11:29.312-0500 7fe18c6d0700 1 == req done
> req=0x7fe2ccb54680 op status=-22 http_status=400 latency=0.00883s ==
> 2023-02-16T01:11:29.312-0500 7fe18c6d0700 1 beast: 0x7fe2ccb54680:
> 10.20.104.178 - authentik [16/Feb/2023:01:11:29.304 -0500] "POST /
> HTTP/1.1" 400 189 - "Boto3/1.26.71 Python/3.11.1
> Linux/6.0.6-76060006-generic Botocore/1.29.72" - latency=0.00883s
>
> Its much less clear what is going on here, it just returns 400. Boto
> raises this exception, "botocore.exceptions.ClientError: An error occurred
> (Unknown) when calling the CreateOpenIDConnectProvider operation: Unknown".
>
> Has anyone seen this before and know how to setup the correct objects for
> OpenidConnect?
>
> Version info
> ==
> ceph version 17.2.5 (e04241aa9b639588fa6c864845287d2824cb6b55) quincy
> (stable)
>
>
> Examples below
> ==
>
> # creating the client works fine - I can see my user authenticate in the
> radosgw logs
> access_key_id = 'L70QT3LN71SQXWHS97Y4'
> secret_access_key = 'QEXLa5V0Zm38068n3goDtm8V6WlaDwxVmAq9W2XV'
> iam = boto3.client('iam',
> aws_access_key_id=access_key_id,
> aws_secret_access_key=secret_access_key,
> region_name="default",
> endpoint_url="https://s3.lab;)
>
> # First error
> providers_response = iam.list_open_id_connect_providers()
>
> # Second Error
> oidc_response = iam.create_open_id_connect_provider(
> # Issuer URL
> Url="https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/;,
> ClientIDList=['authentik'],
>
> ThumbprintList=['BDCC44F40254E7E1258DA4698833FFE2E8AECA3D3799044D8A1F97F7DFF20511'])
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph RADOSGW with Keycloak ODIC
Hi,
When you list the roles, the Condition element of the trust policy in the
role doesn't seem quite right:
"Condition": {
>"StringEquals": {
>"localhost:8080/auth/realms/demo:myclient
": "account"
>}
But what you have mentioned in the policy_document just above is correct:
"Condition":{"StringEquals":{"localhost:8080/auth/realms/demo:app_id":"account"}}
Is the value of 'aud' field in the access token that you generated, set to
"account"?
Another thing to check would be to see that the clientid (myclient) that
you have set in clientIdList as part of create_openid_connect_provider()
call, matches with the value of either clientId or client_id field in the
access token.
Or you can also check rgw logs and see what error is being logged for
AssumeRoleWithWebIdentity.
Thanks,
Pritha
On Sat, Mar 19, 2022 at 12:21 AM Seth Cagampang
wrote:
> Hello,
>
>
>
> It seems like Pritha is the Ceph RGW expert in this forum. I am currently
> trying to integrate CephRGW object storage with KeyCloak as the OIDC
> provider. I am running ceph version 16.2.7 Pacific stable.
>
>
>
> At this point, I am just trying to get a POC working with the python
> scripts provided in the example in these docs <
> https://docs.ceph.com/en/latest/radosgw/STS/#sts-configuration> . Here are
> some step by step instructions on how I set up the ceph cluster and
> KeyCloak server:
>
>
>
> *Set up keycloak server*:
>
> 1. Create new Realm 'demo'
>
> 2. Create 'testuser' and add credentials. Verify that I am able to login to
> the realm using the new credentials.
>
> 3. Create a client 'myclient' and set Access Type as 'confidential' to
> generate client secret
>
> 4. Add a keycloak-oidc provider using the client credentials.
>
> 5. On the client set 'Authorization Enabled' to ON and 'Service Accounts
> Enabled' to ON.
>
>
>
> We should now be able to get the access tokens from the OIDC provider. To
> do this I used the sample curl calls from these docs <
> https://docs.ceph.com/en/latest/radosgw/keycloak/#setting-up-keycloak>
> which I put into scripts:
>
> access_token.sh
>
> #!/bin/bash
>
> KC_REALM=demo
>
> KC_CLIENT=myclient
>
> KC_CLIENT_SECRET=620b31fa----
>
> KC_SERVER=localhost:8080 to
> write a script to generate the thumbprints:
>
> # Get the 'x5c' from this response to turn into an IDP-cert
>
> KEY1_RESPONSE=$(curl -k -v \
>
> -X GET \
>
> -H "Content-Type: application/x-www-form-urlencoded" \
>
> "http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs
> "
> \
>
> | jq -r .keys[0].x5c)
>
>
>
> KEY2_RESPONSE=$(curl -k -v \
>
> -X GET \
>
> -H "Content-Type: application/x-www-form-urlencoded" \
>
> "http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs
> "
> \
>
> | jq -r .keys[1].x5c)
>
>
>
> echo
>
> echo "Assembling Certificates"
>
>
>
> # Assemble Cert1
>
> echo '-BEGIN CERTIFICATE-' > certificate1.crt
>
> echo $(echo $KEY1_RESPONSE) | sed
> 's/^.//;s/.$//;s/^.//;s/.$//;s/^.//;s/.$//' >>
[ceph-users] Re: Keycloack with Radosgw
Hi Simone,
The condition element will be:
"StringEquals": {"mykeycloak.org.com/auth/realms/myrealm:app_id":"radosgw"}
Thanks,
Pritha
On Wed, Mar 16, 2022 at 9:44 PM wrote:
> Hi Pritha,
>
>
>
> I will test APIs with suggested tools.
>
>
>
> What is not clear to me is the aud and app_id.
>
>
>
> In the token I have an “aud” field like this:
> "aud": [
>
> "radosgw",
>
> "account"
>
> ],
>
>
>
> I need to add into this field the value “app_id” and use this Condition?
>
>
> "StringEquals": {"mykeycloak.org.com/auth/realms/myrealm:app_id":"aud"}
>
>
>
>
>
> Could you please share a concrete example?
>
>
>
> Best
>
> Simone
>
> *Da:* Pritha Srivastava
> *Inviato:* mercoledì 16 marzo 2022 16:39
> *A:* simone.becc...@staff.aruba.it
> *Cc:* ceph-users
> *Oggetto:* Re: [ceph-users] Keycloack with Radosgw
>
>
>
> The value of the 'aud' field in the token must be set in the Condition
> element, checking it against 'app_id'. There is no need to add a custom
> field 'app_id'.
>
>
>
> The Ceph STS APIs have been tested using standard AWS tools (boto3 and
> aws), so I'd suggest you to use them.
>
>
>
> Thanks,
>
> Pritha
>
>
>
> On Wed, Mar 16, 2022 at 8:45 PM wrote:
>
> Hi Pritha,
>
>
>
> I need to add a custom field to the access-token named “app_id” ?
>
>
>
> Looking at the RGW logs files I see the curl request coming correctly, the
> point is that i receive a “listBucket” response instead a set of temporary
> credentials.
>
>
>
> Best
>
> Simone
>
> *Da:* Pritha Srivastava
> *Inviato:* mercoledì 16 marzo 2022 13:15
> *A:* simone.becc...@staff.aruba.it
> *Cc:* ceph-users
> *Oggetto:* Re: [ceph-users] Keycloack with Radosgw
>
>
>
> Please correct the trust policy with the condition element that I pointed
> out before. Also, Can you please try using AWS tools - boto3 or AWS STS
> apis to make the AssumeRoleWithWebIdentity call. You can check the RGW log
> files to see whether the call reaches RGW with the curl command.
>
>
>
> Thanks,
>
> Pritha
>
> On Wednesday, March 16, 2022, wrote:
>
> Hi Pritha,
>
> in step 4) I created a role with the Trust Policy:
>
> radosgw-admin role create --role-name=S3Access2
> --assume-role-policy-doc=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Allow\",\"Principal\":\{\"Federated\":\"arn:aws:iam:::oidc-provider/
> mykeycloak.org.com/auth/realms/myrealm\
> <http://mykeycloak.org.com/auth/realms/myrealm/>
> "\},\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Condition\":\{\"StringEquals\":\{\"
> mykeycloak.org.com/auth/realms/myrealm:clientId\
> <http://mykeycloak.org.com/auth/realms/myrealm:clientId/>
> ":\"radosgw\"\}\}\}\]\}
>
> Is this not correct?
>
> To call the AssumeRoleWithWebIdentity I used curl:
>
> curl -k -v -X GET "
> http://X.X.X.X:423/?Action=AssumeRoleWithWebIdentity=3600=
> arn:aws:iam:::role/S3Access =XXX"
>
>
> Best
> Simone
>
> -Messaggio originale-
> Da: Pritha Srivastava
> Inviato: mercoledì 16 marzo 2022 11:11
> A: simone.becc...@staff.aruba.it
> Cc: ceph-users
> Oggetto: [ceph-users] Re: Keycloack with Radosgw
>
> Hi Simone,
>
> There is a step that I see missing here - have you created a role? For
> creating a role, you need to attach 'roles' caps to the user that you
> created.
> Also, what tool have you used to make the AssumeRoleWithWebIdentity call?
> An example using boto3 is outlined in the documentation here:
> https://docs.ceph.com/en/pacific/radosgw/STS/#sts-configuration. Also in
> Pacific the trust policy supports only app_id in the Condition element:
>
> From the documentation:
>
> 'The app_id in the condition above must match the ‘aud’ claim of the
> incoming token.'
>
> Thanks,
> Pritha
>
> On Wed, Mar 16, 2022 at 3:05 PM wrote:
>
> > Hi all,
> >
> >
> >
> > I'm trying to setup Keycloak as OpenID Connect Provider for Rados
> > Gateway without success (I'm using Ceph version pacific).
> >
> >
> >
> > Following the documentation I made these steps:
> >
> >
> >
> > 1) Added to /etc/ceph/ceph.conf the following options:
> >
> >
> >
> > [client]
> >
> > rgw sts key = 1234abcd5678efgh
> >
> > rgw s3 auth use sts = true
> >
> >
> >
&
[ceph-users] Re: Keycloack with Radosgw
The value of the 'aud' field in the token must be set in the Condition
element, checking it against 'app_id'. There is no need to add a custom
field 'app_id'.
The Ceph STS APIs have been tested using standard AWS tools (boto3 and
aws), so I'd suggest you to use them.
Thanks,
Pritha
On Wed, Mar 16, 2022 at 8:45 PM wrote:
> Hi Pritha,
>
>
>
> I need to add a custom field to the access-token named “app_id” ?
>
>
>
> Looking at the RGW logs files I see the curl request coming correctly, the
> point is that i receive a “listBucket” response instead a set of temporary
> credentials.
>
>
>
> Best
>
> Simone
>
> *Da:* Pritha Srivastava
> *Inviato:* mercoledì 16 marzo 2022 13:15
> *A:* simone.becc...@staff.aruba.it
> *Cc:* ceph-users
> *Oggetto:* Re: [ceph-users] Keycloack with Radosgw
>
>
>
> Please correct the trust policy with the condition element that I pointed
> out before. Also, Can you please try using AWS tools - boto3 or AWS STS
> apis to make the AssumeRoleWithWebIdentity call. You can check the RGW log
> files to see whether the call reaches RGW with the curl command.
>
>
>
> Thanks,
>
> Pritha
>
> On Wednesday, March 16, 2022, wrote:
>
> Hi Pritha,
>
> in step 4) I created a role with the Trust Policy:
>
> radosgw-admin role create --role-name=S3Access2
> --assume-role-policy-doc=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Allow\",\"Principal\":\{\"Federated\":\"arn:aws:iam:::oidc-provider/
> mykeycloak.org.com/auth/realms/myrealm\
> <http://mykeycloak.org.com/auth/realms/myrealm/>
> "\},\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Condition\":\{\"StringEquals\":\{\"
> mykeycloak.org.com/auth/realms/myrealm:clientId\
> <http://mykeycloak.org.com/auth/realms/myrealm:clientId/>
> ":\"radosgw\"\}\}\}\]\}
>
> Is this not correct?
>
> To call the AssumeRoleWithWebIdentity I used curl:
>
> curl -k -v -X GET "
> http://X.X.X.X:423/?Action=AssumeRoleWithWebIdentity=3600=
> arn:aws:iam:::role/S3Access =XXX"
>
>
> Best
> Simone
>
> -Messaggio originale-
> Da: Pritha Srivastava
> Inviato: mercoledì 16 marzo 2022 11:11
> A: simone.becc...@staff.aruba.it
> Cc: ceph-users
> Oggetto: [ceph-users] Re: Keycloack with Radosgw
>
> Hi Simone,
>
> There is a step that I see missing here - have you created a role? For
> creating a role, you need to attach 'roles' caps to the user that you
> created.
> Also, what tool have you used to make the AssumeRoleWithWebIdentity call?
> An example using boto3 is outlined in the documentation here:
> https://docs.ceph.com/en/pacific/radosgw/STS/#sts-configuration. Also in
> Pacific the trust policy supports only app_id in the Condition element:
>
> From the documentation:
>
> 'The app_id in the condition above must match the ‘aud’ claim of the
> incoming token.'
>
> Thanks,
> Pritha
>
> On Wed, Mar 16, 2022 at 3:05 PM wrote:
>
> > Hi all,
> >
> >
> >
> > I'm trying to setup Keycloak as OpenID Connect Provider for Rados
> > Gateway without success (I'm using Ceph version pacific).
> >
> >
> >
> > Following the documentation I made these steps:
> >
> >
> >
> > 1) Added to /etc/ceph/ceph.conf the following options:
> >
> >
> >
> > [client]
> >
> > rgw sts key = 1234abcd5678efgh
> >
> > rgw s3 auth use sts = true
> >
> >
> >
> >
> >
> > 2) Created a user on radosgw with caps:
> >
> >
> >
> > radosgw-admin --uid MYUSER --display-name "MyUser" --access_key MYUSER
> > --secret test123 user create
> >
> > radosgw-admin caps add --uid=" MYUSER" --caps="oidc-provider=*"
> >
> >
> >
> >
> >
> > 3) Added an OpenID Connect Provider from aws s3 libs:
> >
> >
> >
> > aws --profile=ceph-lab --endpoint=http://X.X.X.X:423 iam
> > create-open-id-connect-provider --url
> > https://mykeycloak.org.com/auth/realms/myrealm --thumbprint-list
> >
> >
> >
> >
> >
> >
> > 4) Added this trust policy named "S3Access"
> >
> >
> >
> > {
> >
> > "Version": "2012-10-17",
> >
> > "Statement": [
> >
> > {
> >
> > "Effect": "Allow",
> >
> > "Principal": {"Federated":
&
[ceph-users] Re: RGW STS AssumeRoleWithWebIdentity Multi-Tenancy
Alternatively, if you want to restrict access to s3 resources for different
groups of users, then you can do so by creating a role in a tenant, and
then create s3 resources and attach tags to them and then use ABAC/ tags to
allow a user to access a particular resource (bucket/ object). Details can
be found here: https://docs.ceph.com/en/latest/radosgw/session-tags/
Thanks,
Pritha
On Wed, Mar 9, 2022 at 9:54 AM Pritha Srivastava
wrote:
> Hi Mark,
>
> On Wed, Mar 9, 2022 at 6:57 AM Mark Selby wrote:
>
>> I am not sure that what I would like to do is even possible. I was hoping
>> there is someone out there who could chime in on this.
>>
>>
>>
>> We use Ceph RBD and Ceph FS somewhat extensively and are starting on our
>> RGW journey.
>>
>>
>>
>> We have a couple of different groups that would like to be their own
>> tenants and have a namespace to themselves. They would also like to use STS
>> and our internal iDP for short lived credentials.
>>
>>
>>
>> We have working POC in the most simple manner using RGW and STS with
>> AssumeRoleWithWebIdentity.
>>
>>
>>
>> With the Role and Policy shown below I create a bucket and end up with
>> the default ownership and ACL as also shown below.
>>
>>
>>
>> Ceph RGW ends up creating the user
>> "$oidc$cc1cfa5f-b3b2-4370-b80b-7d9492d52dfc" as a map to my iDP user which
>> lives in the default no name tenant. All buckets created by this user end
>> up in the default no name tenant.
>>
>>
>>
>> What I really want to do is assign these “dynamic” users to tenants based
>> on Roles/Policies such that I can use iDP aud/sub/azp fields for the
>> assignment.
>>
>> A role always provides access to resources in the tenant in which it is
> created. So if you want to have (shadow/dynamic) users in different
> tenants, then you will have to create roles in different tenants.
> Additionally you will also have to create openid connect providers in each
> tenant (since in AWS an OIDC provider belongs to an account and in RGW a
> tenant roughly maps to an account.). And then you can have a role trust
> policy for every role that will allow a user to assume the role based on
> its aud/azp/sub field.
>
>>
>> Is there any way using STS and AssumeRoleWithWebIdentity to assign users
>> to tenants or I am stuck with the default tenant. Could it be that I am
>> simply not correct Policy statements to attach the users to the correct
>> tenanacy.
>>
>>
>>
>> I have done a through web search and cannot find any/many examples of
>> people trying to use this paradigm
>>
>>
>>
>> Thanks!
>>
>>
>>
>> #
>>
>> # Ownership
>>
>> #
>>
>> {
>>
>> "Buckets": [
>>
>> {
>>
>> "Name": "mybuck",
>>
>> "CreationDate": "2022-02-15T20:30:32.855000+00:00"
>>
>> }
>>
>> ],
>>
>> "Owner": {
>>
>> "DisplayName": "",
>>
>> "ID": "$oidc$cc1cfa5f-b3b2-4370-b80b-7d9492d52dfc"
>>
>> }
>>
>> }
>>
>>
>>
>> #
>>
>> # ACL
>>
>> #
>>
>> {
>>
>> "Owner": {
>>
>> "ID": "$oidc$cc1cfa5f-b3b2-4370-b80b-7d9492d52dfc"
>>
>> },
>>
>> "Grants": [
>>
>> {
>>
>> "Grantee": {
>>
>> "ID": "$oidc$cc1cfa5f-b3b2-4370-b80b-7d9492d52dfc",
>>
>> "Type": "CanonicalUser"
>>
>> },
>>
>> "Permission": "FULL_CONTROL"
>>
>> }
>>
>> ]
>>
>> }
>>
>>
>>
>> #
>>
>> # Role
>>
>> #
>>
>>
>>
>> {
>>
>> "Version" :"2012-10-17",
>>
>> "Statement": [
>>
>> {
>>
>> "Effect": "Allow",
>>
>> "Principal": {
>>
>> "Federated": [
>>
>> "arn:aws:iam:::oidc-provider/
>> keycloak.acmewidgets.net:8443/auth/realms/acme"
>>
>> ]
>>
>> },
>>
>> "Action": [
>>
>> "sts:AssumeRoleWithWebIdentity"
>>
>> ],
>>
>> "Condition": {
>>
>> "StringEquals": {
>>
>> "keycloak.acmewidgets.net:8443/auth/realms/acme:app_id
>> ":"app001"
>>
>> }
>>
>> }
>>
>> }
>>
>> ]
>>
>> }
>>
>>
>>
>> #
>>
>> # Policy
>>
>> #
>>
>>
>>
>> {
>>
>> "Version": "2012-10-17",
>>
>> "Statement": {
>>
>> "Effect": "Allow",
>>
>> "Action": "s3:*",
>>
>> "Resource": "arn:aws:s3:::*"
>>
>> }
>>
>> }
>>
>>
>>
>> --
>>
>> Mark Selby
>>
>> Sr Linux Administrator, The Voleon Group
>>
>> mse...@voleon.com
>>
>>
>>
>> This email is subject to important conditions and disclosures that are
>> listed on this web page: https://voleon.com/disclaimer/.
>>
>>
>> Thanks,
> Pritha
>
>>
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW STS AssumeRoleWithWebIdentity Multi-Tenancy
Hi Mark,
On Wed, Mar 9, 2022 at 6:57 AM Mark Selby wrote:
> I am not sure that what I would like to do is even possible. I was hoping
> there is someone out there who could chime in on this.
>
>
>
> We use Ceph RBD and Ceph FS somewhat extensively and are starting on our
> RGW journey.
>
>
>
> We have a couple of different groups that would like to be their own
> tenants and have a namespace to themselves. They would also like to use STS
> and our internal iDP for short lived credentials.
>
>
>
> We have working POC in the most simple manner using RGW and STS with
> AssumeRoleWithWebIdentity.
>
>
>
> With the Role and Policy shown below I create a bucket and end up with the
> default ownership and ACL as also shown below.
>
>
>
> Ceph RGW ends up creating the user
> "$oidc$cc1cfa5f-b3b2-4370-b80b-7d9492d52dfc" as a map to my iDP user which
> lives in the default no name tenant. All buckets created by this user end
> up in the default no name tenant.
>
>
>
> What I really want to do is assign these “dynamic” users to tenants based
> on Roles/Policies such that I can use iDP aud/sub/azp fields for the
> assignment.
>
> A role always provides access to resources in the tenant in which it is
created. So if you want to have (shadow/dynamic) users in different
tenants, then you will have to create roles in different tenants.
Additionally you will also have to create openid connect providers in each
tenant (since in AWS an OIDC provider belongs to an account and in RGW a
tenant roughly maps to an account.). And then you can have a role trust
policy for every role that will allow a user to assume the role based on
its aud/azp/sub field.
>
> Is there any way using STS and AssumeRoleWithWebIdentity to assign users
> to tenants or I am stuck with the default tenant. Could it be that I am
> simply not correct Policy statements to attach the users to the correct
> tenanacy.
>
>
>
> I have done a through web search and cannot find any/many examples of
> people trying to use this paradigm
>
>
>
> Thanks!
>
>
>
> #
>
> # Ownership
>
> #
>
> {
>
> "Buckets": [
>
> {
>
> "Name": "mybuck",
>
> "CreationDate": "2022-02-15T20:30:32.855000+00:00"
>
> }
>
> ],
>
> "Owner": {
>
> "DisplayName": "",
>
> "ID": "$oidc$cc1cfa5f-b3b2-4370-b80b-7d9492d52dfc"
>
> }
>
> }
>
>
>
> #
>
> # ACL
>
> #
>
> {
>
> "Owner": {
>
> "ID": "$oidc$cc1cfa5f-b3b2-4370-b80b-7d9492d52dfc"
>
> },
>
> "Grants": [
>
> {
>
> "Grantee": {
>
> "ID": "$oidc$cc1cfa5f-b3b2-4370-b80b-7d9492d52dfc",
>
> "Type": "CanonicalUser"
>
> },
>
> "Permission": "FULL_CONTROL"
>
> }
>
> ]
>
> }
>
>
>
> #
>
> # Role
>
> #
>
>
>
> {
>
> "Version" :"2012-10-17",
>
> "Statement": [
>
> {
>
> "Effect": "Allow",
>
> "Principal": {
>
> "Federated": [
>
> "arn:aws:iam:::oidc-provider/
> keycloak.acmewidgets.net:8443/auth/realms/acme"
>
> ]
>
> },
>
> "Action": [
>
> "sts:AssumeRoleWithWebIdentity"
>
> ],
>
> "Condition": {
>
> "StringEquals": {
>
> "keycloak.acmewidgets.net:8443/auth/realms/acme:app_id":"app001"
>
> }
>
> }
>
> }
>
> ]
>
> }
>
>
>
> #
>
> # Policy
>
> #
>
>
>
> {
>
> "Version": "2012-10-17",
>
> "Statement": {
>
> "Effect": "Allow",
>
> "Action": "s3:*",
>
> "Resource": "arn:aws:s3:::*"
>
> }
>
> }
>
>
>
> --
>
> Mark Selby
>
> Sr Linux Administrator, The Voleon Group
>
> mse...@voleon.com
>
>
>
> This email is subject to important conditions and disclosures that are
> listed on this web page: https://voleon.com/disclaimer/.
>
>
> Thanks,
Pritha
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW support IAM user authentication
Thanks Michael, suggestions for doc changes have been noted.
Pritha
On Wed, Nov 24, 2021 at 5:12 PM Michael Breen <
michael.br...@vikingenterprise.com> wrote:
> Thank you, Pritha, I am most grateful.
>
> As you have "{sts key for encrypting the session token}" with {} marking
> something that needs to be replaced, perhaps the doc could be changed to
> also add a "{name}" in the section header? Putting it under [global] also
> worked.
>
> Best regards,
> Michael
>
>
> On Wed, 24 Nov 2021 at 10:55, Pritha Srivastava
> wrote:
>
>>
>>
>> On Wed, Nov 24, 2021 at 3:41 PM Michael Breen <
>> michael.br...@vikingenterprise.com> wrote:
>>
>>> Thank you, Pritha!
>>> aud != client_id was the immediate problem (there is another...). For
>>> anyone else who comes across this thread, go directly to
>>>
>>>
>>> https://issues.redhat.com/browse/KEYCLOAK-8954?focusedCommentId=13979543=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13979543
>>>
>>> It would be great if the reason for this kind of failure was included in
>>> the Ceph debug log, then someone could do a search and might find the above
>>> page. A lot of debug output has been added (one of the reasons I went to
>>> the master branch to try to figure out this problem) but messages giving
>>> the specific reason for such a rejection would be very helpful. Regarding
>>> IAM-related code not in any released version, do a grep for e.g.
>>> principal_tags.
>>>
>>
>> The code related to principal_tags is on master only.
>>
>>>
>>> The problem that remains - I hope the last one - is
>>>
>>> sts:assume_role_web_identity ERROR: Invalid rgw sts key, please ensure
>>> its length is 16
>>>
>>> Some others seem to have encountered this
>>> https://stackoverflow.com/questions/65420090/how-to-config-ceph-rgw-sts-key
>>> but no solution is described there. I have tried various things - the one
>>> at https://docs.ceph.com/en/latest/radosgw/STS/
>>>
>>> [client.radosgw.gateway]
>>> rgw sts key = abcdefghijklmnop
>>> rgw s3 auth use sts = true
>>>
>>> Also
>>>
>>> [client.radosgw.gateway]
>>> rgw_s3_auth_use_sts = true
>>> rgw_sts_key = "1234567890123456"
>>>
>>> and some others. It's something simple again I expect, but not obvious
>>> to me. Any ideas?
>>>
>>
>> How have you named your rgw? You will have to ensure that your rgw
>> section name is correct, else for testing you can add it to the global
>> section - it should work. The one given in the documentation:
>> https://docs.ceph.com/en/latest/radosgw/STS/ works.
>>
>> Thanks,
>> Pritha
>>
>>> Best regards,
>>> Michael
>>>
>>>
>>> On Wed, 24 Nov 2021 at 03:50, Pritha Srivastava
>>> wrote:
>>>
>>>> Hi Michael,
>>>>
>>>> My responses are inline:
>>>>
>>>> On Tue, Nov 23, 2021 at 10:07 PM Michael Breen <
>>>> michael.br...@vikingenterprise.com> wrote:
>>>>
>>>>> Hi Pritha - or anyone who knows,
>>>>>
>>>>> I too have problems with IAM, in particular
>>>>> with AssumeRoleWithWebIdentity.
>>>>>
>>>>> I am running the master branch version of Ceph because it looks like
>>>>> it includes code related to the functionality described at
>>>>> https://docs.ceph.com/en/latest/radosgw/STS/ - code which is not in
>>>>> any released version, even 17.0.
>>>>>
>>>>> Looking at the code on that page, there appear to be at least two
>>>>> errors:
>>>>> (1) an instance of "client" should be "sts_client" (or vice versa)
>>>>> (2) an access key and secret key are specified when creating
>>>>> sts_client, which is unnecessary and therefore confusing: only the access
>>>>> token is used or should be required for assume_role_with_web_identity
>>>>>
>>>>
>>>> I agree, these can be corrected. However I don't understand what you
>>>> mean by saying that this - "code which is not in any released version,
>>>> even 17.0."
>>>>
>>>>>
>>>>> But I still cannot get the AssumeRoleWithWebIdentity code example to
>>>>> work. The RGW debug logs show
>>
[ceph-users] Re: GCed (as in tail objects already deleted from the data pool) objects remain in the GC queue forever
Can you please open a tracker issue and attach detailed rgw logs for
objects that you know aren't getting removed from the gc queue, like the
one that you had pasted above:
2021-11-23T14:54:00.061+0100 7f6afa7fc700 20 garbage collection:
RGWGC::process iterating over entry
tag='23d143e2-d02d-4481-ba81-e783696ec99f.93072205.26537934^@',
time=2021-11-21T12:01:08.225897+0100, chain.objs.size()=3
2021-11-23T14:54:00.061+0100 7f6afa7fc700 5 garbage collection:
RGWGC::process removing
default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1
2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection:
RGWGC::process removing
default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_2
2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection:
RGWGC::process removing
default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_3
If you have corresponding osd logs, please attach them as well. Also
please add other details like - the settings that you have used. Did
you change these settings after upgrading from nautilus to
octopus?(Seems like you didn't). And also how many days after
upgrading did you start seeing this problem?
I will take a look at it asap.
Thanks,
Pritha
On Wed, Nov 24, 2021 at 4:54 PM Jaka Močnik wrote:
> hi, pritha,
>
> On Wed, 2021-11-24 at 16:41 +0530, Pritha Srivastava wrote:
>
> On Wed, Nov 24, 2021 at 4:11 PM Jaka Močnik wrote:
>
> [...]
>
> after a bit of investigation it turned out that many of the objects in
> the gc queue were already garbage collected. i.e. rgw has deleted them
> from the rados rgw data pool, but has failed to remove them from the gc
> queue.
>
> How did you diagnose this?
>
> by means dumping the gc queue via radosgw-admin gc list --include all,
> checking the logs that objects still in that list were being deleted before
> the dump, and looking at the rgw logs.
>
> here is an example for one such rados object:
>
>
>
> rados object
>
>
> 23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1
>
>
> has been deleted at least once. logs:
>
>
> 2021-11-23T14:54:00.061+0100 7f6afa7fc700 20 garbage collection:
> RGWGC::process iterating over entry
> tag='23d143e2-d02d-4481-ba81-e783696ec99f.93072205.26537934^@',
> time=2021-11-21T12:01:08.225897+0100, chain.objs.size()=3
>
> 2021-11-23T14:54:00.061+0100 7f6afa7fc700 5 garbage collection:
> RGWGC::process removing
> default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1
>
> 2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection:
> RGWGC::process removing
> default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_2
>
> 2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection:
> RGWGC::process removing
> default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_3
>
>
> object indeed does not exist in the data pool anymore:
>
>
> # rados -p default.rgw.buckets.data get
> 23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1
> out.bin
>
> error getting
> default.rgw.buckets.data/23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1:
> (2) No such file or directory
>
>
> however, it is still present in gc queue list made after time of deletion
>
>
> {
>
> "tag": "23d143e2-d02d-4481-ba81-e783696ec99f.93072205.26537934\u",
>
> "time": "2021-11-21T12:01:08.225897+0100",
>
> "objs": [
>
> {
>
> "pool": "default.rgw.buckets.data",
>
> "oid":
> "23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1",
>
> "key": "",
>
> "instance": ""
>
> },
>
> {
>
> "pool": "default.rgw.buckets.data",
>
> "oid":
> "23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_2",
>
> "key": "",
>
> "instance": ""
>
> },
>
> {
>
> "pool": "default.rgw.buckets.data",
>
> &q
[ceph-users] Re: GCed (as in tail objects already deleted from the data pool) objects remain in the GC queue forever
Hi Jaka, On Wed, Nov 24, 2021 at 4:11 PM Jaka Močnik wrote: > hi, > > running an octopus cluster (upgraded from nautilus a few months ago) of > some 0.5PB capacity. it is used exclusively as an object storage via > rgw (clients use the swift API), 6 rgw instances are used to cater to > this. the cluster has been running for a bit over two years. > > it is subject to quite a heavy delete load (think in the order of > magnitude of 1M deletes per day). > > until recently this was handled w/o any problems, however, some 10 days > ago, our monitoring alerted us that the rgw gc queue was holding some > 20k rgw objects dispersed over ~700k rados objects. while such peaks > were common before, they were usually cleared very quickly. however, > this situation has not cleared since. in fact every day, some 100-200k > extra rados objects are added to the gc queue. > > after a bit of investigation it turned out that many of the objects in > the gc queue were already garbage collected. i.e. rgw has deleted them > from the rados rgw data pool, but has failed to remove them from the gc > queue. > > How did you diagnose this? the logs (debug_rgw = 20) do not show anything unusual. deletes > succeed. even when deleting an already deleted rgw object (i.e. its > rados tail objects), there are no complaints in the log (even though > deletes of rados objects must fail as the objects are not present > anymore). however, even after n-th deletion, the objects are not > removed from the gc queue. > > so, can someone help with the following: > - any pointers on where to start debugging this? I am at a loss since > rgws seems happy enough according to the logs. > - any ideas on how to remedy this situation? it will become a problem > in a week or two, according to the trends. > Have you tried running radosgw-admin gc list command? Are some entries always there, past their expiration time? There is a flag --include-all which can also be used to list all expired and unexpired entries. Also in the logs - do you see this "RGWGC::process removing entries, marker: "? Are the markers getting repeated? with regard to remedy in case we cannot diagnose the cause and fix it > soon enough, I was thinking about: > - stopping deletes to rgws for a short while, > - dumping the gc queue contents, > - stopping rgws, > - clearing or recreating the rgw gc queue structures on rados pools, > - restarting rgws and deletes, > - manually deleting the rados objects in the old gc queue dump. > > is that a sound plan? > > if so, what exactly does the "clearing or recreating the rgw gc queue > structures on rados pools" entail? > > I am under the impression that the gc queue is stored in gc. > objects in the GC namespace in the default.rgw.log pool. > > would just deleting these and starting rgw do the trick? or do I need > to somehow recreate empty objects in their stead? > > Have you tried using the command: radosgw-admin gc process, to clear the expired entries and with --include-all to clear all entries? > best regards, > Jaka > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW support IAM user authentication
On Wed, Nov 24, 2021 at 3:41 PM Michael Breen <
michael.br...@vikingenterprise.com> wrote:
> Thank you, Pritha!
> aud != client_id was the immediate problem (there is another...). For
> anyone else who comes across this thread, go directly to
>
>
> https://issues.redhat.com/browse/KEYCLOAK-8954?focusedCommentId=13979543=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13979543
>
> It would be great if the reason for this kind of failure was included in
> the Ceph debug log, then someone could do a search and might find the above
> page. A lot of debug output has been added (one of the reasons I went to
> the master branch to try to figure out this problem) but messages giving
> the specific reason for such a rejection would be very helpful. Regarding
> IAM-related code not in any released version, do a grep for e.g.
> principal_tags.
>
The code related to principal_tags is on master only.
>
> The problem that remains - I hope the last one - is
>
> sts:assume_role_web_identity ERROR: Invalid rgw sts key, please ensure its
> length is 16
>
> Some others seem to have encountered this
> https://stackoverflow.com/questions/65420090/how-to-config-ceph-rgw-sts-key
> but no solution is described there. I have tried various things - the one
> at https://docs.ceph.com/en/latest/radosgw/STS/
>
> [client.radosgw.gateway]
> rgw sts key = abcdefghijklmnop
> rgw s3 auth use sts = true
>
> Also
>
> [client.radosgw.gateway]
> rgw_s3_auth_use_sts = true
> rgw_sts_key = "1234567890123456"
>
> and some others. It's something simple again I expect, but not obvious to
> me. Any ideas?
>
How have you named your rgw? You will have to ensure that your rgw section
name is correct, else for testing you can add it to the global section - it
should work. The one given in the documentation:
https://docs.ceph.com/en/latest/radosgw/STS/ works.
Thanks,
Pritha
> Best regards,
> Michael
>
>
> On Wed, 24 Nov 2021 at 03:50, Pritha Srivastava
> wrote:
>
>> Hi Michael,
>>
>> My responses are inline:
>>
>> On Tue, Nov 23, 2021 at 10:07 PM Michael Breen <
>> michael.br...@vikingenterprise.com> wrote:
>>
>>> Hi Pritha - or anyone who knows,
>>>
>>> I too have problems with IAM, in particular
>>> with AssumeRoleWithWebIdentity.
>>>
>>> I am running the master branch version of Ceph because it looks like it
>>> includes code related to the functionality described at
>>> https://docs.ceph.com/en/latest/radosgw/STS/ - code which is not in any
>>> released version, even 17.0.
>>>
>>> Looking at the code on that page, there appear to be at least two errors:
>>> (1) an instance of "client" should be "sts_client" (or vice versa)
>>> (2) an access key and secret key are specified when creating sts_client,
>>> which is unnecessary and therefore confusing: only the access token is used
>>> or should be required for assume_role_with_web_identity
>>>
>>
>> I agree, these can be corrected. However I don't understand what you mean
>> by saying that this - "code which is not in any released version, even
>> 17.0."
>>
>>>
>>> But I still cannot get the AssumeRoleWithWebIdentity code example to
>>> work. The RGW debug logs show
>>>
>>> debug 2021-11-23T15:51:22.247+ 7fad6e351700 0 evaluating policy for
>>> op: 93 returned deny/pass
>>>
>>> In my case, the policy_document and role_policy are
>>>
>>> policy_document =
>>> '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":["arn:aws:iam:::oidc-provider/proteus.ves.corp/auth/realms/cno"]},"Action":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"proteus.ves.corp/auth/realms/cno:app_id":"ceph_rgw"}}}]}'''
>>> role_policy =
>>> '''{"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}}'''
>>>
>>>
>>
>>> (I assume it is only the former that may be relevant here, but maybe I'm
>>> wrong.)
>>>
>>
>> The former is only relevant here. And have you created an openid connect
>> provider in RGW? ( I am assuming you must have, since it is there in the
>> examples given). You have set the condition as app_id: ceph_rgw. Whereas
>> the documentation says that - "The app_
[ceph-users] Re: RGW support IAM user authentication
Hi Michael,
My responses are inline:
On Tue, Nov 23, 2021 at 10:07 PM Michael Breen <
michael.br...@vikingenterprise.com> wrote:
> Hi Pritha - or anyone who knows,
>
> I too have problems with IAM, in particular with AssumeRoleWithWebIdentity.
>
> I am running the master branch version of Ceph because it looks like it
> includes code related to the functionality described at
> https://docs.ceph.com/en/latest/radosgw/STS/ - code which is not in any
> released version, even 17.0.
>
> Looking at the code on that page, there appear to be at least two errors:
> (1) an instance of "client" should be "sts_client" (or vice versa)
> (2) an access key and secret key are specified when creating sts_client,
> which is unnecessary and therefore confusing: only the access token is used
> or should be required for assume_role_with_web_identity
>
I agree, these can be corrected. However I don't understand what you mean
by saying that this - "code which is not in any released version, even
17.0."
>
> But I still cannot get the AssumeRoleWithWebIdentity code example to work.
> The RGW debug logs show
>
> debug 2021-11-23T15:51:22.247+ 7fad6e351700 0 evaluating policy for
> op: 93 returned deny/pass
>
> In my case, the policy_document and role_policy are
>
> policy_document =
> '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":["arn:aws:iam:::oidc-provider/proteus.ves.corp/auth/realms/cno"]},"Action":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"proteus.ves.corp/auth/realms/cno:app_id":"ceph_rgw"}}}]}'''
> role_policy =
> '''{"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}}'''
>
>
> (I assume it is only the former that may be relevant here, but maybe I'm
> wrong.)
>
The former is only relevant here. And have you created an openid connect
provider in RGW? ( I am assuming you must have, since it is there in the
examples given). You have set the condition as app_id: ceph_rgw. Whereas
the documentation says that - "The app_id in the condition above must match
the ‘aud’ claim of the incoming token." in the example which uses "app_id"
as a condition element, and the value of "aud" is "account" in the web
token. So please modify the condition accordingly. Also note that other
claims can also now be used in the condition element of the trust policy.
>
> In /etc/ceph/ceph.conf I have
> [client.radosgw.gateway]
> rgw sts key = abcdefghijklmnop
> rgw s3 auth use sts = true
>
> In the debug I can see the token from Keycloak looks like (after
> formatting it)
>
> {
> "exp": 1637677729,
> "iat": 1637677429,
> "jti": "06e5422e-8395-4727-9366-a851c3f5930f",
> "iss": "https://proteus.ves.corp/auth/realms/cno;,
> "aud": "account",
> "sub": "f45bae70-1517-48f6-9d75-af7f421f4a0c",
> "typ": "Bearer",
> "azp": "ceph_rgw",
> "session_state": "1413beec-9785-4e63-947f-72eb26da9daf",
> "acr": "1",
> "allowed-origins": [
> "*"
> ],
> "realm_access": {
> "roles": [
> "offline_access",
> "uma_authorization"
> ]
> },
> "resource_access": {
> "ceph_rgw": {
> "roles": [
> "arn:aws:iam:::role/S3Access",
> "S3Access"
> ]
> },
> "account": {
> "roles": [
> "manage-account",
> "manage-account-links",
> "view-profile"
> ]
> }
> },
> "scope": "openid profile email",
> "email_verified": true,
> "name": "testuser",
> "preferred_username": "testuser",
> "given_name": "testuser",
> "email": "test-u...@help-me-please.com"
> }
>
> Please, if you are familiar with this, can you tell me what step is
> missing? There is no description on that page of what should be done at
> Keycloak, so I'm guessing the problem may be there. (Keycloak screens are
> shown elsewhere, but for a different example.) I have spent a good deal of
> time trying to understand this, so if you could help I would greatly
> appreciate
[ceph-users] Re: RGW support IAM user authentication
Hi Nio, Can you provide more details around what you are trying to do? RGW supports attaching IAM policies to users that aid in managing their permissions. Thanks, Pritha On Tue, Nov 23, 2021 at 11:43 AM nio wrote: > hi，all: > In the process of using RGW, I still cannot authenticate users through > IAM. In the near future, will RGW support IAM to manage user permissions > and authentication functions? > > > Looking forward to your reply > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW STS - MalformedPolicyDocument
I tried the aws iam create role on master today and it worked for me. I've
used your policy file with "Main" corrected to "Principal", like below:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"AWS":[
"arn:aws:iam:::user/someuser"
]
},
"Action":[
"sts:AssumeRole"
]
}
]
}
Thanks,
Pritha
On Sun, Sep 5, 2021 at 9:11 PM Marcelo Mariano Miziara <
marcelo.mizi...@serpro.gov.br> wrote:
> Hi Pritha, thanks for the answer.
>
> Even changing to Principal I still get the MalformedPolicyDocument. I
> tested with aws cli versions 1 and 2, both returning the error message. I
> put JSON in several validators to see if there were any errors and it seems
> to be okay. I don't know if I'm missing something in the aws cli
> configuration, my credentials are:
> [default]
> aws_access_key_id = < admin-api-user key >
> aws_secret_access_key = < admin-api-user secret >
>
> And my config:
> [default]
> region = US
> output = json
>
> And I checked that no rule with the same same exists.
>
> With the example python from the page
> https://docs.ceph.com/en/latest/radosgw/STS/ it worked (it has an extra
> "]" that needs to be removed in the policy_document variable).
>
> Thanks again, Marcelo.
>
>
> De: "Pritha Srivastava"
> Para: "marcelo.miziara serpro"
> Cc: "ceph-users"
> Enviadas: Domingo, 5 de setembro de 2021 7:07:14
> Assunto: Re: [ceph-users] RGW STS - MalformedPolicyDocument
>
> Hi Marcelo,
>
> Your trust policy has an error:
>
> {
> "Version":"2012-10-17",
> "Statement":[
> {
> "Effect":"Allow",
> " Main ":{
> "AWS":[
> "arn:aws:iam:::user/someuser"
> ]
> },
> "Action":[
> "sts:AssumeRole"
> ]
> }
> ]
> }
>
> In place of 'Main', use 'Principal' as you have done for the radosgw-admin
> role create command.
>
> Thanks,
> Pritha
>
> On Fri, Sep 3, 2021 at 9:30 PM Marcelo Mariano Miziara < [
> https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=OTM1Q0I0OEYwODkuQTc4N0Q==b70b7db3446fb5c45272034c5174fcaa=bWFpbHRvOm1hcmNlbG8ubWl6aWFyYUBzZXJwcm8uZ292LmJy
> | marcelo.mizi...@serpro.gov.br ] > wrote:
>
>
> Hello all!
>
> I'm having a hard time trying to get the STS to work. I want to give a
> user "someuser" the ability to assumerole. I don't know if I got it wrong
> how to do it, or if my json is spelled wrong.
>
> I've done tests on the latest versions of nautilus, octopus and pacific,
> and I always get the same message.
>
> In RGW I added the following settings:
> -
> rgw_s3_auth_use_sts = true
> rgw_sts_key = "abcdefghijklmnop"
> -
>
> Then I create a user "admin-api-user", giving the following caps:
> -
> # radosgw-admin caps add --uid admin-api-user --caps
> "users=*;buckets=*;metadata=*;usage=*;roles=*;user-policy=*"
> -
>
> But when I try to create a role using aws cli, I get an error message:
> -
> # aws --endpoint= [
> https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=OTM1Q0I0OEYwODkuQTc4N0Q==b70b7db3446fb5c45272034c5174fcaa=aHR0cDovLzEwLjc5LjM1LjI0NTo3NDgw
> | http://10.79.35.245:7480 ] iam create-role --role-name=role1
> --assume-role-policy-document file://policy_document.json
>
> An error occurred (Unknown) when calling the CreateRole operation: Unknown
> -
>
> Running the above command with debug, on one of the lines comes the
> following message:
> -
> 2021-09-02 10:07:56,138 - MainThread - botocore.parsers - DEBUG - Response
> body:
> b' encoding="UTF-8"?>MalformedPolicyDocumenttx01-006130ccac-b3b82-defaultb3b82-default
> -default'
> -
>
> My policy_document.json is like this:
> -
> {
> "Version":"2012-10-17",
> "Statement":[
> {
> "Effect":"Allow",
> "Main":{
> "AWS":[
> "arn:aws:iam:::user/someuser"
> ]
> },
> "Action":[
> "sts:AssumeRole"
> ]
> }
> ]
> }
> -
>
> If I run the the radosgw-admin command with the same JSON (but with
> escaped characters), it works:
> -
> # radosgw-admin role create --role-name=role1
> --assume-role-policy-doc=\{\"Version\":\"2
[ceph-users] Re: RGW STS - MalformedPolicyDocument
Hi Marcelo,
Your trust policy has an error:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"*Main*":{
"AWS":[
"arn:aws:iam:::user/someuser"
]
},
"Action":[
"sts:AssumeRole"
]
}
]
}
In place of 'Main', use 'Principal' as you have done for the radosgw-admin
role create command.
Thanks,
Pritha
On Fri, Sep 3, 2021 at 9:30 PM Marcelo Mariano Miziara <
marcelo.mizi...@serpro.gov.br> wrote:
> Hello all!
>
> I'm having a hard time trying to get the STS to work. I want to give a
> user "someuser" the ability to assumerole. I don't know if I got it wrong
> how to do it, or if my json is spelled wrong.
>
> I've done tests on the latest versions of nautilus, octopus and pacific,
> and I always get the same message.
>
> In RGW I added the following settings:
> -
> rgw_s3_auth_use_sts = true
> rgw_sts_key = "abcdefghijklmnop"
> -
>
> Then I create a user "admin-api-user", giving the following caps:
> -
> # radosgw-admin caps add --uid admin-api-user --caps
> "users=*;buckets=*;metadata=*;usage=*;roles=*;user-policy=*"
> -
>
> But when I try to create a role using aws cli, I get an error message:
> -
> # aws --endpoint=http://10.79.35.245:7480 iam create-role
> --role-name=role1 --assume-role-policy-document file://policy_document.json
>
> An error occurred (Unknown) when calling the CreateRole operation: Unknown
> -
>
> Running the above command with debug, on one of the lines comes the
> following message:
> -
> 2021-09-02 10:07:56,138 - MainThread - botocore.parsers - DEBUG - Response
> body:
> b' encoding="UTF-8"?>MalformedPolicyDocumenttx01-006130ccac-b3b82-defaultb3b82-default
> -default'
> -
>
> My policy_document.json is like this:
> -
> {
> "Version":"2012-10-17",
> "Statement":[
> {
> "Effect":"Allow",
> "Main":{
> "AWS":[
> "arn:aws:iam:::user/someuser"
> ]
> },
> "Action":[
> "sts:AssumeRole"
> ]
> }
> ]
> }
> -
>
> If I run the the radosgw-admin command with the same JSON (but with
> escaped characters), it works:
> -
> # radosgw-admin role create --role-name=role1
> --assume-role-policy-doc=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Allow\",\"Principal\":\{\"AWS\":\[\"arn:aws:iam:::user/someuser\"\]\},\"Action\":\[\"sts:AssumeRole\"\]\}\]\}
>
> {
> "RoleId": "007ed38e-a072-43a4-94f3-2958e5a19408",
> "RoleName": "role1",
> "Path": "/",
> "Arn": "arn:aws:iam:::role/role1",
> "CreateDate": "2021-09-02T13:19:39.721Z",
> "MaxSessionDuration": 3600,
> "AssumeRolePolicyDocument":
> "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/someuser\"]},\"Action\":[\"sts:AssumeRole\"]}]}"
>
> }
> -
>
>
> Does anyone have any idea where I might be going wrong? I did a test on
> aws, with the same JSON (adding my arn account) and it worked.
>
> Thanks, Marcelo!
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Creating a role in another tenant seems to be possible
Hi Daniel,
Yes, it looks like a bug in the way the role name is being parsed in the
code. Please open a tracker issue for the same, and I'll fix it when I can.
Thanks,
Pritha
On Thu, Jun 10, 2021 at 5:09 PM Daniel Iwan wrote:
> Hi Pritha
>
> y answers inline.
> Forgot to add I'm on Ceph 1.2.1
>
>
>> How did you check whether the role was created in tenant1 or tenant2?
>> It shouldn't be created in tenant2, if it is, then it's a bug, please
>> open a tracker issue for it.
>>
>
> I checked that with
> radosgw-admin role list --tenant tenant1
>
> Example commands with output
> User creating roles has in this case roles:* capability.
>
> When creating without tenant prefix role is created in the tenant user
> belongs to
>
> aws --profile=user-from-tenant1 --endpoint=$HOST_S3_API --region="" iam
> create-role --role-name=TemporaryRole --assume-role-policy-document
> file://json/trust-policy-assume-role.json
>
> {
> "Role": {
> "Path": "/",
> "RoleName": "TemporaryRole",
> "RoleId": "507f990e-46cd-418c-ad4e-cc59276500dc",
> "Arn": "arn:aws:iam::tenant1:role/TemporaryRole",
> "CreateDate": "2021-06-10T11:17:15.638000+00:00",
> "AssumeRolePolicyDocument": {
> "Version": "2012-10-17",
> "Statement": [
> {
> "Effect": "Allow",
> "Principal": {
> "Federated": [
> "arn:aws:iam:::oidc-provider/
> localhost.ceph-om-vm-node3.com:8443/auth/realms/tenant1"
> ]
> },
> "Action": [
> "sts:AssumeRoleWithWebIdentity"
> ],
> "Condition": {
> "StringEquals": {
> "
> localhost.ceph-om-vm-node3.com:8443/auth/realms/tenant1:app_id": "account"
> }
> }
> }
> ]
> },
> "MaxSessionDuration": 3600
> }
> }
>
> root@:~# radosgw-admin role list --tenant tenant1
> [
> {
> "RoleId": "507f990e-46cd-418c-ad4e-cc59276500dc",
> "RoleName": "TemporaryRole",
> "Path": "/",
> "Arn": "arn:aws:iam::tenant1:role/TemporaryRole",
> "CreateDate": "2021-06-10T11:17:15.638Z",
> "MaxSessionDuration": 3600,
> "AssumeRolePolicyDocument":
> "{\n\t\"Version\":\"2012-10-17\",\n\t\"Statement\":[\n\t\t{\n\t\t\t\"Effect\":\"Allow\",\n\t\t\t\"Principal\":{\n\t\t\t\t\"Federated\":[\n\t\t\t\t\t\"arn:aws:iam:::oidc-provider/
> localhost.ceph-om-vm-node3.com:8443/auth/realms/tenant1\
>
> "\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"Action\":[\n\t\t\t\t\"sts:AssumeRoleWithWebIdentity\"\n\t\t\t],\n\t\t\t\"Condition\":{\n\t\t\t\t\"StringEquals\":{\n\t\t\t\t\t\"
> localhost.ceph-om-vm-node3.com:8443/auth/realms/tenant1:app_id\
>
> ":\"account\"\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t]\n}"
> }
> ]
>
> then created with another tenant name
>
> aws --profile=user-from-tenant1 --endpoint=$HOST_S3_API --region="" iam
> create-role --role-name="tenant2\$TemporaryRole"
> --assume-role-policy-document file://json/trust-policy-assume-role.json
> {
> "Role": {
> "Path": "/",
> "RoleName": "TemporaryRole",
> "RoleId": "9086dc3c-3654-465c-9524-dd60cee6ec09",
> "Arn": "arn:aws:iam::tenant2:role/TemporaryRole",
> "CreateDate": "2021-06-10T11:17:52.11+00:00",
> "AssumeRolePolicyDocument": {
> "Version": "2012-10-17",
> "Statement": [
> {
> "Effect": "Allow",
> "Principal": {
> "Federated": [
> "arn:aws:iam:::oidc-provider/
> localhost.ceph-om-vm-node3.com:8443/auth/realms/tenant1"
> ]
> },
> "Action": [
> "sts:AssumeRoleWithWebIdentity"
> ],
> "Condition": {
> "StringEquals": {
> "
> localhost.ceph-om-vm-node3.com:8443/auth/realms/tenant1:app_id": "account"
> }
> }
> }
> ]
> },
> "MaxSessionDuration": 3600
> }
> }
>
> root@:~# radosgw-admin role list --tenant tenant2
> [
> {
> "RoleId": "9086dc3c-3654-465c-9524-dd60cee6ec09",
> "RoleName": "TemporaryRole",
> "Path": "/",
> "Arn": "arn:aws:iam::tenant2:role/TemporaryRole",
> "CreateDate": "2021-06-10T11:17:52.110Z",
> "MaxSessionDuration": 3600,
> "AssumeRolePolicyDocument":
>
[ceph-users] Re: Creating a role in another tenant seems to be possible
On Fri, Jun 4, 2021 at 5:06 PM Daniel Iwan wrote:
> Hi
>
> It seems that with command like this
>
> aws --profile=my-user-tenant1 --endpoint=$HOST_S3_API --region="" iam
> create-role --role-name="tenant2\$TemporaryRole"
> --assume-role-policy-document file://json/trust-policy-assume-role.json
>
> I can create a role in another tenant.
> Executing user have roles:* capability which I think is necessary to be
> able to create roles, but at the same time it seems to be a global ability,
> for all tenants.
>
>
How did you check whether the role was created in tenant1 or tenant2?
It shouldn't be created in tenant2, if it is, then it's a bug, please open
a tracker issue for it.
Similarly, a federated user who assumes a role with iam:CreateRole
> permission
> can create an arbitrary role like below.
>
> aws --endpoint=$HOST_S3_API --region="" iam create-role
> --role-name="tenant2\$TemporaryRole" --assume-role-policy-document
> file://json/trust-policy-assume-role.json
>
> Example permission policy
> {
> "Statement":[
> {"Effect":"Allow","Action":["iam:GetRole"]},
> {"Effect":"Allow","Action":["iam:CreateRole"]}
> ]
> }
>
> What entity is this permission policy attached to? The user making the
CreateRole call?
Capability roles:* is not needed in this case, which I think is correct,
> because only permission policy of the assumed role is checked.
>
> Getting information about a role from other tenants is possible with
> iam:GetRole.
> This is less controversial but I would still expect it to be scoped to the
> user's tenant unless explicit tenant name is stated in the policy like this
>
>
> {"Effect":"Allow","Action":["iam:GetRole"],"Resource":"arn:aws:iam::tenant2:*"}
>
> Possibly I'm missing something.
> Why is crossing tenants possible?
>
> Regards
> Daniel
>
Thanks,
Pritha
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW federated user cannot access created bucket
Yes, that is correct. Thanks, Pritha On Thu, May 13, 2021 at 4:07 PM Daniel Iwan wrote: > Thanks, that explains it. > This is in combination with permissions given via bucket policies of > course? > > Daniel > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Using ID of a federated user in a bucket policy in RGW
It would be the second one:
arn:aws:iam::mytenant:user/oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b
Thanks,
Pritha
On Thu, May 13, 2021 at 3:54 PM Daniel Iwan wrote:
> Hi Pritha
>
> This is probably the only combination I haven't tried.
> I was convinced it would have to be $oidc$
> It worked brilliantly with the arn you suggested, thank you for that.
>
> So does it mean that for a non-empty tenant we would have tenant name twice
> like
>
> arn:aws:iam::mytenant:user/mytenant$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b
>
> or would it be a single tenant name?
> arn:aws:iam::mytenant:user/oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b
>
> If it's the second one, it relies on the fact that IDs are unique, which in
> turns depends on the sub field in the token.
>
> Regards
> Daniel
>
> On Wed, 12 May 2021 at 13:31, Pritha Srivastava
> wrote:
>
> > Hi,
> >
> > Can you try with the following ARN:
> >
> > arn:aws:iam:::user/oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b
> >
> > The format of the user id is: $$ , and in
> > $oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b, the '$' before oidc is a
> > separator for a tenant which is empty here, and ARN for a user is of the
> > format: arn:aws:iam:::user/, and hence the ARN here will
> > be arn:aws:iam:::user/oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b
> > Thanks,
> > Pritha
> >
> > On Wed, May 12, 2021 at 4:02 PM Daniel Iwan
> wrote:
> >
> >> Hi all
> >>
> >> I'm working on the following scenario
> >> User is authenticated with OIDC and tries to access a bucket which it
> does
> >> not own.
> >> How to specify user ID etc. to give access to such a user?
> >>
> >> By trial and error I found out that principal can be specified as
> >> "Principal": {"Federated":["arn:aws:sts:::assumed-role/MySession"]},
> >>
> >> but I want to use shadow user ID or something similar as the principal
> >>
> >> Docs
> >> https://docs.ceph.com/en/latest/radosgw/STS/
> >> states:
> >> 'A shadow user is created corresponding to every federated user. The
> user
> >> id is derived from the ‘sub’ field of the incoming web token. The user
> is
> >> created in a separate namespace - ‘oidc’ such that the user id doesn’t
> >> clash with any other user ids in rgw. The format of the user id is -
> >> $$ where user-namespace is ‘oidc’ for users
> >> that authenticate with oidc providers.'
> >>
> >> I see a shadow user in Web UI as e.g.
> 7f71c7c5-c24f-418e-87ac-aa8fe271289b
> >> but I cannot work out the syntax of a user id, I was expecting something
> >> like
> >>
> >> "arn:aws:iam:::user/$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b"
> >>
> >> but when trying to list content of a bucket I get AccessDenied.
> >> If bucket policy has Principal "*" the my authenticated user can access
> >> the
> >> bucket
> >>
> >> Is this possible?
> >> Regards
> >> Daniel
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW federated user cannot access created bucket
The federated user will be allowed to perform only those s3 actions that
are explicitly allowed by the role's permission policy. The permission
policy is there for someone to exercise finer grained control over what s3
action is allowed and what is not, hence it differs from what regular users
are allowed to do.
Thanks,
Pritha
On Wed, May 12, 2021 at 4:04 PM Daniel Iwan wrote:
> Hi all
>
> Scenario is as follows
> Federated user assumes a role via AssumeRoleWithWebIdentity, which gives
> permission to create a bucket.
> User creates a bucket and becomes an owner (this is visible in Ceph's web
> ui as Owner $oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b).
> User cannot list the content of the bucket however, because role's policy
> does not give access to the bucket.
> Later on when user re-authenticates and assumes the same role again.
> At this point user cannot access a bucket it owns for the reason as above
> I'm assuming.
> Bucket's ACL after creation
>
> radosgw-admin policy --bucket my-bucket
> {
> "acl": {
> "acl_user_map": [
> {
> "user": "$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b",
> "acl": 15
> }
> ],
> "acl_group_map": [],
> "grant_map": [
> {
> "id": "$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b",
> "grant": {
> "type": {
> "type": 0
> },
> "id": "$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b",
> "email": "",
> "permission": {
> "flags": 15
> },
> "name": "",
> "group": 0,
> "url_spec": ""
> }
> }
> ]
> },
> "owner": {
> "id": "$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b",
> "display_name": ""
> }
> }
>
> This seems inconsistent with buckets created by regular users
> Is this expected behaviour?
>
> Regards
> Daniel
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Using ID of a federated user in a bucket policy in RGW
Hi,
Can you try with the following ARN:
arn:aws:iam:::user/oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b
The format of the user id is: $$ , and in
$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b, the '$' before oidc is a
separator for a tenant which is empty here, and ARN for a user is of the
format: arn:aws:iam:::user/, and hence the ARN here will
be arn:aws:iam:::user/oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b
Thanks,
Pritha
On Wed, May 12, 2021 at 4:02 PM Daniel Iwan wrote:
> Hi all
>
> I'm working on the following scenario
> User is authenticated with OIDC and tries to access a bucket which it does
> not own.
> How to specify user ID etc. to give access to such a user?
>
> By trial and error I found out that principal can be specified as
> "Principal": {"Federated":["arn:aws:sts:::assumed-role/MySession"]},
>
> but I want to use shadow user ID or something similar as the principal
>
> Docs
> https://docs.ceph.com/en/latest/radosgw/STS/
> states:
> 'A shadow user is created corresponding to every federated user. The user
> id is derived from the ‘sub’ field of the incoming web token. The user is
> created in a separate namespace - ‘oidc’ such that the user id doesn’t
> clash with any other user ids in rgw. The format of the user id is -
> $$ where user-namespace is ‘oidc’ for users
> that authenticate with oidc providers.'
>
> I see a shadow user in Web UI as e.g. 7f71c7c5-c24f-418e-87ac-aa8fe271289b
> but I cannot work out the syntax of a user id, I was expecting something
> like
>
> "arn:aws:iam:::user/$oidc$7f71c7c5-c24f-418e-87ac-aa8fe271289b"
>
> but when trying to list content of a bucket I get AccessDenied.
> If bucket policy has Principal "*" the my authenticated user can access the
> bucket
>
> Is this possible?
> Regards
> Daniel
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph OIDC Integration
Hello, The next Octopus release should be there in 3-4 weeks. In Octopus, shadow users aren't created ((for federated oidc users). But we later realised that shadow users are needed to maintain user stats, hence the code for the same is under the process of being added as of now and should be available in Pacific release. We have also done away with the token introspection url in the latest code and have switched over to offline token validation using the IDP's certs since using token introspection url wouldn't scale well for multiple clients. There is a related Ceph Tech Talk here, that you can watch: https://www.youtube.com/watch?v=Lc32meILfNI=410s Thanks, Pritha On Mon, Oct 19, 2020 at 8:30 PM wrote: > Dear Pritha, thanks a lot for your feedback and apologies for missing your > comment about the backporting. Would you have a rough estimate on the next > Octopus release by any chance? > > On another note on the same subject, would you be able to give us some > feedback on how the users will be created in Ceph? (for example when we > used ldap, an ldap user used to be created in Ceph for "mapping", will it > be the same in this case) > > If we have multiple tenants (unique usernames "emails" in KeyCloak) how > will the introspect url's be defined for different tenants? > > Thanks in advance > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph OIDC Integration
Hello again, The issues that you are seeing are because as I mentioned in my previous email, I missed backporting some commits to Octopus (apologies for the same), and I have opened a backport PR (( https://github.com/ceph/ceph/pull/37640) and this should be available in the next Octopus release. Thanks, Pritha On Wed, Oct 14, 2020 at 11:48 PM wrote: > Hi Pritha and thanks again for your reply. Unfortunately we are still > stuck at the AssumeRoleWithWebIdentity API call as shown below: > > 2020-10-14T08:24:26.314+ 7ff6600ff700 1 == starting new request > req=0x7ff6b69496b0 = > 2020-10-14T08:24:26.314+ 7ff6600ff700 2 req 7 0s initializing for > trans_id = tx7-005f86b5ba-1e4f8-ZylaS3 > 2020-10-14T08:24:26.314+ 7ff6600ff700 10 rgw api priority: s3=8 > s3website=7 > 2020-10-14T08:24:26.314+ 7ff6600ff700 10 host=192.168.35.28 > 2020-10-14T08:24:26.314+ 7ff6600ff700 20 subdomain= domain= > in_hosted_domain=0 in_hosted_domain_s3website=0 > 2020-10-14T08:24:26.314+ 7ff6600ff700 20 final domain/bucket > subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 > s->info.domain= s->info.request_uri=/ > 2020-10-14T08:24:26.314+ 7ff6600ff700 20 req 7 0s get_handler > handler=26RGWHandler_REST_Service_S3 > 2020-10-14T08:24:26.314+ 7ff6600ff700 10 > handler=26RGWHandler_REST_Service_S3 > 2020-10-14T08:24:26.314+ 7ff6600ff700 2 req 7 0s getting op 4 > 2020-10-14T08:24:26.314+ 7ff6600ff700 10 Content of POST: > Action=AssumeRoleWithWebIdentity=2011-06-15=arn%3Aaws%3Aiam%3A%3A%3Arole%2FS3Access=KC-Client=3600= > 2020-10-14T08:24:26.315+ 7ff6600ff700 10 req 7 0.00105s > sts:assume_role_web_identity scheduling with dmclock client=0 cost=1 > 2020-10-14T08:24:26.315+ 7ff6600ff700 10 > op=31RGWSTSAssumeRoleWithWebIdentity > 2020-10-14T08:24:26.315+ 7ff6600ff700 2 req 7 0.00105s > sts:assume_role_web_identity verifying requester > 2020-10-14T08:24:26.315+ 7ff6600ff700 20 req 7 0.00105s > sts:assume_role_web_identity rgw::auth::sts::DefaultStrategy: trying > rgw::auth::sts::WebTokenEngine > 2020-10-14T08:24:26.315+ 7ff6600ff700 20 req 7 0.00105s > sts:assume_role_web_identity rgw::auth::sts::WebTokenEngine denied with > reason=-13 > 2020-10-14T08:24:26.315+ 7ff6600ff700 5 req 7 0.00105s > sts:assume_role_web_identity Failed the auth strategy, reason=-13 > 2020-10-14T08:24:26.315+ 7ff6600ff700 10 failed to authorize request > 2020-10-14T08:24:26.315+ 7ff6600ff700 1 op->ERRORHANDLER: err_no=-13 > new_err_no=-13 > 2020-10-14T08:24:26.315+ 7ff6600ff700 2 req 7 0.00105s > sts:assume_role_web_identity op status=0 > 2020-10-14T08:24:26.315+ 7ff6600ff700 2 req 7 0.00105s > sts:assume_role_web_identity http status=403 > 2020-10-14T08:24:26.315+ 7ff6600ff700 1 == req done > req=0x7ff6b69496b0 op status=0 http_status=403 latency=0.00105s == > 2020-10-14T08:24:26.315+ 7ff6600ff700 1 beast: 0x7ff6b69496b0: > 192.168.35.116 - - [2020-10-14T08:24:26.315479+] "POST / HTTP/1.1" 403 > 185 - "Boto3/1.15.16 Python/3.8.5 Linux/5.4.0-48-generic Botocore/1.18.16" - > 2020-10-14T08:24:26.383+ 7ff66690c700 20 failed to read header: end of > stream > > to get the web token we are using the below script and pasting the output > in the AssumeRoleWithWebIdentity API call: > > #!/bin/bash > > KC_REALM=myrealm > KC_CLIENT=rgw-app > KC_CLIENT_SECRET= > KC_SERVER=: > KC_CONTEXT=auth > > # Request Tokens for credentials > KC_RESPONSE=$( \ > curl -k --no-progress-meter -X POST \ > -H "Content-Type: application/x-www-form-urlencoded" \ > -d "scope=openid" \ > -d "grant_type=client_credentials" \ > -d "client_id=$KC_CLIENT" \ > -d "client_secret=$KC_CLIENT_SECRET" \ > "https://$KC_SERVER/$KC_CONTEXT/realms/$KC_REALM/protocol/openid-connect/token; > \ > | jq . > ) > > KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) > echo $KC_ACCESS_TOKEN > > For the above token request to work we had to turn "ON" the option > "Service Accounts Enabled" in the settings tab of the client > > I suspect that the issue is more in relation to the capabilities of the > user when calling the STS engine for the AssumeRoleWithWebIdentity API call. > > If anyone can help... You are more than welcome :-) > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph OIDC Integration
Hello,
rgw sts key should be a key of length 16 since we use AES 128 for
encryption (e.g. rgw sts key = abcdefghijklmnop)
Yes it should be 'sts_client' and not 'client'. The errors in documentation
have been noted and will be corrected.
Also please note that the backport to octopus of the new changes is
underway (https://github.com/ceph/ceph/pull/37640), and this should be
available in the next Octopus release.
Thanks,
Pritha
On Tue, Oct 13, 2020 at 9:22 PM wrote:
> Hi Pritha and thanks for your reply. We are using Ceph Octopus and we have
> switched to Keycloak from dexIdP.
>
> Having said that we have followed the guide from
> https://docs.ceph.com/en/octopus/radosgw/STS/ but we are constantly
> having an issue with the AssumeRoleWithWebIdentity example.
>
> We are using 2 different accounts for role creation and policy creation
> and those 2 parts of the example script are working fine but when we move
> over to the assume_role_with_web_identity part we have forbidden error from
> Ceph.
>
> We have used cephadm to install Ceph which is at:
> # ceph --version
> ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus
> (stable)
>
> We used the following command to add the role capabilities for both users:
> radosgw-admin caps add --uid="TESTER" --caps="roles=*"
> radosgw-admin caps add --uid="TESTER1" --caps="roles=*"
>
> We have set the capabilities for the 2 users mentioned above as shown here:
> buckets (*)
> metadata (*)
> roles (*)
> usage (*)
> user-policy (*)
> users (*)
> zone (*)
>
> ---
>
> Can you please confirm that the key values have actually spaces in them or
> are they missing an underscore?
> [client.radosgw.gateway]
> rgw sts key = {sts key for encrypting the session token}
> rgw s3 auth use sts = true
>
> ---
>
> We are also getting "NameError: name 'client' is not defined" error from
> AssumeRoleWithWebIdentity example in this part shown below. Shouldn't it be
> "sts_client.assume_role_with_web_identity" from
> "client.assume_role_with_web_identity" as it is being defined as sts_client
> in the code above it?
>
> sts_client = boto3.client('sts',
> aws_access_key_id=,
> aws_secret_access_key=,
> endpoint_url=,
> region_name='',
> )
>
> response = client.assume_role_with_web_identity(
> RoleArn=role_response['Role']['Arn'],
> RoleSessionName='Bob',
> DurationSeconds=3600,
> WebIdentityToken=
> )
>
> Can you or anyone give us some pointers to this issue please?
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph OIDC Integration
Hello, If it is possible for the uid that has been used for LDAP users to be the same for OIDC users (which is based off the 'sub' field of the OpenID connect token), then there are no extra migration steps needed. Which version of Ceph are you using? In octopus, offline token validation has been introduced, where an incoming web token is validated using the certificate of the IDP. Uptil Octopus, there were no shadow users for OIDC users, but we have introduced shadow user creation in the 'master' branch, and that is done automatically when an AssumeRoleWithWebIdentity call is made. So the metadata to look at right now would be $$buckets which stores the user stats and make sure that the same uid is being used across both LDAP and OIDC (if that is possible), else there is a radosgw-admin user rename command that will rename the user and update all other metadata. Also, please note that currently AssumeRoleWithWebIdentity has been tested only with Keycloak. The documentation for STS in Octopus is here: https://docs.ceph.com/en/octopus/radosgw/STS/ Thanks, Pritha On Mon, Oct 5, 2020 at 9:56 PM wrote: > Hello, we have integrated Ceph's RGW with LDAP and have authenticated > users using the mail attribute successfully. We would like to shift to SSO > and are evaluating the new OIDC feature in Ceph together with dexIdP with > an LDAP connector as an upstream IdP. > > We are trying to understand the flow of the user authentication and how it > will effect my current LDAP users buckets which are already created in Ceph > as LDAP users. > > Will the Ceph RGW be able to pass the token to be verified to the IdP and > what type of user will then be created in Ceph? Is this the intended way of > OIDC integration? > > Thanks for any assistance > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ACL for user in another teant
Hi Vishwas,
In the following bucket policy:
Policy:{
"Version": "2012-10-17",
"Statement": [
{
"Principal": {"AWS": ["arn:aws:iam::tenant1:user/Tom"]},
"Action": ["s3:ListBucket"],
"Effect": "Allow",
"Resource": "s3://tenant2/jerry-bucket"
}
]
}
'Resource' should follow the AWS ARN format
(arn:aws:s3::tenant2:jerry-bucket)
Also, you won't be able to pass in a tenant name with bucket name using
s3cmd. You can use boto for the same with bucket names of the format
'tenant:bucket' and disable bucket name validation using
s3client.meta.events.unregister('before-parameter-build.s3',
validate_bucket_name, if you plan to use boto3.
Thanks,
Pritha
On Thu, May 14, 2020 at 2:01 PM Vishwas Bm wrote:
> When I tried as below also, similar error is coming:
>
> [root@vishwas-test cluster]# s3cmd --access_key=GY40PHWVK40A2G4XQH2D
> --secret_key=bKq36rs5t1nZEL3MedAtDY3JCfBoOs1DEou0xfOk ls
> s3://tenant2/jerry-bucket
> ERROR: Bucket 'tenant2' does not exist
> ERROR: S3 error: 404 (NoSuchBucket)
>
>
> [root@vishwas-test cluster]# s3cmd --access_key=GY40PHWVK40A2G4XQH2D
> --secret_key=bKq36rs5t1nZEL3MedAtDY3JCfBoOs1DEou0xfOk ls
> s3://tenant2:jerry-bucket
> ERROR: S3 error: 403 (SignatureDoesNotMatch)
>
>
> *Thanks & Regards,*
>
> *Vishwas *
>
>
> On Thu, May 14, 2020 at 1:54 PM Vishwas Bm wrote:
>
>> Hi Pritha,
>>
>> Thanks for the reply. Please find the user list, bucket list and also the
>> command which I have used.
>>
>> [root@vishwas-test cluster]# radosgw-admin user list
>> [
>> "tenant2$Jerry",
>> "tenant1$Tom"
>> ]
>>
>> [root@vishwas-test cluster]# radosgw-admin bucket list
>> [
>> "tenant2/jerry-bucket"
>> ]
>>
>> [root@vishwas-test cluster]# s3cmd info
>> --access_key=HVTKORMH8LLDF76TKQGI
>> --secret_key=9XFcvgMm4yBncA8D9SguEMVSBsUkhuuRLSbyuUPp s3://jerry-bucket
>> s3://jerry-bucket/ (bucket):
>>Location: default
>>Payer: BucketOwner
>>Expiration Rule: none
>>Policy:{
>> "Version": "2012-10-17",
>> "Statement": [
>> {
>> "Principal": {"AWS": ["arn:aws:iam::tenant1:user/Tom"]},
>> "Action": ["s3:ListBucket"],
>> "Effect": "Allow",
>> "Resource": "s3://tenant2/jerry-bucket"
>> }
>> ]
>> }
>>CORS: none
>>ACL: Jerry: FULL_CONTROL
>>
>>
>> When I try to list using Tom access keys, I get below error:
>> [root@vishwas-test cluster]# s3cmd --access_key=GY40PHWVK40A2G4XQH2D
>> --secret_key=bKq36rs5t1nZEL3MedAtDY3JCfBoOs1DEou0xfOk ls s3://jerry-bucket
>>
>> *ERROR: Bucket 'jerry-bucket' does not existERROR: S3 error: 404
>> (NoSuchBucket)*
>>
>>
>> *Thanks & Regards,*
>>
>> *Vishwas *
>>
>>
>> On Thu, May 14, 2020 at 11:54 AM Pritha Srivastava
>> wrote:
>>
>>> Hi Vishwas,
>>>
>>> Bucket policy should let you access buckets in another tenant.
>>> What exact command are you using?
>>>
>>> Thanks,
>>> Pritha
>>>
>>> On Thursday, May 14, 2020, Vishwas Bm wrote:
>>>
>>>> > Hi,
>>>> >
>>>> > I have two users both belong to different tenant.
>>>> >
>>>> > Can I give permission for the user in another tenant to access the
>>>> bucket
>>>> > using setacl or setPolicy command ?
>>>> > I tried the setacl command and setpolicy command, but it was not
>>>> working ?
>>>> > It used to say bucket not found, when the grantee tried to access.
>>>> >
>>>> > Is this supported ?
>>>> >
>>>> > *Thanks & Regards,*
>>>> > *Vishwas *
>>>> >
>>>>
>>>> >
>>>> ___
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>>>
>>>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ACL for user in another teant
Hi Vishwas, Bucket policy should let you access buckets in another tenant. What exact command are you using? Thanks, Pritha On Thursday, May 14, 2020, Vishwas Bm wrote: > > Hi, > > > > I have two users both belong to different tenant. > > > > Can I give permission for the user in another tenant to access the bucket > > using setacl or setPolicy command ? > > I tried the setacl command and setpolicy command, but it was not working > ? > > It used to say bucket not found, when the grantee tried to access. > > > > Is this supported ? > > > > *Thanks & Regards,* > > *Vishwas * > > > > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW STS Support in Nautilus ?
Matching other fields in the token as part of the Condition Statement is
work in progress, but isnt there in nautilus.
Thanks,
Pritha
On Tue, May 12, 2020 at 10:21 PM Wyllys Ingersoll <
wyllys.ingers...@keepertech.com> wrote:
> Does STS support using other fields from the token as part of the
> Condition statement? For example looking for specific "sub" identities or
> matching on custom token fields like lists of roles?
>
>
>
> On Tue, May 12, 2020 at 11:50 AM Matt Benjamin
> wrote:
>
>> yay! thanks Wyllys, Pritha
>>
>> Matt
>>
>> On Tue, May 12, 2020 at 11:38 AM Wyllys Ingersoll
>> wrote:
>> >
>> >
>> > Thanks for the hint, I fixed my keycloak configuration for that
>> application client so the token only includes a single audience value and
>> now it works fine.
>> >
>> > thanks!!
>> >
>> >
>> > On Tue, May 12, 2020 at 11:11 AM Wyllys Ingersoll <
>> wyllys.ingers...@keepertech.com> wrote:
>> >>
>> >> The "aud" field in the introspection result is a list, not a single
>> string.
>> >>
>> >> On Tue, May 12, 2020 at 11:02 AM Pritha Srivastava <
>> prsri...@redhat.com> wrote:
>> >>>
>> >>> app_id must match with the 'aud' field in the token introspection
>> result (In the example the value of 'aud' is 'customer-portal')
>> >>>
>> >>> Thanks,
>> >>> Pritha
>> >>>
>> >>> On Tue, May 12, 2020 at 8:16 PM Wyllys Ingersoll <
>> wyllys.ingers...@keepertech.com> wrote:
>> >>>>
>> >>>>
>> >>>> Running Nautilus 14.2.9 and trying to follow the STS example given
>> here: https://docs.ceph.com/docs/master/radosgw/STS/ to setup a policy
>> for AssumeRoleWithWebIdentity using KeyCloak (8.0.1) as the OIDC provider.
>> I am able to see in the rgw debug logs that the token being passed from the
>> client is passing the introspection check, but it always ends up failing
>> the final authorization to access the requested bucket resource and is
>> rejected with a 403 status "AccessDenied".
>> >>>>
>> >>>> I configured my policy as described in the 2nd example on the STS
>> page above. I suspect the problem is with the "StringEquals" condition
>> statement in the AssumeRolePolicy document (I could be wrong though).
>> >>>>
>> >>>> The example shows using the keycloak URI followed by ":app_id"
>> matching with the name of the keycloak client application
>> ("customer-portal" in the example). My keycloak setup does not have any
>> such field in the introspection result and I can't seem to figure out how
>> to make this all work.
>> >>>>
>> >>>> I cranked up the logging to 20/20 and still did not see any hints as
>> to what part of the policy is causing the access to be denied.
>> >>>>
>> >>>> Any suggestions?
>> >>>>
>> >>>> -Wyllys Ingersoll
>> >>>>
>> >>>> ___
>> >>>> Dev mailing list -- d...@ceph.io
>> >>>> To unsubscribe send an email to dev-le...@ceph.io
>> >
>> > ___
>> > Dev mailing list -- d...@ceph.io
>> > To unsubscribe send an email to dev-le...@ceph.io
>>
>>
>>
>> --
>>
>> Matt Benjamin
>> Red Hat, Inc.
>> 315 West Huron Street, Suite 140A
>> Ann Arbor, Michigan 48103
>>
>> http://www.redhat.com/en/technologies/storage
>>
>> tel. 734-821-5101
>> fax. 734-769-8938
>> cel. 734-216-5309
>>
>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW STS Support in Nautilus ?
app_id must match with the 'aud' field in the token introspection result
(In the example the value of 'aud' is 'customer-portal')
Thanks,
Pritha
On Tue, May 12, 2020 at 8:16 PM Wyllys Ingersoll <
wyllys.ingers...@keepertech.com> wrote:
>
> Running Nautilus 14.2.9 and trying to follow the STS example given here:
> https://docs.ceph.com/docs/master/radosgw/STS/ to setup a policy
> for AssumeRoleWithWebIdentity using KeyCloak (8.0.1) as the OIDC provider.
> I am able to see in the rgw debug logs that the token being passed from the
> client is passing the introspection check, but it always ends up failing
> the final authorization to access the requested bucket resource and is
> rejected with a 403 status "AccessDenied".
>
> I configured my policy as described in the 2nd example on the STS page
> above. I suspect the problem is with the "StringEquals" condition statement
> in the AssumeRolePolicy document (I could be wrong though).
>
> The example shows using the keycloak URI followed by ":app_id" matching
> with the name of the keycloak client application ("customer-portal" in the
> example). My keycloak setup does not have any such field in the
> introspection result and I can't seem to figure out how to make this all
> work.
>
> I cranked up the logging to 20/20 and still did not see any hints as to
> what part of the policy is causing the access to be denied.
>
> Any suggestions?
>
> -Wyllys Ingersoll
>
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: radosgw garbage collection error
lis/c=5394/5394 les/c/f=5395/5395/0 sis=5394 pruub=12.023579210s) > [15,21,26] r=0 lpr=5394 crt=5395'481460 lcod 5395'481461 mlcod 5395'481461 > active+clean] do_osd_op 5:9a6dd8c0:gc::gc.0:head [sync_read 4096~1024] > 2020-05-06T08:06:33.925+0200 7f73b554a700 10 osd.15 pg_epoch: 5395 pg[5.9( > v 5395'481462 (5387'478000,5395'481462] local-lis/les=5394/5395 n=48 > ec=67/67 lis/c=5394/5394 les/c/f=5395/5395/0 sis=5394 pruub=12.023579210s) > [15,21,26] r=0 lpr=5394 crt=5395'481460 lcod 5395'481461 mlcod 5395'481461 > active+clean] do_osd_op sync_read 4096~1024 > 2020-05-06T08:06:33.925+0200 7f73b554a700 10 osd.15 pg_epoch: 5395 pg[5.9( > v 5395'481462 (5387'478000,5395'481462] local-lis/les=5394/5395 n=48 > ec=67/67 lis/c=5394/5394 les/c/f=5395/5395/0 sis=5394 pruub=12.023579210s) > [15,21,26] r=0 lpr=5394 crt=5395'481460 lcod 5395'481461 mlcod 5395'481461 > active+clean] read got 1024 / 1024 bytes from obj 5:9a6dd8c0:gc::gc.0:head > 2020-05-06T08:06:33.925+0200 7f73b554a700 10 osd.15 pg_epoch: 5395 pg[5.9( > v 5395'481462 (5387'478000,5395'481462] local-lis/les=5394/5395 n=48 > ec=67/67 lis/c=5394/5394 les/c/f=5395/5395/0 sis=5394 pruub=12.023579210s) > [15,21,26] r=0 lpr=5394 crt=5395'481460 lcod 5395'481461 mlcod 5395'481461 > active+clean] method called response length=0 > 2020-05-06T08:06:33.925+0200 7f73b554a700 10 osd.15 pg_epoch: 5395 pg[5.9( > v 5395'481462 (5387'478000,5395'481462] local-lis/les=5394/5395 n=48 > ec=67/67 lis/c=5394/5394 les/c/f=5395/5395/0 sis=5394 pruub=12.023579210s) > [15,21,26] r=0 lpr=5394 crt=5395'481460 lcod 5395'481461 mlcod 5395'481461 > active+clean] do_osd_ops error: (22) Invalid argument > 2020-05-06T08:06:33.925+0200 7f73b554a700 10 osd.15 5395 dequeue_op > 0x555f68a5a580 finish > > Can you draw any conclusions from these logs? > > Regards, > James > > > On 6 May 2020, at 05:03, Pritha Srivastava wrote: > > > > Hi James, > > > > Does radosgw-admin gc list --include-all, give the same error? If yes, > can > > you please open a tracker issue and share rgw and osd logs? > > > > Thanks, > > Pritha > > > > On Wed, May 6, 2020 at 12:22 AM James, GleSYS > > wrote: > > > >> Hi, > >> > >> We’ve recently installed a new Ceph cluster running Octopus 15.2.1, and > >> we’re using RGW with an erasure coded backed pool. > >> > >> I started to get a suspicion that deleted objects were not getting > cleaned > >> up properly, and I wanted to verify this by checking the garbage > collector. > >> > >> That’s when I discovered that when I run “radosgw-admin gc list”, I get > >> the following error: > >> "ERROR: failed to list objs: (22) Invalid argument” > >> > >> When running the command with the debug-rgw=20 flag, I see a bit more > >> information: > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=0 > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=1 > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=2 > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=3 > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=4 > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=5 > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=6 > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=7 > >> 2020-05-05T18:39:19.455+ 7f3312d82080 2 all 8 watchers are set, > >> enabling cache > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 check_secure_mon_conn(): > auth > >> registy supported: methods=[2,1] modes=[2,1] > >> 2020-05-05T18:39:19.455+ 7f3312d82080 20 check_secure_mon_conn(): > >> method 1 is insecure > >> 2020-05-05T18:39:19.455+ 7f32d4fd9700 2 > >> RGWDataChangesLog::ChangesRenewThread: start > >> 2020-05-05T18:39:19.519+ 7f3246ffd700 20 reqs_thread_entry: start > >> 2020-05-05T18:39:19.519+ 7f3312d82080 20 init_complete bucket index > >> max shards: 11 > >> 2020-05-05T18:39:19.519+ 7f3244ff9700 20 reqs_thread_entry: start > >> 2020-05-05T18:39:19.519+ 7f323affd700 20 reqs_thread_entry: start > >> ERROR: failed to list objs: (22) Invalid argument > >> 2020-05-05T18:39:19.523+ 7f32d4fd9700 2 > >> RGWDataChangesLog::ChangesRenewThread: start > >> 2020-05-05T18:39:19.523+ 7f3312d82080 20 remove_watcher() i=0 > >> 2020-05-05T18:39:19.523+ 7f3312d82080 2 removed watcher, disabling > >> cache > >> 2020-05-05T18:39:19.523+ 7f3312d82080 20 remove_watcher() i=1 > >> 2020-05-05T18:39:19.523+ 7f3312d82080 20 remove_watcher() i=2 > >> 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() i=3 > >> 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() i=4 > >> 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() i=5 > >> 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() i=6 > >> 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() I=7 > >> > >> I find very little information regarding this error, so I wondered if > >> someone here could help me troubleshoot the issue? > >> > >> Thanks, > >> James. > >> ___ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-le...@ceph.io > >> > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: radosgw garbage collection error
Hi James, Does radosgw-admin gc list --include-all, give the same error? If yes, can you please open a tracker issue and share rgw and osd logs? Thanks, Pritha On Wed, May 6, 2020 at 12:22 AM James, GleSYS wrote: > Hi, > > We’ve recently installed a new Ceph cluster running Octopus 15.2.1, and > we’re using RGW with an erasure coded backed pool. > > I started to get a suspicion that deleted objects were not getting cleaned > up properly, and I wanted to verify this by checking the garbage collector. > > That’s when I discovered that when I run “radosgw-admin gc list”, I get > the following error: > "ERROR: failed to list objs: (22) Invalid argument” > > When running the command with the debug-rgw=20 flag, I see a bit more > information: > 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=0 > 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=1 > 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=2 > 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=3 > 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=4 > 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=5 > 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=6 > 2020-05-05T18:39:19.455+ 7f3312d82080 20 add_watcher() i=7 > 2020-05-05T18:39:19.455+ 7f3312d82080 2 all 8 watchers are set, > enabling cache > 2020-05-05T18:39:19.455+ 7f3312d82080 20 check_secure_mon_conn(): auth > registy supported: methods=[2,1] modes=[2,1] > 2020-05-05T18:39:19.455+ 7f3312d82080 20 check_secure_mon_conn(): > method 1 is insecure > 2020-05-05T18:39:19.455+ 7f32d4fd9700 2 > RGWDataChangesLog::ChangesRenewThread: start > 2020-05-05T18:39:19.519+ 7f3246ffd700 20 reqs_thread_entry: start > 2020-05-05T18:39:19.519+ 7f3312d82080 20 init_complete bucket index > max shards: 11 > 2020-05-05T18:39:19.519+ 7f3244ff9700 20 reqs_thread_entry: start > 2020-05-05T18:39:19.519+ 7f323affd700 20 reqs_thread_entry: start > ERROR: failed to list objs: (22) Invalid argument > 2020-05-05T18:39:19.523+ 7f32d4fd9700 2 > RGWDataChangesLog::ChangesRenewThread: start > 2020-05-05T18:39:19.523+ 7f3312d82080 20 remove_watcher() i=0 > 2020-05-05T18:39:19.523+ 7f3312d82080 2 removed watcher, disabling > cache > 2020-05-05T18:39:19.523+ 7f3312d82080 20 remove_watcher() i=1 > 2020-05-05T18:39:19.523+ 7f3312d82080 20 remove_watcher() i=2 > 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() i=3 > 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() i=4 > 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() i=5 > 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() i=6 > 2020-05-05T18:39:19.527+ 7f3312d82080 20 remove_watcher() I=7 > > I find very little information regarding this error, so I wondered if > someone here could help me troubleshoot the issue? > > Thanks, > James. > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Deletion of objects and garbage collector
Yes, those head objects are deleted inline and only the tail objects are
added to gc to be deleted once they expire.
Thanks,
Pritha
On Wed, Apr 15, 2020 at 4:25 PM Priya Sehgal wrote:
> Hi Pritha,
> I tried running radosgw-admin gc list --include-all and it shows the
> shadow objects.
>
> One question - the above list only has list of shadow and multi-part
> objects. So, objects that are less than 4MB - will they be not queued -
> since they are part of the HEAD and the HEAD gets deleted when we issue
> s3cmd del command?
>
> Thanks,
> Priya
>
> On Wed, Apr 15, 2020 at 3:33 PM Pritha Srivastava
> wrote:
>
>> Hi Priya,
>>
>> Did you try to list the objects using radosgw-admin gc list immediately
>> after deleting them?
>>
>> Try using the command: radosgw-admin gc list --include-all, this lists
>> all the expired and non expired entries.
>> radosgw-admin gc list, only lists entries that have expired. Objects that
>> have been gc'ed expire after a duration that is determined by
>> rgw_gc_obj_min_wait the default value of which is 2 hrs.
>>
>> Thanks,
>> Pritha
>>
>> On Wed, Apr 15, 2020 at 3:25 PM Priya Sehgal
>> wrote:
>>
>>> Hi,
>>> I read at many places that when an object is deleted from ceph it is
>>> queued
>>> for deletion with the garbage collector (GC). However, when I delete
>>> objects of various sizes (both less than 4MB and large sized greater than
>>> 4MB i.e. MPU) I always find gc list as empty.
>>> I tried disabling GC also to make sure that it does not run and delete
>>> it,
>>> yet I did not find any object when I run the command. Here's the output:
>>> *radosgw-admin gc list*
>>>
>>> [
>>> {
>>> "tag": "2~KsaJkJwSGeuVzeKpkHAe_5vJ3JqZmKc",
>>> "time": "2020-04-10 18:25:23.0.769037s",
>>> "objs": []
>>> }
>>> ]
>>>
>>> NOTE: I tried to delete an object as on 14th April and 15th April.
>>>
>>> I issued "s3cmd del" command.
>>> Ceph Version I am using in Luminous.
>>>
>>> Please let me know how object deletes work.
>>>
>>> --
>>> Regards,
>>> Priya
>>> ___
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>>
>>>
>
> --
> Regards,
> Priya
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Deletion of objects and garbage collector
Hi Priya,
Did you try to list the objects using radosgw-admin gc list immediately
after deleting them?
Try using the command: radosgw-admin gc list --include-all, this lists all
the expired and non expired entries.
radosgw-admin gc list, only lists entries that have expired. Objects that
have been gc'ed expire after a duration that is determined by
rgw_gc_obj_min_wait the default value of which is 2 hrs.
Thanks,
Pritha
On Wed, Apr 15, 2020 at 3:25 PM Priya Sehgal wrote:
> Hi,
> I read at many places that when an object is deleted from ceph it is queued
> for deletion with the garbage collector (GC). However, when I delete
> objects of various sizes (both less than 4MB and large sized greater than
> 4MB i.e. MPU) I always find gc list as empty.
> I tried disabling GC also to make sure that it does not run and delete it,
> yet I did not find any object when I run the command. Here's the output:
> *radosgw-admin gc list*
>
> [
> {
> "tag": "2~KsaJkJwSGeuVzeKpkHAe_5vJ3JqZmKc",
> "time": "2020-04-10 18:25:23.0.769037s",
> "objs": []
> }
> ]
>
> NOTE: I tried to delete an object as on 14th April and 15th April.
>
> I issued "s3cmd del" command.
> Ceph Version I am using in Luminous.
>
> Please let me know how object deletes work.
>
> --
> Regards,
> Priya
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph (version 14.2.7) RGW STS AccessDenied
Hi,
Is the ACCESSKEY_STRING and SECRETKEY_STRING of user trying to assume role
'S3Access' same as that of user 'sr'? (that is the user specified in the
assume_role_policy_document)
Thanks,
Pritha
On Sun, Mar 8, 2020 at 7:54 PM 曹 海旺 wrote:
> Hi ,I want to use the sts to get a temporary credentials,but I got 403 .
>
> The user info :
> {
> "user_id": "sr",
> "display_name": "shiran",
> "email": "",
> "suspended": 0,
> "max_buckets": 1000,
> "subusers": [],
> "keys": [
> {
> "user": "sr",
> "access_key": "FBGYU7CUBUACP2OYBE76",
> "secret_key": "RoovZBBNdlOvVS4LzuuuCsZ1QPA6W81mbNXvonYD"
> }
> ],
> "swift_keys": [],
> "caps": [
> {
> "type": "roles",
> "perm": "*"
> },
> {
> "type": "user-policy",
> "perm": "*"
> }
> ],
> "op_mask": "read, write, delete",
> "default_placement": "",
> "default_storage_class": "",
> "placement_tags": [],
> "bucket_quota": {
> "enabled": false,
> "check_on_raw": false,
> "max_size": -1,
> "max_size_kb": 0,
> "max_objects": -1
> },
> "user_quota": {
> "enabled": false,
> "check_on_raw": false,
> "max_size": -1,
> "max_size_kb": 0,
> "max_objects": -1
> },
> "temp_url_keys": [],
> "type": "rgw",
> "mfa_ids": []
> }
>
>
> and the role info :
>
> {
> "id": "c983c0df-a49c-4e36-957d-7d709f65e292",
> "name": "S3Access",
> "path": "/application_abc/component_xyz/",
> "arn": "arn:aws:iam:::role/application_abc/component_xyz/S3Access",
> "create_date": "2020-03-04T08:19:41.24Z",
> "max_session_duration": 3600,
> "assume_role_policy_document":
> "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/sr\"]},\"Action\":[\"sts:AssumeRole\"]}]}"
> },
>
> I use AWA java sdk to AssumeRole ,code like:
>
>
>
> public static final String ACCESSKEY_STRING = “xx";
> public static final String BUCKETNAME_STRING = "osstest";
> public static final String SECRETKEY_STRING = “x";
>
> public static final String ENDPOINT_STRING = "http://192.168.3.250”;//
> haproxy
>
>
> String clientRegion = ENDPOINT_STRING;
> String roleARN =
> "arn:aws:iam:::role/application_abc/component_xyz/S3Access";
> String roleSessionName = "bob";
> String bucketName = ENDPOINT_STRING;
>
> AWSCredentials credentials = new BasicAWSCredentials(ACCESSKEY_STRING,
> SECRETKEY_STRING);
> ClientConfiguration clientConfig = new ClientConfiguration();
>
> clientConfig.setProtocol(Protocol.HTTP);
> clientConfig.setSignerOverride("AWS3SignerType");
>
> AWSSecurityTokenService sts_client =
> AWSSecurityTokenServiceClientBuilder.standard()
> .withEndpointConfiguration(new
> AwsClientBuilder.EndpointConfiguration(ENDPOINT_STRING, ""))
> .withCredentials(new
> AWSStaticCredentialsProvider(credentials)).withClientConfiguration(clientConfig)
> .build();
> GetSessionTokenRequest session_token_request = new
> GetSessionTokenRequest();
> AssumeRoleRequest roleRequest = new AssumeRoleRequest()
> .withRoleArn(roleARN)
> .withRoleSessionName(roleSessionName).withTokenCode("sts");
> session_token_request.setDurationSeconds(3600);
> AssumeRoleResult roleResponse = sts_client.assumeRole(roleRequest);
> Credentials sessionCredentials = roleResponse.getCredentials();
>
>
> I got the response:
>
> encoding="UTF-8"?>AccessDeniedtx00016-005e64fa9e-b5b47-defaultb5b47-default-default"
>
> The debug info in the log file like this :
>
>
> 1 == starting new request req=0x5592abefc910 =
> 2020-03-08 22:01:02.984 7f2548f95700 2 req 22 0.000s initializing for
> trans_id = tx00016-005e64fa9e-b5b47-default
> 2020-03-08 22:01:02.984 7f2548f95700 10 rgw api priority: s3=7 s3website=6
> 2020-03-08 22:01:02.984 7f2548f95700 10 host=rgw1
> 2020-03-08 22:01:02.984 7f2548f95700 20 subdomain= domain=
> in_hosted_domain=0 in_hosted_domain_s3website=0
> 2020-03-08 22:01:02.984 7f2548f95700 20 final domain/bucket subdomain=
> domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain=
> s->info.request_uri=/
> 2020-03-08 22:01:02.984 7f2548f95700 10 meta>> HTTP_X_AMZ_DATE
> 2020-03-08 22:01:02.984 7f2548f95700 10 meta>> HTTP_X_AMZN_AUTHORIZATION
> 2020-03-08 22:01:02.984 7f2548f95700 10 x>> x-amz-date:Sun, 08 Mar 2020
> 14:01:02 GMT
> 2020-03-08 22:01:02.984 7f2548f95700 10 x>> x-amzn-authorization:AWS3
> AWSAccessKeyId=FBGYU7CUBUACP2OYBE76,Algorithm=HmacSHA256,SignedHeaders=Host;X-Amz-Date,Signature=D7p74AFsJn4J823e6Up4Zo2eRV/n5PFmyQ2/w+kOaqY=
> 2020-03-08 22:01:02.984 7f2548f95700 20 get_handler
> handler=26RGWHandler_REST_Service_S3
> 2020-03-08 22:01:02.984 7f2548f95700 10
> handler=26RGWHandler_REST_Service_S3
> 2020-03-08 22:01:02.984 7f2548f95700 2 req 22
[ceph-users] Re: s3curl putuserpolicy get 405
Hi,
Did you add admin caps to the user trying to attach User Policy to User1?
like
radosgw-admin caps add --uid="TESTER" --caps="user-policy=*".
TESTER is the user trying to attach user policy to TESTER1.
Thanks,
Pritha
On Wed, Dec 25, 2019 at 2:00 PM 黄明友 wrote:
> hi,all
>
>
> I follow the https://docs.ceph.com/docs/master/radosgw/STSLite/ ; run
>
> s3curl.pl --debug --id admin -- -s -v -X POST
> "http://localhost:8000/?Action=PutUserPolicy=Policy1=TESTER1=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Deny\",\"Action\":\"s3:*\",\"Resource\":\[\"*\"\],\"Condition\":\{\"BoolIfExists\":\{\"sts:authentication\":\"false\"\}\}\},\{\"Effect\":\"Allow\",\"Action\":\"sts:GetSessionToken\",\"Resource\":\"*\",\"Condition\":\{\"BoolIfExists\":\{\"sts:authentication\":\"false\"\}\}\}\]\}=2010-05-08;
>
>
> get error like this,
>
> encoding="UTF-8"?>MethodNotAllowedtx01273-005e031dca-e11e-wx6e11e-wx6-wx
>
> the ceph verion is
>
>
> ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus
> (stable)
>
>
>
> *黄明友 *
> IT基础架构部经理
> V.Photos 云摄影
> 移动电话: +86 13540630430
> 客服电话：400 - 806 - 5775
> 电子邮件: hmy@v.photos
> 官方网址: www.v.photos
> 上海 黄浦区中山东二路88号外滩SOHO3Q F栋 2层
> 北京 朝阳区光华路9号光华路SOHO二期南二门SOHO3Q 1层
> 广州 天河区林和中路136号天誉花园二期3Wcoffice 天誉青创社区
> 深圳 南山区蛇口网谷科技大厦二期A座102网谷双创街 1层
> 成都 成华区建设路世贸广场 7层
>
>
>
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
