i've opened https://tracker.ceph.com/issues/63485 to allow admin/system users to override policy parsing errors like this. i'm not sure yet where this parsing regression was introduced. in reef, https://github.com/ceph/ceph/pull/49395 added better error messages here, along with a rgw_policy_reject_invalid_principals option to be strict about principal names
to remove a bucket policy that fails to parse with "Error reading IAM Policy", you can follow these steps: 1. find the bucket's instance id using the 'bucket stats' command $ radosgw-admin bucket stats --bucket {bucketname} | grep id 2. use the rados tool to remove the bucket policy attribute (user.rgw.iam-policy) from the bucket instance metadata object $ rados -p default.rgw.meta -N root rmxattr .bucket.meta.{bucketname}:{bucketid} user.rgw.iam-policy 3. radosgws may be caching the existing bucket metadata and xattrs, so you'd either need to restart them or clear their metadata caches $ ceph daemon client.rgw.xyz cache zap On Wed, Nov 8, 2023 at 9:06 AM Jayanth Reddy <jayanthreddy5...@gmail.com> wrote: > > Hello Wesley, > Thank you for the response. I tried the same but ended up with 403. > > Regards, > Jayanth > > On Wed, Nov 8, 2023 at 7:34 PM Wesley Dillingham <w...@wesdillingham.com> > wrote: >> >> Jaynath: >> >> Just to be clear with the "--admin" user's key's you have attempted to >> delete the bucket policy using the following method: >> https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket-policy.html >> >> This is what worked for me (on a 16.2.14 cluster). I didn't attempt to >> interact with the affected bucket in any way other than "aws s3api >> delete-bucket-policy" >> >> Respectfully, >> >> Wes Dillingham >> w...@wesdillingham.com >> LinkedIn >> >> >> On Wed, Nov 8, 2023 at 8:30 AM Jayanth Reddy <jayanthreddy5...@gmail.com> >> wrote: >>> >>> Hello Casey, >>> >>> We're totally stuck at this point and none of the options seem to work. >>> Please let us know if there is something in metadata or index to remove >>> those applied bucket policies. We downgraded to v17.2.6 and encountering >>> the same. >>> >>> Regards, >>> Jayanth >>> >>> On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <jayanthreddy5...@gmail.com> >>> wrote: >>>> >>>> Hello Casey, >>>> >>>> And on further inspection, we identified that there were bucket policies >>>> set from the initial days; we were in v16.2.12. >>>> We upgraded the cluster to v17.2.7 two days ago and it seems obvious that >>>> the IAM error logs are generated the next minute rgw daemon upgraded from >>>> v16.2.12 to v17.2.7. Looks like there is some issue with parsing. >>>> >>>> I'm thinking to downgrade back to v17.2.6 and earlier, please let me know >>>> if this is a good option for now. >>>> >>>> Thanks, >>>> Jayanth >>>> ________________________________ >>>> From: Jayanth Reddy <jayanthreddy5...@gmail.com> >>>> Sent: Tuesday, November 7, 2023 11:59:38 PM >>>> To: Casey Bodley <cbod...@redhat.com> >>>> Cc: Wesley Dillingham <w...@wesdillingham.com>; ceph-users >>>> <ceph-users@ceph.io>; Adam Emerson <aemer...@redhat.com> >>>> Subject: Re: [ceph-users] Re: owner locked out of bucket via bucket policy >>>> >>>> Hello Casey, >>>> >>>> Thank you for the quick response. I see >>>> `rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please >>>> let me know. >>>> >>>> Regards >>>> Jayanth >>>> >>>> On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbod...@redhat.com> wrote: >>>> >>>> On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy >>>> <jayanthreddy5...@gmail.com> wrote: >>>> > >>>> > Hello Wesley and Casey, >>>> > >>>> > We've ended up with the same issue and here it appears that even the >>>> > user with "--admin" isn't able to do anything. We're now unable to >>>> > figure out if it is due to bucket policies, ACLs or IAM of some sort. >>>> > I'm seeing these IAM errors in the logs >>>> > >>>> > ``` >>>> > >>>> > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851 >>>> > 0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due >>>> > to Handler error. >>>> > >>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 >>>> > 0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing >>>> > due to Handler error. >>>> >>>> it's failing to parse the bucket policy document, but the error >>>> message doesn't say what's wrong with it >>>> >>>> disabling rgw_policy_reject_invalid_principals might help if it's >>>> failing on the Principal >>>> >>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 >>>> > 0.000000000s s3:list_bucket init_permissions on >>>> > :window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, >>>> > ret=-13 >>>> > Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req 13293029267332025583 >>>> > 0.000000000s op->ERRORHANDLER: err_no=-13 new_err_no=-13 >>>> > >>>> > ``` >>>> > >>>> > Please help what's wrong here. We're in Ceph v17.2.7. >>>> > >>>> > Regards, >>>> > Jayanth >>>> > >>>> > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham >>>> > <w...@wesdillingham.com> wrote: >>>> >> >>>> >> Thank you, this has worked to remove the policy. >>>> >> >>>> >> Respectfully, >>>> >> >>>> >> *Wes Dillingham* >>>> >> w...@wesdillingham.com >>>> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham> >>>> >> >>>> >> >>>> >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbod...@redhat.com> wrote: >>>> >> >>>> >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham >>>> >> > <w...@wesdillingham.com> >>>> >> > wrote: >>>> >> > > >>>> >> > > Thank you, I am not sure (inherited cluster). I presume such an >>>> >> > > admin >>>> >> > user created after-the-fact would work? >>>> >> > >>>> >> > yes >>>> >> > >>>> >> > > Is there a good way to discover an admin user other than iterate >>>> >> > > over >>>> >> > all users and retrieve user information? (I presume radosgw-admin >>>> >> > user info >>>> >> > --uid=<user>" would illustrate such administrative access? >>>> >> > >>>> >> > not sure there's an easy way to search existing users, but you could >>>> >> > create a temporary admin user for this repair >>>> >> > >>>> >> > > >>>> >> > > Respectfully, >>>> >> > > >>>> >> > > Wes Dillingham >>>> >> > > w...@wesdillingham.com >>>> >> > > LinkedIn >>>> >> > > >>>> >> > > >>>> >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbod...@redhat.com> >>>> >> > > wrote: >>>> >> > >> >>>> >> > >> if you have an administrative user (created with --admin), you >>>> >> > >> should >>>> >> > >> be able to use its credentials with awscli to delete or overwrite >>>> >> > >> this >>>> >> > >> bucket policy >>>> >> > >> >>>> >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham < >>>> >> > w...@wesdillingham.com> wrote: >>>> >> > >> > >>>> >> > >> > I have a bucket which got injected with bucket policy which >>>> >> > >> > locks the >>>> >> > >> > bucket even to the bucket owner. The bucket now cannot be >>>> >> > >> > accessed >>>> >> > (even >>>> >> > >> > get its info or delete bucket policy does not work) I have >>>> >> > >> > looked in >>>> >> > the >>>> >> > >> > radosgw-admin command for a way to delete a bucket policy but do >>>> >> > >> > not >>>> >> > see >>>> >> > >> > anything. I presume I will need to somehow remove the bucket >>>> >> > >> > policy >>>> >> > from >>>> >> > >> > however it is stored in the bucket metadata / omap etc. If >>>> >> > >> > anyone can >>>> >> > point >>>> >> > >> > me in the right direction on that I would appreciate it. Thanks >>>> >> > >> > >>>> >> > >> > Respectfully, >>>> >> > >> > >>>> >> > >> > *Wes Dillingham* >>>> >> > >> > w...@wesdillingham.com >>>> >> > >> > LinkedIn <http://www.linkedin.com/in/wesleydillingham> >>>> >> > >> > _______________________________________________ >>>> >> > >> > ceph-users mailing list -- ceph-users@ceph.io >>>> >> > >> > To unsubscribe send an email to ceph-users-le...@ceph.io >>>> >> > >> > >>>> >> > >> >>>> >> > >>>> >> > >>>> >> _______________________________________________ >>>> >> ceph-users mailing list -- ceph-users@ceph.io >>>> >> To unsubscribe send an email to ceph-users-le...@ceph.io >>>> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io