i've opened https://tracker.ceph.com/issues/63485 to allow
admin/system users to override policy parsing errors like this. i'm
not sure yet where this parsing regression was introduced. in reef,
https://github.com/ceph/ceph/pull/49395 added better error messages
here, along with a rgw_policy_reject_invalid_principals option to be
strict about principal names
to remove a bucket policy that fails to parse with "Error reading IAM
Policy", you can follow these steps:
1. find the bucket's instance id using the 'bucket stats' command
$ radosgw-admin bucket stats --bucket {bucketname} | grep id
2. use the rados tool to remove the bucket policy attribute
(user.rgw.iam-policy) from the bucket instance metadata object
$ rados -p default.rgw.meta -N root rmxattr
.bucket.meta.{bucketname}:{bucketid} user.rgw.iam-policy
3. radosgws may be caching the existing bucket metadata and xattrs, so
you'd either need to restart them or clear their metadata caches
$ ceph daemon client.rgw.xyz cache zap
On Wed, Nov 8, 2023 at 9:06 AM Jayanth Reddy <jayanthreddy5...@gmail.com> wrote:
>
> Hello Wesley,
> Thank you for the response. I tried the same but ended up with 403.
>
> Regards,
> Jayanth
>
> On Wed, Nov 8, 2023 at 7:34 PM Wesley Dillingham <w...@wesdillingham.com>
> wrote:
>>
>> Jaynath:
>>
>> Just to be clear with the "--admin" user's key's you have attempted to
>> delete the bucket policy using the following method:
>> https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket-policy.html
>>
>> This is what worked for me (on a 16.2.14 cluster). I didn't attempt to
>> interact with the affected bucket in any way other than "aws s3api
>> delete-bucket-policy"
>>
>> Respectfully,
>>
>> Wes Dillingham
>> w...@wesdillingham.com
>> LinkedIn
>>
>>
>> On Wed, Nov 8, 2023 at 8:30 AM Jayanth Reddy <jayanthreddy5...@gmail.com>
>> wrote:
>>>
>>> Hello Casey,
>>>
>>> We're totally stuck at this point and none of the options seem to work.
>>> Please let us know if there is something in metadata or index to remove
>>> those applied bucket policies. We downgraded to v17.2.6 and encountering
>>> the same.
>>>
>>> Regards,
>>> Jayanth
>>>
>>> On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <jayanthreddy5...@gmail.com>
>>> wrote:
>>>>
>>>> Hello Casey,
>>>>
>>>> And on further inspection, we identified that there were bucket policies
>>>> set from the initial days; we were in v16.2.12.
>>>> We upgraded the cluster to v17.2.7 two days ago and it seems obvious that
>>>> the IAM error logs are generated the next minute rgw daemon upgraded from
>>>> v16.2.12 to v17.2.7. Looks like there is some issue with parsing.
>>>>
>>>> I'm thinking to downgrade back to v17.2.6 and earlier, please let me know
>>>> if this is a good option for now.
>>>>
>>>> Thanks,
>>>> Jayanth
>>>> ________________________________
>>>> From: Jayanth Reddy <jayanthreddy5...@gmail.com>
>>>> Sent: Tuesday, November 7, 2023 11:59:38 PM
>>>> To: Casey Bodley <cbod...@redhat.com>
>>>> Cc: Wesley Dillingham <w...@wesdillingham.com>; ceph-users
>>>> <ceph-users@ceph.io>; Adam Emerson <aemer...@redhat.com>
>>>> Subject: Re: [ceph-users] Re: owner locked out of bucket via bucket policy
>>>>
>>>> Hello Casey,
>>>>
>>>> Thank you for the quick response. I see
>>>> `rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please
>>>> let me know.
>>>>
>>>> Regards
>>>> Jayanth
>>>>
>>>> On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbod...@redhat.com> wrote:
>>>>
>>>> On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy
>>>> <jayanthreddy5...@gmail.com> wrote:
>>>> >
>>>> > Hello Wesley and Casey,
>>>> >
>>>> > We've ended up with the same issue and here it appears that even the
>>>> > user with "--admin" isn't able to do anything. We're now unable to
>>>> > figure out if it is due to bucket policies, ACLs or IAM of some sort.
>>>> > I'm seeing these IAM errors in the logs
>>>> >
>>>> > ```
>>>> >
>>>> > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851
>>>> > 0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due
>>>> > to Handler error.
>>>> >
>>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
>>>> > 0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing
>>>> > due to Handler error.
>>>>
>>>> it's failing to parse the bucket policy document, but the error
>>>> message doesn't say what's wrong with it
>>>>
>>>> disabling rgw_policy_reject_invalid_principals might help if it's
>>>> failing on the Principal
>>>>
>>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583
>>>> > 0.000000000s s3:list_bucket init_permissions on
>>>> > :window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed,
>>>> > ret=-13
>>>> > Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req 13293029267332025583
>>>> > 0.000000000s op->ERRORHANDLER: err_no=-13 new_err_no=-13
>>>> >
>>>> > ```
>>>> >
>>>> > Please help what's wrong here. We're in Ceph v17.2.7.
>>>> >
>>>> > Regards,
>>>> > Jayanth
>>>> >
>>>> > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham
>>>> > <w...@wesdillingham.com> wrote:
>>>> >>
>>>> >> Thank you, this has worked to remove the policy.
>>>> >>
>>>> >> Respectfully,
>>>> >>
>>>> >> *Wes Dillingham*
>>>> >> w...@wesdillingham.com
>>>> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>>>> >>
>>>> >>
>>>> >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbod...@redhat.com> wrote:
>>>> >>
>>>> >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham
>>>> >> > <w...@wesdillingham.com>
>>>> >> > wrote:
>>>> >> > >
>>>> >> > > Thank you, I am not sure (inherited cluster). I presume such an
>>>> >> > > admin
>>>> >> > user created after-the-fact would work?
>>>> >> >
>>>> >> > yes
>>>> >> >
>>>> >> > > Is there a good way to discover an admin user other than iterate
>>>> >> > > over
>>>> >> > all users and retrieve user information? (I presume radosgw-admin
>>>> >> > user info
>>>> >> > --uid=<user>" would illustrate such administrative access?
>>>> >> >
>>>> >> > not sure there's an easy way to search existing users, but you could
>>>> >> > create a temporary admin user for this repair
>>>> >> >
>>>> >> > >
>>>> >> > > Respectfully,
>>>> >> > >
>>>> >> > > Wes Dillingham
>>>> >> > > w...@wesdillingham.com
>>>> >> > > LinkedIn
>>>> >> > >
>>>> >> > >
>>>> >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbod...@redhat.com>
>>>> >> > > wrote:
>>>> >> > >>
>>>> >> > >> if you have an administrative user (created with --admin), you
>>>> >> > >> should
>>>> >> > >> be able to use its credentials with awscli to delete or overwrite
>>>> >> > >> this
>>>> >> > >> bucket policy
>>>> >> > >>
>>>> >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham <
>>>> >> > w...@wesdillingham.com> wrote:
>>>> >> > >> >
>>>> >> > >> > I have a bucket which got injected with bucket policy which
>>>> >> > >> > locks the
>>>> >> > >> > bucket even to the bucket owner. The bucket now cannot be
>>>> >> > >> > accessed
>>>> >> > (even
>>>> >> > >> > get its info or delete bucket policy does not work) I have
>>>> >> > >> > looked in
>>>> >> > the
>>>> >> > >> > radosgw-admin command for a way to delete a bucket policy but do
>>>> >> > >> > not
>>>> >> > see
>>>> >> > >> > anything. I presume I will need to somehow remove the bucket
>>>> >> > >> > policy
>>>> >> > from
>>>> >> > >> > however it is stored in the bucket metadata / omap etc. If
>>>> >> > >> > anyone can
>>>> >> > point
>>>> >> > >> > me in the right direction on that I would appreciate it. Thanks
>>>> >> > >> >
>>>> >> > >> > Respectfully,
>>>> >> > >> >
>>>> >> > >> > *Wes Dillingham*
>>>> >> > >> > w...@wesdillingham.com
>>>> >> > >> > LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>>>> >> > >> > _______________________________________________
>>>> >> > >> > ceph-users mailing list -- ceph-users@ceph.io
>>>> >> > >> > To unsubscribe send an email to ceph-users-le...@ceph.io
>>>> >> > >> >
>>>> >> > >>
>>>> >> >
>>>> >> >
>>>> >> _______________________________________________
>>>> >> ceph-users mailing list -- ceph-users@ceph.io
>>>> >> To unsubscribe send an email to ceph-users-le...@ceph.io
>>>>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
