I went through multiple deletions and rebuilds but the steps I usually took were: - ceph orch rm the rgw daemons - remove secondary realm - commit period - Remove primary realm - Remove zongroup on both - Remove realm on both - Delete realm related pools
On the last delete when I wanted to ensure I removed everything and attempt a completely fresh start I also deleted the .rgw.root and any default realm pools (that I never used) on both clusters. Though I believe I skipped a couple steps after the realm removal because I thought deleting the .rgw.root pools would give it a fresh start. Kevin On Sun, Sep 14, 2025 at 2:48 AM Eugen Block <[email protected]> wrote: > How exactly did you clean up the RGW config? > > Zitat von Kevin Hrpcek <[email protected]>: > > > Hey all, > > > > I've been having some problems with a relatively basic rgw multisite set > > up. I initially had it working a few weeks ago with a simple config but > > rebuilt it with ssl enabled for the endpoints and then started having > some > > problems to work through. I eventually had that working but then tore it > > all down so I could verify the recreation steps but now I'm having > problems > > with the multisite. There are a few different errors.... > > > > I set up the master zone without problems, but if I use the rgw module > with > > a token and spec it fails to sync the sync user and the primary zone > can't > > sync back from the secondary zone. A permission denied error for data > > syncing primary->secondary but secondary-> primary works because it has > the > > credentials. > > > > So i used to be able to manually create the user on the secondary zone > and > > get data syncing but the metadata wouldn't sync with the master showing > an > > error message but when I check the mdlog on the primary it looks okay. > > > > 2025-09-11T15:15:18.771+0000 7fc29e96c640 1 req 14061042141473454829 > > 0.001000005s failed to read mdlog history: (2) No such file or directory > > > > I ended up trying a full clean reset of rgw on both clusters so I deleted > > the daemons and all related pools. The primary starts up but I still have > > the problem of the sync user not being created on the secondary and now > > when I try to create a user on the secondary for syncing or just for the > > dashboard to work I get errors that I've tracked down to > > > https://github.com/ceph/ceph/blob/62bcf65e8c0995783bb3e368909716346874ad62/src/rgw/radosgw-admin/radosgw-admin.cc#L4631 > > but am not quite following whats all going on in the source to determine > > why. > > > > # radosgw-admin user create --uid=dashboard --display-name=dashboard > > --system > > couldn't init storage provider > > # radosgw-admin user create --uid=sysuser-asdf > --display-name=sysuser-asdf > > --access-key=xxxxxxxxxxxxx --secret-key=yyyyyyyyyyyyyyy --system > > --rgw-realm=asdf > > Unable to initialize site config. > > > > Any thoughts? It seems like I may have messed something up in the ceph > > config database from multiple tear downs and restarts of the multisite > > setup but I'm not seeing what it could be. > > > > ceph 19.2.1 using cephadm on alma 9 > > > > Thanks! > > Kevin > > _______________________________________________ > > ceph-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
