[ceph-users] ceph-radosgw: Initialization timeout, failed to initialize
We have a Red Hat installation of Luminuous (full packages version: 12.2.8-128.1). We're experiencing an issue where the ceph-radosgw service will timeout during initialization and cycle through attempts every five minutes until it seems to just give up. Every other ceph service starts successfully. I tried looking at the health of the cluster, but anytime I run a command, whether ceph or radosgw-admin just to see a list of users, it seems to time out as well. I've used strace when attempting to start radosgw directly and was presented with a missing keyring error. I would be inclined to think that might be the problem, but wouldn't that also impact all of the other services? I haven't been able to find anything in the logs that would lead me down any paths. Everything I've looked at (journalctl, /var/log/messages, /var/log/ceph/ceph-rgw-server.log) all just say the same thing: the service attempted to start, it failed to initialize, entered a failed state, service stopped. This repeats. I only get pulled in to look at Ceph every now and then so I don't have enough knowledge to know how the various components interact or if anything external is having an impact. Is there anywhere that might be holding a bit of information? Thank you, Mathew Sent with [ProtonMail](https://protonmail.com) Secure Email. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph-radosgw: Initialization timeout, failed to initialize
I increased the debug level to 20. There isn't anything additional being written: 2021-02-23 16:26:38.736642 7£2c45£3700 -1 Initialization timeout, failed to initialize 2021-02-23 16:26:38.931400 7f4d7bf4a000 0 deferred set uid:gid to 167:167 (ceph:ceph) 2021-02-23 16:26:38.931707 7f4d7bf4a000 0 ceph version 12.2.8-128.1. TEST. bz1742993.el7cp (87ba41fa7b3bcb79d916fb0ca41a9dd90eb877f8) luminous (stable), process radosgw, pid 7169 2021-02-23 16:31:38.931973 7f4d6b8e5700 -1 Initialization timeout, failed to initialize 2021-02-23 16:31:39.191898 7fa2fae31000 deferred set uid:gid to 167:167 (ceph:ceph) 2021-02-23 16:31:39.192119 7fa2fae31000 0 ceph version 12.2.8-128.1. TEST.bz1742993.el7cp (87ba41fa7b3bcb79d916fb0ca41a9dd90eb877f8) luminous (stable), process radosgw, pid 7204 Thank you, Mathew Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Tuesday, February 23, 2021 10:57 AM, Janne Johansson wrote: > Den tis 23 feb. 2021 kl 16:53 skrev Mathew Snyder > mathew.sny...@protonmail.com: > > > We have a Red Hat installation of Luminuous (full packages version: > > 12.2.8-128.1). We're experiencing an issue where the ceph-radosgw service > > will timeout during initialization and cycle through attempts every five > > minutes until it seems to just give up. Every other ceph service starts > > successfully. > > I tried looking at the health of the cluster, but anytime I run a command, > > whether ceph or radosgw-admin just to see a list of users, it seems to time > > out as well. > > I've used strace when attempting to start radosgw directly and was > > presented with a missing keyring error. I would be inclined to think that > > might be the problem, but wouldn't that also impact all of the other > > services? > > No, a missing rgw key would stop only it, and the radosgw-admin > command (if run on a box without a global admin key) > > > I haven't been able to find anything in the logs that would lead me down > > any paths. Everything I've looked at (journalctl, /var/log/messages, > > /var/log/ceph/ceph-rgw-server.log) all just say the same thing: the service > > attempted to start, it failed to initialize, entered a failed state, > > service stopped. This repeats. > > See if you can bump the debug log level of the radosgw when starting it. > > https://access.redhat.com/solutions/2085183 > and > https://docs.ceph.com/en/latest/rados/troubleshooting/log-and-debug/#rados-gateway > > --- > > May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] radosgw-admin error: "could not fetch user info: no user info saved"
I'm running into a problem that I've found around the Internet, but for which I'm unable to find a solution: $ sudo radosgw-admin user info could not fetch user info: no user info saved I've seen others report this issue, but the troubleshooting has either failed to pinpoint the issue, or has gone from "I have a problem" to "I solved it" with no information as to how it was solved. Unfortunately, due to the nature of our network, I'm unable to provide debug information en masse. I can try to provide individual lines of output if someone knows what I should look for. I am able to run the user creation command: $ sudo radosgw-admin user create --uid="testuser2" --display-name="testuser2" { "user_id": "testuser2", "display_name": "testuser2", "email": "", "suspended": 0, "max_buckets": 1000, ... } After doing this, I'm still unable to see the user information instead getting the same "no user info saved" output. That said, I know the user is added: $ sudo radosgw-admin metadata list user [ "devtest1", "testuser", "testuser2", .. ] I'm pretty new to this system and Ceph in general. I looked at /etc/ceph/ceph.conf but I don't see anything that would have anything to do with users. I also don't have an account on the Ansible server that controls the configurations so I can't see if anything is defined there. I know this isn't particularly detailed in its description of the issue, but I can't find any other useful help via search. Thank you, Mathew Sent with [ProtonMail](https://protonmail.com) Secure Email. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: radosgw-admin error: "could not fetch user info: no user info saved"
As simple as that, eh? Cripes. Thank you, Mathew Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐ Original Message ‐‐‐ On Thursday, April 16, 2020 4:40 AM, Janne Johansson wrote: > Den ons 15 apr. 2020 kl 21:01 skrev Mathew Snyder > : > >> I'm running into a problem that I've found around the Internet, but for >> which I'm unable to find a solution: >> $ sudo radosgw-admin user info >> could not fetch user info: no user info saved > > radosgw-admin user info --uid="name of user goes here" > > -- > May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: radosgw-admin error: "could not fetch user info: no user info saved"
Was able to do what I needed to do. Thank you, Mathew Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐ Original Message ‐‐‐ On Thursday, April 16, 2020 4:40 AM, Janne Johansson wrote: > Den ons 15 apr. 2020 kl 21:01 skrev Mathew Snyder > : > >> I'm running into a problem that I've found around the Internet, but for >> which I'm unable to find a solution: >> $ sudo radosgw-admin user info >> could not fetch user info: no user info saved > > radosgw-admin user info --uid="name of user goes here" > > -- > May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io