** Description changed:
+ [ Impact ]
+
+ * The Ceph RADOS gateway is consistently crashing in version 20.2.0,
+ on Ubuntu resolute, when used keystone for authentication.
+
+ * The new version, 20.2.1, fixes this issue by reverting the behaviour of
+ the http handler to a way more in line with the implementation in
+ Squid (19.2.3).
+
+ [ Test Plan ]
+
+ * The issue can be reproduced by running a basic S3 workflow with the
+ RADOS gateway and using keystone for authentication.
+
+ * We will verify that the issue is no longer present by running the
functional
+ test suite for the Ceph charms, which include appropriate tests for RGW.
+
+ [ Where problems could occur ]
+
+ * The new version reverts part of the behaviour in the HTTP handling of the
+ RADOS gateway. Since this is an integral part of the gateway, any issues
+ that are introduced by these changes should be immediately visible.
+
+ * The new version uses the same implementation as what it was in the previous
+ packages (19.2.3). No issues with RADOS gateway were found with them.
+
+ [ Original Description ]
+
On Ubuntu resolute radosgw from 20.2.0-0ubuntu2 reproducibly segfaults
when an S3 request is authenticated with Keystone EC2 credentials. The
client sees "502 Bad Gateway" and systemd restarts radosgw.
This is the Keystone-backed S3 auth path. Local RGW users can work when
local auth is tried first, but a request using Keystone EC2 credentials
still crashes RGW.
This can be reproduced using the CI bundle with adaptation for resolute
and zaza tests
Stack trace
```
Thread 602 "http_manager" received signal SIGSEGV, Segmentation fault.
0x00005e52632bf3b6 in
ceph::async::detail::CompletionImpl<boost::asio::any_io_executor,
boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void
(boost::system::error_code), void>, void,
boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&)
()
Thread 602 (Thread 0x753657aea6c0 (LWP 63226) "http_manager"):
- #0 0x00005e52632bf3b6 in
ceph::async::detail::CompletionImpl<boost::asio::any_io_executor,
boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void
(boost::system::error_code), void>, void,
boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&)
()
- #1 0x00005e5263345d18 in RGWHTTPManager::finish_request(rgw_http_req_data*,
int, long) ()
- #2 0x00005e5263346c5a in RGWHTTPManager::reqs_thread_entry() ()
- #3 0x00005e52633475e1 in RGWHTTPManager::ReqsThread::entry() ()
- #4 0x000075365eca40da in start_thread (arg=<optimized out>) at
./nptl/pthread_create.c:454
- #5 0x000075365ed377ac in __GI___clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
+ #0 0x00005e52632bf3b6 in
ceph::async::detail::CompletionImpl<boost::asio::any_io_executor,
boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void
(boost::system::error_code), void>, void,
boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&)
()
+ #1 0x00005e5263345d18 in RGWHTTPManager::finish_request(rgw_http_req_data*,
int, long) ()
+ #2 0x00005e5263346c5a in RGWHTTPManager::reqs_thread_entry() ()
+ #3 0x00005e52633475e1 in RGWHTTPManager::ReqsThread::entry() ()
+ #4 0x000075365eca40da in start_thread (arg=<optimized out>) at
./nptl/pthread_create.c:454
+ #5 0x000075365ed377ac in __GI___clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
```
Juju status
```
# juju status
- Model Controller Cloud/Region Version SLA
Timestamp
- zaza-9bef958d671a lxd localhost/localhost 3.6.23 unsupported
11:39:43Z
+ Model Controller Cloud/Region Version SLA Timestamp
+ zaza-9bef958d671a lxd localhost/localhost 3.6.23 unsupported 11:39:43Z
- App Version Status Scale Charm Channel Rev
Exposed Message
- ceph-mon 20.2.0 active 3 ceph-mon 0
no Unit is ready and clustered
- ceph-osd 20.2.0 active 3 ceph-osd 15
no Unit is ready (1 OSD)
- ceph-radosgw 20.2.0 active 1 ceph-radosgw 26
no Unit is ready
- keystone 25.0.0 active 1 keystone latest/edge 774
no Application Ready
- mysql 8.0.44-0ubun... active 1 mysql 8.0/stable 444
no
- vault 1.8.8 active 1 vault latest/edge 363
no Unit is ready (active: true, mlock: disabled)
+ App Version Status Scale Charm Channel Rev Exposed Message
+ ceph-mon 20.2.0 active 3 ceph-mon 0 no Unit is ready and clustered
+ ceph-osd 20.2.0 active 3 ceph-osd 15 no Unit is ready (1 OSD)
+ ceph-radosgw 20.2.0 active 1 ceph-radosgw 26 no Unit is ready
+ keystone 25.0.0 active 1 keystone latest/edge 774 no Application Ready
+ mysql 8.0.44-0ubun... active 1 mysql 8.0/stable 444 no
+ vault 1.8.8 active 1 vault latest/edge 363 no Unit is ready (active: true,
mlock: disabled)
- Unit Workload Agent Machine Public address
Ports Message
- ceph-mon/0* active idle 4 10.33.157.168
Unit is ready and clustered
- ceph-mon/1 active idle 5 10.33.157.25
Unit is ready and clustered
- ceph-mon/2 active idle 6
fd42:34da:d2af:4f10:216:3eff:fea3:1a7b Unit is ready and
clustered
- ceph-osd/0* active idle 1
fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 Unit is ready (1 OSD)
- ceph-osd/1 active idle 2 10.33.157.197
Unit is ready (1 OSD)
- ceph-osd/2 active idle 3 10.33.157.131
Unit is ready (1 OSD)
- ceph-radosgw/0* active idle 0
fd42:34da:d2af:4f10:216:3eff:fece:c073 80/tcp Unit is ready
- keystone/0* active idle 8 10.33.157.200
5000/tcp Unit is ready
- mysql/0* active idle 7 10.33.157.76
3306,33060/tcp Primary
- vault/0* active idle 9
fd42:34da:d2af:4f10:216:3eff:fe01:376d 8200/tcp Unit is ready (active:
true, mlock: disabled)
+ Unit Workload Agent Machine Public address Ports Message
+ ceph-mon/0* active idle 4 10.33.157.168 Unit is ready and clustered
+ ceph-mon/1 active idle 5 10.33.157.25 Unit is ready and clustered
+ ceph-mon/2 active idle 6 fd42:34da:d2af:4f10:216:3eff:fea3:1a7b Unit is ready
and clustered
+ ceph-osd/0* active idle 1 fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 Unit is
ready (1 OSD)
+ ceph-osd/1 active idle 2 10.33.157.197 Unit is ready (1 OSD)
+ ceph-osd/2 active idle 3 10.33.157.131 Unit is ready (1 OSD)
+ ceph-radosgw/0* active idle 0 fd42:34da:d2af:4f10:216:3eff:fece:c073 80/tcp
Unit is ready
+ keystone/0* active idle 8 10.33.157.200 5000/tcp Unit is ready
+ mysql/0* active idle 7 10.33.157.76 3306,33060/tcp Primary
+ vault/0* active idle 9 fd42:34da:d2af:4f10:216:3eff:fe01:376d 8200/tcp Unit
is ready (active: true, mlock: disabled)
- Machine State Address Inst id Base
AZ Message
- 0 started fd42:34da:d2af:4f10:216:3eff:fece:c073 juju-a24339-0
[email protected] sunkern Running
- 1 started fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 juju-a24339-1
[email protected] sunkern Running
- 2 started 10.33.157.197 juju-a24339-2
[email protected] sunkern Running
- 3 started 10.33.157.131 juju-a24339-3
[email protected] sunkern Running
- 4 started 10.33.157.168 juju-a24339-4
[email protected] sunkern Running
- 5 started 10.33.157.25 juju-a24339-5
[email protected] sunkern Running
- 6 started fd42:34da:d2af:4f10:216:3eff:fea3:1a7b juju-a24339-6
[email protected] sunkern Running
- 7 started 10.33.157.76 juju-a24339-7
[email protected] sunkern Running
- 8 started 10.33.157.200 juju-a24339-8
[email protected] sunkern Running
- 9 started fd42:34da:d2af:4f10:216:3eff:fe01:376d juju-a24339-9
[email protected] sunkern Running
+ Machine State Address Inst id Base AZ Message
+ 0 started fd42:34da:d2af:4f10:216:3eff:fece:c073 juju-a24339-0 [email protected]
sunkern Running
+ 1 started fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 juju-a24339-1 [email protected]
sunkern Running
+ 2 started 10.33.157.197 juju-a24339-2 [email protected] sunkern Running
+ 3 started 10.33.157.131 juju-a24339-3 [email protected] sunkern Running
+ 4 started 10.33.157.168 juju-a24339-4 [email protected] sunkern Running
+ 5 started 10.33.157.25 juju-a24339-5 [email protected] sunkern Running
+ 6 started fd42:34da:d2af:4f10:216:3eff:fea3:1a7b juju-a24339-6 [email protected]
sunkern Running
+ 7 started 10.33.157.76 juju-a24339-7 [email protected] sunkern Running
+ 8 started 10.33.157.200 juju-a24339-8 [email protected] sunkern Running
+ 9 started fd42:34da:d2af:4f10:216:3eff:fe01:376d juju-a24339-9 [email protected]
sunkern Running
```
versions
```
Ubuntu 26.04 LTS (resolute)
- radosgw: 20.2.0-0ubuntu2
- ceph-common: 20.2.0-0ubuntu2
- librados2: 20.2.0-0ubuntu2
+ radosgw: 20.2.0-0ubuntu2
+ ceph-common: 20.2.0-0ubuntu2
+ librados2: 20.2.0-0ubuntu2
radosgw --version:
ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
(stable - RelWithDebInfo)
```
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2154304
Title:
ceph radosgw segv with keystone auth
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2154304/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
