[VERIFICATION BIONIC]
I have deployed a Ceph cluster using juju deploy and then have updated
the entire cluster[1] to the ceph packages found in bionic-proposed
(built against libssl1.0.0).
On the rgw node, I have setup a ssl certificate, and instruct civetweb
in /etc/ceph/ceph.conf to use ssl[2].
radosgw is now running just fine[3][4] and civetweb LISTEN on port 443
as it should[5].
[1] Ceph cluster:
.......
Unit Workload Agent Machine Public address Ports Message
ceph-mon/0* active idle 0 10.5.0.4 Unit is ready
and clustered
ceph-osd/0* active idle 1 10.5.0.5 Unit is ready
(1 OSD)
ceph-osd/1 active idle 2 10.5.0.27 Unit is ready
(1 OSD)
ceph-osd/2 active idle 3 10.5.0.6 Unit is ready
(1 OSD)
ceph-rgw/0* active idle 4 10.5.0.18 80/tcp Unit is ready
......
[2] /etc/ceph/ceph.conf
[client.rgw.<HOSTNAME>]
......
rgw_frontends = civetweb port=443s ssl_certificate=/etc/ssl/server.pem
.......
[3] sudo systemctl status ceph-radosgw@rgw.`hostname -s`
● ceph-rado...@rgw.juju-521d82-default-4.service - Ceph rados gateway
Loaded: loaded (/lib/systemd/system/ceph-radosgw@.service; indirect; vendor
preset: enabled)
Active: active (running) since Wed 2019-05-01 19:51:55 UTC; 10min ago
Main PID: 4225 (radosgw)
Tasks: 580
CGroup:
/system.slice/system-ceph\x2dradosgw.slice/ceph-rado...@rgw.juju-521d82-default-4.service
└─4225 /usr/bin/radosgw -f --cluster ceph --name
client.rgw.juju-521d82-default-4 --setuser ceph --setgroup
May 01 19:59:59 juju-521d82-default-4 radosgw[4225]: 2019-05-01 19:59:59.208671
7f19095f0700 2 RGWDataChangesLog::Cha
May 01 20:00:21 juju-521d82-default-4 radosgw[4225]: 2019-05-01 20:00:21.208946
7f19095f0700 2 RGWDataChangesLog::Cha
May 01 20:00:43 juju-521d82-default-4 radosgw[4225]: 2019-05-01 20:00:43.209214
7f19095f0700 2 RGWDataChangesLog::Cha
May 01 20:01:05 juju-521d82-default-4 radosgw[4225]: 2019-05-01 20:01:05.209332
7f19095f0700 2 RGWDataChangesLog::Cha
May 01 20:01:27 juju-521d82-default-4 radosgw[4225]: 2019-05-01 20:01:27.209500
7f19095f0700 2 RGWDataChangesLog::Cha
May 01 20:01:49 juju-521d82-default-4 radosgw[4225]: 2019-05-01 20:01:49.209716
7f19095f0700 2 RGWDataChangesLog::Cha
May 01 20:01:56 juju-521d82-default-4 radosgw[4225]: 2019-05-01 20:01:56.129879
7f1907ded700 2 object expiration: sta
May 01 20:02:11 juju-521d82-default-4 radosgw[4225]: 2019-05-01 20:02:11.209902
7f19095f0700 2 RGWDataChangesLog::Cha
May 01 20:02:12 juju-521d82-default-4 radosgw[4225]: 2019-05-01 20:02:12.346598
7f1907ded700 2 object expiration: sto
May 01 20:02:33 juju-521d82-default-4 radosgw[4225]: 2019-05-01 20:02:33.210102
7f19095f0700 2 RGWDataChangesLog::Cha
[4] logs
May 1 19:51:56 juju-521d82-default-4 radosgw: 2019-05-01 19:51:56.115874
7f1924299000 0 starting handler: civetweb
May 1 19:51:56 juju-521d82-default-4 radosgw: 2019-05-01 19:51:56.186842
7f1924299000 1 mgrc service_daemon_register rgw.juju-521d82-default-4 metadata
{arch=x86_64,ceph_version=ceph version 12.2.11
(26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable),cpu=Intel Xeon
E312xx (Sandy Bridge, IBRS update),distro=ubuntu,distro_description=Ubuntu
18.04.2 LTS,distro_version=18.04,frontend_config#0=civetweb port=443s
ssl_certificate=/etc/ssl/server.pem,frontend_type#0=civetweb,hostname=juju-521d82-default-4,kernel_description=#50-Ubuntu
SMP Wed Mar 13 10:44:52 UTC
2019,kernel_version=4.15.0-47-generic,mem_swap_kb=0,mem_total_kb=2041224,num_handles=1,os=Linux,pid=4225,zone_id=be9d4d4f-725f-490d-acf6-c0a713e03da4,zone_name=default,zonegroup_id=ab35965a-0856-4671-906a-fe7aedcb92ca,zonegroup_name=default}
[5] netstat -anputa | grep -i radosgw | grep 443
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
4225/radosgw
- Eric
** Description changed:
[Impact]
Since the introduction of OpenSSL 1.1.1 in 18.04 LTS:
https://launchpad.net/bugs/1797386
This is breaking Ceph cluster https service.
# logs:
2019-04-02 16:40:14.846313 7ff8c1736000 0 starting handler: civetweb
2019-04-02 16:40:14.846397 7ff8c1736000 0 civetweb: 0x56114520d620: load_dll:
libcrypto.so.1.1: cannot find CRYPTO_num_locks
2019-04-02 16:40:14.846424 7ff8c1736000 -1 ERROR: failed run
[Test Case]
1) Generate a self-signed certificate or use whatever existing SSL
certificate already in place.
If one want to create a PEM file for civetweb, instructions can be found here
:
https://github.com/civetweb/civetweb/blob/master/docs/OpenSSL.md
** Note: "CivetWeb requires one certificate file in PEM format" **
2) Enable logging and debugging in "/etc/ceph/ceph.conf"
Example:
------
log to syslog = true
err to syslog = true
clog to syslog = true
debug rgw = 10/5
debug civetweb = 1/10
------
http://docs.ceph.com/docs/mimic/rados/troubleshooting/log-and-debug/
3) From the radosgw node, modify "/etc/ceph/ceph.conf" as follow:
rgw_frontends = civetweb port=443s
ssl_certificate=/<path_to_PEM_FILE>/<PEM_FILE>
4) Restart the daemon:
systemctl restart ceph-radosgw@rgw.`hostname -s`
5) Look logs:
2019-04-10 12:02:53.535133 7fcd20c4e000 0 civetweb: 0x562d710ed620: load_dll:
libcrypto.so.1.1: cannot find CRYPTO_num_locks
6) Look radosgw which should FAILED to start.
systemctl status ceph-radosgw@rgw.`hostname -s`
What we are looking for here is radosgw to be 'Active' and to have a
LISTEN port on 443 as follow :
$ netstat -anputa | grep LISTEN | grep 443 # or any port mentioned in the
configuration above.
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
10153/radosgw
[Potential Regression]
* Same downgrade approach has been made for 'nodejs' via LP: #1798367
+ * The proposed packages has been tested on at least 2 different Ceph
+ clusters impacted by the issue, and have been tested at various level
+ (no package update problem, radosgw is now working fine when civetweb is
+ configure over ssl, ...)
+
* Nothing can be worst than current situation, considering that civetweb
- is non-functionnal when SSL is in used due to the incompatibility with
+ is non-functional when SSL is in used due to the incompatibility with
1.1 and make radosgw daemon to fail.
* libssl1.0 and libssl1.1 are coinstallable ABIs so it shouldn't be a
problem here.
* See discussion IRC discussion (xnox/jamespage) on comment #11
* All autopkgtest 'passed'
http://autopkgtest.ubuntu.com/packages/ceph
[Other Information]
* Adding the OpenSSL 1.1 support has been explored and revealed to be
non-trivial :
https://github.com/civetweb/civetweb/pull/384/commits
https://github.com/civetweb/civetweb/commit/adac9c916fa892ec5edce7b565803f1e62d304a2
https://github.com/civetweb/civetweb/commit/5d83900fd29fb6fa1cd604676cb0562dc984dcc9
http://docs.ceph.com/docs/bobtail/radosgw/troubleshooting/
See discussion IRC discussion on comment #11
[Original Description]
Bionic's radosgw package (Version 12.2.11-0ubuntu0.18.04.1 ) can't run
on Bionic, because the version of civetweb in Luminous is incompatible
with libssl1.1, but it's built against libssl1.1.
This has been known about upstream for a while now, and as noted in the
bug-tracker (https://tracker.ceph.com/issues/20696), it can be fixed by
building Luminous in an environment that has only libssl1.0 available
(or, in a more invasive manner, by incorporating a newer civetweb). A
patch is in the tracker.ceph.com issue.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1822872
Title:
Bionic: Luminous radosgw incompatible with libssl1.1
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1822872/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
