Thank you Gilles for updating the ticket with the info, One last thing that could shed some light is to see the cert used by Prometheus (then let's continue the discussion on the tracker),
openssl x509 -in /var/lib/ceph/<fsid>/prometheus.<host>/etc/prometheus/prometheus.crt -text (Plz omit any sensitive information for your lab from the output) BTW: there's a Slack channel dedicated for cephadm that can be used for a more interactive dicussion https://ceph-storage.slack.com/archives/C04SNUBD2M6 Best, Redouane. On Wed, Dec 10, 2025 at 5:06 PM Gilles Mocellin < [email protected]> wrote: > Thank you Redouane, > > I have updated the ticket with all what you made me do and what you > asked. > Except that "--include-cephadm-signed" does not exist on my version for > `ceph orch certmgr cert ls`. > > https://tracker.ceph.com/issues/74164 > > Le 2025-12-10 15:36, Redouane Kachach a écrit : > > Thanks for the feedback Gilles. > > > > Using your own cert for the mgmt-gateway is Okay and shouldn't be an > > issue. > > Errors arise when you use custom certs for the Dashboard itself as > > bcz when the mgmt-gateway is enabled mTLS is enforced for internal > > communications (Dashboard <-> Prometheus and Alertmanager <-> Dashbaord > > i.e) > > > > From what you are commenting it seems like Grafana is not able to get > > metrics from Prometheus. > > > > Can you plz update the ticket with the output of the command (obviously > > omitting any private/sensitive data): > > > >> ceph orch certmgr cert ls --include-cephadm-signed > > > > This would show all the certificates in use for the different services. > > > > It may also help see the config of Grafana: > > > > cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/grafana.ini > > cat > > > /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/provisioning/datasources/ceph-dashboard.yml > > > > > > Thanks, > > Redouane. > > > > > > > > On Wed, Dec 10, 2025 at 2:33 PM Gilles Mocellin < > > [email protected]> wrote: > > > >> Better but not completely. > >> > >> I don't have errors anymore concerning alertmanager, and I can see > >> alerts, thereis on concerning mgr/prometheus module not responding. > >> Even if I disable/enable it. > >> > >> In the MGR logs, I see certificate problem : > >> > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 > >> > ceph-8ec7575a-7de5-11f0-a78a-246e96bd90a4-mgr-fidcl-lyo1-sto-sds-lab-01-gtcifn[7494]: > >> > >> [10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, > >> 0, > >> 0) lost — peer dropped the TLS connection suddenly, during handshake: > >> (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > >> (_ssl.c:1147)') > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[7789]: [prometheus > >> INFO cherrypy.error] [10/Dec/2025:13:27:40] ENGINE Client > >> ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS > >> connection suddenly, during handshake: (1, '[SSL: > >> SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > >> (_ssl.c:1147)') > >> > >> Grafana Dashboard still don't work. > >> In embeded mode, I still have Unauthorized messages. > >> If I open directly Grafana, with the DNS name, I can list the > >> Dashboard, > >> but in them, there is still "origin not allowed" messages. > >> I cannot log in, with also the message "origin not allowed". > >> > >> > >> Le 2025-12-10 14:15, Gilles Mocellin a écrit : > >> > Hi Redouane, > >> > > >> > Yes, I'm using my own certificates, signed by our internal PKI. > >> > > >> > I will try, but I still want a valid certificate for my DNS entry, > >> > certificate guven to the mgmt-gateway service. > >> > > >> > Le 10 décembre 2025 13:28:45 GMT+01:00, Redouane Kachach > >> > <[email protected]> a écrit : > >> >> Hi Gilles, > >> >> > >> >> What certificate are you using on your Dashboard? > >> >> > >> >> Just to rule out any certificate issues: when using mgmt-gateway, the > >> >> dashboard must be configured to use cephadm-signed certificates. > >> >> This is normally done automatically, but if you were using custom > >> >> certificates for your dashboard, those will not be overwritten by > >> >> cephadm. > >> >> If this is the case then plz just follow this procedure to fix them: > >> >> > >> >> 1) generate cephadm-signed certs for the dashboard (from cephadm > >> >> shell): > >> >> > >> >> ceph orch certmgr generate-certificates dashboard \ > >> >> | tee \ > >> >> >(jq -r '.cert' > dashboard.cert.pem) \ > >> >> >(jq -r '.key' > dashboard.key.pem) \ > >> >> > /dev/null > >> >> > >> >> 2) The above will generate two files: dashboard.cert.pem and > >> >> dashboard.key.pem, now we have to use them to set the dashboard > >> >> cert/key: > >> >> > >> >> ceph dashboard set-ssl-certificate -i dashboard.cert.pem > >> >> ceph dashboard set-ssl-certificate-key -i dashboard.key.pem > >> >> > >> >> 3) Reload the dashboard, so it starts using the new populated > >> >> certificates: > >> >> > >> >> ceph mgr module disable dashboard; ceph mgr module enable dashboard > >> >> --force > >> >> > >> >> Best, > >> >> Redouane. > >> >> > >> >> > >> >> On Wed, Dec 10, 2025 at 11:31 AM Gilles Mocellin via ceph-users < > >> >> [email protected]> wrote: > >> >> > >> >>> Hello ! > >> >>> > >> >>> Does someone manage to use mgmt-gateway, with a FQDN, and the > >> >>> monitoring > >> >>> stack working ? > >> >>> > >> >>> In my LAB, after upgrading from 10.2.3 to 20.2.0, > >> >>> My dashboard was working fine, with grafana dashboards. > >> >>> > >> >>> I decided to test mgmt-gateway. > >> >>> I use a VIP with a specific DNS name pointing at that VIP. > >> >>> I have a certificate generated with our internal PKI, valid for the > >> >>> DNS > >> >>> name and the VIP. > >> >>> > >> >>> Dashboard works, but no monitoring anymore. > >> >>> - Popups concerning alermanager : > >> >>> > >> >>> Failed to execute prometheus > >> >>> Error parsing Prometheus Alertmanager response: Expecting value > >> >>> 9/12/25 04:07 PM > >> >>> > >> >>> - No graphs in dashboard, with errors "origin not allowed" in each > >> >>> panel > >> >>> > >> >>> If I open Grafana directly, I see : > >> >>> - that the VIP is used, not my DNS name, but none works. > >> >>> - Grafana works but I can't see dashboards, and if I put a URL that > >> >>> should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I > have a > >> >>> message : > >> >>> > >> >>> Failed to load dashboard > >> >>> Unauthorized > >> >>> > >> >>> Some more infos : > >> >>> > >> >>> https://$DNS/grafana/dashboards => show the list of Dashboards > >> >>> Open a dashboard show errors 'origin not allowed" on panels > >> >>> > >> >>> https://$VIP/grafana/dashboards => cannot list the Dashboards, > error > >> >>> Unauthorized > >> >>> > >> >>> Concerning the Altermanger errors, I see thoses XHR calls in my > >> >>> browser, > >> >>> ending in HTTP error 400 Bad Request : > >> >>> https://$DNS/api/prometheus?cluster_filter=true > >> >>> > >> >>> I opened an issue here : https://tracker.ceph.com/issues/74164 > >> >>> > >> >>> -- > >> >>> Gilles > >> >>> _______________________________________________ > >> >>> ceph-users mailing list -- [email protected] > >> >>> To unsubscribe send an email to [email protected] > >> >>> > >> >>> > >> > >> > > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
