Hi Gilles, I was able to reproduce a similar problem locally (the error related to "origin not allowed). As of my local testing (and observations) Grafana was failing with “origin not allowed” only when user login was enabled (user/password), but everything worked fine when using anonymous access. I think this difference is due to how Grafana’s CSRF protection works:
With anonymous access: - You are effectively a non-authenticated user. - Most requests are simple GETs and there is no session cookie to protect. - Grafana’s CSRF checks are much less strict, so it doesn’t really care about the Origin header. With user login enabled: - After logging in, the browser sends requests with a session cookie and an Origin header like: Origin: https://<your-fqdn-domain> - Because this is now an authenticated session, Grafana enforces strict CSRF checks. - If the Origin does not exactly match what Grafana considers its own URL or trusted origins (Grafana is configured with an internal hostname, while you access it via the mgmt-gateway FQDN), Grafana rejects the request with: “origin not allowed”. I think that explains why the problem only appears once user authentication is enabled. The problem is that Grafana is deployed in this case behind mgmt-gateway (which acts as reverse-proxy). So Grafana has no idea what's the fqdn in use as it comes from the mgmt-gateway. When oauth2-proxy is also used so authentication goes through SSO this issue doesn't happen. To solve the problem, grafana provides a config field csrf_trusted_origins which can be used to indicate manually your trusted origins. In this case, you would need to add something like: [security] csrf_trusted_origins = <your-fqdn-domain> For example: [security] csrf_trusted_origins = my-cloud.ceph-node-0.com In your grafana.ini (located at /var/lib/ceph/<fsid>/grafana.<your-grafana-node>/etc/grafana/grafana.ini To test this quickly you can edit .ini manually and then run "ceph orch restart grafana" from the cephadm shell. This change is unfortunately not permanent, if you redeploy your grafana or move it to another node, you will lose the change. To make it permanent you have to adjust the grafana Jinja template, used by cephadm to generate the ini file. For that you have to modify the template to add the above entry and store it for example in a grafana.ini.j2 file then run the following commands: ceph config-key set mgr/cephadm/services/grafana/grafana.ini -i grafana.ini.j2 ceph orch reconfig grafana To get the template you can either find it in github, for tentacle (plz adjust the path to your exact release): https://github.com/ceph/ceph/blob/tentacle/src/pybind/mgr/cephadm/templates/services/grafana/grafana.ini.j2 or directly from the ceph-mgr container: /usr/share/ceph/mgr/cephadm/templates/services/grafana/grafana.ini.j2 Best, Redouane. On Thu, Dec 11, 2025 at 9:27 AM Redouane Kachach via ceph-users < [email protected]> wrote: > Thank you Gilles for updating the ticket with the info, > > One last thing that could shed some light is to see the cert used by > Prometheus (then let's continue the discussion on the tracker), > > openssl x509 -in > /var/lib/ceph/<fsid>/prometheus.<host>/etc/prometheus/prometheus.crt -text > > (Plz omit any sensitive information for your lab from the output) > > BTW: there's a Slack channel dedicated for cephadm that can be used for a > more interactive dicussion > https://ceph-storage.slack.com/archives/C04SNUBD2M6 > > Best, > Redouane. > > > > On Wed, Dec 10, 2025 at 5:06 PM Gilles Mocellin < > [email protected]> wrote: > > > Thank you Redouane, > > > > I have updated the ticket with all what you made me do and what you > > asked. > > Except that "--include-cephadm-signed" does not exist on my version for > > `ceph orch certmgr cert ls`. > > > > https://tracker.ceph.com/issues/74164 > > > > Le 2025-12-10 15:36, Redouane Kachach a écrit : > > > Thanks for the feedback Gilles. > > > > > > Using your own cert for the mgmt-gateway is Okay and shouldn't be an > > > issue. > > > Errors arise when you use custom certs for the Dashboard itself as > > > bcz when the mgmt-gateway is enabled mTLS is enforced for internal > > > communications (Dashboard <-> Prometheus and Alertmanager <-> Dashbaord > > > i.e) > > > > > > From what you are commenting it seems like Grafana is not able to get > > > metrics from Prometheus. > > > > > > Can you plz update the ticket with the output of the command (obviously > > > omitting any private/sensitive data): > > > > > >> ceph orch certmgr cert ls --include-cephadm-signed > > > > > > This would show all the certificates in use for the different services. > > > > > > It may also help see the config of Grafana: > > > > > > cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/grafana.ini > > > cat > > > > > > /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/provisioning/datasources/ceph-dashboard.yml > > > > > > > > > Thanks, > > > Redouane. > > > > > > > > > > > > On Wed, Dec 10, 2025 at 2:33 PM Gilles Mocellin < > > > [email protected]> wrote: > > > > > >> Better but not completely. > > >> > > >> I don't have errors anymore concerning alertmanager, and I can see > > >> alerts, thereis on concerning mgr/prometheus module not responding. > > >> Even if I disable/enable it. > > >> > > >> In the MGR logs, I see certificate problem : > > >> > > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 > > >> > > > ceph-8ec7575a-7de5-11f0-a78a-246e96bd90a4-mgr-fidcl-lyo1-sto-sds-lab-01-gtcifn[7494]: > > >> > > >> [10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, > > >> 0, > > >> 0) lost — peer dropped the TLS connection suddenly, during handshake: > > >> (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > > >> (_ssl.c:1147)') > > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[7789]: [prometheus > > >> INFO cherrypy.error] [10/Dec/2025:13:27:40] ENGINE Client > > >> ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS > > >> connection suddenly, during handshake: (1, '[SSL: > > >> SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > > >> (_ssl.c:1147)') > > >> > > >> Grafana Dashboard still don't work. > > >> In embeded mode, I still have Unauthorized messages. > > >> If I open directly Grafana, with the DNS name, I can list the > > >> Dashboard, > > >> but in them, there is still "origin not allowed" messages. > > >> I cannot log in, with also the message "origin not allowed". > > >> > > >> > > >> Le 2025-12-10 14:15, Gilles Mocellin a écrit : > > >> > Hi Redouane, > > >> > > > >> > Yes, I'm using my own certificates, signed by our internal PKI. > > >> > > > >> > I will try, but I still want a valid certificate for my DNS entry, > > >> > certificate guven to the mgmt-gateway service. > > >> > > > >> > Le 10 décembre 2025 13:28:45 GMT+01:00, Redouane Kachach > > >> > <[email protected]> a écrit : > > >> >> Hi Gilles, > > >> >> > > >> >> What certificate are you using on your Dashboard? > > >> >> > > >> >> Just to rule out any certificate issues: when using mgmt-gateway, > the > > >> >> dashboard must be configured to use cephadm-signed certificates. > > >> >> This is normally done automatically, but if you were using custom > > >> >> certificates for your dashboard, those will not be overwritten by > > >> >> cephadm. > > >> >> If this is the case then plz just follow this procedure to fix > them: > > >> >> > > >> >> 1) generate cephadm-signed certs for the dashboard (from cephadm > > >> >> shell): > > >> >> > > >> >> ceph orch certmgr generate-certificates dashboard \ > > >> >> | tee \ > > >> >> >(jq -r '.cert' > dashboard.cert.pem) \ > > >> >> >(jq -r '.key' > dashboard.key.pem) \ > > >> >> > /dev/null > > >> >> > > >> >> 2) The above will generate two files: dashboard.cert.pem and > > >> >> dashboard.key.pem, now we have to use them to set the dashboard > > >> >> cert/key: > > >> >> > > >> >> ceph dashboard set-ssl-certificate -i dashboard.cert.pem > > >> >> ceph dashboard set-ssl-certificate-key -i dashboard.key.pem > > >> >> > > >> >> 3) Reload the dashboard, so it starts using the new populated > > >> >> certificates: > > >> >> > > >> >> ceph mgr module disable dashboard; ceph mgr module enable > dashboard > > >> >> --force > > >> >> > > >> >> Best, > > >> >> Redouane. > > >> >> > > >> >> > > >> >> On Wed, Dec 10, 2025 at 11:31 AM Gilles Mocellin via ceph-users < > > >> >> [email protected]> wrote: > > >> >> > > >> >>> Hello ! > > >> >>> > > >> >>> Does someone manage to use mgmt-gateway, with a FQDN, and the > > >> >>> monitoring > > >> >>> stack working ? > > >> >>> > > >> >>> In my LAB, after upgrading from 10.2.3 to 20.2.0, > > >> >>> My dashboard was working fine, with grafana dashboards. > > >> >>> > > >> >>> I decided to test mgmt-gateway. > > >> >>> I use a VIP with a specific DNS name pointing at that VIP. > > >> >>> I have a certificate generated with our internal PKI, valid for > the > > >> >>> DNS > > >> >>> name and the VIP. > > >> >>> > > >> >>> Dashboard works, but no monitoring anymore. > > >> >>> - Popups concerning alermanager : > > >> >>> > > >> >>> Failed to execute prometheus > > >> >>> Error parsing Prometheus Alertmanager response: Expecting value > > >> >>> 9/12/25 04:07 PM > > >> >>> > > >> >>> - No graphs in dashboard, with errors "origin not allowed" in each > > >> >>> panel > > >> >>> > > >> >>> If I open Grafana directly, I see : > > >> >>> - that the VIP is used, not my DNS name, but none works. > > >> >>> - Grafana works but I can't see dashboards, and if I put a URL > that > > >> >>> should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I > > have a > > >> >>> message : > > >> >>> > > >> >>> Failed to load dashboard > > >> >>> Unauthorized > > >> >>> > > >> >>> Some more infos : > > >> >>> > > >> >>> https://$DNS/grafana/dashboards => show the list of Dashboards > > >> >>> Open a dashboard show errors 'origin not allowed" on panels > > >> >>> > > >> >>> https://$VIP/grafana/dashboards => cannot list the Dashboards, > > error > > >> >>> Unauthorized > > >> >>> > > >> >>> Concerning the Altermanger errors, I see thoses XHR calls in my > > >> >>> browser, > > >> >>> ending in HTTP error 400 Bad Request : > > >> >>> https://$DNS/api/prometheus?cluster_filter=true > > >> >>> > > >> >>> I opened an issue here : https://tracker.ceph.com/issues/74164 > > >> >>> > > >> >>> -- > > >> >>> Gilles > > >> >>> _______________________________________________ > > >> >>> ceph-users mailing list -- [email protected] > > >> >>> To unsubscribe send an email to [email protected] > > >> >>> > > >> >>> > > >> > > >> > > > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
