Hi all!
I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to
AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all
ceph related fields but, for example, the nodes information is still fine
(Host Detail Dashboard).
I have redeployed the mgr service with cephadm, disabled and re-enabled mgr
prometheus module , but nothing changed. Digging into the problem, I
accessed the prometheus interface. When I access prometheus, and found this
error[image: Screen Shot 2024-06-04 at 15.22.37.png]
When I access the node shown as down, it reports
503 Service Unavailable
No cached data available yet
Traceback (most recent call last):
File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line
638, in respond
self._do_respond(path_info)
File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line
697, in _do_respond
response.body = self.handler()
File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line
219, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line
54, in __call__
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics
return self._metrics(_global_instance)
File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics
raise cherrypy.HTTPError(503, 'No cached data available yet')
cherrypy._cperror.HTTPError: (503, 'No cached data available yet')
I checked the mgr prometheus address and port
[ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr
::
[ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port
9283
It seems to be ok.
When I check the master manager node for the port, I found
[root@ceph-hn01 ~]# netstat -natup | grep 9283
tcp6 0 0 :::9283 :::* LISTEN
2453/ceph-mgr
tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130
ESTABLISHED 2453/ceph-mgr
I don't understand why it is showing as IPv6, the node doesn't have a dual
stack.
I also tried to use a newer version of the prometheus container image, the
1.6.0, but it keeps reporting the same, so I rolled it back to the original
one.
Has anyone experienced an issue like this?
Where can I look for more information about it?
Thanks in advance.
Regards.
--
Dario Graña
PIC (Port d'Informació Científica)
Campus UAB, Edificio D
E-08193 Bellaterra, Barcelona
http://www.pic.es
Avis - Aviso - Legal Notice: http://legal.ifae.es
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
