Hello All,
I recently installed Ceph (v 16.2.4 pacific stable). I know Ceph creates
and exposes two Prometheus instances (from what I've witnessed).
To that end, I installed Grafana in a docker container, and am
attempting to pull metrics from Ceph (Cluster Health, OSD information,
etc), but I'm running into issues.
If I setup a Prometheus data source to $ServerIP:9100 (Ceph internal
metrics, CPU/RAM usage, etc), no issue arises. It automatically parses
the URL to http://$ServerIP:9100/api/v1/query, which is expected. If I
go to that URL and look at the /metrics endpoint, I can see the raw metrics.
If I setup a Prometheus data source to $ServerIP:9283 (Ceph Cluster
Health, OSD info, etc), Grafana produces an error in the Web UI stating
"Unknown error during query transaction. Please check JS console logs.".
Now, I decided to enable debug logging in Grafana, and checked the logs
during setup of both the data source into Ceph internal metrics, and
Ceph Cluster Health metrics. The only difference is that it shoots out a
301 status, which doesn't make sense, because the URLs are the same, as
are the /metrics endpoints.
Log Excerpts are below:
*URL pointing to Ceph Node Metrics:*
t=2021-06-17T16:09:08+0000 lvl=dbug msg="Received command to update data
source" logger=datasources url=http://$ServerIP:9100/
t=2021-06-17T16:09:08+0000 lvl=dbug msg="Applying default URL parsing
for this data source type" logger=datasource type=prometheus
url=http://$ServerIP:9100/
t=2021-06-17T16:09:08+0000 lvl=dbug msg="Querying for data source via
SQL store" logger=datasources id=1 orgId=1
t=2021-06-17T16:09:08+0000 lvl=dbug msg="Applying default URL parsing
for this data source type" logger=datasource type=prometheus
url=http://$ServerIP:9100/
t=2021-06-17T16:09:08+0000 lvl=info msg=Requesting logger=data-proxy-log
url=http://172.16.168.3:9100/api/v1/query
*URL Pointing to Ceph Cluster Metrics:*
sg="Received command to update data source" logger=datasources
url=http://$ServerIP:9283
t=2021-06-17T16:06:32+0000 lvl=dbug msg="Applying default URL parsing
for this data source type" logger=datasource type=prometheus
url=http://$ServerIP:9283
t=2021-06-17T16:06:33+0000 lvl=dbug msg="Querying for data source via
SQL store" logger=datasources id=1 orgId=1
t=2021-06-17T16:06:33+0000 lvl=dbug msg="Applying default URL parsing
for this data source type" logger=datasource type=prometheus
url=http://$ServerIP:9283
t=2021-06-17T16:06:33+0000 lvl=info msg=Requesting logger=data-proxy-log
url=http://$ServerIP:9283/api/v1/query
t=2021-06-17T16:06:33+0000 lvl=info msg="Request Completed"
logger=context userId=1 orgId=1 uname=admin method=POST
path=/api/datasources/proxy/1/api/v1/query status=301 remote_addr=$LANIP
time_ms=3 size=131 referer=http://$ServerIP:3001/datasources/edit/yh9QZAg7k
I've ran out of ideas to sort this, because I'd really like to get
Grafana to show Ceph cluster metrics.
Thanks,
Preston
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
