Hello, I've been facing the exact same issue today and its driving me equally crazy. I tried running prometheus as root but still:
level=error ts=2020-11-15T21:45:35.983Z caller=refresh.go:98 component="discovery manager scrape" discovery=dockerswarm msg="Unable to refresh target groups" err="error while listing swarm *services*: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" level=error ts=2020-11-15T21:45:35.984Z caller=refresh.go:98 component="discovery manager scrape" discovery=dockerswarm msg="Unable to refresh target groups" err="error while listing swarm *nodes*: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" I think I've exhausted all the options I could try by myself and would gladly appreciate any help at this point. Le dimanche 15 novembre 2020 à 21:54:26 UTC+1, Julien Pivotto a écrit : > Can you run prometheus as nobody:docker? > On 15 Nov 12:23, Carlos Colaço wrote: > > sorry .. also tried changing the permissions which changed nothing... > > > > ``` > > # chmod +r /var/run/docker.sock > > # ls -la /var/run/docker.sock > > srw-rw-r--. 1 root docker 0 Nov 15 20:12 /var/run/docker.sock > > # docker service update --force monitor_private > > ``` > > > > On Sunday, November 15, 2020 at 9:20:07 PM UTC+1 Carlos Colaço wrote: > > > > > Hi all .. Having the same issue... > > > > > > https://github.com/prometheus/prometheus/issues/8185 > > > > > > > > > Also don't think changing permissions on docker sock is a good option > .. > > > that way you are giving permissions to anyone to access it and that is > > > something not desirable ... > > > > > > What i also tried to do instead ... since prometheus runs as Nobody ( > uid: > > > 65534 ) ... i added it to the Docker group which changed nothing =/ > > > > > > Any hints or solutions for this? driving me crazy trying different > > > approaches and solutions.. nothing seems to work ... > > > > > > On Tuesday, August 11, 2020 at 7:03:12 AM UTC+2 [email protected] > wrote: > > > > > >> Thanks Julien and Tom, > > >> > > >> I got the problem which i was facing, actually when we change the > > >> permissions to read-write for docker.sock, permissions are only > changed > > >> till the docker daemon or docker service is restarted. Once the > > >> docker/daemon is restarted then the permissions for docker sock > changes > > >> back to the original one. > > >> > > >> Is there any way using which we can make permanent changes for the > > >> permission of docker.sock or do we need to file a issue for the same, > as > > >> docker/daemon might be restarted for various reasons > > >> > > >> > > >> ? > > >> > > >> On Monday, 10 August 2020 12:40:17 UTC+5:30, Umang Goel wrote: > > >>> > > >>> Hello Julien, > > >>> > > >>> group_add is not allowed in docker swarm. Do you have any other > > >>> workaround for this? > > >>> > > >>> -- > > >>> Umang > > >>> > > >>> On Monday, 10 August 2020 12:20:51 UTC+5:30, Julien Pivotto wrote: > > >>>> > > >>>> > > >>>> Can you use: > > >>>> > > >>>> --group-add docker? > > >>>> > > >>>> or in compose v2 file: > > >>>> > > >>>> version: "2.4" > > >>>> services: > > >>>> prometheus: > > >>>> group_add: > > >>>> - docker > > >>>> > > >>>> > > >>>> On 09 Aug 22:48, Umang Goel wrote: > > >>>> > ls -l /var/run/docker.sock > > >>>> > > > >>>> > - srwxrw-rw- 1 root docker 0 Aug 7 11:31 /var/run/docker.sock > > >>>> after > > >>>> > making changes as per Tom, > > >>>> > > > >>>> > On Sunday, 9 August 2020 02:16:28 UTC+5:30, Julien Pivotto wrote: > > >>>> > > > > >>>> > > On 07 Aug 04:36, Umang Goel wrote: > > >>>> > > > Hello Tom, > > >>>> > > > > > >>>> > > > Even this is not working, I am still facing the same issue. > Can > > >>>> you help > > >>>> > > me > > >>>> > > > how did you implement it. > > >>>> > > > > >>>> > > > > >>>> > > What are you current permissions on the /var/run/docker.sock ? > > >>>> > > > > >>>> > > ls -l /var/run/docker.sock > > >>>> > > > > >>>> > > > > > >>>> > > > On Friday, 7 August 2020 16:47:23 UTC+5:30, Tom Kun wrote: > > >>>> > > > > > > >>>> > > > > Hello Umang, > > >>>> > > > > > > >>>> > > > > What are you current permissions on the > /var/run/docker.sock ? > > >>>> > > > > > > >>>> > > > > I faced the same issue, and to start and no rebuild the > > >>>> Prometheus > > >>>> > > image > > >>>> > > > > with the appropriate user. > > >>>> > > > > I put the rights to read and write the docker.socket. > > >>>> > > > > > > >>>> > > > > sudo chmod 766 /var/run/docker.sock > > >>>> > > > > > > >>>> > > > > I hope this gonna help you. > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On Friday, 7 August 2020 11:59:32 UTC+2, Umang Goel wrote: > > >>>> > > > >> > > >>>> > > > >> Hello Community, > > >>>> > > > >> > > >>>> > > > >> I tired using Docker Swarm Service Discovery in > prometheus, > > >>>> but > > >>>> > > facing > > >>>> > > > >> problems using it. I followed the docker swarm support > > >>>> documentation > > >>>> > > > >> <https://prometheus.io/docs/guides/dockerswarm/>. Created > a > > >>>> > > daemon.json > > >>>> > > > >> file and mounted /var/run/docker.sock in prometheus > container. > > >>>> > > Container is > > >>>> > > > >> giving permission denied error as prometheus is running as > > >>>> nobody and > > >>>> > > > >> doesn't have access to mounted /var/run/docker.sock. Below > is > > >>>> my > > >>>> > > > >> prometheus.yml. > > >>>> > > > >> Prometheus Version : v2.20.1 > > >>>> > > > >> > > >>>> > > > >> prometheus: > > >>>> > > > >> image: prom/prometheus > > >>>> > > > >> networks: > > >>>> > > > >> - monitor > > >>>> > > > >> ports: > > >>>> > > > >> - "9090:9090" > > >>>> > > > >> command: > > >>>> > > > >> - '--config.file=/etc/prometheus/prometheus.yml' > > >>>> > > > >> - '--storage.tsdb.path=/prometheus' > > >>>> > > > >> - > > >>>> '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-24h}' > > >>>> > > > >> volumes: > > >>>> > > > >> - prometheus:/prometheus > > >>>> > > > >> - /home/efs/devops/dsm:/etc/prometheus:ro > > >>>> > > > >> - /var/run/docker.sock:/var/run/docker.sock:ro > > >>>> > > > >> deploy: > > >>>> > > > >> mode: replicated > > >>>> > > > >> replicas: 1 > > >>>> > > > >> resources: > > >>>> > > > >> limits: > > >>>> > > > >> memory: 1024M > > >>>> > > > >> reservations: > > >>>> > > > >> memory: 128M > > >>>> > > > >> > > >>>> > > > >> Prometheus.yml > > >>>> > > > >> > > >>>> > > > >> scrape_configs: > > >>>> > > > >> - job_name: 'docker' > > >>>> > > > >> dockerswarm_sd_configs: > > >>>> > > > >> - host: unix:///var/run/docker.sock > > >>>> > > > >> role: nodes > > >>>> > > > >> > > >>>> > > > >> Error: > > >>>> > > > >> [email protected] > > >>>> <javascript:> | > > >>>> > > level=error > > >>>> > > > >> ts=2020-08-06T07:21:19.106Z caller=refresh.go:98 > > >>>> component="discovery > > >>>> > > > >> manager scrape" discovery=dockerswarm msg="Unable to > refresh > > >>>> target > > >>>> > > groups" > > >>>> > > > >> err="error while listing swarm nodes: Got permission > denied > > >>>> while > > >>>> > > trying to > > >>>> > > > >> connect to the Docker daemon socket at > > >>>> unix:///var/run/docker.sock: > > >>>> > > Get > > >>>> > > > >> \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/nodes\": dial > unix > > >>>> > > > >> /var/run/docker.sock: connect: permission denied > > >>>> > > > >> > > >>>> > > > > > > >>>> > > > > > >>>> > > > -- > > >>>> > > > You received this message because you are subscribed to the > > >>>> Google > > >>>> > > Groups "Prometheus Users" group. > > >>>> > > > To unsubscribe from this group and stop receiving emails from > it, > > >>>> send > > >>>> > > an email to [email protected] <javascript:>. > > >>>> > > > To view this discussion on the web visit > > >>>> > > > > >>>> > https://groups.google.com/d/msgid/prometheus-users/e5e55a73-7cc1-4c0c-99e3-0a09270df62bo%40googlegroups.com. > > > > >>>> > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > -- > > >>>> > > Julien Pivotto > > >>>> > > @roidelapluie > > >>>> > > > > >>>> > > > >>>> > -- > > >>>> > You received this message because you are subscribed to the > Google > > >>>> Groups "Prometheus Users" group. > > >>>> > To unsubscribe from this group and stop receiving emails from it, > > >>>> send an email to [email protected]. > > >>>> > To view this discussion on the web visit > > >>>> > https://groups.google.com/d/msgid/prometheus-users/e5614621-f57a-466e-befd-269bf77d69c8o%40googlegroups.com. > > > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Julien Pivotto > > >>>> @roidelapluie > > >>>> > > >>> > > > > -- > > You received this message because you are subscribed to the Google > Groups "Prometheus Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/e058c64f-3db4-45c2-9550-c8db557d2a2cn%40googlegroups.com > . > > > -- > Julien Pivotto > @roidelapluie > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com.
