Interesting! This comment helped me a lot. Under the new assumption that the address was wrong, I searched where cadvisor metrics could be scraped for prometheus in GKE. I found a blog post with an example kubernetes-cadvisor config that worked for me.
https://medium.com/htc-research-engineering-blog/monitoring-kubernetes-clusters-with-grafana-e2a413febefd config pasted below for posterity: ``` - job_name: 'kubernetes-cadvisor' scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc.cluster.local:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor ``` Now I see the metrics appearing in prometheus. Thanks again for the hint! Now on to trying to get the grafana dashboards working. No rest for the wicked. On Tuesday, September 1, 2020 at 10:38:07 AM UTC-7 b.ca...@pobox.com wrote: > :10255 is on a test server running microk8s snap: > > root@nuc1:~# netstat -natp | grep kubelet > tcp 0 0 127.0.0.1:10248 0.0.0.0:* > LISTEN 4129/kubelet > tcp 0 0 127.0.0.1:37888 127.0.0.1:16443 > ESTABLISHED 4129/kubelet > tcp6 0 0 :::10255 :::* > LISTEN 4129/kubelet > tcp6 0 0 :::10250 :::* > LISTEN 4129/kubelet > root@nuc1:~# > > Looking at a production k8s 1.16 cluster (installed using kubectl) I don't > see this port open. And I'm afraid I don't know where it's configured, > even in microk8s. > > I do note that the kubelet command line includes "--insecure-port=0", but > I don't think that's it; even with a reboot it comes up on the same ports. > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ee8054df-a6a5-45c2-a9b3-b9383efedf59n%40googlegroups.com.
