Brian, Thank you, that was very helpful - this is the config I ended up with (copy/paste breaks the yaml spacing): scrape_configs: - job_name: 'blackbox-ba-[% key "global/config/env" %]' metrics_path: /probe params: target: ['mineralsoft.dev.ba.drillinginfo.com'] consul_sd_configs: - services: ['blackbox-exporter'] relabel_configs: - source_labels: [__param_target] target_label: instance - source_labels: ['__meta_consul_address','__meta_consul_service_port'] separator: ':' target_label: __address__
On Wednesday, November 11, 2020 at 3:58:46 PM UTC-6 Brian Brazil wrote: > On Wed, 11 Nov 2020 at 21:46, david goodine <[email protected]> wrote: > >> >> Hi, >> We are running blackbox exporter as a Nomad allocation, so its ip:port is >> dynamic. We had been running a prometheus instance, also in Nomad, and it >> was configured with consul-template to dynamically set the __address__: >> >> - target_label: __address__ >> replacement: [% range $i, $s := service "blackbox-exporter" >> %][% if eq $i 0 %][%.Address%]:[%.Port%][% end %][% end %] >> >> Now, we have migrated to using grafana cloud agent instead of a >> standalone prometheus. We run the agent on every Nomad client, to collect >> logs and metrics with its built in integrations. I want to update the >> config so that it will spawn a prometheus instance to scrape any local >> blackbox exporter services. However, instead of probing the URL that I >> expect, I get these targets: (from curl of http://<grafana agent >> ip>:53000/agent/api/v1/targets) >> >> { >> "instance": "f5d8368b42ae35f965c73d1be161364c", >> "target_group": "blackbox-ba-dev", >> "endpoint": "http://<bbexip>:21699/probe?target=<bbexip>%3A21699", >> "state": "up", >> "labels": { >> "instance": "10.25.194.212:21699", >> "job": "blackbox-ba-dev" >> }, >> "last_scrape": "2020-11-11T21:31:30.45482873Z", >> "scrape_duration_ms": 1, >> "scrape_error": "" >> }, >> { >> "instance": "f5d8368b42ae35f965c73d1be161364c", >> "target_group": "blackbox-ba-dev", >> "endpoint": "http://:/probe?target= >> mineralsoft.dev.ba.drillinginfo.com", >> "state": "down", >> "labels": { >> "instance": "mineralsoft.dev.ba.drillinginfo.com", >> "job": "blackbox-ba-dev" >> }, >> "last_scrape": "2020-11-11T21:31:29.843730608Z", >> "scrape_duration_ms": 0, >> "scrape_error": "Get \"http:///probe?target= >> mineralsoft.dev.ba.drillinginfo.com\": http: no Host in request URL" >> } >> >> What I need is a single target, with endpoint = http://<bbexip>:<bbex >> port>/probe?target=mineralsoft.dev.ba.drillinginfo.com >> >> Here is my prometheus config: >> prometheus: >> wal_directory: /tmp/wal >> global: >> scrape_interval: 60s >> configs: >> - name: 'ba-[% key "global/config/env" %]-prometheus-gca' >> host_filter: false >> remote_write: >> - url: >> https://prometheus-us-central1.grafana.net/api/prom/push >> basic_auth: >> [%with secret "di-secrets/prometheus/credentials"%] >> username: [%.Data.data.USERNAME%] >> password: [%.Data.data.APIKEY%] >> [%end%] >> scrape_configs: >> - job_name: 'blackbox-ba-[% key "global/config/env" %]' >> > > Job labels should not vary by environment, you should use external labels > for this. > > >> metrics_path: /probe >> static_configs: >> - targets: >> - mineralsoft.dev.ba.drillinginfo.com # Target to >> probe with http. >> > consul_sd_configs: >> - services: ['blackbox-exporter'] >> > > You can't mix SDs like this, they get all merged together. Use this like a > normal exporter, remove the static_configs and put the single URL to be > probed in as a params. > > Brian > > >> relabel_configs: >> - source_labels: [__address__] >> target_label: __param_target >> - source_labels: [__param_target] >> target_label: instance >> - source_labels: >> ['__meta_consul_address','__meta_consul_service_port'] >> separator: ':' >> target_label: __address__ >> >> >> Does anyone see what I am doing wrong? >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/f4a4b496-3c1a-4cd9-890d-0496ca09ee98n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/f4a4b496-3c1a-4cd9-890d-0496ca09ee98n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Brian Brazil > www.robustperception.io > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/24852988-870b-432c-9f6f-de17504d4f43n%40googlegroups.com.
