I am building a demo application to show off Prometheus Exemplars and OpenTelemetry Tracing abilities with Java / Spring applications. However, I cannot get exemplars to appear in the metrics HTTP endpoint. I'm using the following curl command and I get metrics in the OpenMetrics format, but no exemplars.
$ curl -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' http://localhost:8081/metrics I've got the Prometheus client_java 0.12.0 loaded and I'm using that to directly instrument. In build.gradle: dependencies { ... implementation 'io.prometheus:simpleclient:0.12.0' implementation 'io.prometheus:simpleclient_hotspot:0.12.0' implementation 'io.prometheus:simpleclient_httpserver:0.12.0' implementation 'io.prometheus:simpleclient_tracer_otel:0.12.0' implementation 'io.prometheus:simpleclient_tracer_otel_agent:0.12.0' implementation('io.opentelemetry:opentelemetry-api:1.0.0') implementation('io.opentelemetry:opentelemetry-extension-annotations:1.0.0') } I setup my Prometheus metrics in the class constructor: prometheusErrors = io.prometheus.client.Counter.build() .namespace("custommetricsdemo") .name("errors") .help("Test Prometheus Client Library errors counter") .withExemplars() .register(); prometheusTimer = io.prometheus.client.Summary.build() .namespace("custommetricsdemo") .name("latency_timer") .help("Test Prometheus Client Library latency summary") .quantile(0.5, 0.01) .quantile(0.95, 0.01) .register(); prometheusHistogram = io.prometheus.client.Histogram.build() .namespace("custommetricsdemo") .name("histogram") .help("Test Prometheus Client Library latency histogram") .withExemplars() .register(); try { io.prometheus.client.exporter.HTTPServer server = new io.prometheus.client.exporter.HTTPServer(8081); } catch (Exception e) { logger.error("Failed to setup Prometheus HTTP server", e); } This code runs a ScheduledTask at a random interval which sleeps for a random time. Thanks to running with the OpenTelemetry Agent, I can see trace_id and span_id in the logback logs. That part works well. I can also query the span IDs in code: logger.info("{}, {}", span.getSpanContext().getSpanId(), span.getSpanContext().getTraceId()); I could not get the auto-exemplars to work, so I began to observe them in code: prometheusErrors.incWithExemplar("span_id", span.getSpanContext().getSpanId(), "trace_id", span.getSpanContext().getTraceId()); I also tested with different exemplar tag names: prometheusHistogram.observeWithExemplar(sw.getTotalTimeSeconds(), "span_foo", span.getSpanContext().getSpanId(), "trace_bar", span.getSpanContext().getTraceId()); However, no exemplars ever appear in the output. $ curl -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' http://localhost:8081/metrics # TYPE custommetricsdemo_histogram histogram # HELP custommetricsdemo_histogram Test Prometheus Client Library latency histogram custommetricsdemo_histogram_bucket{le="0.005"} 3.0 custommetricsdemo_histogram_bucket{le="0.01"} 3.0 custommetricsdemo_histogram_bucket{le="0.025"} 3.0 custommetricsdemo_histogram_bucket{le="0.05"} 4.0 custommetricsdemo_histogram_bucket{le="0.075"} 4.0 custommetricsdemo_histogram_bucket{le="0.1"} 4.0 custommetricsdemo_histogram_bucket{le="0.25"} 6.0 custommetricsdemo_histogram_bucket{le="0.5"} 9.0 custommetricsdemo_histogram_bucket{le="0.75"} 11.0 custommetricsdemo_histogram_bucket{le="1.0"} 11.0 custommetricsdemo_histogram_bucket{le="2.5"} 11.0 custommetricsdemo_histogram_bucket{le="5.0"} 11.0 custommetricsdemo_histogram_bucket{le="7.5"} 11.0 custommetricsdemo_histogram_bucket{le="10.0"} 11.0 custommetricsdemo_histogram_bucket{le="+Inf"} 11.0 custommetricsdemo_histogram_count 11.0 custommetricsdemo_histogram_sum 2.732210076 custommetricsdemo_histogram_created 1.632231018649E9 What am I missing here? Is something interfering with the Spring Boot libraries perhaps? Jack Neely -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/7a09c0bc-a002-4643-a8fa-8cb6a24db87dn%40googlegroups.com.