+1 with OTEL implementation of MetricsReporter, but have you considered a span-based implementation instead of/in addition to a metrics-based implementation?
High cardinality metrics should be avoided and (schema_name, table_name) attributes can be high cardinality depending on your workload. Spans do not have problems with high cardinality. For context, we built a metrics-based MetricsReporter, ran into high cardinality cost issues with thousands of tables, then switched to a span-based MetricsReporter. On Mon, May 25, 2026 at 2:08 AM Noritaka Sekiyama via dev < [email protected]> wrote: > Hi JB, and all, > > Thanks for the suggestion. Pushed efc48d429 which adds an > OtelMetricsReporter section to docs/docs/metrics-reporting.md. It documents > the host's responsibility for packaging the OpenTelemetry API, SDK, and a > metric exporter (Gradle plus a spark-submit --packages example), the > programmatic SDK registration path, exporter-wiring examples for the > OpenTelemetry Collector, Prometheus (pull and push), and Amazon CloudWatch > via the sigv4auth Collector extension, plus the emitted metric names and > attribute set. > > Verified end-to-end against the Prometheus pull pattern from the docs > (host SDK with PrometheusHttpServer + OtelMetricsReporter reporting > synthetic ScanReport/CommitReport, all 12 iceberg.* series visible on > /metrics with the documented attribute set); each Collector YAML in the > docs was otelcol-contrib validate-checked. > > Looking forward to your PR review. > > Thanks, > Nori > > On Mon, May 25, 2026 at 3:00 PM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi, >> >> I think this is a great proposal. >> >> I would suggest documenting how to package the exporter, as I believe it >> is up to the user to package the specific OpenTelemetry exporter they need. >> >> I will take a look at the PR. >> >> Regards, >> JB >> >> On Thu, May 21, 2026 at 3:39 AM Noritaka Sekiyama via dev < >> [email protected]> wrote: >> >>> Hi all, >>> >>> I'd like to propose adding an OpenTelemetry-based MetricsReporter to >>> iceberg-core that exports ScanReport and CommitReport to any OTLP-compatible >>> backend. >>> >>> # Background >>> Iceberg ships three built-in MetricsReporter implementations today: >>> LoggingMetricsReporter, InMemoryMetricsReporter (Spark-internal), and >>> RESTMetricsReporter (REST catalog only). >>> None of them give users an out-of-the-box way to ship scan/commit >>> metrics to an external observability platform. >>> The gap applies to Spark users on non-REST catalogs and to all non-Spark >>> engines (Trino, Flink, etc.). >>> >>> # Motivation >>> OpenTelemetry is the vendor-neutral CNCF standard for telemetry, >>> supported by every major observability backend (Prometheus, CloudWatch, >>> Datadog, Grafana Cloud, etc.). >>> A single OTLP-based MetricsReporter in Iceberg lets users reach all of >>> these without per-vendor integrations in the project. >>> This is complementary to #14360, which adds OTel support to HTTPClient >>> at the REST-catalog HTTP layer; this proposal covers the Iceberg-level >>> ScanReport / CommitReport layer. >>> >>> # Proposal >>> Issue: https://github.com/apache/iceberg/issues/16169 >>> PR: https://github.com/apache/iceberg/pull/16250 >>> >>> The reporter follows the same SDK-ownership philosophy as #14360 - the >>> host application (Spark/Flink/Trino/...) registers an OpenTelemetrySdk via >>> GlobalOpenTelemetry, and the reporter just looks up a Meter from it. >>> The reporter has zero Iceberg-specific catalog properties; everything >>> else is owned by the host. >>> >>> The PR has been validated end-to-end against two unrelated OTLP backends >>> (Databricks Zerobus and Amazon CloudWatch) - full procedures and queries >>> are linked from the PR. >>> >>> # On dependencies >>> Given the current sensitivity around new runtime dependencies in 1.11, >>> the PR adds only opentelemetry-api to iceberg-core as compileOnly. >>> The OpenTelemetry SDK and OTLP exporters are not added to the runtime >>> classpath >>> - they come from the host application. >>> opentelemetry-sdk / -sdk-testing are testImplementation only. >>> >>> # Questions for the community >>> >>> Q1. Any objection to taking the opentelemetry-api compileOnly >>> dependency in iceberg-core? >>> Q2. Module placement: iceberg-core (current PR), or a >>> separate iceberg-opentelemetry module? >>> >>> Thanks, >>> Noritaka Sekiyama, Databricks >>> >>
