GitHub user Zhuoxi2000 edited a discussion: [Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)
## Motivation #900 defines how Agent Traces are recorded in the Event Log, and deliberately lists OTel export as a non-goal of the recording scope. In that thread, @joeyutong agreed that OpenTelemetry export should be a supported follow-up once the recording contract settles, with the exact GenAI attribute mapping to be worked out as part of the exporter design. This discussion is that follow-up: a config-gated exporter that maps recorded Execution Events to the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), so any OTLP-compatible backend (Jaeger, Grafana Tempo, Langfuse, Phoenix, vendor APMs) can visualize and analyze agent runs without flink-agents shipping its own UI. ## Proposed mapping Trace topology, aligned with the #900 recording model: | flink-agents | OTel GenAI | |---|---| | Input run (`input_run_id`) | Trace (one run = one trace) | | Execution (`execution_id` / `parent_execution_id`) | Span / parent Span | | Agent execution | span `invoke_agent {agent_name}`; `gen_ai.operation.name = invoke_agent`, `gen_ai.agent.name` | | Tool execution | span `execute_tool {tool_name}`; `gen_ai.operation.name = execute_tool`, `gen_ai.tool.name` | | Model call | span `chat {model}`; `gen_ai.operation.name = chat`, `gen_ai.request.model`, `gen_ai.provider.name` | | `business_key` | `gen_ai.conversation.id` | | Lifecycle events (started/finished) | span start/end timestamps | | `status` / `problem_category` | span status + `error.type` | | Token usage | `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` | **ID derivation.** Framework IDs stay opaque, as preferred in #900; the exporter derives fixed-width OTel IDs deterministically, e.g. `trace_id = truncate(SHA-256(input_run_id), 128 bits)` and `span_id = truncate(SHA-256(execution_id), 64 bits)`. Deterministic derivation keeps re-exports idempotent. ## Design points to agree on 1. **Where the exporter runs.** Two candidate shapes: - **(a) In-process**: an Execution Event listener in the runtime feeds an async OTLP exporter (batching, bounded queue, drop-on-overflow so it can never backpressure the job). - **(b) Out-of-band**: a separate consumer reads the Event Log and exports — zero impact on the running job, works retroactively, but requires event-log trace recording (#924) to be enabled. I lean toward (b) first, since it derives everything from the recording contract and adds no runtime risk, keeping (a) as a later option. Opinions welcome. 2. **Config surface.** `event-log.trace.otel.enabled` (default `false`, mirroring the recording flag), OTLP endpoint/protocol/headers, `service.name`. Content capture (prompt/response bodies) stays off by default and is opted into separately — gen_ai content capture has privacy and volume implications. 3. **Semconv version.** The `gen_ai.*` conventions are still `development` stability; we should pin the exported `schema_url` and document the target version. I'm engaging in the OTel GenAI SIG, so we can track upstream changes as they stabilize. 4. **Scope for 0.4.** Phase 1 = Java, spans-only, out-of-band exporter + config; Phase 2 = usage-metrics enrichment; Phase 3 = Python parity. Is Phase 1 realistic for the 0.4 feature freeze (Sept 15)? ## What I'd deliver I've already implemented the inverse mapping (OTel GenAI → storage) in a side project, so the attribute table above is grounded in working code. If maintainers agree with the general shape, I'll break this into concrete issues and start with Phase 1. --- **Updates (2026-08-07)** — incorporating the review feedback above: **Rejected alternative: Flink's `OpenTelemetryTraceReporterFactory`.** Flink 2.3 ships a native OTel trace reporter, but it cannot serve as the export path for agent traces: it does not preserve caller-derived trace/span ids or span status, it requires related spans to be reported together, and it is not consistently available across the 1.20 / 2.0 / 2.1 distributions this project ships for. A standalone converter avoids all three constraints. **Scope clarifications:** - Phase 1 is a **batch, offline converter** over already-collected Event Log files — not a continuous tailer. Durability, partition discovery, checkpoints, and retention are properties of a durable Event Log sink, which deserves its own design track; a continuous OTel consumer would build on that. Delivery is at-least-once; deterministic ids make duplicates identifiable, and idempotent coalescing is a backend property. - Since the converter is an external process, its settings are plain converter arguments (endpoint, protocol, service.name) — the earlier `event-log.trace.otel.*` naming is dropped. - The contract is the serialized Event Log record, which both runtimes produce through the same writer, so **one converter serves both Java and Python agents**. "Java" refers only to the converter's implementation language; there is no separate Python-parity phase. - The converter lives in an **optional standalone module that is not bundled into dist**, with its OTel dependencies aligned through the `opentelemetry-bom`, keeping them off the distribution classpath entirely. - Attribute-quality gaps (resolved model vs resource alias, provider identity, per-call token usage) are best addressed by recording them in `entityMetadata` at recording time, as raised above — tracked as part of the recording-contract conversation on #924. GitHub link: https://github.com/apache/flink-agents/discussions/929 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
