GitHub user pavelekshin created a discussion: Airflow 2.10.X and standalone OpenTelemetry Collector integration
I'm trying to integrate Airflow 2.10.5 with a standalone OpenTelemetry Collector version 0.70, following the suggested setup script https://github.com/apache/airflow/blob/2.10.2/scripts/ci/docker-compose/integration-otel.yml. However, I'm encountering a 400 Bad Request error. The OpenTelemetry Collector logs don't show any entries related to the HTTP request. Any ideas on what might be going wrong or how to debug this further? ```logs [Metric Exporter] Connecting to OpenTelemetry Collector at http://otel-collector:4318/v1/metrics [2026-03-17T10:16:03.577+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request [2026-03-17T10:16:03.698+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request [2026-03-17T10:16:08.585+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request [2026-03-17T10:16:08.708+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request ``` I also try gRPC on 4317, but airflow continuously send HTTP requests ```logs Airflow side: [2026-03-17T09:06:14.263+0000] {connectionpool.py:241} DEBUG - Starting new HTTP connection (1): otel-collector:4317 [2026-03-17T09:06:14.267+0000] {connectionpool.py:241} DEBUG - Starting new HTTP connection (2): otel-collector:4317 [2026-03-17T09:06:14.267+0000] {__init__.py:266} WARNING - Transient error ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) encountered while exporting metrics batch, retrying in 0.98s Otel side: Closing: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/metrics HTTP/1.\"" ``` This works and return 200, but nothing on otel logs ```logs curl -v -X POST http://otel-collector:4318/v1/metrics -H "Content-Type: application/json" -d '{"resourceMetrics": [{"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "test"}}]}}]}' ``` I use following airflow ENV: ```env - AIRFLOW__METRICS__OTEL_ON=True - AIRFLOW__METRICS__OTEL_HOST=otel-collector - AIRFLOW__METRICS__OTEL_PORT=4318 - AIRFLOW__METRICS__OTEL_INTERVAL_MILLISECONDS=30000 - AIRFLOW__TRACES__OTEL_ON=True - AIRFLOW__TRACES__OTEL_HOST=otel-collector - AIRFLOW__TRACES__OTEL_PORT=4318 - AIRFLOW__TRACES__OTEL_DEBUGGING_ON=False - AIRFLOW__TRACES__OTEL_TASK_LOG_EVENT=True ``` and installed otel plugin via apache-airflow[otel] ```bash opentelemetry-api 1.40.0 opentelemetry-exporter-otlp 1.40.0 opentelemetry-exporter-otlp-proto-common 1.40.0 opentelemetry-exporter-otlp-proto-grpc 1.40.0 opentelemetry-exporter-otlp-proto-http 1.40.0 opentelemetry-exporter-prometheus 0.61b0 opentelemetry-proto 1.40.0 opentelemetry-sdk 1.40.0 opentelemetry-semantic-conventions 0.61b0 ``` otel/prometheus setup ```yml otel-collector: image: otel/opentelemetry-collector-contrib:0.70.0 command: [--config=/etc/otel-collector-config.yml] volumes: - ./otel-collector-config.yml:/etc/otel-collector-config.yml ports: - "24318:4318" - "28889:8889" prometheus: image: prom/prometheus user: "0" ports: - "29090:9090" volumes: - ./prometheus-config.yml:/etc/prometheus/prometheus.yml ``` otel config: ```yml receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 grpc: endpoint: 0.0.0.0:4317 processors: batch: exporters: logging: verbosity: detailed prometheus: endpoint: 0.0.0.0:8889 service: telemetry: logs: level: debug development: true pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [logging, prometheus] traces: receivers: [otlp] processors: [batch] exporters: [logging] ``` I also tested with the latest OpenTelemetry Collector (v0.146). The error changed from 400 to 500 Internal Server Error, but the collector logs remain completely empty. No entries appear even with debug logging enabled." Also tried to manipulate exporter settings on airflow using protocol settings - https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/ OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, etc but without any success GitHub link: https://github.com/apache/airflow/discussions/63808 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
