Ma77Ball commented on code in PR #5377:
URL: https://github.com/apache/texera/pull/5377#discussion_r4358422121
##########
bin/single-node/docker-compose.yml:
##########
@@ -530,6 +530,189 @@ services:
command: >
sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash
/examples/load-examples.sh'
+ # ========================================================================
+ # Part 5: Observability stack (PR 6).
Review Comment:
Removed the PR-number notes; the comments now only say what each service
does.
##########
bin/single-node/docker-compose.yml:
##########
@@ -530,6 +530,189 @@ services:
command: >
sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash
/examples/load-examples.sh'
+ # ========================================================================
+ # Part 5: Observability stack (PR 6).
+ #
+ # All six services live in this single compose file (no separate
+ # observability compose file, no monolithic observability profile).
+ # Per-signal profiles let an operator drop one without disturbing
+ # the others — see bin/single-node/up.sh and bin/single-node/.env
+ # (COMPOSE_PROFILES default includes every observability profile so
+ # `docker compose up` runs the whole stack).
+ #
+ # Network posture: every receiver/HTTP port binds to loopback
+ # (127.0.0.1) on the host or stays inside the texera-single-node
+ # bridge network entirely. No 0.0.0.0 host bindings, no Ingress.
+ # ========================================================================
+
+ # OpenTelemetry Collector — the single OTLP ingress for all three
+ # signals. Reads bin/observability/otel-collector/config.yaml.
+ # Apache-2.0; pinned to the contrib distribution.
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.153.0
+ container_name: texera-otel-collector
+ profiles: [observability-collector]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ -
../observability/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro,z
Review Comment:
The release workflow now archives bin/observability next to bin/single-node
and rewrites the ../observability mount paths to ./observability in the bundle,
so the collector config ships and the mount resolves.
##########
bin/single-node/.env:
##########
@@ -96,3 +96,48 @@ LLM_ENDPOINT=http://nginx:8080
TEXERA_DASHBOARD_SERVICE_ENDPOINT=http://dashboard-service:8080
WORKFLOW_COMPILING_SERVICE_ENDPOINT=http://workflow-compiling-service:9090
WORKFLOW_EXECUTION_SERVICE_ENDPOINT=http://workflow-runtime-coordinator-service:8085
+
+# ============================================================================
+# Observability (PR 6)
+# ============================================================================
+# Each backend lives behind its own docker-compose profile. By default
+# COMPOSE_PROFILES enables every observability profile so `docker compose up`
+# brings the whole stack online — there is no separate observability compose
+# file or monolithic observability profile.
+#
+# To disable a signal:
+# * Prefer using bin/single-node/up.sh, which translates the
+# TEXERA_OBSERVABILITY_* env vars below into the right COMPOSE_PROFILES.
+# * Or edit COMPOSE_PROFILES directly here.
+#
+# Disable env-var conventions (consumed by up.sh):
Review Comment:
Each signal is its own docker compose profile and COMPOSE_PROFILES is empty
by default, so a plain install runs only core Texera; observability is opt-in
by listing the profiles you want.
##########
bin/single-node/.env:
##########
@@ -96,3 +96,48 @@ LLM_ENDPOINT=http://nginx:8080
TEXERA_DASHBOARD_SERVICE_ENDPOINT=http://dashboard-service:8080
WORKFLOW_COMPILING_SERVICE_ENDPOINT=http://workflow-compiling-service:9090
WORKFLOW_EXECUTION_SERVICE_ENDPOINT=http://workflow-runtime-coordinator-service:8085
+
+# ============================================================================
+# Observability (PR 6)
+# ============================================================================
+# Each backend lives behind its own docker-compose profile. By default
+# COMPOSE_PROFILES enables every observability profile so `docker compose up`
+# brings the whole stack online — there is no separate observability compose
+# file or monolithic observability profile.
+#
+# To disable a signal:
+# * Prefer using bin/single-node/up.sh, which translates the
+# TEXERA_OBSERVABILITY_* env vars below into the right COMPOSE_PROFILES.
+# * Or edit COMPOSE_PROFILES directly here.
+#
+# Disable env-var conventions (consumed by up.sh):
+# TEXERA_OBSERVABILITY_LOGS=disabled drops victorialogs
+# TEXERA_OBSERVABILITY_METRICS=disabled drops victoriametrics
+# TEXERA_OBSERVABILITY_TRACES=disabled drops jaeger
+# TEXERA_OBSERVABILITY_PROFILES=disabled drops parca + parca-agent
+# TEXERA_OBSERVABILITY_COLLECTOR=disabled drops the otel-collector (rare)
Review Comment:
Dropped the custom TEXERA_OBSERVABILITY_* variables in favor of docker
compose's native COMPOSE_PROFILES, which both launchers already pass through,
and fixed a leftover mention of the old variable in parca-agent.env.
##########
bin/single-node/.env:
##########
@@ -96,3 +96,48 @@ LLM_ENDPOINT=http://nginx:8080
TEXERA_DASHBOARD_SERVICE_ENDPOINT=http://dashboard-service:8080
WORKFLOW_COMPILING_SERVICE_ENDPOINT=http://workflow-compiling-service:9090
WORKFLOW_EXECUTION_SERVICE_ENDPOINT=http://workflow-runtime-coordinator-service:8085
+
+# ============================================================================
+# Observability (PR 6)
+# ============================================================================
+# Each backend lives behind its own docker-compose profile. By default
+# COMPOSE_PROFILES enables every observability profile so `docker compose up`
+# brings the whole stack online — there is no separate observability compose
+# file or monolithic observability profile.
+#
+# To disable a signal:
+# * Prefer using bin/single-node/up.sh, which translates the
+# TEXERA_OBSERVABILITY_* env vars below into the right COMPOSE_PROFILES.
+# * Or edit COMPOSE_PROFILES directly here.
+#
+# Disable env-var conventions (consumed by up.sh):
+# TEXERA_OBSERVABILITY_LOGS=disabled drops victorialogs
+# TEXERA_OBSERVABILITY_METRICS=disabled drops victoriametrics
+# TEXERA_OBSERVABILITY_TRACES=disabled drops jaeger
+# TEXERA_OBSERVABILITY_PROFILES=disabled drops parca + parca-agent
+# TEXERA_OBSERVABILITY_COLLECTOR=disabled drops the otel-collector (rare)
+#
+# Parca eBPF agent needs Linux + privileged container; macOS/Windows
+# developers must set TEXERA_OBSERVABILITY_PROFILES=disabled.
+COMPOSE_PROFILES=observability-collector,observability-logs,observability-metrics,observability-traces,observability-profiles
+
+# Query endpoints the dashboard gateway calls to read each signal. The
+# defaults in observability-gateway.conf are host-local (127.0.0.1), which
+# is what a natively-run backend (sbt / IntelliJ) needs. Inside this compose
+# the gateway runs in a container, so it must reach the backends by their
+# bridge-network service names — these overrides do that.
+TEXERA_OBS_LOGS_URL=http://victorialogs:9428
+TEXERA_OBS_METRICS_URL=http://victoriametrics:8428
+TEXERA_OBS_TRACES_URL=http://jaeger:16686
+TEXERA_OBS_PROFILES_URL=http://parca:7070
+
+# Emission side: the Scala services (via OtelInit) push OTLP to the bundled
+# collector over the bridge network. OtelInit validates the endpoint host
+# against an allowlist (localhost only by default), so the collector's compose
+# service name has to be allowlisted explicitly, or the SDK rejects it and
+# emits nothing. These are read by every service that loads this .env; only the
+# OtelInit-enabled Scala services act on them. When the observability-collector
+# profile is disabled the collector is absent and OtelInit fails quietly
Review Comment:
Telemetry is turned on or off explicitly through OTEL_SDK_DISABLED (disabled
by default), not by whether the collector connection succeeds, so a failed
connection never silently disables it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
