Ma77Ball commented on code in PR #5377: URL: https://github.com/apache/texera/pull/5377#discussion_r4233113961
########## bin/single-node/.env: ########## @@ -96,3 +96,48 @@ LLM_ENDPOINT=http://nginx:8080 TEXERA_DASHBOARD_SERVICE_ENDPOINT=http://dashboard-service:8080 WORKFLOW_COMPILING_SERVICE_ENDPOINT=http://workflow-compiling-service:9090 WORKFLOW_EXECUTION_SERVICE_ENDPOINT=http://workflow-runtime-coordinator-service:8085 + +# ============================================================================ +# Observability (PR 6) +# ============================================================================ +# Each backend lives behind its own docker-compose profile. By default +# COMPOSE_PROFILES enables every observability profile so `docker compose up` +# brings the whole stack online — there is no separate observability compose +# file or monolithic observability profile. +# +# To disable a signal: +# * Prefer using bin/single-node/up.sh, which translates the +# TEXERA_OBSERVABILITY_* env vars below into the right COMPOSE_PROFILES. +# * Or edit COMPOSE_PROFILES directly here. +# +# Disable env-var conventions (consumed by up.sh): +# TEXERA_OBSERVABILITY_LOGS=disabled drops victorialogs +# TEXERA_OBSERVABILITY_METRICS=disabled drops victoriametrics +# TEXERA_OBSERVABILITY_TRACES=disabled drops jaeger +# TEXERA_OBSERVABILITY_PROFILES=disabled drops parca + parca-agent +# TEXERA_OBSERVABILITY_COLLECTOR=disabled drops the otel-collector (rare) +# +# Parca eBPF agent needs Linux + privileged container; macOS/Windows +# developers must set TEXERA_OBSERVABILITY_PROFILES=disabled. +COMPOSE_PROFILES=observability-collector,observability-logs,observability-metrics,observability-traces,observability-profiles + +# Query endpoints the dashboard gateway calls to read each signal. The +# defaults in observability-gateway.conf are host-local (127.0.0.1), which +# is what a natively-run backend (sbt / IntelliJ) needs. Inside this compose +# the gateway runs in a container, so it must reach the backends by their +# bridge-network service names — these overrides do that. +TEXERA_OBS_LOGS_URL=http://victorialogs:9428 +TEXERA_OBS_METRICS_URL=http://victoriametrics:8428 +TEXERA_OBS_TRACES_URL=http://jaeger:16686 +TEXERA_OBS_PROFILES_URL=http://parca:7070 + +# Emission side: the Scala services (via OtelInit) push OTLP to the bundled +# collector over the bridge network. OtelInit validates the endpoint host +# against an allowlist (localhost only by default), so the collector's compose +# service name has to be allowlisted explicitly, or the SDK rejects it and +# emits nothing. These are read by every service that loads this .env; only the +# OtelInit-enabled Scala services act on them. When the observability-collector +# profile is disabled the collector is absent and OtelInit fails quietly Review Comment: Emission is now gated by OTEL_SDK_DISABLED (default disabled), the OTel SDK's documented off switch, instead of connection failure. ########## bin/single-node/docker-compose.yml: ########## @@ -530,6 +530,189 @@ services: command: > sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash /examples/load-examples.sh' + # ======================================================================== + # Part 5: Observability stack (PR 6). + # + # All six services live in this single compose file (no separate + # observability compose file, no monolithic observability profile). + # Per-signal profiles let an operator drop one without disturbing + # the others — see bin/single-node/up.sh and bin/single-node/.env + # (COMPOSE_PROFILES default includes every observability profile so + # `docker compose up` runs the whole stack). + # + # Network posture: every receiver/HTTP port binds to loopback + # (127.0.0.1) on the host or stays inside the texera-single-node + # bridge network entirely. No 0.0.0.0 host bindings, no Ingress. + # ======================================================================== + + # OpenTelemetry Collector — the single OTLP ingress for all three + # signals. Reads bin/observability/otel-collector/config.yaml. + # Apache-2.0; pinned to the contrib distribution. + otel-collector: + image: otel/opentelemetry-collector-contrib:0.153.0 + container_name: texera-otel-collector + profiles: [observability-collector] + restart: always + user: "10001:10001" + read_only: true + security_opt: + - no-new-privileges:true + volumes: + - ../observability/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro,z + command: ["--config=/etc/otelcol-contrib/config.yaml"] + # Local dev override: publish OTLP receiver ports on the host + # loopback so a Scala backend running outside docker (sbt / + # IntelliJ) can emit telemetry. In compose-only deploys, services + # talk to otel-collector:4317/:4318 via the bridge network. + ports: + - "127.0.0.1:4317:4317" + - "127.0.0.1:4318:4318" + deploy: + resources: + limits: + memory: 768M + cpus: "1.0" + + # VictoriaLogs — Apache-2.0 log store. LogsQL query API on 9428. + victorialogs: + image: victoriametrics/victoria-logs:v1.50.0 + container_name: texera-victorialogs + profiles: [observability-logs] + restart: always + user: "10002:10002" + read_only: true + security_opt: + - no-new-privileges:true + command: + - "-storageDataPath=/data" + - "-retentionPeriod=30d" + - "-httpListenAddr=:9428" + volumes: + - victorialogs_data:/data + # Loopback-only host binding so an operator can curl the query + # API from the host for ad-hoc debugging without exposing it to + # the network. + ports: + - "127.0.0.1:9428:9428" + deploy: + resources: + limits: + memory: 1G + cpus: "1.0" + + # VictoriaMetrics — Apache-2.0 metrics store. Accepts Prometheus + # remote-write from the collector at /api/v1/write, MetricsQL on + # /api/v1/query. + victoriametrics: + image: victoriametrics/victoria-metrics:v1.144.0 + container_name: texera-victoriametrics + profiles: [observability-metrics] + restart: always + user: "10003:10003" + read_only: true + security_opt: + - no-new-privileges:true + command: + - "-storageDataPath=/data" + - "-retentionPeriod=90d" + - "-httpListenAddr=:8428" + volumes: + - victoriametrics_data:/data + ports: + - "127.0.0.1:8428:8428" + deploy: + resources: + limits: + memory: 1G + cpus: "1.0" + + # Jaeger v2 — Apache-2.0 traces backend + UI. v2 accepts OTLP + # natively. The query API lives at :16686, OTLP ingest at :4317 + # (kept inside the bridge network only — only the collector talks + # to it). + jaeger: + image: jaegertracing/jaeger:2.18.0 + container_name: texera-jaeger + profiles: [observability-traces] + restart: always + security_opt: + - no-new-privileges:true + # In-memory storage for the single-node deployment — restarts + # wipe traces. Any deploy needing trace persistence must swap + # in Cassandra or OpenSearch-backed storage here. + ports: + - "127.0.0.1:16686:16686" + deploy: + resources: + limits: + memory: 1G + cpus: "1.0" + + # Parca server — Apache-2.0 profiles backend. Reads + # bin/observability/parca/parca.yaml (PR 5). FILESYSTEM storage. + parca: + image: ghcr.io/parca-dev/parca:v0.28.0 + container_name: texera-parca + profiles: [observability-profiles] + restart: always + security_opt: + - no-new-privileges:true + volumes: + - ../observability/parca/parca.yaml:/parca.yaml:ro,z + - parca_data:/var/lib/parca + command: + - "/parca" + - "--config-path=/parca.yaml" + ports: + - "127.0.0.1:7070:7070" + deploy: + resources: + limits: + memory: 1G + cpus: "1.0" + + # Parca eBPF agent — Apache-2.0. Linux-only, privileged. + # See bin/observability/parca/README.md for the full deploy + # posture. The privileged + bind-mount block here is the ONLY + # observability service that requires elevated permissions; the + # surface is documented and reviewed. + parca-agent: Review Comment: Parca and the eBPF agent are now opt-in behind the observability-profiles profile, off by default; removed the "documented and reviewed" wording and documented the host-wide scope and privilege/platform implications before enabling. ########## bin/single-node/docker-compose.yml: ########## @@ -530,6 +530,189 @@ services: command: > sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash /examples/load-examples.sh' + # ======================================================================== + # Part 5: Observability stack (PR 6). + # + # All six services live in this single compose file (no separate + # observability compose file, no monolithic observability profile). + # Per-signal profiles let an operator drop one without disturbing + # the others — see bin/single-node/up.sh and bin/single-node/.env + # (COMPOSE_PROFILES default includes every observability profile so + # `docker compose up` runs the whole stack). + # + # Network posture: every receiver/HTTP port binds to loopback + # (127.0.0.1) on the host or stays inside the texera-single-node + # bridge network entirely. No 0.0.0.0 host bindings, no Ingress. + # ======================================================================== + + # OpenTelemetry Collector — the single OTLP ingress for all three + # signals. Reads bin/observability/otel-collector/config.yaml. + # Apache-2.0; pinned to the contrib distribution. + otel-collector: + image: otel/opentelemetry-collector-contrib:0.153.0 + container_name: texera-otel-collector + profiles: [observability-collector] + restart: always + user: "10001:10001" + read_only: true + security_opt: + - no-new-privileges:true + volumes: + - ../observability/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro,z + command: ["--config=/etc/otelcol-contrib/config.yaml"] + # Local dev override: publish OTLP receiver ports on the host + # loopback so a Scala backend running outside docker (sbt / + # IntelliJ) can emit telemetry. In compose-only deploys, services + # talk to otel-collector:4317/:4318 via the bridge network. + ports: + - "127.0.0.1:4317:4317" + - "127.0.0.1:4318:4318" + deploy: + resources: + limits: + memory: 768M + cpus: "1.0" + + # VictoriaLogs — Apache-2.0 log store. LogsQL query API on 9428. + victorialogs: + image: victoriametrics/victoria-logs:v1.50.0 + container_name: texera-victorialogs + profiles: [observability-logs] + restart: always + user: "10002:10002" + read_only: true + security_opt: + - no-new-privileges:true + command: + - "-storageDataPath=/data" + - "-retentionPeriod=30d" + - "-httpListenAddr=:9428" + volumes: + - victorialogs_data:/data + # Loopback-only host binding so an operator can curl the query + # API from the host for ad-hoc debugging without exposing it to + # the network. + ports: + - "127.0.0.1:9428:9428" + deploy: + resources: + limits: + memory: 1G + cpus: "1.0" + + # VictoriaMetrics — Apache-2.0 metrics store. Accepts Prometheus + # remote-write from the collector at /api/v1/write, MetricsQL on + # /api/v1/query. + victoriametrics: + image: victoriametrics/victoria-metrics:v1.144.0 + container_name: texera-victoriametrics + profiles: [observability-metrics] + restart: always + user: "10003:10003" + read_only: true + security_opt: + - no-new-privileges:true + command: + - "-storageDataPath=/data" + - "-retentionPeriod=90d" + - "-httpListenAddr=:8428" + volumes: + - victoriametrics_data:/data + ports: + - "127.0.0.1:8428:8428" + deploy: + resources: + limits: + memory: 1G + cpus: "1.0" + + # Jaeger v2 — Apache-2.0 traces backend + UI. v2 accepts OTLP + # natively. The query API lives at :16686, OTLP ingest at :4317 + # (kept inside the bridge network only — only the collector talks + # to it). + jaeger: + image: jaegertracing/jaeger:2.18.0 + container_name: texera-jaeger + profiles: [observability-traces] + restart: always + security_opt: + - no-new-privileges:true + # In-memory storage for the single-node deployment — restarts + # wipe traces. Any deploy needing trace persistence must swap + # in Cassandra or OpenSearch-backed storage here. + ports: + - "127.0.0.1:16686:16686" + deploy: + resources: + limits: + memory: 1G + cpus: "1.0" + + # Parca server — Apache-2.0 profiles backend. Reads + # bin/observability/parca/parca.yaml (PR 5). FILESYSTEM storage. + parca: + image: ghcr.io/parca-dev/parca:v0.28.0 + container_name: texera-parca + profiles: [observability-profiles] + restart: always + security_opt: + - no-new-privileges:true + volumes: + - ../observability/parca/parca.yaml:/parca.yaml:ro,z + - parca_data:/var/lib/parca + command: + - "/parca" + - "--config-path=/parca.yaml" + ports: + - "127.0.0.1:7070:7070" + deploy: + resources: + limits: + memory: 1G + cpus: "1.0" + + # Parca eBPF agent — Apache-2.0. Linux-only, privileged. + # See bin/observability/parca/README.md for the full deploy + # posture. The privileged + bind-mount block here is the ONLY + # observability service that requires elevated permissions; the + # surface is documented and reviewed. + parca-agent: + image: ghcr.io/parca-dev/parca-agent:v0.47.1 + container_name: texera-parca-agent + profiles: [observability-profiles] + restart: always + depends_on: + parca: + condition: service_started + env_file: + - ../observability/parca/parca-agent.env + # eBPF requires CAP_SYS_ADMIN-class privileges. macOS / Windows + # developers cannot run this — they should drop the + # observability-profiles profile from COMPOSE_PROFILES. + privileged: true + pid: "host" + # Bind-mount kernel state read-only — the agent reads /proc and + # /sys for stack-trace symbolization but cannot write to either. Review Comment: Reworded the comment to state only the kernel-enforced read-only (:ro) behavior of the /proc and /sys mounts. ########## bin/observability/parca/README.md: ########## @@ -0,0 +1,98 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Parca profiles (PR 5) + +This directory holds configuration for the **profiles** signal in the +Texera observability stack. The compose service definitions that +consume these files land in PR 6; PR 5 ships the configuration only, +so the agent's deploy posture can be reviewed in isolation from the +broader compose changes. + +Components — both Apache-2.0 (see +[`docs/observability/LICENSING.md`](../../../docs/observability/LICENSING.md)): + +| File | Component | Image | +|---|---|---| +| `parca.yaml` | Parca server v0.28.0 | `ghcr.io/parca-dev/parca:v0.28.0` | +| `parca-agent.env` | Parca eBPF agent v0.47.1 | `ghcr.io/parca-dev/parca-agent:v0.47.1` | + +## Deploy posture + +The Parca agent uses eBPF to sample stack traces from running +processes. That puts a few non-negotiable requirements on the host: + +- **Linux only.** eBPF is a Linux kernel feature. macOS and Windows + developers cannot run the agent; the rest of the observability + stack (logs, metrics, traces) works on all platforms. +- **Privileged container.** The agent needs `CAP_SYS_ADMIN`-class + permissions to load eBPF programs and mount the perf-event + facility. The PR 6 compose service will set `privileged: true` + and bind-mount `/sys/kernel/debug`, `/proc`, and `/sys` read-only + into the container. +- **Read-only on host filesystems.** The bind-mounts above are + `ro` — the agent reads kernel state but cannot write to it. No + network exposure outside the cluster: the agent only opens an + outbound connection to the bundled Parca server on + `parca:7070`. + +## Opt-out + +For developers on non-Linux dev machines, or for any deploy that +chooses not to run profiles, set this in the host environment before +`docker compose up`: + +``` +TEXERA_OBSERVABILITY_PROFILES=disabled +``` + +PR 6's compose file gates the `parca-agent` (and optionally the +`parca` server too) on this flag — the rest of the stack continues +to run with `disabled` panels in the UI. + +## What gets profiled + +The agent's default behaviour is to discover and profile every +process on the host. We attach two static labels via +`parca-agent.env`: + +- `deployment=texera` +- `cluster=local` (override per env) Review Comment: Corrected the README: the agent profiles every process on the host, not only Texera; labels only filter what a query displays, not what is collected. ########## bin/observability/parca/README.md: ########## @@ -0,0 +1,98 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Parca profiles (PR 5) + +This directory holds configuration for the **profiles** signal in the +Texera observability stack. The compose service definitions that +consume these files land in PR 6; PR 5 ships the configuration only, +so the agent's deploy posture can be reviewed in isolation from the +broader compose changes. + +Components — both Apache-2.0 (see +[`docs/observability/LICENSING.md`](../../../docs/observability/LICENSING.md)): + +| File | Component | Image | +|---|---|---| +| `parca.yaml` | Parca server v0.28.0 | `ghcr.io/parca-dev/parca:v0.28.0` | +| `parca-agent.env` | Parca eBPF agent v0.47.1 | `ghcr.io/parca-dev/parca-agent:v0.47.1` | + +## Deploy posture + +The Parca agent uses eBPF to sample stack traces from running +processes. That puts a few non-negotiable requirements on the host: + +- **Linux only.** eBPF is a Linux kernel feature. macOS and Windows + developers cannot run the agent; the rest of the observability + stack (logs, metrics, traces) works on all platforms. +- **Privileged container.** The agent needs `CAP_SYS_ADMIN`-class + permissions to load eBPF programs and mount the perf-event + facility. The PR 6 compose service will set `privileged: true` + and bind-mount `/sys/kernel/debug`, `/proc`, and `/sys` read-only + into the container. +- **Read-only on host filesystems.** The bind-mounts above are + `ro` — the agent reads kernel state but cannot write to it. No + network exposure outside the cluster: the agent only opens an + outbound connection to the bundled Parca server on + `parca:7070`. + +## Opt-out + +For developers on non-Linux dev machines, or for any deploy that +chooses not to run profiles, set this in the host environment before +`docker compose up`: + +``` +TEXERA_OBSERVABILITY_PROFILES=disabled +``` + +PR 6's compose file gates the `parca-agent` (and optionally the +`parca` server too) on this flag — the rest of the stack continues +to run with `disabled` panels in the UI. + +## What gets profiled + +The agent's default behaviour is to discover and profile every +process on the host. We attach two static labels via +`parca-agent.env`: + +- `deployment=texera` +- `cluster=local` (override per env) + +When the PR 7 Texera query gateway runs Parca queries, it filters on Review Comment: Removed all inter-PR references from the README. ########## bin/observability/parca/README.md: ########## @@ -0,0 +1,98 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Parca profiles (PR 5) + +This directory holds configuration for the **profiles** signal in the +Texera observability stack. The compose service definitions that +consume these files land in PR 6; PR 5 ships the configuration only, +so the agent's deploy posture can be reviewed in isolation from the +broader compose changes. + +Components — both Apache-2.0 (see +[`docs/observability/LICENSING.md`](../../../docs/observability/LICENSING.md)): + +| File | Component | Image | +|---|---|---| +| `parca.yaml` | Parca server v0.28.0 | `ghcr.io/parca-dev/parca:v0.28.0` | +| `parca-agent.env` | Parca eBPF agent v0.47.1 | `ghcr.io/parca-dev/parca-agent:v0.47.1` | + +## Deploy posture + +The Parca agent uses eBPF to sample stack traces from running +processes. That puts a few non-negotiable requirements on the host: + +- **Linux only.** eBPF is a Linux kernel feature. macOS and Windows + developers cannot run the agent; the rest of the observability + stack (logs, metrics, traces) works on all platforms. +- **Privileged container.** The agent needs `CAP_SYS_ADMIN`-class + permissions to load eBPF programs and mount the perf-event + facility. The PR 6 compose service will set `privileged: true` + and bind-mount `/sys/kernel/debug`, `/proc`, and `/sys` read-only + into the container. +- **Read-only on host filesystems.** The bind-mounts above are + `ro` — the agent reads kernel state but cannot write to it. No + network exposure outside the cluster: the agent only opens an + outbound connection to the bundled Parca server on + `parca:7070`. + +## Opt-out + +For developers on non-Linux dev machines, or for any deploy that +chooses not to run profiles, set this in the host environment before +`docker compose up`: + +``` +TEXERA_OBSERVABILITY_PROFILES=disabled +``` + +PR 6's compose file gates the `parca-agent` (and optionally the +`parca` server too) on this flag — the rest of the stack continues +to run with `disabled` panels in the UI. + +## What gets profiled + +The agent's default behaviour is to discover and profile every +process on the host. We attach two static labels via +`parca-agent.env`: + +- `deployment=texera` +- `cluster=local` (override per env) + +When the PR 7 Texera query gateway runs Parca queries, it filters on +`deployment=texera` so the dashboard only ever shows Texera-process +profiles, never the operator's other workloads. + +We do **not** label profiles with `workflow.id` / `execution.id`. As +with metrics, those are unbounded identifiers and would blow up +Parca's storage cardinality. Per-execution profile views are reached +by joining on `trace_id` at query time (the Parca query API supports Review Comment: Corrected the README: the agent samples stack traces and does not read application-level OpenTelemetry trace ids, so profiles cannot be joined to individual traces by trace_id. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
