Ma77Ball commented on code in PR #5377:
URL: https://github.com/apache/texera/pull/5377#discussion_r4358423560
##########
bin/single-node/docker-compose.yml:
##########
@@ -530,6 +530,189 @@ services:
command: >
sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash
/examples/load-examples.sh'
+ # ========================================================================
+ # Part 5: Observability stack (PR 6).
+ #
+ # All six services live in this single compose file (no separate
+ # observability compose file, no monolithic observability profile).
+ # Per-signal profiles let an operator drop one without disturbing
+ # the others — see bin/single-node/up.sh and bin/single-node/.env
+ # (COMPOSE_PROFILES default includes every observability profile so
+ # `docker compose up` runs the whole stack).
+ #
+ # Network posture: every receiver/HTTP port binds to loopback
+ # (127.0.0.1) on the host or stays inside the texera-single-node
+ # bridge network entirely. No 0.0.0.0 host bindings, no Ingress.
+ # ========================================================================
+
+ # OpenTelemetry Collector — the single OTLP ingress for all three
+ # signals. Reads bin/observability/otel-collector/config.yaml.
+ # Apache-2.0; pinned to the contrib distribution.
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.153.0
+ container_name: texera-otel-collector
+ profiles: [observability-collector]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ -
../observability/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro,z
+ command: ["--config=/etc/otelcol-contrib/config.yaml"]
+ # Local dev override: publish OTLP receiver ports on the host
+ # loopback so a Scala backend running outside docker (sbt /
+ # IntelliJ) can emit telemetry. In compose-only deploys, services
+ # talk to otel-collector:4317/:4318 via the bridge network.
+ ports:
+ - "127.0.0.1:4317:4317"
Review Comment:
Moved the host-facing collector ports into the local-dev override so the
local-dev configuration lives in the local-dev file.
##########
bin/single-node/docker-compose.yml:
##########
@@ -530,6 +530,189 @@ services:
command: >
sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash
/examples/load-examples.sh'
+ # ========================================================================
+ # Part 5: Observability stack (PR 6).
+ #
+ # All six services live in this single compose file (no separate
+ # observability compose file, no monolithic observability profile).
+ # Per-signal profiles let an operator drop one without disturbing
+ # the others — see bin/single-node/up.sh and bin/single-node/.env
+ # (COMPOSE_PROFILES default includes every observability profile so
+ # `docker compose up` runs the whole stack).
+ #
+ # Network posture: every receiver/HTTP port binds to loopback
+ # (127.0.0.1) on the host or stays inside the texera-single-node
+ # bridge network entirely. No 0.0.0.0 host bindings, no Ingress.
+ # ========================================================================
+
+ # OpenTelemetry Collector — the single OTLP ingress for all three
+ # signals. Reads bin/observability/otel-collector/config.yaml.
+ # Apache-2.0; pinned to the contrib distribution.
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.153.0
+ container_name: texera-otel-collector
+ profiles: [observability-collector]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ -
../observability/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro,z
+ command: ["--config=/etc/otelcol-contrib/config.yaml"]
+ # Local dev override: publish OTLP receiver ports on the host
+ # loopback so a Scala backend running outside docker (sbt /
+ # IntelliJ) can emit telemetry. In compose-only deploys, services
+ # talk to otel-collector:4317/:4318 via the bridge network.
+ ports:
+ - "127.0.0.1:4317:4317"
+ - "127.0.0.1:4318:4318"
+ deploy:
+ resources:
+ limits:
+ memory: 768M
+ cpus: "1.0"
+
+ # VictoriaLogs — Apache-2.0 log store. LogsQL query API on 9428.
+ victorialogs:
+ image: victoriametrics/victoria-logs:v1.50.0
+ container_name: texera-victorialogs
+ profiles: [observability-logs]
+ restart: always
+ user: "10002:10002"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ command:
+ - "-storageDataPath=/data"
+ - "-retentionPeriod=30d"
+ - "-httpListenAddr=:9428"
+ volumes:
+ - victorialogs_data:/data
+ # Loopback-only host binding so an operator can curl the query
+ # API from the host for ad-hoc debugging without exposing it to
+ # the network.
+ ports:
+ - "127.0.0.1:9428:9428"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # VictoriaMetrics — Apache-2.0 metrics store. Accepts Prometheus
+ # remote-write from the collector at /api/v1/write, MetricsQL on
+ # /api/v1/query.
+ victoriametrics:
+ image: victoriametrics/victoria-metrics:v1.144.0
+ container_name: texera-victoriametrics
+ profiles: [observability-metrics]
+ restart: always
+ user: "10003:10003"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ command:
+ - "-storageDataPath=/data"
+ - "-retentionPeriod=90d"
+ - "-httpListenAddr=:8428"
+ volumes:
+ - victoriametrics_data:/data
+ ports:
+ - "127.0.0.1:8428:8428"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # Jaeger v2 — Apache-2.0 traces backend + UI. v2 accepts OTLP
+ # natively. The query API lives at :16686, OTLP ingest at :4317
+ # (kept inside the bridge network only — only the collector talks
+ # to it).
+ jaeger:
+ image: jaegertracing/jaeger:2.18.0
+ container_name: texera-jaeger
+ profiles: [observability-traces]
+ restart: always
+ security_opt:
+ - no-new-privileges:true
+ # In-memory storage for the single-node deployment — restarts
+ # wipe traces. Any deploy needing trace persistence must swap
+ # in Cassandra or OpenSearch-backed storage here.
+ ports:
+ - "127.0.0.1:16686:16686"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # Parca server — Apache-2.0 profiles backend. Reads
+ # bin/observability/parca/parca.yaml (PR 5). FILESYSTEM storage.
+ parca:
+ image: ghcr.io/parca-dev/parca:v0.28.0
+ container_name: texera-parca
+ profiles: [observability-profiles]
+ restart: always
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ - ../observability/parca/parca.yaml:/parca.yaml:ro,z
+ - parca_data:/var/lib/parca
+ command:
+ - "/parca"
+ - "--config-path=/parca.yaml"
+ ports:
+ - "127.0.0.1:7070:7070"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # Parca eBPF agent — Apache-2.0. Linux-only, privileged.
+ # See bin/observability/parca/README.md for the full deploy
+ # posture. The privileged + bind-mount block here is the ONLY
+ # observability service that requires elevated permissions; the
+ # surface is documented and reviewed.
+ parca-agent:
Review Comment:
The eBPF agent is behind its own opt-in profile (observability-profiles),
off by default, with the privilege and whole-host scope spelled out in the
compose comments and bin/observability/parca/README.md; macOS and Windows just
leave that profile out. The host-JVM collector ports now live in the local-dev
override the launcher actually loads.
##########
bin/single-node/docker-compose.yml:
##########
@@ -530,6 +530,189 @@ services:
command: >
sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash
/examples/load-examples.sh'
+ # ========================================================================
+ # Part 5: Observability stack (PR 6).
+ #
+ # All six services live in this single compose file (no separate
+ # observability compose file, no monolithic observability profile).
+ # Per-signal profiles let an operator drop one without disturbing
+ # the others — see bin/single-node/up.sh and bin/single-node/.env
+ # (COMPOSE_PROFILES default includes every observability profile so
+ # `docker compose up` runs the whole stack).
+ #
+ # Network posture: every receiver/HTTP port binds to loopback
+ # (127.0.0.1) on the host or stays inside the texera-single-node
+ # bridge network entirely. No 0.0.0.0 host bindings, no Ingress.
+ # ========================================================================
+
+ # OpenTelemetry Collector — the single OTLP ingress for all three
+ # signals. Reads bin/observability/otel-collector/config.yaml.
+ # Apache-2.0; pinned to the contrib distribution.
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.153.0
+ container_name: texera-otel-collector
+ profiles: [observability-collector]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ -
../observability/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro,z
+ command: ["--config=/etc/otelcol-contrib/config.yaml"]
+ # Local dev override: publish OTLP receiver ports on the host
+ # loopback so a Scala backend running outside docker (sbt /
+ # IntelliJ) can emit telemetry. In compose-only deploys, services
+ # talk to otel-collector:4317/:4318 via the bridge network.
+ ports:
+ - "127.0.0.1:4317:4317"
+ - "127.0.0.1:4318:4318"
+ deploy:
+ resources:
+ limits:
+ memory: 768M
+ cpus: "1.0"
+
+ # VictoriaLogs — Apache-2.0 log store. LogsQL query API on 9428.
+ victorialogs:
+ image: victoriametrics/victoria-logs:v1.50.0
+ container_name: texera-victorialogs
+ profiles: [observability-logs]
+ restart: always
+ user: "10002:10002"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ command:
+ - "-storageDataPath=/data"
+ - "-retentionPeriod=30d"
+ - "-httpListenAddr=:9428"
+ volumes:
+ - victorialogs_data:/data
+ # Loopback-only host binding so an operator can curl the query
+ # API from the host for ad-hoc debugging without exposing it to
+ # the network.
+ ports:
+ - "127.0.0.1:9428:9428"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # VictoriaMetrics — Apache-2.0 metrics store. Accepts Prometheus
+ # remote-write from the collector at /api/v1/write, MetricsQL on
+ # /api/v1/query.
+ victoriametrics:
+ image: victoriametrics/victoria-metrics:v1.144.0
+ container_name: texera-victoriametrics
+ profiles: [observability-metrics]
+ restart: always
+ user: "10003:10003"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ command:
+ - "-storageDataPath=/data"
+ - "-retentionPeriod=90d"
+ - "-httpListenAddr=:8428"
+ volumes:
+ - victoriametrics_data:/data
+ ports:
+ - "127.0.0.1:8428:8428"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # Jaeger v2 — Apache-2.0 traces backend + UI. v2 accepts OTLP
+ # natively. The query API lives at :16686, OTLP ingest at :4317
+ # (kept inside the bridge network only — only the collector talks
+ # to it).
+ jaeger:
+ image: jaegertracing/jaeger:2.18.0
+ container_name: texera-jaeger
+ profiles: [observability-traces]
+ restart: always
+ security_opt:
+ - no-new-privileges:true
+ # In-memory storage for the single-node deployment — restarts
+ # wipe traces. Any deploy needing trace persistence must swap
+ # in Cassandra or OpenSearch-backed storage here.
+ ports:
+ - "127.0.0.1:16686:16686"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # Parca server — Apache-2.0 profiles backend. Reads
+ # bin/observability/parca/parca.yaml (PR 5). FILESYSTEM storage.
+ parca:
+ image: ghcr.io/parca-dev/parca:v0.28.0
+ container_name: texera-parca
+ profiles: [observability-profiles]
+ restart: always
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ - ../observability/parca/parca.yaml:/parca.yaml:ro,z
+ - parca_data:/var/lib/parca
+ command:
+ - "/parca"
+ - "--config-path=/parca.yaml"
+ ports:
+ - "127.0.0.1:7070:7070"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # Parca eBPF agent — Apache-2.0. Linux-only, privileged.
+ # See bin/observability/parca/README.md for the full deploy
+ # posture. The privileged + bind-mount block here is the ONLY
+ # observability service that requires elevated permissions; the
+ # surface is documented and reviewed.
+ parca-agent:
+ image: ghcr.io/parca-dev/parca-agent:v0.47.1
+ container_name: texera-parca-agent
+ profiles: [observability-profiles]
+ restart: always
+ depends_on:
+ parca:
+ condition: service_started
+ env_file:
+ - ../observability/parca/parca-agent.env
+ # eBPF requires CAP_SYS_ADMIN-class privileges. macOS / Windows
+ # developers cannot run this — they should drop the
+ # observability-profiles profile from COMPOSE_PROFILES.
+ privileged: true
+ pid: "host"
+ # Bind-mount kernel state read-only — the agent reads /proc and
+ # /sys for stack-trace symbolization but cannot write to either.
Review Comment:
Reworded that block to the factual version: the agent needs
CAP_SYS_ADMIN-class privileges, runs on Linux only, and profiles every process
on the host.
##########
bin/observability/parca/README.md:
##########
@@ -0,0 +1,98 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Parca profiles (PR 5)
+
+This directory holds configuration for the **profiles** signal in the
+Texera observability stack. The compose service definitions that
+consume these files land in PR 6; PR 5 ships the configuration only,
+so the agent's deploy posture can be reviewed in isolation from the
+broader compose changes.
+
+Components — both Apache-2.0 (see
+[`docs/observability/LICENSING.md`](../../../docs/observability/LICENSING.md)):
+
+| File | Component | Image |
+|---|---|---|
+| `parca.yaml` | Parca server v0.28.0 | `ghcr.io/parca-dev/parca:v0.28.0` |
+| `parca-agent.env` | Parca eBPF agent v0.47.1 |
`ghcr.io/parca-dev/parca-agent:v0.47.1` |
+
+## Deploy posture
+
+The Parca agent uses eBPF to sample stack traces from running
+processes. That puts a few non-negotiable requirements on the host:
+
+- **Linux only.** eBPF is a Linux kernel feature. macOS and Windows
+ developers cannot run the agent; the rest of the observability
+ stack (logs, metrics, traces) works on all platforms.
+- **Privileged container.** The agent needs `CAP_SYS_ADMIN`-class
+ permissions to load eBPF programs and mount the perf-event
+ facility. The PR 6 compose service will set `privileged: true`
+ and bind-mount `/sys/kernel/debug`, `/proc`, and `/sys` read-only
+ into the container.
+- **Read-only on host filesystems.** The bind-mounts above are
+ `ro` — the agent reads kernel state but cannot write to it. No
+ network exposure outside the cluster: the agent only opens an
+ outbound connection to the bundled Parca server on
+ `parca:7070`.
+
+## Opt-out
+
+For developers on non-Linux dev machines, or for any deploy that
+chooses not to run profiles, set this in the host environment before
+`docker compose up`:
+
+```
+TEXERA_OBSERVABILITY_PROFILES=disabled
+```
+
+PR 6's compose file gates the `parca-agent` (and optionally the
+`parca` server too) on this flag — the rest of the stack continues
+to run with `disabled` panels in the UI.
+
+## What gets profiled
+
+The agent's default behaviour is to discover and profile every
+process on the host. We attach two static labels via
+`parca-agent.env`:
+
+- `deployment=texera`
+- `cluster=local` (override per env)
Review Comment:
The README's 'What gets profiled' section states the agent profiles every
process on the host, not only Texera's, and that the static labels only change
which profiles are shown, not what is collected.
##########
bin/observability/parca/README.md:
##########
@@ -0,0 +1,98 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Parca profiles (PR 5)
+
+This directory holds configuration for the **profiles** signal in the
+Texera observability stack. The compose service definitions that
+consume these files land in PR 6; PR 5 ships the configuration only,
+so the agent's deploy posture can be reviewed in isolation from the
+broader compose changes.
+
+Components — both Apache-2.0 (see
+[`docs/observability/LICENSING.md`](../../../docs/observability/LICENSING.md)):
+
+| File | Component | Image |
+|---|---|---|
+| `parca.yaml` | Parca server v0.28.0 | `ghcr.io/parca-dev/parca:v0.28.0` |
+| `parca-agent.env` | Parca eBPF agent v0.47.1 |
`ghcr.io/parca-dev/parca-agent:v0.47.1` |
+
+## Deploy posture
+
+The Parca agent uses eBPF to sample stack traces from running
+processes. That puts a few non-negotiable requirements on the host:
+
+- **Linux only.** eBPF is a Linux kernel feature. macOS and Windows
+ developers cannot run the agent; the rest of the observability
+ stack (logs, metrics, traces) works on all platforms.
+- **Privileged container.** The agent needs `CAP_SYS_ADMIN`-class
+ permissions to load eBPF programs and mount the perf-event
+ facility. The PR 6 compose service will set `privileged: true`
+ and bind-mount `/sys/kernel/debug`, `/proc`, and `/sys` read-only
+ into the container.
+- **Read-only on host filesystems.** The bind-mounts above are
+ `ro` — the agent reads kernel state but cannot write to it. No
+ network exposure outside the cluster: the agent only opens an
+ outbound connection to the bundled Parca server on
+ `parca:7070`.
+
+## Opt-out
+
+For developers on non-Linux dev machines, or for any deploy that
+chooses not to run profiles, set this in the host environment before
+`docker compose up`:
+
+```
+TEXERA_OBSERVABILITY_PROFILES=disabled
+```
+
+PR 6's compose file gates the `parca-agent` (and optionally the
+`parca` server too) on this flag — the rest of the stack continues
+to run with `disabled` panels in the UI.
+
+## What gets profiled
+
+The agent's default behaviour is to discover and profile every
+process on the host. We attach two static labels via
+`parca-agent.env`:
+
+- `deployment=texera`
+- `cluster=local` (override per env)
+
+When the PR 7 Texera query gateway runs Parca queries, it filters on
Review Comment:
Removed the PR-number references from the READMEs and checked the other
observability READMEs and compose comments for the same wording.
##########
bin/observability/parca/README.md:
##########
@@ -0,0 +1,98 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Parca profiles (PR 5)
+
+This directory holds configuration for the **profiles** signal in the
+Texera observability stack. The compose service definitions that
+consume these files land in PR 6; PR 5 ships the configuration only,
+so the agent's deploy posture can be reviewed in isolation from the
+broader compose changes.
+
+Components — both Apache-2.0 (see
+[`docs/observability/LICENSING.md`](../../../docs/observability/LICENSING.md)):
+
+| File | Component | Image |
+|---|---|---|
+| `parca.yaml` | Parca server v0.28.0 | `ghcr.io/parca-dev/parca:v0.28.0` |
+| `parca-agent.env` | Parca eBPF agent v0.47.1 |
`ghcr.io/parca-dev/parca-agent:v0.47.1` |
+
+## Deploy posture
+
+The Parca agent uses eBPF to sample stack traces from running
+processes. That puts a few non-negotiable requirements on the host:
+
+- **Linux only.** eBPF is a Linux kernel feature. macOS and Windows
+ developers cannot run the agent; the rest of the observability
+ stack (logs, metrics, traces) works on all platforms.
+- **Privileged container.** The agent needs `CAP_SYS_ADMIN`-class
+ permissions to load eBPF programs and mount the perf-event
+ facility. The PR 6 compose service will set `privileged: true`
+ and bind-mount `/sys/kernel/debug`, `/proc`, and `/sys` read-only
+ into the container.
+- **Read-only on host filesystems.** The bind-mounts above are
+ `ro` — the agent reads kernel state but cannot write to it. No
+ network exposure outside the cluster: the agent only opens an
+ outbound connection to the bundled Parca server on
+ `parca:7070`.
+
+## Opt-out
+
+For developers on non-Linux dev machines, or for any deploy that
+chooses not to run profiles, set this in the host environment before
+`docker compose up`:
+
+```
+TEXERA_OBSERVABILITY_PROFILES=disabled
+```
+
+PR 6's compose file gates the `parca-agent` (and optionally the
+`parca` server too) on this flag — the rest of the stack continues
+to run with `disabled` panels in the UI.
+
+## What gets profiled
+
+The agent's default behaviour is to discover and profile every
+process on the host. We attach two static labels via
+`parca-agent.env`:
+
+- `deployment=texera`
+- `cluster=local` (override per env)
+
+When the PR 7 Texera query gateway runs Parca queries, it filters on
+`deployment=texera` so the dashboard only ever shows Texera-process
+profiles, never the operator's other workloads.
+
+We do **not** label profiles with `workflow.id` / `execution.id`. As
+with metrics, those are unbounded identifiers and would blow up
+Parca's storage cardinality. Per-execution profile views are reached
+by joining on `trace_id` at query time (the Parca query API supports
Review Comment:
Corrected the README: the agent samples stack traces and does not read
application-level OpenTelemetry trace ids, so profiles cannot be joined to
individual traces by trace_id, only by the coarse labels and by time.
##########
bin/single-node/docker-compose.yml:
##########
@@ -590,6 +590,227 @@ services:
command: >
sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash
/examples/load-examples.sh'
+ # ========================================================================
+ # Observability stack: collector + logs, metrics, traces, profiles.
+ #
+ # Each signal is a separate compose profile; enable them via
+ # COMPOSE_PROFILES in bin/single-node/.env. Every port binds to
+ # loopback (127.0.0.1) or stays on the bridge network; no 0.0.0.0
+ # host bindings.
+ # ========================================================================
+
+ # OpenTelemetry Collector, the single OTLP ingress for all three
+ # signals. Reads bin/observability/otel-collector/config.yaml.
+ # Apache-2.0; pinned to the contrib distribution.
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.153.0
+ container_name: texera-otel-collector
+ profiles: [observability-collector]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ -
../observability/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro,z
+ command: ["--config=/etc/otelcol-contrib/config.yaml"]
+ # In-compose services reach the OTLP receivers over the bridge
+ # network as otel-collector:4317/:4318, so the base file publishes
+ # no host ports. For a backend running outside docker (sbt /
+ # IntelliJ), docker-compose.override.yml publishes them on the host
+ # loopback in local dev; the released bundle omits that override.
+ deploy:
+ resources:
+ limits:
+ memory: 768M
+ cpus: "1.0"
+
+ # Loki (AGPLv3) log store. OTLP ingest at /otlp, LogQL query API on
+ # 3100. AGPLv3, so it is an optional operator-run component and is NOT
+ # shipped in any Texera release artifact (see bin/observability/README.md).
+ # Reads bin/observability/loki/config.yaml.
+ loki:
+ image: grafana/loki:3.2.0
+ container_name: texera-loki
+ profiles: [observability-logs]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ - ../observability/loki/config.yaml:/etc/loki/config.yaml:ro,z
+ - loki_data:/loki
+ # Read-only root filesystem, so Loki's scratch space is an in-memory
+ # tmpfs; all durable state lives on the loki_data volume under /loki.
+ tmpfs:
+ - /tmp
+ command: ["-config.file=/etc/loki/config.yaml"]
+ # Loopback-only host binding so an operator can curl the query
+ # API from the host for ad-hoc debugging without exposing it to
+ # the network.
+ ports:
+ - "127.0.0.1:3100:3100"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # Grafana (AGPLv3) dashboard/UI. Reads the Apache-2.0 backends below
+ # (Loki, VictoriaMetrics, Jaeger) as pre-provisioned datasources. The UI
+ # is meant to be opened by a human, but like the backends its port is
+ # bound to loopback only (http://localhost:3000). AGPLv3, so it is an
+ # optional operator-run component and is NOT shipped in any Texera
+ # release artifact (see bin/observability/README.md).
+ grafana:
+ image: grafana/grafana:11.2.0
+ container_name: texera-grafana
+ profiles: [observability-grafana]
+ restart: always
+ security_opt:
+ - no-new-privileges:true
+ depends_on:
+ - loki
+ - victoriametrics
+ - jaeger
+ environment:
+ # Admin login. Defaults are weak dev defaults; override in .env for
+ # any deployment reachable beyond localhost.
+ - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
+ - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-texera}
+ # No anonymous access: the UI requires the admin login above.
+ - GF_AUTH_ANONYMOUS_ENABLED=false
+ # Users cannot self-register.
+ - GF_USERS_ALLOW_SIGN_UP=false
+ volumes:
+ # Datasource provisioning (Loki + VictoriaMetrics + Jaeger), read-only.
+ - ../observability/grafana/provisioning:/etc/grafana/provisioning:ro,z
+ - grafana_data:/var/lib/grafana
+ # Loopback-only host binding, like the backends: a human opens
+ # http://localhost:3000, and the UI is never reachable from the
+ # network by default.
+ ports:
+ - "127.0.0.1:${GRAFANA_PORT:-3000}:3000"
+ deploy:
+ resources:
+ limits:
+ memory: 512M
+ cpus: "0.5"
+
+ # VictoriaMetrics, Apache-2.0 metrics store. Accepts Prometheus
+ # remote-write from the collector at /api/v1/write, MetricsQL on
+ # /api/v1/query.
+ victoriametrics:
+ image: victoriametrics/victoria-metrics:v1.144.0
+ container_name: texera-victoriametrics
+ profiles: [observability-metrics]
+ restart: always
+ user: "10003:10003"
Review Comment:
Added a one-shot victoriametrics-init step that chowns the data volume to
10003:10003 before VictoriaMetrics starts, so a fresh volume no longer fails
with permission denied.
##########
bin/single-node/docker-compose.override.yml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Local-dev override, auto-loaded by `docker compose` when run from this
+# directory. It publishes the OTel collector's OTLP receiver ports on the
+# host loopback so a backend running outside docker (sbt / IntelliJ) can
+# emit telemetry to the collector. The released bundle does not ship this
+# file, so the base compose stays network-clean in a deploy.
+name: texera-single-node
+
+services:
+ otel-collector:
+ ports:
Review Comment:
Added the collector's 4317 and 4318 loopback port mappings to
bin/local-dev/docker-compose.override.yml, the file the local-dev launcher
actually loads, so a host JVM can reach the collector, and updated the compose
comment to point there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
