zuozhiw commented on code in PR #5377:
URL: https://github.com/apache/texera/pull/5377#discussion_r4346785591
##########
bin/single-node/docker-compose.yml:
##########
@@ -590,6 +590,227 @@ services:
command: >
sh -c 'apk add --no-cache curl jq bash > /dev/null 2>&1 && bash
/examples/load-examples.sh'
+ # ========================================================================
+ # Observability stack: collector + logs, metrics, traces, profiles.
+ #
+ # Each signal is a separate compose profile; enable them via
+ # COMPOSE_PROFILES in bin/single-node/.env. Every port binds to
+ # loopback (127.0.0.1) or stays on the bridge network; no 0.0.0.0
+ # host bindings.
+ # ========================================================================
+
+ # OpenTelemetry Collector, the single OTLP ingress for all three
+ # signals. Reads bin/observability/otel-collector/config.yaml.
+ # Apache-2.0; pinned to the contrib distribution.
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.153.0
+ container_name: texera-otel-collector
+ profiles: [observability-collector]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ -
../observability/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml:ro,z
+ command: ["--config=/etc/otelcol-contrib/config.yaml"]
+ # In-compose services reach the OTLP receivers over the bridge
+ # network as otel-collector:4317/:4318, so the base file publishes
+ # no host ports. For a backend running outside docker (sbt /
+ # IntelliJ), docker-compose.override.yml publishes them on the host
+ # loopback in local dev; the released bundle omits that override.
+ deploy:
+ resources:
+ limits:
+ memory: 768M
+ cpus: "1.0"
+
+ # Loki (AGPLv3) log store. OTLP ingest at /otlp, LogQL query API on
+ # 3100. AGPLv3, so it is an optional operator-run component and is NOT
+ # shipped in any Texera release artifact (see bin/observability/README.md).
+ # Reads bin/observability/loki/config.yaml.
+ loki:
+ image: grafana/loki:3.2.0
+ container_name: texera-loki
+ profiles: [observability-logs]
+ restart: always
+ user: "10001:10001"
+ read_only: true
+ security_opt:
+ - no-new-privileges:true
+ volumes:
+ - ../observability/loki/config.yaml:/etc/loki/config.yaml:ro,z
+ - loki_data:/loki
+ # Read-only root filesystem, so Loki's scratch space is an in-memory
+ # tmpfs; all durable state lives on the loki_data volume under /loki.
+ tmpfs:
+ - /tmp
+ command: ["-config.file=/etc/loki/config.yaml"]
+ # Loopback-only host binding so an operator can curl the query
+ # API from the host for ad-hoc debugging without exposing it to
+ # the network.
+ ports:
+ - "127.0.0.1:3100:3100"
+ deploy:
+ resources:
+ limits:
+ memory: 1G
+ cpus: "1.0"
+
+ # Grafana (AGPLv3) dashboard/UI. Reads the Apache-2.0 backends below
+ # (Loki, VictoriaMetrics, Jaeger) as pre-provisioned datasources. The UI
+ # is meant to be opened by a human, but like the backends its port is
+ # bound to loopback only (http://localhost:3000). AGPLv3, so it is an
+ # optional operator-run component and is NOT shipped in any Texera
+ # release artifact (see bin/observability/README.md).
+ grafana:
+ image: grafana/grafana:11.2.0
+ container_name: texera-grafana
+ profiles: [observability-grafana]
+ restart: always
+ security_opt:
+ - no-new-privileges:true
+ depends_on:
+ - loki
+ - victoriametrics
+ - jaeger
+ environment:
+ # Admin login. Defaults are weak dev defaults; override in .env for
+ # any deployment reachable beyond localhost.
+ - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
+ - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-texera}
+ # No anonymous access: the UI requires the admin login above.
+ - GF_AUTH_ANONYMOUS_ENABLED=false
+ # Users cannot self-register.
+ - GF_USERS_ALLOW_SIGN_UP=false
+ volumes:
+ # Datasource provisioning (Loki + VictoriaMetrics + Jaeger), read-only.
+ - ../observability/grafana/provisioning:/etc/grafana/provisioning:ro,z
+ - grafana_data:/var/lib/grafana
+ # Loopback-only host binding, like the backends: a human opens
+ # http://localhost:3000, and the UI is never reachable from the
+ # network by default.
+ ports:
+ - "127.0.0.1:${GRAFANA_PORT:-3000}:3000"
+ deploy:
+ resources:
+ limits:
+ memory: 512M
+ cpus: "0.5"
+
+ # VictoriaMetrics, Apache-2.0 metrics store. Accepts Prometheus
+ # remote-write from the collector at /api/v1/write, MetricsQL on
+ # /api/v1/query.
+ victoriametrics:
+ image: victoriametrics/victoria-metrics:v1.144.0
+ container_name: texera-victoriametrics
+ profiles: [observability-metrics]
+ restart: always
+ user: "10003:10003"
Review Comment:
On a fresh deployment, VictoriaMetrics exits because UID/GID 10003 cannot
write to the new `victoriametrics_data` volume. It cannot create
`/data/flock.lock` and fails with `permission denied`, leaving the stack
without a running metrics database. For example, a developer starting the
observability profile for the first time would have nowhere to store the
exported workflow metrics, even if the collector is running.
Please initialize the volume ownership for UID/GID 10003, or use a writable
data-directory/user arrangement supported by the image. The service can remain
non-root with a read-only root filesystem; its mounted `/data` directory needs
to be writable.
Please include a startup check with a fresh volume. Reusing an existing
volume with suitable ownership can hide this first-start failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
