zuozhiw commented on code in PR #5377: URL: https://github.com/apache/texera/pull/5377#discussion_r4346796690
########## bin/observability/README.md: ########## @@ -0,0 +1,131 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Texera Observability Stack + +Texera's backend services are instrumented with [OpenTelemetry](https://opentelemetry.io/) +and export OTLP (logs, metrics, and traces). This directory is an **optional, +bundled reference stack** that an operator can start alongside a single-node +deployment to collect and visualize that telemetry. It is opt-in: nothing here +runs unless you enable the observability compose profiles (see below), and +Texera emits no telemetry unless `OTEL_SDK_DISABLED=false` is set. + +## Topology + +A single OpenTelemetry Collector is the only OTLP ingress. It fans each signal +out to a dedicated backend, and Grafana sits on top as the query UI: + +``` + Texera services + │ OTLP/gRPC (4317) or OTLP/HTTP (4318) + ▼ + otel-collector + ┌──────────────┬──────┴────────┬───────────────┐ + OTLP │ remote-write OTLP │ │ (eBPF agent push) + ▼ ▼ ▼ ▼ + Loki VictoriaMetrics Jaeger Parca + (logs) (metrics) (traces) (profiles) + │ │ │ + └──────────────┼────────────────┘ + ▼ + Grafana + (reads Loki + VictoriaMetrics + Jaeger) +``` + +- **Loki** stores logs; the collector exports OTLP to `http://loki:3100/otlp`. +- **VictoriaMetrics** stores metrics; the collector exports Prometheus + remote-write to `http://victoriametrics:8428/api/v1/write`. +- **Jaeger** stores traces; the collector exports OTLP to `jaeger:4317`. +- **Parca** stores continuous profiles, pushed by its eBPF agent (not read by + Grafana; use the Parca UI on `:7070`). +- **Grafana** reads Loki, VictoriaMetrics (via the Prometheus query API), and + Jaeger as pre-provisioned datasources. + +## Starting it + +The stack is driven by docker-compose profiles. Enable the signals you want in +`bin/single-node/.env` via `COMPOSE_PROFILES`, then bring the deployment up: + +``` +# bin/single-node/.env +COMPOSE_PROFILES=observability-collector,observability-logs,observability-metrics,observability-traces,observability-grafana +``` + +| Profile | Service(s) | +|---------------------------|-----------------------------------------| +| `observability-collector` | otel-collector (OTLP ingress; required) | +| `observability-logs` | loki | +| `observability-metrics` | victoriametrics | +| `observability-traces` | jaeger | +| `observability-profiles` | parca + privileged eBPF parca-agent | +| `observability-grafana` | grafana | + +Also set `OTEL_SDK_DISABLED=false` so the services actually emit telemetry, then: + +``` +cd bin/single-node && docker compose up -d +# or the dedicated entry point: +cd bin/observability && docker compose up -d +``` + +Reach **Grafana at http://localhost:3000**. The default login is +`admin` / `texera` (set via `GRAFANA_ADMIN_USER` / `GRAFANA_ADMIN_PASSWORD` in +`.env`); change it for anything reachable beyond localhost. + +### Pre-provisioned datasources + +Grafana comes with three datasources already wired up Review Comment: Could we add dashboards and alerts in a follow-up PR so operators can use the collected data to diagnose problems? Configured datasources alone still leave them writing queries from scratch. For example, if workflows become slower, they need to compare execution latency and failures with host CPU/memory and JVM heap/GC activity to investigate whether the slowdown comes from the workflow or resource pressure on the service running it. Please include: - Host CPU/memory and JVM heap/GC/thread metrics. - Maintained dashboards for workflow counts, failures, and latency, filterable by environment, service, and instance. - A small set of example alerts, such as rising workflow failure rates or disk usage approaching capacity. The OTel Collector's host-metrics and Kubernetes kubelet-stats receivers can collect infrastructure metrics; Texera should expose application/runtime metrics. Database metrics can follow later. Texera-authored dashboards and configuration can live in this repository under its Apache license. Keep backends replaceable and Grafana/Loki optional, following the licensing boundary documented here. This can be a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
