Musxeto opened a new issue, #6272: URL: https://github.com/apache/texera/issues/6272
### Feature Summary Texera's Amber engine already collects rich runtime statistics internally (tuple counts, processing times, worker counts, execution duration) via `StatisticsManager` and `ExecutionStatsService` — but these are only visible in the UI during an active session and aren't available for alerting, historical analysis, or system-level correlation. **This feature adds:** - A **Prometheus `/metrics` endpoint** on the coordinator service exposing workflow, operator, and JVM metrics - **Pre-built Grafana dashboards** for workflow overview, operator performance, and system health - **Docker Compose integration** (opt-in `observability` profile) for one-command setup of Prometheus + Grafana **Problems this solves:** - No way to monitor execution health outside the UI - No alerting for stalled workflows or resource exhaustion - No historical trend analysis for capacity planning - JVM/system metrics (heap, GC, threads) are completely invisible ### Proposed Solution or Design **Instrument the existing stats pipeline** with [Micrometer](https://micrometer.io/) (JVM metrics facade) and expose a standard Prometheus scrape endpoint. No changes to the existing stats interfaces — just an additional export path. **High-level approach:** 1. Add Micrometer & Prometheus registry dependency to `amber/build.sbt` 2. Create a `MetricsService` singleton wiring existing `StatisticsManager` / `ExecutionStatsService` data into Micrometer meters 3. Expose `GET /api/metrics` via a new Dropwizard resource 4. Add Prometheus + Grafana to `bin/single-node/docker-compose.yml` under an optional `observability` profile 5. Ship 3 pre-provisioned Grafana dashboards (Workflow Overview, Operator Deep-Dive, System Health) **Key metrics:** - Workflow: active count, execution duration, state transitions - Operator: input/output tuple rates, processing time breakdown, worker count - System: JVM heap, GC pauses, thread count (via Micrometer's built-in JVM binders) **Contributors:** We (@musxeto and @thegamingbat) would like to work on this. Happy to discuss the design further before coding. ### Affected Area Workflow Engine (Amber), Deployment / Infrastructure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
