davidzollo opened a new issue, #11351: URL: https://github.com/apache/seatunnel/issues/11351
## Background SeaTunnel already exposes useful low-level realtime metrics for running jobs, and the current UI can show vertex and edge metric tables. Recent work such as PR #11215 and PR #11244 is also strengthening the observability foundation. However, SeaTunnel still does not provide a first-class **runtime execution graph** that lets users diagnose a live pipeline directly on the DAG itself. Today, when a job becomes slow, stalls, or shows uneven throughput, users usually have to switch between: - topology understanding from the static job graph - vertex metric tables - edge metric tables - checkpoint history - logs That makes hotspot diagnosis slower than it should be. ## Problem to solve We need a live runtime graph that overlays execution signals on the pipeline topology, so users can answer questions like: - which vertex is actually busy, idle, or blocked right now - which edge is filling up and creating backpressure - whether the bottleneck is source read, transform processing, sink write, or checkpoint/commit work - where the latest slowdown started, and how it propagated downstream ## Proposed scope Add a runtime execution graph view on the job detail path. Suggested first-delivery scope: - render the job DAG with live operator- and edge-level metrics - show node state such as source busy/idle, transform busy, sink busy, and recent record throughput - show edge state such as queue fill ratio, blocked ratio, and queue size/capacity - support drill-down from a node or edge into the existing detailed metrics table or panel - surface the latest checkpoint status and latest task-level error signal without leaving the graph view - keep the graph usable for both small and medium-sized pipelines, with a clear fallback for very large DAGs ## Why this should be a dedicated feature This is more than a UI polish task. It changes the user-facing observability contract across: - engine-side metric aggregation - REST data shape and refresh behavior - UI rendering and interaction model - memory and persistence boundaries for high-frequency snapshots ## STIP requirement before implementation Because this feature affects cross-module observability contracts and may add sustained sampling/rendering cost, **the claimant should submit a STIP design first and get maintainer agreement before starting implementation**. The STIP should make the following decisions explicit: - the canonical runtime-graph data model - refresh frequency, retention window, and degradation strategy for large jobs - how to combine topology, metric snapshots, checkpoint status, and error signals - which fields are best-effort observability signals and which are treated as stable API/UI contracts - how this feature avoids re-introducing persistence/write-amplification problems for runtime metric snapshots ## Acceptance criteria - A job detail page provides a dedicated runtime graph view. - The graph shows live node and edge health using meaningful visual encoding rather than a static topology only. - Users can identify the hottest vertex and most blocked edge without switching across multiple pages. - REST/API and UI behavior are documented clearly. - English and Chinese docs are updated. ## Non-goals for the first version - full distributed tracing - historical replay for arbitrarily long time ranges - connector-specific custom graph widgets in the first delivery ## Related work - PR #11215 - PR #11244 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
