GitHub user joeyutong created a discussion: [Discussion][Observability 2/2] Built-in Operational Metrics for Flink Agents
This proposal is the second in a two-part Flink Agents observability series: 1. [**Recording Agent Traces in the Event Log**](https://github.com/apache/flink-agents/discussions/900) 2. **Built-in Operational Metrics for Flink Agents** (this proposal) Part 1 defines the per-run recording needed to reconstruct an Agent Trace. This proposal defines bounded-cardinality metrics for observing aggregate health and behavior across runs. The two capabilities share runtime observation boundaries but produce independent outputs: ```text Agent Trace = per-run recording Operational metrics = aggregate measurement ``` ## 1. Context and Relationship to Part 1 Flink already provides job, operator, checkpoint, backpressure, and resource metrics. These metrics describe the Flink execution layer, but they do not directly answer Agent-specific operational questions: - Are input runs succeeding, and is their end-to-end latency increasing? - Are inputs waiting behind other inputs for the same key? - Is an Action waiting to be scheduled or spending too long on asynchronous work? - Are failures or latency concentrated in a particular model resource, Tool, Skill, or MCP Server? - Are model retries or token consumption increasing unexpectedly? Part 1 proposes two runtime Event categories for per-run observation: - **Business Events** describe workflow occurrences and Event-to-Action causality. - **Execution Events** describe the lifecycle of Action, LLM, Parser, and Tool executions. Those Events also provide natural boundaries for aggregate metrics. Trace logging and metric aggregation can consume the same Execution Events independently, so Action, LLM, and Tool metrics follow the lifecycle semantics recorded in each trace. Execution Events cover only work represented as executions. Queue depth, `ActionTask` scheduling, token usage, and retry statistics instead arise from runtime state or call results and are recorded at their own boundaries. The complete signal model is therefore: ```mermaid flowchart LR BE["Business Event"] --> TRACE["Event Log<br/>per-run Agent Trace"] BE --> EVENT_METRICS["Event-driven metrics<br/>Business Event / Action / LLM / Tool"] EE["Execution Event"] --> TRACE EE --> EVENT_METRICS RUN_STATE["Run and queue state"] --> RUN_METRICS["Input Run and scheduling metrics"] MODEL_RESULT["Model call result"] --> MODEL_METRICS["Token usage and retry metrics"] ``` This model aligns Agent Trace and execution metrics on lifecycle, entity, outcome, and metadata semantics while keeping the two outputs independent. ## 2. Goals and Non-goals ### Goals - Define built-in metrics for Input Run health, Action scheduling, and Agent execution entities. - Align Action, LLM, and Tool metric boundaries with the Execution Event model from Part 1. - Define stable metric names, types, scopes, and outcome semantics. - Keep metric dimensions bounded and suitable for aggregation. - Preserve the distinction between configured resources and provider-returned model names. - Keep Java and Python metric semantics aligned. - Define metric behavior across task attempts and recovery. ### Non-goals - Dashboard layout, visualization, or alert thresholds. - Reporter-specific queries or Prometheus expressions. - Per-run diagnosis through metric labels. - Exactly-once metric values across task attempts. - Provider-attempt-level LLM metrics. - Evaluation datasets, experiments, or model-quality metrics. ## 3. Metric Model ### 3.1 Scope Hierarchy The Agent name identifies the Flink operator that runs the Agent. Agent-specific dimensions are registered as key-value `MetricGroup` scopes. Metric reporters that support dimensions, such as Prometheus, can expose these scopes as labels; other reporters may encode them in the metric identifier. ```text Agent operator ├─ Agent-level metrics └─ action=<action_name> ├─ Action-level metrics ├─ model_resource=<resource_name> ├─ tool=<tool_name> ├─ skill=<skill_name> ├─ mcp_server=<server_name> └─ model=<model_name> ``` | Scope | Meaning | | --- | --- | | Agent | Aggregate behavior of the Agent operator, without selecting an Action | | Action | Scheduling and execution behavior of one Action | | Model Resource | The ChatModel resource name referenced by the Agent plan; used for call health and retries | | Model | The model name returned by the provider; used for token consumption | | Tool | One named Tool; used for Tool-call health | | Skill | The Skill name carried by an explicit `load_skill` Tool call | | MCP Server | The MCP Server name carried by an MCP Tool execution | `model_resource` and `model` belong to independent branches: - Model Resource metrics answer whether a configured ChatModel resource is healthy. - Model metrics attribute token consumption to the model name returned by the provider. When one resource is shared by multiple Actions, each Action produces a separate raw metric series. Consumers can retain the Action scope to locate the caller or aggregate across that scope to view resource-wide health. ### 3.2 Metric Types and Naming The proposal uses standard Flink metric types: | Type | Semantics | | --- | --- | | Counter | A cumulative count since the current task attempt started | | Meter | A rate derived from a Counter | | Histogram | A distribution of latency samples observed in the current task attempt | | Gauge | A current value that can increase or decrease | Metric names follow the existing Flink Agents style: - Counts use `numOf...`. - Rates use `...PerSec`. - Millisecond latency distributions use `...LatencyMs`. - Existing domain-specific names for token usage and retries are retained for compatibility. - Scope keys use snake_case, such as `model_resource` and `mcp_server`. ### 3.3 Cardinality Operational metrics intentionally exclude per-run and per-execution identifiers. Fields such as `input_run_id`, `execution_id`, `business_key`, and `event_id` are useful for Agent Trace queries but would create unbounded metric cardinality. Metric scopes are limited to reusable Agent entities and resources: Action, Model Resource, model name, Tool, Skill, and MCP Server. Metrics identify where aggregate behavior is concentrated; Agent Trace remains the per-run diagnostic surface. ## 4. Built-in Metrics The existing Event throughput, Action throughput, and token metrics remain part of the built-in metric surface. This proposal adds Input Run, Action scheduling, and execution-entity health metrics while preserving existing names and semantics. ### 4.1 Existing Baseline | Scope | Metric | Type | Meaning | | --- | --- | --- | --- | | Agent | `numOfEventProcessed` | Counter | Processed business Events | | Agent | `numOfEventProcessedPerSec` | Meter | Business Event processing rate | | Agent | `numOfActionsExecuted` | Counter | Completed Actions | | Agent | `numOfActionsExecutedPerSec` | Meter | Action completion rate | | Action | `numOfActionsExecuted` | Counter | Completed invocations of one Action | | Action | `numOfActionsExecutedPerSec` | Meter | Completion rate of one Action | | Agent | `eventLogTruncatedEvents` | Counter | Event Log records whose payload was truncated | | Model | `promptTokens` | Counter | Prompt tokens consumed | | Model | `completionTokens` | Counter | Completion tokens consumed | | Model | `totalTokens` | Counter | Total tokens consumed | ### 4.2 Input Run An **Input Run** is one attempt to process an input for a business key. Metric tracking begins when the runtime receives the Input Event. The run's processing phase begins later, when that Event leaves the pending queue. Tracking ends when all associated Actions complete or an unhandled exception terminates the run. | Scope | Metric | Type | Meaning | | --- | --- | --- | --- | | Agent | `numOfInputRunsSucceeded` | Counter | Input Runs that reached the normal completion boundary | | Agent | `numOfInputRunsFailed` | Counter | Input Runs terminated by an unhandled exception, including failures before run context creation | | Agent | `inputRunLatencyMs` | Histogram | Input receipt to run completion or failure | | Agent | `inputRunQueueLatencyMs` | Histogram | Input receipt to the start of run processing, primarily reflecting same-key queueing | | Agent | `inputRunProcessingLatencyMs` | Histogram | Start of run processing to completion or failure | | Agent | `numOfPendingInputEvents` | Gauge | Input Events waiting in same-key queues | | Agent | `numOfActiveInputRuns` | Gauge | Runs selected for processing but not yet complete | The latency boundaries are: ```text Input Event received │ ├─ inputRunQueueLatencyMs │ Input Run starts processing │ ├─ inputRunProcessingLatencyMs │ Input Run completed or failed inputRunLatencyMs = received → completed or failed ``` Here, processing starts when the Input Event leaves the pending queue and enters Event routing. It does not imply that the first Action has already begun executing. A child execution failure does not by itself make the Input Run fail. If the Agent handles the failure and reaches its normal completion boundary, the run is successful. ### 4.3 Action Action metrics distinguish between a physical `ActionTask` and a logical Action execution. One logical execution may suspend while waiting for asynchronous work and later resume through another `ActionTask`. | Scope | Metric | Type | Meaning | | --- | --- | --- | --- | | Action | `actionSchedulingLatencyMs` | Histogram | Initial `ActionTask` enqueue to its first execution | | Action | `actionExecutionLatencyMs` | Histogram | Action execution `started` to `finished` or `failed`, including asynchronous waiting | | Action | `numOfPendingActionTasks` | Gauge | Physical `ActionTask` instances waiting for execution, including continuations | | Action | `numOfActiveActionExecutions` | Gauge | Logical Action executions that started but have not reached a terminal state | Only the initial `ActionTask` contributes a scheduling-latency sample. A continuation updates the pending-task Gauge but does not create another scheduling sample. ### 4.4 LLM and Token Usage LLM health metrics use the configured ChatModel resource scope. Each sample represents one logical model call, not an individual provider attempt. | Scope | Metric | Type | Meaning | | --- | --- | --- | --- | | Model Resource | `numOfLlmCallsSucceeded` | Counter | Logical LLM calls that ultimately succeeded | | Model Resource | `numOfLlmCallsFailed` | Counter | Logical LLM calls that ultimately failed | | Model Resource | `llmCallLatencyMs` | Histogram | Logical call latency, including retry attempts and retry wait time | | Model Resource | `retryCount` | Counter | Provider retries performed | | Model Resource | `retryWaitSec` | Counter | Total retry wait time in seconds | | Model | `promptTokens` | Counter | Prompt tokens attributed to the provider-returned model name | | Model | `completionTokens` | Counter | Completion tokens attributed to the provider-returned model name | | Model | `totalTokens` | Counter | Total tokens attributed to the provider-returned model name | Structured-output parsing is a separate execution in Agent Trace. If the model call succeeds but parsing fails, the LLM call remains successful. The Parser failure is diagnosed through Agent Trace rather than counted as an LLM failure. ### 4.5 Tool, Skill, and MCP Server Each Tool call is one Tool execution. Skill and MCP Server metrics are additional aggregate projections derived from explicit Tool execution metadata; they do not introduce separate execution levels. | Scope | Metric | Type | Meaning | | --- | --- | --- | --- | | Tool | `numOfToolCallsSucceeded` | Counter | Successful Tool calls | | Tool | `numOfToolCallsFailed` | Counter | Failed Tool calls | | Tool | `toolCallLatencyMs` | Histogram | Tool-call latency | | Skill | `numOfSkillLoads` | Counter | Explicit `load_skill` calls that reached a terminal state | | Skill | `skillLoadLatencyMs` | Histogram | Explicit `load_skill` call latency | | MCP Server | `numOfMcpToolCallsSucceeded` | Counter | Successful Tool calls exposed by one MCP Server | | MCP Server | `numOfMcpToolCallsFailed` | Counter | Failed Tool calls exposed by one MCP Server | | MCP Server | `mcpToolCallLatencyMs` | Histogram | Latency of Tool calls exposed by one MCP Server | Every named Tool execution contributes to Tool metrics. Only an explicit `load_skill` Tool execution contributes to Skill metrics. The runtime does not infer that subsequent Tool calls belong to the loaded Skill. A Tool execution carrying MCP Server metadata contributes to both its Tool and MCP Server scopes. ## 5. Recording Semantics ### 5.1 Event-Driven Metrics Metrics derived from Business Events and Execution Events use the same semantic boundaries as Agent Trace: - A processed business Event increments Event throughput metrics. - Action Execution Events update Action execution latency and active-execution state. - Terminal LLM and Tool Execution Events update outcome counters and latency. - Tool execution metadata adds Skill and MCP Server aggregate projections where applicable. `execution_id` may be used internally to pair `started` and terminal Events, but it is never exposed as a metric scope or label. Metric aggregation consumes these in-process Events directly rather than reading them back from the Event Log. ### 5.2 Runtime State Metrics Input Run and scheduling metrics require state transitions that are not fully represented by Execution Events: - Input receipt, pending-queue enqueue, and pending-queue dequeue define run queue latency and pending counts. - Run processing start and terminal boundaries define processing latency and active-run counts. - `ActionTask` enqueue and dequeue define scheduling latency and pending-task counts. These observations extend the metric model without turning queue operations into public Event types. ### 5.3 Model Call Metrics Token usage and retry information are available only at the model-call boundary: - Retry metrics record the actual number of retries and cumulative wait time for the logical call. - Token metrics use usage data and the model name returned in the final response. These values complement the LLM Execution Event without expanding its lifecycle schema with provider-specific payloads. ### 5.4 Outcome Boundaries | Scenario | Metric outcome | | --- | --- | | An LLM call succeeds after retries | One successful LLM call; latency includes retries and wait; retry counters increase | | An LLM call succeeds but structured-output parsing fails | LLM success; Parser failure remains visible in Agent Trace | | A Tool returns an error response | Failed Tool execution, even if the containing Action handles the response | | A child execution fails but the Agent handles it | Child failure metrics increase; the Input Run can still succeed | | An Action suspends and resumes through a continuation | One logical Action execution; continuation affects pending tasks but not scheduling sample count | ## 6. Task Attempt and Recovery Semantics Flink metrics are scoped to the current task attempt. Counter, Meter, and Histogram values are not restored from checkpoints and do not provide exactly-once values across failures. Current-value Gauges may be reconstructed from restored runtime state when that state identifies pending or active work. Historical start timestamps cannot be reconstructed: - A latency sample is recorded only when the current attempt observes both the required start and terminal boundaries. - Restored pending or active work contributes to current-value Gauges without fabricating a historical start time. - LLM and Tool terminal outcomes observed in the current attempt may update counters even when no latency sample can be produced. Consumers must aggregate Gauge values across subtasks for a job-level view. Histograms include only samples observed by the current task attempts. ## 7. Cross-Language and Reporter Semantics Java and Python use the same logical execution entity types, lifecycle outcomes, metadata keys, metric names, and scope hierarchy. This alignment is part of the metric contract, even though the two runtimes observe LLM and Tool calls at different language-specific boundaries. `MetricGroup` scopes are the framework-level representation of dimensions. A reporter may expose them as labels or encode them in a metric identifier, but this does not change their logical meaning. Dashboards and queries must preserve or deliberately aggregate scopes rather than assume identical physical names across reporters. ## 8. Prototype Status The metric model has been prototyped in both the Java and Python runtime paths. The prototype covers: - Input Run outcome, latency, queue, and active-state metrics. - Action scheduling, execution latency, pending-task, and active-execution metrics. - Event-driven LLM and Tool outcome and latency metrics. - Model Resource, model, Tool, Skill, and MCP Server scopes. - Token usage and retry metrics. - Task-attempt and recovery behavior for Gauges and latency samples. - Java and Python semantic alignment through unit and runtime tests. The purpose of this discussion is to review the metric contracts and their relationship to Agent Trace before submitting the implementation upstream. ## 9. Summary Agent Trace and operational metrics answer different questions but share the same runtime semantics. Business Events and Execution Events provide stable observation boundaries for Event throughput and execution health. Run state, queue state, and model-call results provide the additional signals required by metrics. This proposal adds: - Input Run health and latency metrics. - Action scheduling and execution-state metrics. - LLM and Tool success, failure, and latency metrics. - Model Resource, model, Tool, Skill, and MCP Server scopes. - Explicit cardinality and task-attempt semantics. - A shared metric contract for Java and Python. The result is a bounded-cardinality operational metric surface that complements per-run Agent Trace without depending on Event Log storage or introducing per-run identifiers into metrics. GitHub link: https://github.com/apache/flink-agents/discussions/901 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
