Copilot commented on code in PR #29496: URL: https://github.com/apache/flink/pull/29496#discussion_r4290970478
########## docs/content/docs/ops/metrics.md: ########## @@ -2441,6 +2469,23 @@ It is recommended to only use them for debugging purposes. If state.ttl is enabled, the size of the value will include the size of the TTL-related timestamp. The value size of AggregatingState is not accounted for because AggregatingState returns a result processed by a user-defined AggregateFunction, whereas currently, only the actual stored data size in the state can be tracked. +## UDF metrics + +Flink can track the processing time and exceptions of SQL/Table user-defined functions on a per-operator basis. This feature is disabled by default. +To enable it you must set `table.exec.udf-metric-enabled` to `true`. When disabled, nothing is registered and there is no overhead. + +Once enabled, each user-defined function gets two metrics on the operator that runs it, scoped under `udf.<udf_name>`: a `udfProcessingTime` histogram and a `udfExceptionCount` counter. + +Flink samples the processing time every `N` invocations, in which `N` is defined by `table.exec.udf-metric.sample-interval`. +This configuration has a default value of 100. A smaller value will get more accurate results but have a higher performance impact since it is sampled more frequently. +Because only the sampled invocations are timed, the remaining invocations cost a single counter increment, so the default interval keeps the overhead of the enabled path small. +Exceptions are counted on every invocation and are not sampled. + +The metrics cover scalar and table functions that a query calls directly, both synchronous and asynchronous. Not covered: process table functions (`PROCESS_TABLE`), aggregate functions, functions registered through the deprecated `registerFunction` API, and the functions invoked internally by lookup joins, model prediction, and vector search. Review Comment: Please explicitly exclude Python UDFs here and in the Chinese page. `CommonExecPythonCalc` and `CommonExecPythonCorrelate` select dedicated Python operators rather than the instrumented JVM call paths, so enabling this option does not register these metrics for Python scalar or table functions. The current coverage statement gives those users the wrong expectation. ########## docs/content/docs/ops/metrics.md: ########## @@ -2382,6 +2382,34 @@ Metrics below can be used to measure the effectiveness of speculative execution. </tbody> </table> +### UDF + +The following metrics are registered per SQL/Table user-defined function when UDF metrics are enabled (see [UDF metrics](#udf-metrics) below). They are attached to the operator that runs the function and scoped under `udf.<udf_name>`, so the full metric identifier is `<operator_name>.udf.<udf_name>.<metric>`. + +<table class="table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 30%">Metrics</th> + <th class="text-left" style="width: 44%">Description</th> + <th class="text-left" style="width: 8%">Type</th> + </tr> + </thead> + <tbody> + <tr> + <th rowspan="2"><strong>Task/Operator</strong></th> Review Comment: Please correct the identifier and table scope on both pages. `UdfMetrics.register` appends its groups to the complete operator scope, which includes `<subtask_index>` after `<operator_name>` by default. Filters using the documented identifier therefore will not match the default identifiers. Registration is also only on the operator metric group; this page defines `Task/Operator` as registration for both entities. Use `<operator_scope>` for the prefix and `Operator` in the table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
