weiqingy commented on code in PR #29496: URL: https://github.com/apache/flink/pull/29496#discussion_r4291017867
########## docs/content/docs/ops/metrics.md: ########## @@ -2382,6 +2382,34 @@ Metrics below can be used to measure the effectiveness of speculative execution. </tbody> </table> +### UDF + +The following metrics are registered per SQL/Table user-defined function when UDF metrics are enabled (see [UDF metrics](#udf-metrics) below). They are attached to the operator that runs the function and scoped under `udf.<udf_name>`, so the full metric identifier is `<operator_name>.udf.<udf_name>.<metric>`. + +<table class="table table-bordered"> + <thead> + <tr> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 30%">Metrics</th> + <th class="text-left" style="width: 44%">Description</th> + <th class="text-left" style="width: 8%">Type</th> + </tr> + </thead> + <tbody> + <tr> + <th rowspan="2"><strong>Task/Operator</strong></th> Review Comment: Thanks, fixed in b8d05f90891 on both pages. The table now uses the `Operator` scope with a `udf.<udf_name>` infix, following this page's Scope/Infix convention, and the intro now says the identifier is the operator scope (`metrics.scope.operator`) followed by `udf.<udf_name>.<metric>`. I used the page's existing `metrics.scope.operator` term rather than a new `<operator_scope>` placeholder, which the page doesn't define. ########## docs/content/docs/ops/metrics.md: ########## @@ -2441,6 +2469,23 @@ It is recommended to only use them for debugging purposes. If state.ttl is enabled, the size of the value will include the size of the TTL-related timestamp. The value size of AggregatingState is not accounted for because AggregatingState returns a result processed by a user-defined AggregateFunction, whereas currently, only the actual stored data size in the state can be tracked. +## UDF metrics + +Flink can track the processing time and exceptions of SQL/Table user-defined functions on a per-operator basis. This feature is disabled by default. +To enable it you must set `table.exec.udf-metric-enabled` to `true`. When disabled, nothing is registered and there is no overhead. + +Once enabled, each user-defined function gets two metrics on the operator that runs it, scoped under `udf.<udf_name>`: a `udfProcessingTime` histogram and a `udfExceptionCount` counter. + +Flink samples the processing time every `N` invocations, in which `N` is defined by `table.exec.udf-metric.sample-interval`. +This configuration has a default value of 100. A smaller value will get more accurate results but have a higher performance impact since it is sampled more frequently. +Because only the sampled invocations are timed, the remaining invocations cost a single counter increment, so the default interval keeps the overhead of the enabled path small. +Exceptions are counted on every invocation and are not sampled. + +The metrics cover scalar and table functions that a query calls directly, both synchronous and asynchronous. Not covered: process table functions (`PROCESS_TABLE`), aggregate functions, functions registered through the deprecated `registerFunction` API, and the functions invoked internally by lookup joins, model prediction, and vector search. Review Comment: Agreed, fixed in b8d05f90891 on both pages: Python functions are now the first entry under "Not covered". `PythonScalarFunction#eval` and `PythonTableFunction#eval` are placeholders that throw if called, and the planner routes Python calls to the dedicated Python calc/correlate operators, so they never reach the instrumented call sites. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
