weiqingy commented on code in PR #28878: URL: https://github.com/apache/flink/pull/28878#discussion_r3753878793
########## flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/metrics/UdfMetrics.java: ########## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.runtime.operators.metrics; + +import org.apache.flink.metrics.Counter; +import org.apache.flink.metrics.Histogram; +import org.apache.flink.metrics.MetricGroup; +import org.apache.flink.metrics.ThreadSafeSimpleCounter; +import org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogram; +import org.apache.flink.util.Preconditions; + +/** + * Per-operator metrics for a single user-defined function, registered under {@code + * <operator>.udf.<udfName>}: {@code udfProcessingTime} (a latency histogram) and {@code + * udfExceptionCount} (a counter of exceptions escaping the function). + * + * <p>Timing is sampled: only one invocation out of every {@code sampleInterval} is measured, so a + * hot function does not pay {@code System.nanoTime()} on every record. The sample decision advances + * a plain {@code int} and is only ever taken on the task thread, so it needs no synchronization. + * The two registered metrics are safe for the async completion thread to touch: the histogram + * synchronizes internally and the exception counter is {@link ThreadSafeSimpleCounter}. + */ +public final class UdfMetrics { + + private static final int HISTORY_SIZE = 128; + + private final int sampleInterval; Review Comment: Thanks for the review! It's not time, it's a count of calls. At the default of 100, one call in every 100 gets timed; set it to 1 and every call is timed. Fair point that the declaration didn't say that. I've added a line there and made the `@param` spell it out. ########## flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/metrics/UdfMetrics.java: ########## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.runtime.operators.metrics; + +import org.apache.flink.metrics.Counter; +import org.apache.flink.metrics.Histogram; +import org.apache.flink.metrics.MetricGroup; +import org.apache.flink.metrics.ThreadSafeSimpleCounter; +import org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogram; +import org.apache.flink.util.Preconditions; + +/** + * Per-operator metrics for a single user-defined function, registered under {@code + * <operator>.udf.<udfName>}: {@code udfProcessingTime} (a latency histogram) and {@code + * udfExceptionCount} (a counter of exceptions escaping the function). + * + * <p>Timing is sampled: only one invocation out of every {@code sampleInterval} is measured, so a + * hot function does not pay {@code System.nanoTime()} on every record. The sample decision advances + * a plain {@code int} and is only ever taken on the task thread, so it needs no synchronization. + * The two registered metrics are safe for the async completion thread to touch: the histogram + * synchronizes internally and the exception counter is {@link ThreadSafeSimpleCounter}. + */ +public final class UdfMetrics { + + private static final int HISTORY_SIZE = 128; + + private final int sampleInterval; + private final Histogram processingTime; + private final Counter exceptionCount; + + // Sample counter; only touched on the task thread, hence a plain int with no synchronization. + private int invocationCount = 0; + + private UdfMetrics(int sampleInterval, Histogram processingTime, Counter exceptionCount) { + this.sampleInterval = sampleInterval; + this.processingTime = processingTime; + this.exceptionCount = exceptionCount; + } + + /** + * Registers the {@code udfProcessingTime} histogram and {@code udfExceptionCount} counter under + * {@code <operatorMetricGroup>.udf.<udfName>}. + * + * @param sampleInterval measure one invocation out of every {@code sampleInterval}; must be + * {@code >= 1} ({@code 1} measures every invocation) + */ + public static UdfMetrics register( + MetricGroup operatorMetricGroup, String udfName, int sampleInterval) { + Preconditions.checkArgument( + sampleInterval >= 1, + "UDF metric sample interval must be >= 1, but was %s.", + sampleInterval); + MetricGroup group = operatorMetricGroup.addGroup("udf", udfName); + return new UdfMetrics( + sampleInterval, + group.histogram( + "udfProcessingTime", new DescriptiveStatisticsHistogram(HISTORY_SIZE)), + group.counter("udfExceptionCount", new ThreadSafeSimpleCounter())); + } + + /** + * Returns {@code true} for the one invocation in every {@code sampleInterval} whose processing + * time should be measured. Must be called once per invocation, on the task thread only. + */ + public boolean shouldSample() { + if (sampleInterval == 1) { Review Comment: Fair point, the name doesn't tell you that on its own. Would it be worth keeping for consistency though? Flink's state latency tracking does the same kind of sampling with the same word: `state.latency-track.sample-interval` is an int, defaults to 100, and means "track the latency every 100 access requests". Its `MetricsTrackingStateConfig` declares the same `private final int sampleInterval` with the same `>= 1` check. FLIP-485 followed that pattern, and the matching option here, `table.exec.udf-metric.sample-interval`, lands in the next PR. What I'd worry about with a rename is the field no longer matching the option users actually set. So I've made the field say it counts invocations, not time. Does that address it, or would you still prefer a different name? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
