Steven Schlansker created KAFKA-21076:
-----------------------------------------
Summary: Oversize decompressed telemetry payload returns
INVALID_RECORD and permanently disables client telemetry
Key: KAFKA-21076
URL: https://issues.apache.org/jira/browse/KAFKA-21076
Project: Kafka
Issue Type: Bug
Components: clients, core
Affects Versions: 4.3.1
Reporter: Steven Schlansker
Since [PR #22327|https://github.com/apache/kafka/pull/22327] (4.3.1, "MINOR:
Fixed metrics decompression"), \{{ClientTelemetryUtils.decompress}} throws
\{{TelemetryTooLargeException}} when the decompressed payload exceeds
\{{telemetry.max.bytes}}. The call runs inside
\{{ClientMetricsManager.processPushTelemetryRequest}} in the plugin export
block, whose \{{catch (Throwable)}} maps every failure to
\{{Errors.INVALID_RECORD}}:
{code:java}
catch (Throwable exception) {
clientMetricsStats.recordPluginErrorCount(clientInstanceId);
clientInstance.lastKnownError(Errors.INVALID_RECORD);
log.error("Error exporting client metrics to the plugin for client instance
id: {}", clientInstanceId, exception);
return request.errorResponse(0, Errors.INVALID_RECORD);
}
{code}
KIP-714 defines INVALID_RECORD as "Broker failed to decode or validate the
client's encoded metrics. Log an error and stop pushing metrics. This is viewed
as a problem in the client implementation of metrics serialization that is not
likely to be resolved by retrying." The Java client implements exactly that in
\{{ClientTelemetryUtils.maybeFetchErrorIntervalMs}}: on INVALID_RECORD it sets
the push interval to \{{Integer.MAX_VALUE}}, so telemetry from that client
instance stops until the process restarts.
KIP-714 defines TELEMETRY_TOO_LARGE for a payload that is too large, and the
client handles that code by retrying at the normal interval.
Observed effect: after a broker upgrade from 4.3.0 to 4.3.1, every Kafka
Streams stream-thread consumer (payload several MB decompressed, under 1 MiB
compressed) got INVALID_RECORD on its next push and disabled its telemetry. All
\{{org.apache.kafka.stream.\*}} metrics that KIP-1076 routes through the
stream-thread consumers disappeared from the metrics backend in every
environment at the same time. Broker log:
{noformat}
ERROR Error exporting client metrics to the plugin for client instance id:
xkdFrlXGTGig-RPx5SjuQA (org.apache.kafka.server.ClientMetricsManager)
org.apache.kafka.common.errors.TelemetryTooLargeException: Decompressed
telemetry metrics exceed maximum allowed size: 1048576
{noformat}
Two further points:
* KIP-714 documents \{{telemetry.max.bytes}} as "The maximum size (after
compression if compression is used) of telemetry pushed from a client to the
broker." PR #22327 changed the meaning to also bound the decompressed size,
with no KIP update, no release note, and no new configuration. A 4.3.0 cluster
that accepted a client's pushes rejects the same pushes on 4.3.1.
* Before PR #22327 the same code path allocated \{{metrics.limit() \* 2}} bytes
and grew without bound, so a bound is reasonable. The bound needs its own error
code and, given that decompression ratios of 10x or more are normal for OTLP
metrics, its own configuration or a multiple of \{{telemetry.max.bytes}}.
Suggested fix: catch \{{TelemetryTooLargeException}} before the generic
\{{catch (Throwable)}} and return \{{Errors.TELEMETRY_TOO_LARGE}}, so the
client retries at its interval instead of disabling telemetry. Document the
decompressed bound in \{{telemetry.max.bytes}} or add a separate configuration,
and mention the behavior change in the upgrade notes.
Workaround: raise \{{telemetry.max.bytes}} on the brokers and restart affected
clients.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
