Xinyao Zhang created SPARK-60016:
------------------------------------
Summary: Real-time shuffle intermittently fails with oversized
transport frames in Kafka tests
Key: SPARK-60016
URL: https://issues.apache.org/jira/browse/SPARK-60016
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Xinyao Zhang
Real-time Kafka tests intermittently fail in TransportFrameDecoder with an
oversized frame error. The streaming query terminates, but the test continues
waiting for batchId to advance and reports a 60-second timeout instead of the
underlying transport error.
Observed CI failures:
- On upstream commit 5926a10389e8eace9f655dc138f4bb30254f7bed,
KafkaRealTimeModeWindowSuite's "sliding window count" failed with
IllegalArgumentException: Too large frame: 206159609852 in a
streaming-shuffle-writer server thread:
https://github.com/apache/spark/actions/runs/37394387916/job/112291798245
- KafkaRealTimeModeAggregationSuite's "tumbling window min" failed twice on PR
#59249, commit 8d0370aa2da539e9ac2fb6ebd144acb87392f901. Attempt 1 reported the
same frame size, 206159609852; attempt 2 reported 210454577148:
https://github.com/zhangxinyao88/spark/actions/runs/37411255644/attempts/1
https://github.com/zhangxinyao88/spark/actions/runs/37411255644/attempts/2
The stack begins at TransportFrameDecoder.decodeNext (line 149), then
channelRead (line 99). StreamingShuffleServerHandler records the exception, the
shuffle producer stage fails, and MicroBatchExecution logs that the query
terminated with an error. The eventual batchId assertion subsequently times
out. The underlying errors are in the archived Kafka unit-tests.log artifacts;
the test annotations show only the timeout.
The unchanged PR commit passed on the third CI attempt. All four
KafkaRealTimeModeAggregationSuite cases also passed locally on both the PR
commit and its upstream baseline, dd669799743e4579e99618c94c4e3eefa48257e1,
using JDK 17. Local testing used a temporary SBT source filter excluding three
unrelated Parquet suites whose test helpers were unavailable locally.
The failure is intermittent. A deterministic reproducer, the cause of the
unexpected frame length, and any production impact are not yet established. The
independent upstream failure shows that the error can occur without the changes
in PR #59249.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
