superdiaodiao opened a new pull request, #57297: URL: https://github.com/apache/spark/pull/57297
### What changes were proposed in this pull request? This PR updates `KafkaDataConsumer` to log a warning when a Kafka poll returns zero visible records but advances the consumer position. It also fixes the nearby comment from "not invisible" to "invisible". ### Why are the changes needed? When transactional or aborted Kafka messages are invisible under `isolation.level=read_committed`, Spark can poll zero visible records while the offset still advances. This is a valid path, so Spark should not throw, but the current behavior is silent and makes troubleshooting poll delays difficult. The warning gives users the group id, topic partition, poll timeout, and offset movement so they can distinguish this path from out-of-range offsets and true poll timeouts. ### Does this PR introduce _any_ user-facing change? Yes. Spark now emits a warning for the valid empty-poll path where the Kafka consumer position advances despite no visible records being returned. ### How was this patch tested? - `git diff --check` - `python3 dev/structured_logging_style.py --help` (the script completed and reported the structured logging style check passed) Not run locally: - `./dev/scalafmt --test` was attempted, but the script maps to `./build/mvn scalafmt:format ...` and produced no output for over 90 seconds, so I interrupted it. - `./build/mvn -pl connector/kafka-0-10-sql -DskipTests compile` was attempted, but produced no output for over 60 seconds, so I interrupted it. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
