iamrajatrana opened a new pull request, #17950: URL: https://github.com/apache/iceberg/pull/17950
Async micro-batch initial preload currently combines its row and file safety limits with OR, so it continues until both limits are reached. This can preload substantially more snapshots and retain excessive file tasks when rows-per-file is skewed. This changes the condition to stop when either limit is reached and adds regression coverage for both asymmetric cases across Spark 3.5, 4.0, and 4.1. Snapshots are still added atomically, so a limit may be exceeded by one snapshot. Tests: - Spark 3.5 `TestAsyncSparkMicroBatchPlanner` - Spark 4.0 `TestAsyncSparkMicroBatchPlanner` - Spark 4.1 `TestAsyncSparkMicroBatchPlanner` - `./gradlew spotlessApply` --- **AI Disclosure** - Model: GPT-5 (Codex) - Platform/Tool: OpenAI Codex - Human Oversight: unreviewed - Prompt Summary: Identify, validate, and fix excessive initial queue preloading in the asynchronous Spark micro-batch planner. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
