zahed1994 opened a new pull request, #58176: URL: https://github.com/apache/spark/pull/58176
### What changes were proposed in this PR? This PR introduces `VectorizedReaderCapacityOverflowException` thrown by `WritableColumnVector` when the vectorized reader capacity is exhausted or encounters an integer overflow. `DataSourceUtils.shouldIgnoreCorruptFileException` is updated so that `VectorizedReaderCapacityOverflowException` is not ignored when `spark.sql.files.ignoreCorruptFiles=true`. Additionally, invalid negative element capacities (from corrupted file data) throw `IllegalArgumentException` so corrupt files are still safely skipped. ### Why are the changes needed? When `ignoreCorruptFiles=true`, vectorized reader capacity overflow was previously caught as a general `RuntimeException`, silently skipping files and dropping user data without warning. ### Does this PR introduce any user-facing change? No API changes. Fixes incorrect silent file skipping on vectorized reader capacity overflow. ### How was this patch tested? - Added unit tests in `ParquetQuerySuite` asserting `DataSourceUtils.shouldIgnoreCorruptFileException` behavior. - Ran scalastyle (`sql/scalastyle`) and compilation checks (`sql/Test/compile`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
