zahed1994 commented on PR #58176: URL: https://github.com/apache/spark/pull/58176#issuecomment-5519801683
Thanks @krismo-oai. Good catch on the decoder boundary. Updated the patch to validate decoded lengths in `VectorizedDeltaByteArrayReader` before passing them to `WritableColumnVector`: - Checked `prefixLength >= 0`, `suffixLength >= 0`, `prefixLength <= previous.remaining()`, and `(long) prefixLength + suffixLength <= Integer.MAX_VALUE`, throwing `ParquetDecodingException` on invalid values. This keeps malformed delta pages on the corrupt file path when `ignoreCorruptFiles=true`. - Added defensive checks in `WritableColumnVector.reserveAdditional` and `appendBytes` to reject negative arguments with `IllegalArgumentException`, keeping `VectorizedReaderCapacityOverflowException` strictly for genuine capacity exhaustion. - Added decoder unit tests in `ParquetDeltaByteArrayEncodingSuite` for negative prefix and prefix exceeding previous length. - Added an end-to-end file-scan test in `ParquetQuerySuite` (exercised on both V1 and V2) using a mutated 2-row delta byte array fixture (`prefixLength = -2`), verifying it gets skipped under `ignoreCorruptFiles=true` and surfaces `ParquetDecodingException` (not capacity overflow) under `ignoreCorruptFiles=false`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
