iemejia commented on PR #55924: URL: https://github.com/apache/spark/pull/55924#issuecomment-5526610630
Thanks @LuciferYang, both review comments addressed in cee762f73f2: 1. **Corrupt suffix length no longer over-allocates.** `getSuffixLength` now validates the length (negative, and against `in.available()`, which is exact at that call site) before any caller sizes `prevBuf`, so a corrupt page fails fast with a `ParquetDecodingException` rather than an up-to-2GB allocation / `OutOfMemoryError`. Negative `prefixLength` is now guarded in `checkPrefixLength` too. 2. **Geo skip-interleave test can now catch a bug.** New `growingPrefixPolygons()` (counts `8, 12, 12, 16, 30, 30`) mirrors `longPrefixValues`: a read now exercises `readGeoData`'s grow-and-preserve branch with a non-zero prefix, and consecutive equal-length polygons share a longer prefix than values two apart, so a `skipBinary` that stopped maintaining `prevBuf` would decode wrong bytes instead of passing. `sharedPrefixPolygons` (and the corrupt-first-geo / `setPreviousReader` tests) are untouched. New coverage: `corrupt suffix/prefix lengths fail fast without over-allocating`. `ParquetDeltaByteArrayEncodingSuite` (28) + `ParquetDeltaLengthByteArrayEncodingSuite` (14), plus `ParquetEncodingSuite` + `ParquetGeoSuite` (13), all pass. PTAL, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
