dejii opened a new pull request, #38580: URL: https://github.com/apache/beam/pull/38580
## Summary `SerializableDataFile.toByteArrayMap` used `ByteBuffer.array()` to extract bound bytes. `array()` returns the full backing array, ignoring the buffer's `[position, limit)` content window. Iceberg's [Conversions.toByteBuffer](https://github.com/apache/iceberg/blob/56013b72db456f774efb696e237b58de88c2e94e/parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetrics.java#L247-L248) delegates to JDK [CharsetEncoder.encode(CharBuffer)](https://github.com/apache/iceberg/blob/56013b72db456f774efb696e237b58de88c2e94e/api/src/main/java/org/apache/iceberg/types/Conversions.java#L112), which sizes its allocation by `(int)(chars * averageBytesPerChar)`. For UTF-8 the average is 1.1, so a 15-char string lands in a 16-byte buffer with position=0, limit=15, capacity=16. The 16th byte is the zero-initialized tail of the backing array. Reading via `.array()` leaks that tail into the byte[] form, and on the return trip `ByteBuffer.wrap(byte[])` treats it as content. The manifest ends up with a lower bound that is strictly greater than the real minimum byte-lexicographically. Query engines using Iceberg metadata for predicate pushdown decide `value < lower_bound` for any equality filter on that minimum and prune every file, returning 0 rows. ## Why the bug stays quiet for most queries All three must hold: 1. **Encoded length 10–N bytes** (N = `truncate(N)` from metrics config, default 16). Below 10, the JDK encoder allocates exact-size buffers; above N, Iceberg trims the string first, so the bound holds a truncated form that a full-value query naturally sorts past. 2. **Value is the file's actual lower bound** on that column. Other values never compare against the padding byte. 3. **Predicate is eligible for bound pruning** (`=`, `IN`, etc.). ## Fix - Fix by copying the buffer's [position, limit) window. Added a regression test that builds a DataFile with bounds matching that shape, roundtrips through `SerializableDataFile`, and asserts the resulting byte[] match the encoded content rather than the inflated backing array. - New writes will produce correct bounds. Tables already written by the buggy sink keep broken bounds until their manifests are rewritten ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier). To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md) GitHub Actions Tests Status (on master branch) ------------------------------------------------------------------------------------------------ [](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule) See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
