slachiewicz opened a new pull request, #725: URL: https://github.com/apache/hudi-rs/pull/725
## Description > **Stacked on #724.** The diff shown here includes that PR's three commits and will shrink > to the two below once it merges. Review the last two commits only: > `fix(core): decode partition values after splitting, and honour the Hive null sentinel` > and `feat(core): report partition filters that could not be applied`. Two independent correctness problems in partition path handling, plus the guard that would have made #580 visible instead of silent. **Decoding ran before splitting.** `parse_segments` percent-decoded the whole path and then split it on `/` and `=`. Hudi escapes the partition *value* only and leaves the structural separators literal, per `PartitionPathEncodeUtils.escapePathName` in `KeyGenUtils.getRecordPartitionPath`, so a value written as `a%2Fb` was decoded to `a/b` and then torn into two segments. A url-encoded table with any escaped separator in a value could not be read at all. Splitting now happens first and each value is decoded exactly once. The url-encoding test this replaces escaped the separators too, which no Hudi writer does, so it passed against input that cannot occur. The replacements cover escaped `/`, `=`, space and `%` inside real segments, and pin that `%252F` decodes to `%2F` rather than to `/`. **The Hive null sentinel was parsed as a value.** `__HIVE_DEFAULT_PARTITION__` was cast to the partition column's type, which errors on anything numeric or temporal. A parse error is treated as "keep the partition", so such a table quietly degraded to a full scan. It is now read as null and compared under SQL three-valued logic, so a comparison against it is unknown and the partition is retained deliberately rather than by accident. **Unapplied filters are now reported.** A filter that failed to bind to the partition schema was dropped with no record. That is correct for a data-column predicate, which is enforced per row, but it is also how a *partition* predicate disappears: when a key generator cannot be built, the untransformed filter still names the source column, which the partition schema does not contain. The result was a full scan that returned no error and could not be told apart from a successful one. This is the same failure mode as #580, reached by a different route, and #581 fixed only the ordering cause. `PartitionPruner` now records those filters and whether the key generator was unavailable, exposed as `unapplied_filters` and `is_keygen_config_unavailable`, and logs each drop. A caller that needs pruning to actually happen, rather than silently degrade, can assert on them. Worth knowing for the second one: the `hoodie.keygen.timebased.*` options are writer-side and are not guaranteed to reach `hoodie.properties`, so a table can name `TimestampBasedKeyGenerator` while carrying nothing that says which granularity, timezone or format it used. Guessing from the class name is unsafe, so pruning is skipped and reported instead of approximated. Verified: `make format check test` passes, and `cargo clippy --all-targets --all-features --workspace -- -D warnings` is clean. ## How are the changes test-covered - [ ] N/A - [x] Automated tests (unit and/or integration tests) - [ ] Manual tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
