markhoerth opened a new pull request, #14463:
URL: https://github.com/apache/gravitino/pull/14463
Backport to branch-1.3 of the fix for #14461 (same change as the PR to main).
### What changes were proposed in this pull request?
- Incremental scans use `fromSnapshotExclusive`, and a range from a snapshot
to itself is empty.
- A `ValidationException` raised while files are planned (unknown column,
wrong case, wrong value type) is reported as 400 without the table's schema
struct. Unknown `stats-fields` names are refused up front with a plain message.
- `IcebergObjectMapper` accepts the spec's constant filters `{"type":
"true"}` / `{"type": "false"}` anywhere in the filter, as well as Iceberg's
boolean form.
- `checkDeleteFilesCanBeReferenced`: when a plan contains different delete
files with the same location (deletion vectors sharing a Puffin file), the plan
is refused with an error, because Iceberg's response serializer would reference
the wrong deletion vector for every task but one. Wrong results are worse than
an error; the client can plan the scan itself.
If #14458 is merged first, `IcebergObjectMapper` needs a trivial merge (both
register a module after `RESTSerializers.registerAll`).
### Why are the changes needed?
Spec-conformant clients received the start snapshot's files twice, could not
send the spec's constant filters, got 500 for invalid filters, and could read
wrong data from v3 tables with deletion vectors.
Fixes #14461
### Does this PR introduce _any_ user-facing change?
Yes. Incremental plans exclude the start snapshot. Invalid filters return
400. Spec constant filters are accepted. Plans whose deletion vectors cannot be
referenced correctly return an error instead of wrong references.
### How was this patch tested?
`TestIcebergScanPlanning`: exclusive start and the empty same-snapshot
range; unknown column, wrong case, and unknown stats field (400 without the
schema struct or NPE text); constant filters in the spec form and nested;
deletion vectors sharing a file (refused) and a shared delete file (accepted).
`./gradlew :iceberg:iceberg-rest-server:test -PskipITs`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Eu6fb9pLfmrSJzGQDVLi1g
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
