Hi everyone,
I'd like the community's call on how REST servers should send residuals
while
clients migrate to the new expression syntax, and I'm proposing conformance
fixtures in apache/iceberg-verification to track that migration.
Background
The expressions spec (apache/iceberg#17138) changed the preferred predicate
form:
before (deprecated): {"type": "eq", "term": "id", "value": 5}
after (preferred): {"type": "eq", "left": {"type": "reference",
"name": "id"}, "right": 5}
It also adds "literals" objects and "apply". Java, Go and C++ still read
and write
only the deprecated predicate form (Go checked by running it; Java and C++
by
reading the source).
1. Residuals during the migration
The spec says services should omit residuals unless the client supports the
new
syntax, "for example, via client version". But the Java reference server
always
sends residuals, and there's no reliable way to detect support:
X-Client-Version
isn't in the REST spec, and Java sends its library version while iceberg-go
sends
the REST spec version. Options:
a. Always omit residuals. Simple and correct, but clients re-evaluate the
full
filter per task instead of the server's simplified residual.
b. Clients declare support explicitly (an optional PlanTableScanRequest
field
or a spec-defined header); servers send new-form residuals only to
them.
c. Infer from client version. Unreliable, for the reason above.
I'd suggest (b), with readers in every implementation updated before any
writer
switches.
2. Conformance fixtures
Add read-conformance surfaces to apache/iceberg-verification for expression
JSON
(deprecated and preferred forms) and REST scan tasks. Implementations opt
in, and
forms they can't read yet show up as UNSUPPORTED, so the migration is
visible
across implementations. Does this fit the next phase of
apache/iceberg-verification#10? Happy to open the PRs.
Related spec-doc fixes, filed separately:
- Drop {"type":"true"} from OpenAPI:
https://github.com/apache/iceberg/issues/18351
- Hex case for binary/fixed:
https://github.com/apache/iceberg/issues/18352
Thanks,
Revanth
