I would recommend always dropping residuals. They are expensive to calculate and almost never applied.
On Fri, Oct 2, 2026 at 11:18 AM Revanth Ch <[email protected]> wrote: > Hi everyone, > > I'd like the community's call on how REST servers should send residuals > while > clients migrate to the new expression syntax, and I'm proposing conformance > fixtures in apache/iceberg-verification to track that migration. > > Background > > The expressions spec (apache/iceberg#17138) changed the preferred > predicate form: > > before (deprecated): {"type": "eq", "term": "id", "value": 5} > after (preferred): {"type": "eq", "left": {"type": "reference", > "name": "id"}, "right": 5} > > It also adds "literals" objects and "apply". Java, Go and C++ still read > and write > only the deprecated predicate form (Go checked by running it; Java and C++ > by > reading the source). > > 1. Residuals during the migration > > The spec says services should omit residuals unless the client supports > the new > syntax, "for example, via client version". But the Java reference server > always > sends residuals, and there's no reliable way to detect support: > X-Client-Version > isn't in the REST spec, and Java sends its library version while > iceberg-go sends > the REST spec version. Options: > > a. Always omit residuals. Simple and correct, but clients re-evaluate > the full > filter per task instead of the server's simplified residual. > b. Clients declare support explicitly (an optional PlanTableScanRequest > field > or a spec-defined header); servers send new-form residuals only to > them. > c. Infer from client version. Unreliable, for the reason above. > > I'd suggest (b), with readers in every implementation updated before any > writer > switches. > > 2. Conformance fixtures > > Add read-conformance surfaces to apache/iceberg-verification for > expression JSON > (deprecated and preferred forms) and REST scan tasks. Implementations opt > in, and > forms they can't read yet show up as UNSUPPORTED, so the migration is > visible > across implementations. Does this fit the next phase of > apache/iceberg-verification#10? Happy to open the PRs. > > Related spec-doc fixes, filed separately: > - Drop {"type":"true"} from OpenAPI: > https://github.com/apache/iceberg/issues/18351 > - Hex case for binary/fixed: > https://github.com/apache/iceberg/issues/18352 > > Thanks, > Revanth >
