Hi all, I'd like to open a discussion about whether incremental scan semantics (specifically incremental append scans) should be codified in the table spec, and if so, what the contract should be.
The gap: Today the spec is effectively silent on incremental scans. In practice, the Java IncrementalAppendScan is the de facto reference implementation, and everyone else is following it: - PyIceberg -- implemented by following Java github.com/apache/iceberg-python/pull/3512 - iceberg-cpp -- incremental scan API following Java: github.com/apache/iceberg-cpp/pull/559 - iceberg-rust -- I have a PR up that, likewise, just mirrors Java github.com/apache/iceberg-rust/pull/2337 So we have four implementations converging on behavior that lives only in one codebase. That's fragile: any of Java's incidental choices become the unwritten contract, and there's no spec to point to when implementations disagree. There's a good precedent for what the fix could look like. The spec already has a "Point-in-Time Reads (Time Travel)" section (iceberg.apache.org/spec#… <https://iceberg.apache.org/spec/#point-in-time-reads-time-travel>) that specifies read semantics in prose: which metadata to use to resolve the snapshot (snapshot-log), how the two histories can diverge, and what to do when the metadata is missing (raise an informative error). Codifying them would fill a gap right next to something the spec already covers. Why this is now also a catalog concern, not just an engine concern? This used to be purely an engine-side matter. But the REST scan-planning API has changed that. REST spec already exposes start-snapshot-id ("Starting snapshot ID for an incremental scan (exclusive)") and end-snapshot-id ("Ending snapshot ID for an incremental scan (inclusive). Required when start-snapshot-id is specified."), and the planTableScan description states that incremental scans are planned by passing both, and that mixing incremental with point-in-time parameters is invalid. In other words, the REST spec already obligates a catalog to produce an incremental scan plan, but "correct" is defined only by whatever Java happens to do. Two conformant catalogs could return different plans for the same request and both claim REST compliance. It also means boundary conditions (exclusive start / inclusive end) currently live only in the REST documentation, not in the spec, so a non-REST engine has no shared source of truth. The core principle for me is that the result of an incremental scan should not depend on where it was planned. Whether an engine plans the scan locally against table metadata, or delegates to a catalog via planTableScan, it should get the same set of files back. Today that's only true by coincidence because every planner happens to copy Java. If the semantics aren't in the spec, then server-side planning and client-side planning are free to diverge, and the same query returns different data purely based on whether the catalog implements the scan-plan API. That's a correctness property that shouldn't be left to chance I think. Some of the questions I think would need answering: If we do put this in the spec, these are the ambiguities I'd want pinned down (drawn from Java's current behavior): - Range / traversal: is the scan the linear ancestry from end-snapshot-id back to but excluding start-snapshot-id? What's required if start is not an ancestor of end? (Java validates ancestry and errors if the start is not anancestor / parent-ancestor of the end.) - Non-append operations in the range: Java today skips non-append snapshots in the range -- appendsBetween keeps only snapshots whose operation is "append" and ignores the rest (replace/overwrite/delete). - Boundary semantics: exclusive-start / inclusive-end is currently only asserted in the REST spec should it be normative in the spec more generally? - Which schema? Java projects the incremental scan result into the current table schema. But the range spans multiple snapshots, so "the snapshot schema" isn't well-defined the way it is for a point-in-time scan. I don't have a strong predetermined answer on whether this belongs in the spec, I mainly want to surface that it's currently implementation-defined,that multiple implementations are already depending on it, and that the REST scan-plan API has quietly turned it into a conformance question for catalogs too. Interested to hear what folks think. Thanks, Xander
