daviddallakyan2005 opened a new pull request, #3012:
URL: https://github.com/apache/iceberg-rust/pull/3012
## Which issue does this PR close?
- Closes #1690
Stacked on #3011 (slices 1–3: REST plan / fetch / wait client). This is
slice 4: wire `TableScan::plan_files` and decode REST content-file JSON so
auto-route is safe.
## What changes are included in this PR?
- Add a narrow `ScanPlanner` capability and `ScanPlanningMode::{Auto, Local,
Remote}` in the iceberg crate. The `Catalog` trait is unchanged.
- `RestCatalog` implements `ScanPlanner` and injects it onto tables it loads
(`create` / `load` / `register` / `update`).
- Decode REST kebab-case `data-file` JSON into `FileScanTask` (path, size,
format, deletes, partition spec). Empty plans yield an empty task list.
- Expand `plan-tasks` via `fetch_scan_tasks` with a seen-set so a reissued
handle cannot loop.
- Auto mode uses remote planning when `supports_remote_scan_planning()` is
true, falls back to local only on `FeatureUnsupported`, and does not fall back
on other remote errors. Remote mode errors if the planner is missing or
unsupported. Local mode always reads manifests.
- `supports_remote_scan_planning` now returns
`supports_full_remote_scan_planning()` because decode works.
Out of scope (as agreed on #1690): DataFusion (#2671), plan-scoped FileIO /
vended credentials (#2651 / #2932). Filter JSON on the wire is omitted (no
ExpressionParser yet); the scan's bound predicate is stamped onto each task.
## Are these changes tested?
Unit tests with mockito (no docker / iceberg-rest-fixture):
- Completed plan with a real data-file JSON → matching `FileScanTask`
- submitted → `wait_for_plan` → completed
- `plan-task` fan-out via `fetch_scan_tasks`
- Cycle / reissued plan-task handle does not loop
- Empty plan → empty task list, not an error
- Auto without advertised endpoints → local planning (no `POST /plan`)
- Remote without endpoints → `FeatureUnsupported`
- `supports_remote_scan_planning` is true when all four endpoints are
advertised
Native local planning tests keep passing.
## How to test
```bash
cargo test -p iceberg --lib scan
cargo test -p iceberg-catalog-rest --lib
cargo clippy -p iceberg -p iceberg-catalog-rest --all-targets --all-features
-- -D warnings
```
Also run: `cargo test -p iceberg --lib` (1553 passed locally) and `cargo
build -p iceberg-datafusion --all-targets` (compiles; this PR does not change
DataFusion).
## AI Disclosure
This change was drafted with an AI coding assistant. I reviewed the Go
`PlanFiles` / `ScanPlanner` design, the closed #2656 draft, and the resulting
Rust diff (Catalog trait, decode, Auto/Remote fallback, and tests).
Decode allocates `FileScanTask`s from opaque REST `data-file` JSON and a
small seen-set while expanding `plan-task` handles. Polling itself is unchanged
from #3011. Plan-scoped FileIO is not built; scans keep using the table's
existing FileIO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
