gabotechs opened a new pull request, #86: URL: https://github.com/apache/datafusion-iceberg/pull/86
The checked-in taxi table can only model one case: a single-snapshot V2 table with seven files. PRs that needed anything else built it by hand. #67 hand-writes manifests to get a position delete, #55 hand-writes sorted files and rewrites metadata in place, and #65 couldn't test page-index pruning at all because the taxi files have no page indexes. This replaces the committed fixture with tables generated at test time, by a new unpublished crate, `iceberg-datafusion-test-tools` (`crates/test-tools`): - `FixtureCatalog` is an in-memory catalog over a temp dir. `fixtures.table(ident, schema).partition_by(..).property(..).create()` creates a table. - `table.append().writer_properties(..).file(batch)...commit()` commits one snapshot in which every batch becomes exactly one data file. Columns are matched by name and cast to the table's types. - Data files are written with iceberg-rust's writers rather than this crate's insert path, so they carry real column statistics, and a writer bug can't hide a reader bug. - The crate writes the manifest, manifest list, and snapshot itself, so snapshot ids count up from 1 and metadata-table snapshots are deterministic. iceberg-rust's `fast_append` picks random ids and can't commit delete files, so this is also where deletes can be added later. `create_taxi_table` generates a synthetic stand-in for the old fixture with the same schema and layout: 7 days × 25,000 rows, identity-partitioned on `pickup_date`, one file per day, each a single row group with column chunk statistics but no page indexes. Rows come from `rand_pcg`, seeded per day, so they're the same on every run and platform. `rand` and `rand_pcg` were already in the lockfile. `IcebergTestHarness` moves from the published crate's `test_utils` module to `crates/datafusion/tests/common`, which also drops `serde` and `typetag` from `iceberg-datafusion`'s dependencies. Snapshot values change with the new data; plans don't. Git LFS is no longer needed, so the CI checkout no longer fetches it. Generating the table takes about 0.9 s per harness in debug builds and 0.1 s in release, almost all of it Parquet encoding. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
