gabotechs opened a new pull request, #87:
URL: https://github.com/apache/datafusion-iceberg/pull/87
Replaces #86, moved to a branch in this repository so that the follow-up can
be stacked on it.
The checked-in taxi table can only model one case: a single-snapshot V2
table with seven files. PRs that needed anything else built it by hand. #67
hand-writes manifests to get a position delete, #55 hand-writes sorted files
and rewrites metadata in place, and #65 couldn't test page-index pruning at all
because the taxi files have no page indexes.
This replaces the committed fixture with tables generated at test time, by a
new unpublished crate, `iceberg-datafusion-test-tools` (`crates/test-tools`).
It doesn't depend on `iceberg-datafusion`, so nothing in it can affect the
library. The same tools are used by
datafusion-contrib/datafusion-distributed#808, which they were ported to and
back from.
## Table builders
- `FixtureCatalog` is an in-memory catalog over a temp dir.
`fixtures.table(ident, schema).partition_by(..).create()` creates a table.
- `table.append().writer_properties(..).file(batch)...commit()` commits one
snapshot in which every batch becomes exactly one data file. Columns are
matched by name and cast to the table's types.
- Data files are written with iceberg-rust's writers rather than this
crate's insert path, so they carry real column statistics, and a writer bug
can't hide a reader bug.
- The tools write the manifest, manifest list, and snapshot themselves, so
snapshot ids count up from 1 and metadata-table snapshots are deterministic.
iceberg-rust's `fast_append` picks random ids and can't commit delete files, so
this is also where deletes can be added later.
- `create_taxi_table()` generates a synthetic stand-in for the old fixture
with the same schema and layout: `default.taxi`, 7 days × 25,000 rows,
identity-partitioned on `pickup_date`, one file per day, each a single row
group without page indexes. Rows come from `rand_pcg`, seeded per day, so
they're the same on every run and platform.
## Test harness
`IcebergTestHarness` moves from the published crate's `test_utils` module to
the test tools, and each test configures it in one builder chain:
```rust
let (plan, batches) = IcebergTestHarness::builder()
.with_iceberg_table_factory(IcebergTableProviderFactory::new())
.with_table(create_taxi_table().await?)
.query("SELECT COUNT(*) AS trips FROM taxi WHERE passenger_count IS
NULL")
.await?;
```
The harness registers its table with `CREATE EXTERNAL TABLE`, so it's read
through `IcebergStaticTableProvider`. Metadata tables only resolve through a
catalog, so the metadata test also registers an `IcebergCatalogProvider` with
`with_catalog(..)`. The harness now runs the plan it displays instead of
planning twice.
## Other changes
- `serde` and `typetag` leave `iceberg-datafusion`'s dependencies; only
`test_utils` used them.
- Snapshot values change with the new data; plans don't.
- `page_index.rs` now checks that the taxi files have no page indexes, since
its query returns the same rows either way.
- Git LFS is no longer needed, so the CI checkout no longer fetches it, and
`*.parquet` leaves the RAT exclusions.
Generating the table takes about 0.9 s per test in debug builds and 0.1 s in
release, almost all of it Parquet encoding.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
