mbutrovich commented on code in PR #2862:
URL: https://github.com/apache/iceberg-rust/pull/2862#discussion_r3738655889
##########
crates/integrations/datafusion/src/table/mod.rs:
##########
@@ -56,6 +56,31 @@ use crate::physical_plan::scan::IcebergTableScan;
use crate::physical_plan::sort::sort_by_partition;
use crate::physical_plan::write::IcebergWriteExec;
+/// Computes the arrow schema to expose for reading `table` at the given
snapshot:
+/// the table's current schema for `None`, that snapshot's schema for `Some`,
so
+/// a time-travel read plans against the schema in effect at that snapshot.
+pub fn snapshot_arrow_schema(table: &Table, snapshot_id: Option<i64>) ->
Result<ArrowSchemaRef> {
Review Comment:
This is now public at the crate root
(`iceberg_datafusion::snapshot_arrow_schema`, per the `public-api.txt` diff).
Is exposing it to external callers intentional, or is it meant to stay an
internal helper shared by the two provider constructors? If the latter,
`pub(crate)` would keep the public surface matching what's actually meant to be
supported.
##########
crates/integrations/datafusion/src/table/mod.rs:
##########
@@ -88,15 +117,37 @@ impl IcebergTableProvider {
// Load table once to get initial schema
let table = catalog.load_table(&table_ident).await?;
- let schema =
Arc::new(schema_to_arrow_schema(table.metadata().current_schema())?);
+ let schema = snapshot_arrow_schema(&table, None)?;
Ok(IcebergTableProvider {
catalog,
table_ident,
schema,
+ snapshot_id: None,
})
}
+ /// Pins reads to a snapshot for time travel; `None` (the default) reads
the
+ /// current snapshot. A pinned provider is read-only: `insert_into` would
+ /// commit to the current table state, invisible to its own reads, so it
+ /// errors instead.
+ ///
+ /// Reloads metadata from the catalog to validate the snapshot and
re-derive
+ /// `schema()` from it. The reload matters because providers handed out by
+ /// the catalog are cached, so a snapshot committed after construction
would
+ /// otherwise look nonexistent.
+ pub async fn with_snapshot_id(mut self, snapshot_id: Option<i64>) ->
Result<Self> {
+ let table = self.catalog.load_table(&self.table_ident).await?;
+ self.schema = snapshot_arrow_schema(&table, snapshot_id)?;
+ self.snapshot_id = snapshot_id;
+ Ok(self)
+ }
Review Comment:
This reloads table metadata from the catalog purely to validate the snapshot
and recompute the schema, and `scan()` (line 178-183 below) reloads again right
after. That's two catalog round trips before a query that needs one, and it
makes what reads like a builder setter into a fallible, async network call. Per
the comment above it, this cost exists specifically to handle providers the
catalog hands out and caches across calls. Is that caching concern this type's
to own, or should the reload-for-freshness logic live wherever the caching
actually happens (the Ballista scheduler side, by the description upthread)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
