NoahKusaba opened a new issue, #3017:
URL: https://github.com/apache/iceberg-rust/issues/3017
### Is your feature request related to a problem or challenge?
`IcebergStaticTableProvider` holds a `Table`, which owns a live `FileIO` and
is
bound to the catalog it was loaded from. Neither survives serialization.
That blocks distributed execution: engines like DataFusion Ballista plan on
one
node and execute on others, so `IcebergTableScan` has to be shipped to
workers
that then rebuild storage before they can read data files. The provider knows
which catalog it loaded from, but drops that information, leaving consumers
no
way to identify the table well enough to rebuild it elsewhere.
The repo already has the shape needed — `CatalogBuilder::load` takes `name`
and
`props`, and `iceberg-catalog-loader` selects a builder by `type`. That
triple
just isn't recorded on the read path.
### Describe the solution you'd like
Add `IcebergCatalogConfig { type, name, props }` to `iceberg-datafusion`:
plain
data, no live connections, mirroring the loader's inputs. Let
`IcebergStaticTableProvider` record it and expose it alongside
`table_ident()`
and `snapshot_id()`, and let `IcebergTableScan` carry it into the physical
plan.
Together those are enough to rebuild the catalog, load the `Table`, and pin
the
same snapshot. Nothing in the crate would connect using them; the config is
an
`Option` defaulting to `None`, so existing behavior is unchanged.
### Willingness to contribute
I can contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
