jk-shah opened a new issue, #41227:
URL: https://github.com/apache/beam/issues/41227
### Description
Migrating or synchronizing partitioned Delta Lake tables into Apache Iceberg
currently requires full physical data rewriting. For multi-terabyte partitioned
tables, this takes hours of cluster compute and substantial network/storage
costs.
Because both formats use Parquet for data storage, an in-place zero-copy
metadata translation can synthesize Iceberg manifests directly from Delta
transaction logs.
### Proposed Architecture
Add a partition-aware zero-copy metadata bridge in
`apache_beam.io.lakehouse`:
```python
from apache_beam.io.lakehouse import translate_delta_to_iceberg
# Synthesizes Iceberg v2 metadata from Delta partitioned transaction logs
metadata_path = translate_delta_to_iceberg(
delta_table_uri="gs://my-bucket/delta/orders/",
iceberg_warehouse="gs://my-bucket/iceberg/wh/",
table_identifier="analytics.orders"
)
```
### Key Capabilities
1. **Partition Specification Mapping**: Parses Delta `_delta_log/*.json`
partition definitions (e.g., `date=2026-11-11/region=US/`) into matching
Iceberg `PartitionSpec` fields.
2. **Avro Manifest Generation**: Generates Iceberg `ManifestFile` records
pointing to existing Parquet files without copying data.
3. **Direct REST Catalog Registration**: Automatically registers the
synthesized snapshot in Google Cloud BigLake REST Catalog or Polaris in <5
seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
