fightBoxing opened a new issue, #3897:
URL: https://github.com/apache/iceberg-python/issues/3897
## Is your feature request related to a problem? Please describe.
Java Iceberg ships a filesystem-only `HadoopCatalog` / `HadoopTables`, where
table metadata lives under `<warehouse>/<db>.db/<table>/metadata/` with no
external metastore. PyIceberg currently has no equivalent — the available
catalog types are `rest / hive / glue / dynamodb / sql / in-memory / bigquery`.
This gap matters in two ways:
1. **Interop with Java-side HadoopCatalog tables.** Tables created by Java
`HadoopCatalog` (common in lightweight deployments without a metastore) cannot
be opened through any supported PyIceberg catalog. Users must fall back to
`StaticTable.from_metadata` and resolve the latest `metadata.json` themselves,
which loses catalog semantics (no namespace listing, no create/commit).
2. **Downstream projects already assume the module exists.** Daft's
Gravitino integration imports `from pyiceberg.catalog.hadoop import
HadoopCatalog` and calls `HadoopCatalog("gravitino_reader",
props).load_table(table_dir)` to open a table from a storage location
(`daft/catalog/__gravitino/_catalog.py`). Against PyIceberg 0.11.x this raises
`TypeError: HadoopCatalog.__init__() takes 2 positional arguments but 3 were
given`, and against versions without the module it fails at import time.
## Describe the solution you'd like
A `pyiceberg.catalog.hadoop.HadoopCatalog` (subclassing `MetastoreCatalog`)
implementing Java HadoopCatalog semantics:
- `warehouse` property as the root location
- table dir = `<warehouse>/<namespace>/<table>`
- metadata at `<table_dir>/metadata/v{n}.metadata.json` plus a
`version-hint.text` holding the current version
- latest-version resolution: read `version-hint.text`, fall back to scanning
`metadata/` for the max `v{n}` (matching Java behavior)
- namespace/table create/list/commit driven purely by the warehouse
filesystem (no metastore calls)
## Additional context / pitfalls observed while prototyping
Happy to contribute a PR if this is in scope. A few notes from an internal
prototype:
1. **Metadata file naming.** Java HadoopCatalog uses `v{n}.metadata.json` +
`version-hint.text`, but tables created by JDBC/REST catalogs use
`00000-<uuid>.metadata.json` with no version-hint. To open those as well, the
scan fallback should accept both patterns (`v(\d+)\.metadata\.json` and
`\d{5}-.*\.metadata\.json`), or at least document the limitation.
2. **Filesystem abstraction.** `__init__` should derive the filesystem from
the catalog's FileIO (`PyArrowFileIO`) instead of hardcoding
`pyarrow.fs.HadoopFileSystem.from_uri(warehouse)`. The JVM-backed
`HadoopFileSystem` only supports `hdfs://` and fails for object-store schemes
(`s3://`, and custom schemes), so routing through FileIO keeps it
scheme-agnostic.
3. **Atomicity.** `create_table` / `commit_table` use create-if-absent on
`v{n}.metadata.json` for optimistic concurrency — safe on HDFS but not atomic
on plain object stores (S3 has no create-if-absent guarantee). Java has the
same caveat; worth documenting or using a conditional-write primitive where
available.
## References
- Java: `org.apache.iceberg.hadoop.HadoopCatalog` / `HadoopTables`
- Downstream usage that currently breaks:
`daft/catalog/__gravitino/_catalog.py` → `_open_iceberg_table`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
