MisterRaindrop opened a new pull request, #1881: URL: https://github.com/apache/cloudberry/pull/1881
<!-- Thank you for your contribution to Apache Cloudberry (Incubating)! --> ### What does this PR do? Adds `contrib/datalake_fdw`: Apache Iceberg lake tables as a table access method, with no kernel changes. A lake table is an ordinary `CREATE TABLE ... USING iceberg WITH (catalog = '...', volume = '...')`, where `catalog` and `volume` name foreign servers created through two foreign-data wrappers this extension registers. **This is deliberately a skeleton, opened as a draft to get the shape reviewed before the implementation follows.** `CREATE TABLE` and `DROP TABLE` work end to end against a stub metadata engine -- no catalog service, object store, Arrow or JVM involved -- and everything that would touch data reports a clean `iceberg: <operation> is not supported yet`. The interfaces the later work plugs into ship whole so they can be argued about now: the metadata-engine vtable with a capability bitmap the registry validates and dispatches through, the reader/writer format interfaces, and the storage facade. The question this draft is really asking is whether the **seam** is in the right place -- see Additional Context. ### Type of Change - [ ] Bug fix (non-breaking change) - [x] New feature (non-breaking change) - [ ] Breaking change (fix or feature with breaking changes) - [ ] Documentation update ### Test Plan - [ ] Unit tests added/updated - [x] Integration tests added/updated - [x] Passed `make installcheck` - [ ] Passed `make -C src/test installcheck-cbdb-parallel` ### Impact **Performance:** none. Nothing outside the module changes behaviour, and no statement gains work: the utility and object-access hooks match on the access method and return immediately for every other relation. **User-facing changes:** a new extension, plus two lines of registration (`contrib/Makefile`, and a CI matrix entry so the suite actually runs). The extension installs process-wide hooks, so it must be in `shared_preload_libraries`; `_PG_init` refuses to load any other way rather than half-initialising. **Dependencies:** none. The module builds with no new external dependency -- the C++ translation units are stubs today, and the object-store SDK and Arrow arrive with the changes that need them. ### Checklist - [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code) - [ ] Added/updated documentation - [x] Reviewed code for security implications - [x] This PR contains AI-assisted code generation - [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers) ### Additional Context **This is the upstream half of an implementation that already exists.** A datalake_fdw with the same shape -- the same two hooks, the same table access method, largely the same file names -- runs in production elsewhere, and the intent is that this becomes the single source for it rather than a second lookalike. Two consequences a reviewer should see explicitly: *The mapping accessor is deliberately shaped to that implementation.* A lake table's mapping to its catalog and volume servers is read only through `pg_iceberg_get_table_info()`, whose signature and result types (`IcebergTableInfo`, `IcebergTableOptions`) match it field for field. That implementation keeps the same mapping in a system catalog of its own, which an extension cannot add, so here the function reads reloptions instead -- but the difference stops inside the function body, and the layers above it can be the same code on both sides. *Two option names will look wrong, and are chosen anyway.* Catalog servers take `url`, not the Iceberg specification's `uri`, and the REST-based catalog type is named `polaris` rather than a generic `rest`. Both follow the released implementation. One vocabulary across the two trees is worth more than either name, and this side has never been released, so it is the side that yields. A generic `rest` type is deferred rather than rejected; `hadoop` and `s3` are in the vocabulary but refused until something implements them. ### What was and was not verified Stated plainly, because a draft is worth less if you have to guess: **Verified on a three-segment cluster:** `make installcheck` 3/3, and order-independent (each file alone and in reverse order, on a fresh database); zero build warnings; exactly 8 exported symbols; a schema-level `pg_dump`/restore round trip; Apache RAT clean. Separately, an 84-statement smoke pass over ordinary objects -- heap/AO/partitioned/temp/unlogged tables, CTAS, matviews, inherited and typed tables, views, sequences, indexes, functions, another wrapper's servers and user mappings, every ALTER form, COPY, TRUNCATE, VACUUM FULL, CLUSTER, REINDEX -- with the library preloaded, both with and without the extension installed, plus two concurrency probes. Zero errors: the hooks are inert for everything that is not a lake table. **Not verified:** `make -C src/test installcheck-cbdb-parallel` was not run (no configured source tree in the environment used); the CI entry added here has never executed, so this PR is also the first run of it. ### Known limits, stated rather than left to be found - The mapping stores server *names*, so `ALTER SERVER ... RENAME` on a referenced server is refused. Storing OIDs instead would allow it, at the cost of owning dump/restore translation; the accessor above is the only thing that would change. - A full `pg_dump` of a database containing a lake table fails, because dumping table *contents* means scanning. A schema-level dump round-trips. The data belongs to the lake, so it arguably should not be in the dump at all -- but `pg_dump` decides from relkind, which an extension cannot influence, so that needs a separate conversation. - Every `ALTER TABLE` form except `OWNER TO` is refused while the access method is unfinished. `purge_on_drop` is therefore fixed at CREATE time. - The error-detail channel has no regression coverage, because no statement can make the stub engine fail. ### Naming `datalake_fdw` is inherited from the existing implementation and is a poor fit: what this installs is a table access method plus two wrappers, not one FDW. A better name is welcome while it is still cheap to change. --- <!-- Join our community: - Mailing list: [[email protected]](https://lists.apache.org/[email protected]) (subscribe: [email protected]) - Discussions: https://github.com/apache/cloudberry/discussions --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
