mengw15 opened a new issue, #6034: URL: https://github.com/apache/texera/issues/6034
### Task Summary Production uses the REST (Lakekeeper) Iceberg catalog, but `common/config/src/main/resources/storage.conf` still defaults `iceberg.catalog.type` to `postgres`. CI should stay close to production. Flip the default to `rest`, and first make sure the amber **unit** job no longer depends on a live Iceberg catalog so the flip doesn't break it. Scope is ~3 ordered tasks (Task 1 must land before Task 2, or the amber job breaks). Context: prod = REST; local-dev and single-node already override to `rest` via env, so they're unaffected by the default. Separate from #6033 (which only removes the obsolete `bootstrap-lakekeeper.sh`). **Task 1 — Move the live-catalog iceberg specs into integration.** `IcebergDocumentSpec`, `IcebergDocumentConsoleMessagesSpec`, `IcebergTableStatsSpec`, `IcebergUtilSpec` (in `common/workflow-core/src/test/scala/.../storage/result/iceberg/` and `.../util/`) resolve a live catalog via `IcebergCatalogInstance.getInstance()`. Today the amber job runs them against the postgres catalog it provisions; after Task 2 they would try to reach Lakekeeper, which the unit job has no reason to run. Relocate them to `amber/src/test/integration/` (where `IcebergRestCatalogIntegrationSpec` already lives and the integration filter applies). Note: tagging in place is **not** sufficient — `common/workflow-core/build.sbt` does not read `AMBER_TEST_FILTER` (only `amber/build.sbt` does), so `@IntegrationTest` would be ignored there; relocating (or wiring the same filter into `common/workflow-core/build.sbt`) is required. **Task 2 — Flip the default.** Change `iceberg.catalog.type` in `storage.conf` from `postgres` to `rest`. **Task 3 — Make CI integration mirror prod, keep both catalogs covered.** Run `amber-integration` against the REST catalog so the integration suite matches production (Lakekeeper is already provisioned there). Keep postgres-catalog coverage in the integration suite as well — `postgres` stays a supported `iceberg.catalog.type`, so it should still be exercised somewhere, just not in the unit job. The only change to the amber **unit** job is that it stops needing a live Iceberg catalog for these specs; postgres catalog support itself is untouched. ### Task Type - [x] Refactor / Cleanup - [x] DevOps / Deployment / CI - [x] Testing / QA - [ ] Documentation - [ ] Performance - [ ] Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
