This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7740-8be174308d4e597f120253cc68dbd4f21c0af1ed in repository https://gitbox.apache.org/repos/asf/texera.git
commit 51f38e13d66e42070872ccb532b7796b82273be4 Author: Meng Wang <[email protected]> AuthorDate: Wed Aug 19 06:07:25 2026 +0000 feat(storage): give the default Iceberg warehouse its own key prefix (#7740) ### What changes were proposed in this PR? The deployment scripts created the default Lakekeeper warehouse without a `key-prefix`, so it claimed the entire bucket. Per-user warehouses (#6870) live in the same bucket under `user-<uid>-<name>`, which Lakekeeper rejects as an overlapping storage profile — making per-user warehouse creation fail on every standard deployment. The default warehouse now owns an explicit prefix (`shared`), so it and the per-user warehouses are siblings. `WarehouseResource` is unchanged — the mismatch was entirely in deployment configuration. The same prefix goes into the CI warehouse (`build.yml`), so the integration environment matches a real deployment — that is where a future integration test of the warehouse management API would otherwise hit the same overlap. It is a literal there rather than extracted from `storage.conf` like the neighbouring values: the prefix describes how the deployment lays out the bucket and no runtime code reads it, and every key currently in `storage.conf` is read by `StorageConfig`. Both compose payloads carry the prefix, not just the create one: Lakekeeper rejects a storage update whose `key_prefix` differs from the stored value, and an omitted field counts as a change, so the start-up endpoint refresh (#6195) would fail with HTTP 400 against a prefixed warehouse. ### Any related issues, documentation, discussions? Closes #7739. Existing deployments must drop and recreate the default warehouse: Lakekeeper refuses to change `key_prefix` in place. The cost is bounded — `ComputingUnitMaster.recurringCheckExpiredResults` purges results, runtime statistics and logs past `result-cleanup.ttl-in-seconds` (default 86400) — and Texera addresses the warehouse by name, so nothing else changes. ### How was this PR tested? Rendered both compose payloads with the real `.env` values and confirmed each is valid JSON carrying `key-prefix`; `docker compose config` passes. Against a live Lakekeeper: the refresh payload without the prefix returns HTTP 400 (`Field key_prefix cannot be updated`) and with it returns 200; a per-user warehouse creates successfully alongside a prefixed default (HTTP 201), and an execution targeting it writes results, runtime statistics and console messages entirely under that warehouse's prefix while the shared prefix stays empty. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-opus-5) --- .github/workflows/build.yml | 3 +++ bin/k8s/values.yaml | 6 +++--- bin/single-node/.env | 3 +++ bin/single-node/docker-compose.yml | 2 ++ 4 files changed, 11 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index a79f427823..6a2a6cff2c 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -570,6 +570,8 @@ jobs: # storage.conf is followed by a `${?VAR}` env-override line whose # name is globally unique, so anchoring grep on that override line # selects the value unambiguously across nested scopes. + # The key-prefix below is a literal instead: no runtime code reads it, + # so it is not in storage.conf. It must stay non-empty (#7739). run: | CONF=common/config/src/main/resources/storage.conf extract() { @@ -611,6 +613,7 @@ jobs: "storage-profile": { "type": "s3", "bucket": "$S3_BUCKET", + "key-prefix": "shared", "region": "$S3_REGION", "endpoint": "$S3_ENDPOINT", "flavor": "s3-compat", diff --git a/bin/k8s/values.yaml b/bin/k8s/values.yaml index 4651bae694..9c37d3b408 100644 --- a/bin/k8s/values.yaml +++ b/bin/k8s/values.yaml @@ -170,9 +170,9 @@ lakekeeperInit: # Lakekeeper S3 storage-profile flavor: "s3-compat" (MinIO/other; adds # endpoint + path-style-access) or "aws" (real AWS S3; omits both). flavor: s3-compat - # Optional object-key prefix to isolate the Iceberg warehouse within a - # shared bucket. Empty = bucket root. - keyPrefix: "" + # Object-key prefix for this warehouse. Must stay non-empty: a warehouse at the + # bucket root overlaps every per-user warehouse's prefix (#7739). + keyPrefix: "shared" # Part2: configurations of Texera-related micro services texeraImages: diff --git a/bin/single-node/.env b/bin/single-node/.env index dc8877275d..a6f631ae56 100644 --- a/bin/single-node/.env +++ b/bin/single-node/.env @@ -74,6 +74,9 @@ STORAGE_ICEBERG_CATALOG_TYPE=rest STORAGE_ICEBERG_CATALOG_REST_URI=http://texera-lakekeeper:8181/catalog STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME=texera STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET=texera-iceberg +# Must stay non-empty: a warehouse at the bucket root overlaps every per-user +# warehouse's prefix, which Lakekeeper rejects (#7739). +STORAGE_ICEBERG_CATALOG_REST_KEY_PREFIX=shared # Postgres-backed Iceberg catalog STORAGE_ICEBERG_CATALOG_POSTGRES_URI_WITHOUT_SCHEME=texera-postgres:5432/texera_iceberg_catalog diff --git a/bin/single-node/docker-compose.yml b/bin/single-node/docker-compose.yml index 64247d9d66..27ce134bd7 100644 --- a/bin/single-node/docker-compose.yml +++ b/bin/single-node/docker-compose.yml @@ -233,6 +233,7 @@ services: "storage-profile": { "type": "s3", "bucket": "$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET", + "key-prefix": "$$STORAGE_ICEBERG_CATALOG_REST_KEY_PREFIX", "region": "$$STORAGE_S3_REGION", "endpoint": "$$STORAGE_S3_ENDPOINT", "flavor": "s3-compat", @@ -272,6 +273,7 @@ services: "storage-profile": { "type": "s3", "bucket": "$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET", + "key-prefix": "$$STORAGE_ICEBERG_CATALOG_REST_KEY_PREFIX", "region": "$$STORAGE_S3_REGION", "endpoint": "$$STORAGE_S3_ENDPOINT", "flavor": "s3-compat",
