GitHub user mengw15 edited a discussion: Per-user warehouses are feature-complete behind a flag; proposing to turn it on
Per-user warehouses (#6870) now work end to end behind `storage.warehouse.enabled`, which has been off since the first commit. Each user owns their own warehouses — as many as they need — instead of everything sharing one. Everything has landed in `main`: the storage foundation and REST-catalog work, the warehouse REST endpoints and Lakekeeper client, the dashboard tab (#8005), the on-canvas picker (#8551), and the backend requirement that every execution name a warehouse (#8586). I would like to propose turning the flag on, and to hear objections before anyone does. ### Demo https://github.com/user-attachments/assets/e08060e2-2d94-4ffe-aad7-c79bccc329d9 1. The **Warehouses** dashboard tab: creating a warehouse, the row it produces, and the details dialog its name opens. 2. The **warehouse picker** on the canvas, next to the computing-unit selector, choosing where the next run writes. 3. With no warehouse left, the **Run button becomes "Create Warehouse"** and leads straight to the create dialog; once a warehouse exists, Run comes back. 4. A run started from the **form view** with no warehouse selected is refused with a message, rather than quietly writing to shared storage. 5. A normal execution, writing into the selected warehouse. ### What changes for users when it is on - A **Warehouses** tab appears in the dashboard: create, inspect and delete your own warehouses. - A **warehouse picker** appears on the canvas, choosing where the next run writes. - **Every execution must name a warehouse.** Until you have one, Run becomes "Create Warehouse"; runs started elsewhere (form view, run-up-to-operator, replay, and agent-driven runs) are refused with a message rather than silently writing to shared storage. - Nothing is provisioned automatically — each user creates their own. ### What it requires from a deployment The switch only takes effect with the REST catalog: `StorageConfig` keeps the feature off unless the Iceberg catalog type is `rest` (Lakekeeper), because warehouses are Lakekeeper entities. Deployments on another catalog type are unaffected by the flag entirely. ### What it does not change - **Results written before the flip stay readable.** Their storage URIs carry no warehouse segment and continue to resolve to the shared default warehouse. - **The switch is reversible, and fails loudly rather than silently.** Turned back off, a result that lives in a per-user warehouse is refused with an explicit message ("this result is stored in warehouse X; per-user warehouses are disabled in this deployment") instead of surfacing as "table not found", which would be indistinguishable from data loss. Nothing is deleted; re-enabling restores access. - **Where the data lives.** Each warehouse is its own Lakekeeper catalog with its own key prefix inside the deployment's existing storage. Isolation is at the catalog level, not at the bucket level. ### The ask Speak up if enabling this would break something on a deployment you care about — particularly anything affecting existing users, or if the "every execution needs a warehouse" requirement looks too strict as a Phase 0 default. Absent objections, I will turn it on. GitHub link: https://github.com/apache/texera/discussions/8633 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
