bobbai00 opened a new issue, #6195:
URL: https://github.com/apache/texera/issues/6195
### What happened?
When running the stack via `bin/local-dev.sh`, clicking **Run** on a
workflow makes the execution **fail** (it looks like it's stuck in the UI).
`computing-unit-master` logs show:
```
org.apache.iceberg.exceptions.RESTException: Unable to process:
at org.apache.iceberg.rest.RESTCatalog.createTable(RESTCatalog.java:128)
at
org.apache.texera.amber.util.IcebergUtil$.createTable(IcebergUtil.scala:195)
at
org.apache.texera.amber.core.storage.DocumentFactory$.createDocument(DocumentFactory.scala:89)
at
org.apache.texera.web.service.ExecutionStatsService.<init>(ExecutionStatsService.scala:84)
```
Root cause: the Lakekeeper warehouse `texera` stores the MinIO S3 `endpoint`
in its persistent storage-profile. `local-dev` intentionally sets
`STORAGE_S3_ENDPOINT=http://$HOST_LAN_IP:9000` (the host **LAN IP**, not
`localhost`, because Iceberg remote-signing requires both the containerized
Lakekeeper *and* the host JVMs to reach the same MinIO). But `lakekeeper-init`
(in `bin/single-node/docker-compose.yml`, ~L222) is idempotent and **skips the
warehouse when it already exists**:
```
Lakekeeper Warehouse 'texera' already exists. Skipping creation.
```
So once the warehouse is created, its endpoint is **never updated**. When
the host's LAN IP changes (different network / Wi-Fi / DHCP lease — routine for
a laptop), the warehouse keeps vending a stale, unreachable endpoint. Every S3
metadata operation then fails, `createTable` fails, and the workflow execution
is marked FAILED.
In my case the warehouse had `http://192.168.1.162:9000/` baked in while the
machine's current IP was `192.168.50.189`. The Lakekeeper error is an
empty-body response, which surfaces on the client as `No content to map due to
end-of-input` → `RESTException: Unable to process`, giving no hint that a stale
endpoint is the cause.
Expected: `local-dev` should keep the warehouse's storage endpoint in sync
with the current `STORAGE_S3_ENDPOINT`. Suggested fix: on every `up`, if the
warehouse already exists, `POST /management/v1/warehouse/{id}/storage` with the
current endpoint (and credentials) instead of skipping — this is
non-destructive and preserves existing Iceberg tables (table metadata stores
`s3://bucket/...` paths, not the endpoint).
Manual workaround (no restart needed — the profile is fetched per-request):
```bash
WID=$(curl -s http://localhost:8181/management/v1/warehouse | jq -r
'.warehouses[]|select(.name=="texera").id')
IP=$(ipconfig getifaddr en0)
curl -s -X POST -H 'Content-Type: application/json' \
-d
'{"storage-profile":{"type":"s3","bucket":"texera-iceberg","region":"us-west-2","endpoint":"http://'"$IP"':9000","flavor":"s3-compat","path-style-access":true,"sts-enabled":false},"storage-credential":{"type":"s3","credential-type":"access-key","aws-access-key-id":"texera_minio","aws-secret-access-key":"password"}}'
\
"http://localhost:8181/management/v1/warehouse/$WID/storage";
```
### How to reproduce?
1. Run `bin/local-dev.sh up` on network A (warehouse `texera` is created
with `STORAGE_S3_ENDPOINT=http://<IP-on-A>:9000`).
2. Move the machine to network B (or otherwise get a new LAN IP) and run
`bin/local-dev.sh up` again — `lakekeeper-init` prints `Warehouse 'texera'
already exists. Skipping creation.` and the endpoint stays at the old IP.
3. Open the frontend and click Run on any workflow.
4. Execution fails; `computing-unit-master` log shows `RESTException: Unable
to process` from `IcebergUtil.createTable`.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
be99b3661 (observed; init lives in `bin/single-node/docker-compose.yml`,
warehouse-skip logic ~L222)
### Relevant log output
```shell
# computing-unit-master.log
[WARN] org.apache.iceberg.rest.ErrorHandlers - Unable to parse error
response
java.io.UncheckedIOException:
com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map
due to end-of-input
[ERROR] org.apache.texera.web.service.WorkflowService - error during
execution
org.apache.iceberg.exceptions.RESTException: Unable to process:
at org.apache.iceberg.rest.RESTCatalog.createTable(RESTCatalog.java:128)
at
org.apache.texera.amber.util.IcebergUtil$.createTable(IcebergUtil.scala:195)
at
org.apache.texera.amber.core.storage.DocumentFactory$.createDocument(DocumentFactory.scala:89)
at
org.apache.texera.web.service.ExecutionStatsService.<init>(ExecutionStatsService.scala:84)
# warehouse storage-profile actually vended (stale IP):
"storage-profile":{"type":"s3","bucket":"texera-iceberg","endpoint":"http://192.168.1.162:9000/";,
... "remote-signing-enabled":true}
# host's current IP: 192.168.50.189
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
