mengw15 opened a new issue, #8569:
URL: https://github.com/apache/texera/issues/8569
### Problem
On local-dev, `STORAGE_S3_ENDPOINT` has two audiences at once: the
Lakekeeper container dials it for S3 operations **and** bakes it into every
warehouse's storage profile, and the host-native JVMs then receive that same
string back at runtime — the REST catalog vends the S3 settings to clients (see
the note in `IcebergUtil.createRestCatalog`).
The wrapper script satisfies both today by sniffing the host's LAN IP at
startup (see the `lakekeeper-init` note in
`bin/local-dev/docker-compose.override.yml`). That address is only valid until
DHCP hands out a new one. After a network change:
- creating a warehouse fails — the running services still carry the old
address from boot, and Lakekeeper's storage validation cannot reach it;
- existing warehouses keep the old address persisted in their storage
profiles, so reads and writes fail even after a restart;
- recovery is manual every time: re-edit the config, restart the services,
and re-issue `POST /management/v1/warehouse/{id}/storage` for every existing
warehouse.
This has repeatedly bitten local BYO-S3 testing (#6870).
### Observations so far
- `localhost` satisfies only the host JVMs; it is unreachable from the
Lakekeeper container.
- Container DNS (`http://texera-rustfs:9000`) satisfies only the compose
network; the catalog vends it to host JVMs, which cannot resolve it — the
caveat already documented in that override file.
- `host.docker.internal` was tried at one point and did not work end to end.
Two isolated observations from this round: a container on the compose network
could open a TCP connection to `host.docker.internal:9000`, while macOS itself
does not resolve the name (so anything vended to a host process would fail
there). No end-to-end validation of any alternative has been done.
Part of #6870.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
