kevinjqliu opened a new issue, #18043:
URL: https://github.com/apache/iceberg/issues/18043
### Summary
`apache/iceberg-rest-fixture` runs `JdbcCatalog` on a file-backed SQLite db
using sqlite's default rollback journal. That mode only lets one connection
touch the db at a time, so when several clients hit the fixture in parallel,
which is what every integration suite does, one of them eventually fails with
`SQLITE_BUSY ("database is locked")`.
WAL mode lets reads and the write happen side by side, so there's nothing to
fight over. It's a one-line change to the JDBC URI and it should be the
default. Details, stress numbers, and the reasoning against the alternatives
(`busy_timeout`, retries, serializing tests) are in apache/iceberg-rust#3183.
Several repos also set
`CATALOG_URI=jdbc:sqlite:file:/tmp/iceberg_rest_mode=memory`, inherited from
the old
[databricks/iceberg-rest-image](https://github.com/databricks/iceberg-rest-image),
which hard-coded it as its default. `iceberg_rest_mode` isn't an option,
there's no `?` in the URI so it's just part of the filename. Those overrides
should be replaced with the same WAL URI rather than carried along.
### Proposed URI
```
jdbc:sqlite:/tmp/iceberg_catalog.db?journal_mode=WAL
```
sqlite-jdbc parses the query string on both the plain-path and `file:` forms.
### Tasks
Fixture default (covers every consumer that doesn't set `CATALOG_URI`):
- [ ] apache/iceberg: `docker/iceberg-rest-fixture/Dockerfile`, and note WAL
where `docker/iceberg-rest-fixture/README.md` describes the sqlite default
The `latest` tag is rebuilt nightly from main, so consumers on `latest` get
this the day after it merges. Versioned tags only at the next release.
Repos overriding `CATALOG_URI` (a compose `environment` entry replaces the
image default, so each needs its own change):
- [ ] apache/iceberg-rust: `dev/docker-compose.yaml`
(apache/iceberg-rust#3183)
- [ ] apache/iceberg:
`kafka-connect/kafka-connect-runtime/docker/docker-compose.yml`
- [ ] apache/iceberg-cpp:
`src/iceberg/test/resources/iceberg-rest-fixture/docker-compose.yml`
- [ ] apache/iggy:
`core/integration/tests/connectors/fixtures/iceberg/container.rs` sets
`jdbc:sqlite:/tmp/iceberg_catalog.db` explicitly
- [ ] apache/doris:
`docker/thirdparties/docker-compose/iceberg-rest/docker-compose.yaml.tpl`, five
services each on their own `jdbc:sqlite:/tmp/*_catalog.db`
Related:
- [ ] apache/iceberg-go: the sql catalog's sqlite test db has the same
problem (apache/iceberg-go#1793), same fix
Repos that don't set `CATALOG_URI` (iceberg-python, the iceberg-go recipe,
terraform-provider-iceberg, the Flink quickstart) need nothing: they pick up
WAL automatically once they bump to an image with the new default, or can set
the env var themselves in the meantime.
Apache projects still on the old `tabulario/iceberg-rest` image (druid,
texera, a doris sample) have the same backend and would need to move to
`apache/iceberg-rest-fixture` first. Out of scope here, listed for completeness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
