Yicong-Huang opened a new pull request, #6145:
URL: https://github.com/apache/texera/pull/6145
### What changes were proposed in this PR?
jOOQ codegen reads the live postgres at sbt-compile time, so pulling main
after a PR adds a new `sql/updates/N.sql` breaks `bin/local-dev.sh up` with
`not found: value <NEW_TABLE>` until the update is applied by hand (hit today
with `27.sql` / `workflow_cover_image` from #5704). `infra_ensure_db_schema`
only bootstrapped `sql/texera_ddl.sql` on a fresh DB; nothing reconciled an
existing local DB, and the repo's official runner (`sql/docker-compose.yml`,
liquibase) is manual with creds that don't match local-dev's.
This PR teaches local-dev to reconcile the schema before any sbt build, on
every path that builds (full-stack `up`, `auto`, single-service `up`):
- `parse_changelog_changesets` parses `sql/changelog.xml` (BSD-awk
compatible, skips the commented example changeset) into id/author/path triples.
- `infra_apply_sql_updates` diffs those against liquibase's
`public.databasechangelog` and psql-applies the missing updates in document
order, with the same normalization the official liquibase compose uses: strip
the `\c` meta-command (not every file has one — `23.sql` doesn't) and connect
to `texera_db` directly; the files' own `SET search_path` targets the schema.
Applications are recorded in `databasechangelog` (liquibase-created DDL,
`MD5SUM` left `NULL` so a later real liquibase run adopts the rows and fills
checksums instead of re-executing).
- A fresh bootstrap seeds every changeSet as applied — `texera_ddl.sql` is
kept in sync with `sql/updates/*`, so replaying them would be redundant.
- A failing update aborts with a clear message BEFORE the sbt build instead
of letting the compile die on missing generated tables. The all-current fast
path is one psql query + one awk parse.
### Any related issues, documentation, discussions?
Closes #6144. The `--help` header's docker-infra paragraph documents the
behavior. Related context: the official liquibase runner in
`sql/docker-compose.yml` remains the mechanism for non-local deployments (the
`ddl-change` email workflow notifies admins); this PR only automates the local
dev loop while staying compatible with its bookkeeping.
### How was this PR tested?
- `bash bin/local-dev/tests/test_local_dev_sh.sh` — 50 passed, 0 failed. New
cases: the reconcile function exists and is wired into `infra_ensure_db_schema`
on both paths (apply on existing DB, seed on fresh bootstrap), tracks via
`databasechangelog` with `ON_ERROR_STOP`, the changelog parser handles the real
`sql/changelog.xml` (5 changesets, commented example skipped), and every
referenced update file exists on disk.
- Live test against a real local dev DB that had `27.sql` applied manually
and no `databasechangelog` table: first run applied changeSets 23–27
idempotently (and surfaced that `23.sql`'s `notebook` tables had in fact never
been applied to this DB — now healed) and wrote the 5 tracking rows with
correct authors; second run took the fast path ("schema current"). Verified
`notebook` exists in the `texera_db` schema afterwards.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
