The GitHub Actions job "Build and push images" on texera.git/main has failed. Run started by GitHub user bobbai00 (triggered by bobbai00).
Head commit for run: bc9479618953a24b057b4e7ce4d13e9af71775a2 / Tanishq Gandhi <[email protected]> feat(storage): singularize the resource-type prefix on logical paths (#7789) ### What changes were proposed in this PR? The leading segment of a versioned-resource logical path was plural while every name it refers to is singular — the backing tables (`dataset`, `model`), the dashboard search's resource type, and the hub's entity type. This makes the prefix match them. ``` Before: /datasets/[email protected]/twitter/v1/f.csv table: dataset /models/[email protected]/resnet/v1/w.pt table: model After: /dataset/[email protected]/twitter/v1/f.csv table: dataset /model/[email protected]/resnet/v1/w.pt table: model ``` | Layer | Change | | --- | --- | | Enums | `ResourceType` in Scala (`Datasets`/`Models` → `Dataset`/`Model`), TypeScript (`Datasets` → `Dataset`), Python (`DATASETS` → `DATASET`) | | Call sites | `FileResolver`, `FileListerSourceOpExec`, `DocumentFactory`, `LakeFSFileDocument`, `DatasetResource`, `DatasetFileNode`, `dataset-selection-modal`, `dataset_file_document` | | Migration | `36.sql` restored as it shipped; normalization added in a new `39.sql` | | Examples | the two shipped `bin/single-node/examples/workflows/*.json` carried the plural prefix | Two hardcoded `"dataset"` string literals on the DB side now read from the enum, since the prefix and the type name are the same word again: - `SearchQueryBuilder.DATASET_RESOURCE_TYPE`, inlined into the unified search SQL - `EntityType.Dataset.value`, written to `user_action.resource_type` **On the migration.** `36.sql` introduced the prefix in its plural form and is restored here byte-for-byte, so its checksum stays intact; the normalization lives in a new `39.sql` (38 is taken). Correcting 36 in place could not reach the databases that need it: liquibase selects changeSets by id — `bin/local-dev/main.sh:1824` skips on a match in `databasechangelog` without reading the file — so the databases holding plural paths are exactly the ones that recorded id 36 and will never run it again, and `MD5SUM` is left NULL by design, so nothing reports a mismatch and the stale paths surface only when someone opens one of those workflows. `39.sql` normalizes both shapes, so it converges whether or not 36 ran: | Stored value | Action | | --- | --- | | `/[email protected]/ds/v1/f.csv` (unprefixed legacy) | prepend `dataset` | | `/datasets/[email protected]/ds/v1/f.csv` (plural prefix) | rewrite the leading segment | | `/dataset/...` | already current — untouched | Both cases apply only when the path's owner and name segments match a real `(user.email, dataset.name)` pair — read at parts 1 and 2 unprefixed, 2 and 3 prefixed. That guard is what keeps the migration off the plain filesystem paths and URLs this column also holds: `/datasets` is an ordinary directory name, and `FileResolver` tries `localResolveFunc` first, so a `fileName` of `/datasets/imdb/movies.csv` can be a working local mount and is left alone. A path naming a dataset that no longer exists therefore keeps its old form; it cannot resolve either way. `/models/` is out of scope, since `37.sql` creates that table and runs after this changeset. ### Any related issues, documentation, discussions? Additional work on #6495, which introduced the resource-type prefix. ### How was this PR tested? Existing suites, updated where they pin the path form. ```bash sbt "WorkflowCore/testOnly org.apache.texera.amber.storage.FileResolverSpec org.apache.texera.amber.core.storage.DocumentFactorySpec org.apache.texera.amber.core.storage.model.LakeFSFileDocumentSpec" \ "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.dataset.FileListerSourceOpExecSpec" \ "FileService/testOnly org.apache.texera.service.type.DatasetFileNodeSpec" \ "WorkflowExecutionService/testOnly org.apache.texera.web.resource.dashboard.DatasetSearchQueryBuilderSpec org.apache.texera.web.resource.dashboard.DashboardResourceSpec org.apache.texera.web.resource.dashboard.UnifiedResourceSchemaSpec org.apache.texera.web.resource.dashboard.hub.HubResourceSpec org.apache.texera.web.resource.dashboard.user.workflow.WorkflowExecutionsResourceSpec org.apache.texera.web.resource.dashboard.file.DatasetResourceSpec" ``` ```bash pytest amber/src/test/python/pytexera/storage/test_dataset_file_document.py ``` ```bash cd frontend && yarn ng test --watch=false --include='**/datasetVersionFileTree.spec.ts' --include='**/dataset-selection-modal.component.spec.ts' --include='**/dataset-version-selector.component.spec.ts' --include='**/user-dataset-version-filetree.component.spec.ts' --include='**/dataset-detail.component.spec.ts' ``` `sbt Test/compile` and `tsc` on both app and spec sources are clean, so the enum rename has no missed call sites. `39.sql` was run against a seeded Postgres over 13 fixtures, one per case: legacy unprefixed, plural prefix, `/datasets`-rooted local mounts (3-segment, 4-segment, and as `datasetVersionPath`), plural naming a deleted dataset, `/models/`, already-singular, a URL, `/datasets/` appearing mid-path, non-array `operators`, absent property, and a row checking other operator properties, links and operator order survive. Correct in every case, on Postgres 18 and 15.11, and idempotent — a second run reports 0 rows. Tested through liquibase's id tracking rather than only by piping the SQL in, emulating `main.sh` against a real `databasechangelog` table: ``` fresh DB: changeSet 36 ran, 39 rewrote -> /dataset/[email protected]/ds/v1/a.csv recorded id 36: changeSet 36 skipped, 39 rewrote -> /dataset/[email protected]/ds/v1/a.csv ``` Also covered: `39` alone with no `36`, `39` twice, and `39` before `36` out of order — all four converge on the same state, with the local mount untouched throughout. The same statements were also run inside a Kubernetes deployment: `bin/k8s` rendered with `values-development.yaml`, the postgresql resources applied to minikube, and the SQL piped into the pod against the real 39-table schema the chart bootstraps. Row-for-row identical to local, including the local-mount cases staying untouched. Note that the chart's postgres init runs only `texera_ddl.sql` and the three catalog scripts, and `texera_db` has no `databasechangelog` table — no liquibase changeSet executes on a Kubernetes deployment. `sbt scalafmtCheckAll`, `ruff check`, `ruff format --check`, and `prettier --check` are clean. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Claude Opus 5 <[email protected]> Co-authored-by: ali <[email protected]> Report URL: https://github.com/apache/texera/actions/runs/32439950731 With regards, GitHub Actions via GitBox
