This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new bc94796189 feat(storage): singularize the resource-type prefix on
logical paths (#7789)
bc94796189 is described below
commit bc9479618953a24b057b4e7ce4d13e9af71775a2
Author: Tanishq Gandhi <[email protected]>
AuthorDate: Fri Aug 21 00:31:39 2026 +0000
feat(storage): singularize the resource-type prefix on logical paths (#7789)
### What changes were proposed in this PR?
The leading segment of a versioned-resource logical path was plural
while every name it refers to is singular — the backing tables
(`dataset`, `model`), the dashboard search's resource type, and the
hub's entity type. This makes the
prefix match them.
```
Before: /datasets/[email protected]/twitter/v1/f.csv table: dataset
/models/[email protected]/resnet/v1/w.pt table: model
After: /dataset/[email protected]/twitter/v1/f.csv table: dataset
/model/[email protected]/resnet/v1/w.pt table: model
```
| Layer | Change |
| --- | --- |
| Enums | `ResourceType` in Scala (`Datasets`/`Models` →
`Dataset`/`Model`), TypeScript (`Datasets` → `Dataset`), Python
(`DATASETS` → `DATASET`) |
| Call sites | `FileResolver`, `FileListerSourceOpExec`,
`DocumentFactory`, `LakeFSFileDocument`, `DatasetResource`,
`DatasetFileNode`, `dataset-selection-modal`, `dataset_file_document` |
| Migration | `36.sql` restored as it shipped; normalization added in a
new `39.sql` |
| Examples | the two shipped `bin/single-node/examples/workflows/*.json`
carried the plural prefix |
Two hardcoded `"dataset"` string literals on the DB side now read from
the enum, since the prefix and the type name are the same word again:
- `SearchQueryBuilder.DATASET_RESOURCE_TYPE`, inlined into the unified
search SQL
- `EntityType.Dataset.value`, written to `user_action.resource_type`
**On the migration.** `36.sql` introduced the prefix in its plural form
and is restored here byte-for-byte, so its checksum stays intact; the
normalization lives in a new `39.sql` (38 is taken). Correcting 36 in
place could not reach the databases that need it: liquibase selects
changeSets by id — `bin/local-dev/main.sh:1824` skips on a match in
`databasechangelog` without reading the file — so the databases holding
plural paths are exactly the ones that recorded id 36 and will never run
it again, and `MD5SUM` is left NULL by design, so nothing reports a
mismatch and the stale paths surface only when someone opens one of
those workflows.
`39.sql` normalizes both shapes, so it converges whether or not 36 ran:
| Stored value | Action |
| --- | --- |
| `/[email protected]/ds/v1/f.csv` (unprefixed legacy) | prepend `dataset` |
| `/datasets/[email protected]/ds/v1/f.csv` (plural prefix) | rewrite the
leading segment |
| `/dataset/...` | already current — untouched |
Both cases apply only when the path's owner and name segments match a
real `(user.email, dataset.name)` pair — read at parts 1 and 2
unprefixed, 2 and 3 prefixed. That guard is what keeps the migration off
the plain filesystem paths and URLs this column also holds: `/datasets`
is an ordinary directory name, and `FileResolver` tries
`localResolveFunc` first, so a `fileName` of `/datasets/imdb/movies.csv`
can be a working local mount and is left alone. A path naming a dataset
that no longer exists therefore keeps its old form; it cannot resolve
either way. `/models/` is out of scope, since `37.sql` creates that
table and runs after this changeset.
### Any related issues, documentation, discussions?
Additional work on #6495, which introduced the resource-type prefix.
### How was this PR tested?
Existing suites, updated where they pin the path form.
```bash
sbt "WorkflowCore/testOnly org.apache.texera.amber.storage.FileResolverSpec
org.apache.texera.amber.core.storage.DocumentFactorySpec
org.apache.texera.amber.core.storage.model.LakeFSFileDocumentSpec" \
"WorkflowOperator/testOnly
org.apache.texera.amber.operator.source.dataset.FileListerSourceOpExecSpec" \
"FileService/testOnly
org.apache.texera.service.type.DatasetFileNodeSpec" \
"WorkflowExecutionService/testOnly
org.apache.texera.web.resource.dashboard.DatasetSearchQueryBuilderSpec
org.apache.texera.web.resource.dashboard.DashboardResourceSpec
org.apache.texera.web.resource.dashboard.UnifiedResourceSchemaSpec
org.apache.texera.web.resource.dashboard.hub.HubResourceSpec
org.apache.texera.web.resource.dashboard.user.workflow.WorkflowExecutionsResourceSpec
org.apache.texera.web.resource.dashboard.file.DatasetResourceSpec"
```
```bash
pytest amber/src/test/python/pytexera/storage/test_dataset_file_document.py
```
```bash
cd frontend && yarn ng test --watch=false
--include='**/datasetVersionFileTree.spec.ts'
--include='**/dataset-selection-modal.component.spec.ts'
--include='**/dataset-version-selector.component.spec.ts'
--include='**/user-dataset-version-filetree.component.spec.ts'
--include='**/dataset-detail.component.spec.ts'
```
`sbt Test/compile` and `tsc` on both app and spec sources are clean, so
the enum rename has no missed call sites.
`39.sql` was run against a seeded Postgres over 13 fixtures, one per
case: legacy unprefixed, plural prefix, `/datasets`-rooted local mounts
(3-segment, 4-segment, and as `datasetVersionPath`), plural naming a
deleted dataset, `/models/`, already-singular, a URL, `/datasets/`
appearing mid-path, non-array `operators`, absent property, and a row
checking other operator properties, links and operator order survive.
Correct in every case, on Postgres 18 and 15.11, and idempotent — a
second run reports 0 rows.
Tested through liquibase's id tracking rather than only by piping the
SQL in, emulating `main.sh` against a real `databasechangelog` table:
```
fresh DB: changeSet 36 ran, 39 rewrote ->
/dataset/[email protected]/ds/v1/a.csv
recorded id 36: changeSet 36 skipped, 39 rewrote ->
/dataset/[email protected]/ds/v1/a.csv
```
Also covered: `39` alone with no `36`, `39` twice, and `39` before `36`
out of order — all four converge on the same state, with the local mount
untouched throughout.
The same statements were also run inside a Kubernetes deployment:
`bin/k8s` rendered with `values-development.yaml`, the postgresql
resources applied to minikube, and the SQL piped into the pod against
the real 39-table schema the chart bootstraps. Row-for-row identical to
local, including the local-mount cases staying untouched. Note that the
chart's postgres init runs only `texera_ddl.sql` and the three catalog
scripts, and `texera_db` has no `databasechangelog` table — no liquibase
changeSet executes on a Kubernetes deployment.
`sbt scalafmtCheckAll`, `ruff check`, `ruff format --check`, and
`prettier --check`
are clean.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
---------
Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: ali <[email protected]>
---
.../pytexera/storage/dataset_file_document.py | 10 +-
.../main/python/pytexera/storage/resource_type.py | 2 +-
.../resource/dashboard/SearchQueryBuilder.scala | 11 +-
.../web/resource/dashboard/hub/EntityType.scala | 3 +-
.../pytexera/storage/test_dataset_file_document.py | 34 ++-
.../workflow/WorkflowExecutionsResourceSpec.scala | 6 +-
...xample] Data Exploration on Movies Dataset.json | 2 +-
...[Example] Machine Learning on Iris Dataset.json | 2 +-
.../amber/core/storage/DocumentFactory.scala | 8 +-
.../texera/amber/core/storage/FileResolver.scala | 16 +-
.../texera/amber/core/storage/ResourceType.scala | 4 +-
.../core/storage/model/LakeFSFileDocument.scala | 4 +-
.../amber/core/storage/DocumentFactorySpec.scala | 8 +-
.../storage/model/LakeFSFileDocumentSpec.scala | 10 +-
.../texera/amber/storage/FileResolverSpec.scala | 24 +-
.../source/dataset/FileListerSourceOpExec.scala | 6 +-
.../dataset/FileListerSourceOpExecSpec.scala | 26 +--
.../texera/service/resource/DatasetResource.scala | 2 +-
.../service/type/dataset/DatasetFileNode.scala | 16 +-
.../service/type/dataset/DatasetFileNodeSpec.scala | 12 +-
.../app/common/type/datasetVersionFileTree.spec.ts | 8 +-
.../src/app/common/type/datasetVersionFileTree.ts | 2 +-
frontend/src/app/common/type/resource-type.ts | 4 +-
.../dataset-detail.component.spec.ts | 4 +-
...user-dataset-version-filetree.component.spec.ts | 6 +-
.../dataset-selection-modal.component.spec.ts | 8 +-
.../dataset-selection-modal.component.ts | 4 +-
.../dataset-version-selector.component.spec.ts | 8 +-
sql/changelog.xml | 5 +
sql/updates/39.sql | 254 +++++++++++++++++++++
30 files changed, 388 insertions(+), 121 deletions(-)
diff --git a/amber/src/main/python/pytexera/storage/dataset_file_document.py
b/amber/src/main/python/pytexera/storage/dataset_file_document.py
index b8d778d342..2d14f858e4 100644
--- a/amber/src/main/python/pytexera/storage/dataset_file_document.py
+++ b/amber/src/main/python/pytexera/storage/dataset_file_document.py
@@ -59,18 +59,18 @@ class DatasetFileDocument:
:param file_path:
Expected format -
- "/datasets/ownerEmail/datasetName/versionName/fileRelativePath"
+ "/dataset/ownerEmail/datasetName/versionName/fileRelativePath"
Example:
- "/datasets/[email protected]/twitterDataset/v1/california/tw1.csv"
+ "/dataset/[email protected]/twitterDataset/v1/california/tw1.csv"
"""
parts = file_path.strip("/").split("/")
invalid_format = ValueError(
"Invalid file path format. Expected: "
- "/datasets/ownerEmail/datasetName/versionName/fileRelativePath"
+ "/dataset/ownerEmail/datasetName/versionName/fileRelativePath"
)
- # TODO(datasets-prefix): require the prefix once all stored paths are
migrated (36.sql) and ml model support work is completed.
+ # TODO(dataset-prefix): require the prefix once all stored paths are
migrated (36.sql) and ml model support work is completed.
if parts and parts[0] in {t.value for t in ResourceType}:
if len(parts) < 5:
raise invalid_format
@@ -80,7 +80,7 @@ class DatasetFileDocument:
self.version_name = parts[3]
self.file_relative_path = "/".join(parts[4:])
elif len(parts) >= 4:
- self.resource_type = ResourceType.DATASETS
+ self.resource_type = ResourceType.DATASET
self.owner_email = parts[0]
self.dataset_name = parts[1]
self.version_name = parts[2]
diff --git a/amber/src/main/python/pytexera/storage/resource_type.py
b/amber/src/main/python/pytexera/storage/resource_type.py
index 596fb5d745..e77ec9dd49 100644
--- a/amber/src/main/python/pytexera/storage/resource_type.py
+++ b/amber/src/main/python/pytexera/storage/resource_type.py
@@ -21,4 +21,4 @@ from enum import Enum
class ResourceType(str, Enum):
"""The leading segment of a logical file path, identifying the resource
kind"""
- DATASETS = "datasets"
+ DATASET = "dataset"
diff --git
a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala
b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala
index e19755e611..e8a651cb82 100644
---
a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala
+++
b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/SearchQueryBuilder.scala
@@ -19,6 +19,7 @@
package org.apache.texera.web.resource.dashboard
+import org.apache.texera.amber.core.storage.ResourceType
import org.apache.texera.dao.SqlServer
import org.apache.texera.web.resource.dashboard.DashboardResource.{
DashboardClickableFileEntry,
@@ -33,10 +34,18 @@ object SearchQueryBuilder {
SqlServer
.getInstance()
.createDSLContext()
+ // Resource-type discriminators for dashboard search. They are inlined into
the search SQL,
+ // pattern-matched in DashboardResource, and duplicated as bare literals in
the frontend.
+ // DATASET_RESOURCE_TYPE aliases ResourceType.Dataset, whose value doubles
as the leading
+ // segment of a storage logical path, so renaming that prefix silently
changes search results
+ // with no compile error here.
+ // TODO: give every resource type (file, workflow, project, dataset, model)
a single shared
+ // constant/enum instead of these raw strings, so the search contract
has one source of
+ // truth and no longer borrows an unrelated storage constant.
val FILE_RESOURCE_TYPE = "file"
val WORKFLOW_RESOURCE_TYPE = "workflow"
val PROJECT_RESOURCE_TYPE = "project"
- val DATASET_RESOURCE_TYPE = "dataset"
+ val DATASET_RESOURCE_TYPE = ResourceType.Dataset.toString
val ALL_RESOURCE_TYPE = ""
}
diff --git
a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala
b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala
index 0cf1bd4cc8..bd83b7837e 100644
---
a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala
+++
b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/EntityType.scala
@@ -20,6 +20,7 @@
package org.apache.texera.web.resource.dashboard.hub
import com.fasterxml.jackson.annotation.{JsonCreator, JsonValue}
+import org.apache.texera.amber.core.storage.ResourceType
/**
* Defines all supported entity types for Hub resources.
@@ -34,7 +35,7 @@ sealed trait EntityType {
object EntityType {
case object Workflow extends EntityType { val value = "workflow" }
- case object Dataset extends EntityType { val value = "dataset" }
+ case object Dataset extends EntityType { val value: String =
ResourceType.Dataset.toString }
private val values = Seq(Workflow, Dataset)
diff --git
a/amber/src/test/python/pytexera/storage/test_dataset_file_document.py
b/amber/src/test/python/pytexera/storage/test_dataset_file_document.py
index bef134bc74..b90dcce81b 100644
--- a/amber/src/test/python/pytexera/storage/test_dataset_file_document.py
+++ b/amber/src/test/python/pytexera/storage/test_dataset_file_document.py
@@ -47,18 +47,18 @@ def make_response(status_code: int, body=None, content:
bytes = b""):
class TestDatasetFileDocumentInit:
def test_parses_prefixed_path(self, auth_env):
- doc = DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv")
+ doc = DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv")
assert doc.owner_email == "[email protected]"
assert doc.dataset_name == "ds"
assert doc.version_name == "v1"
assert doc.file_relative_path == "file.csv"
def test_joins_nested_relative_path_back_with_slashes(self, auth_env):
- doc = DatasetFileDocument("/datasets/[email protected]/ds/v1/a/b/c/file.csv")
+ doc = DatasetFileDocument("/dataset/[email protected]/ds/v1/a/b/c/file.csv")
assert doc.file_relative_path == "a/b/c/file.csv"
def test_strips_leading_and_trailing_slashes_before_parsing(self,
auth_env):
- doc = DatasetFileDocument("///datasets/[email protected]/ds/v1/file.csv///")
+ doc = DatasetFileDocument("///dataset/[email protected]/ds/v1/file.csv///")
assert doc.owner_email == "[email protected]"
assert doc.file_relative_path == "file.csv"
@@ -80,7 +80,7 @@ class TestDatasetFileDocumentInit:
def test_rejects_prefixed_path_with_too_few_segments(self, auth_env):
with pytest.raises(ValueError, match="Invalid file path format"):
- DatasetFileDocument("/datasets/[email protected]/ds/v1")
+ DatasetFileDocument("/dataset/[email protected]/ds/v1")
def test_rejects_legacy_path_with_too_few_segments(self, auth_env):
with pytest.raises(ValueError, match="Invalid file path format"):
@@ -92,29 +92,29 @@ class TestDatasetFileDocumentInit:
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT
)
with pytest.raises(ValueError, match="JWT token is required"):
- DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv")
+ DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv")
def test_treats_empty_jwt_as_missing(self, monkeypatch):
# An empty string is falsy and should be rejected just like an unset
var.
monkeypatch.setenv("USER_JWT_TOKEN", "")
with pytest.raises(ValueError, match="JWT token is required"):
- DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv")
+ DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv")
def test_falls_back_to_default_endpoint_when_env_missing(self,
monkeypatch):
monkeypatch.setenv("USER_JWT_TOKEN", "tok")
monkeypatch.delenv(
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", raising=False
)
- doc = DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv")
+ doc = DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv")
assert doc.presign_endpoint == DEFAULT_ENDPOINT
def test_uses_explicit_endpoint_from_environment(self, auth_env):
- doc = DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv")
+ doc = DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv")
assert doc.presign_endpoint == CUSTOM_ENDPOINT
class TestGetPresignedUrl:
- def _make_doc(self, monkeypatch,
path="/datasets/[email protected]/ds/v1/file.csv"):
+ def _make_doc(self, monkeypatch,
path="/dataset/[email protected]/ds/v1/file.csv"):
monkeypatch.setenv("USER_JWT_TOKEN", "test-jwt-token")
monkeypatch.setenv(
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT
@@ -144,9 +144,7 @@ class TestGetPresignedUrl:
def test_url_encodes_filepath_query_parameter(self, monkeypatch):
# urllib.parse.quote keeps "/" as safe by default, but encodes "@"
# and " " — pin both pieces so the contract is explicit.
- doc = self._make_doc(
- monkeypatch, path="/datasets/[email protected]/ds/v1/data file.csv"
- )
+ doc = self._make_doc(monkeypatch, path="/dataset/[email protected]/ds/v1/data
file.csv")
with patch(
"pytexera.storage.dataset_file_document.requests.Session.get"
) as mock_get:
@@ -158,16 +156,16 @@ class TestGetPresignedUrl:
assert "bob%40x.com" in file_path
assert file_path.startswith("/")
- def test_sends_datasets_prefixed_filepath(self, monkeypatch):
- # The reconstructed filePath sent to the file-service carries the
"datasets" prefix.
- doc = self._make_doc(monkeypatch,
path="/datasets/[email protected]/ds/v1/file.csv")
+ def test_sends_dataset_prefixed_filepath(self, monkeypatch):
+ # The reconstructed filePath sent to the file-service carries the
"dataset" prefix.
+ doc = self._make_doc(monkeypatch,
path="/dataset/[email protected]/ds/v1/file.csv")
with patch(
"pytexera.storage.dataset_file_document.requests.Session.get"
) as mock_get:
mock_get.return_value = make_response(200, body={"presignedUrl":
"u"})
doc.get_presigned_url()
_, kwargs = mock_get.call_args
- assert kwargs["params"]["filePath"].startswith("/datasets/")
+ assert kwargs["params"]["filePath"].startswith("/dataset/")
def test_calls_configured_endpoint(self, monkeypatch):
doc = self._make_doc(monkeypatch)
@@ -235,7 +233,7 @@ class TestReadFile:
monkeypatch.setenv(
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT
)
- return DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv")
+ return DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv")
def test_returns_bytesio_with_downloaded_content(self, monkeypatch):
doc = self._make_doc(monkeypatch)
@@ -291,7 +289,7 @@ class TestTimeoutsAndRetries:
monkeypatch.setenv(
"FILE_SERVICE_GET_DATASET_PRESIGNED_URL_ENDPOINT", CUSTOM_ENDPOINT
)
- return DatasetFileDocument("/datasets/[email protected]/ds/v1/file.csv")
+ return DatasetFileDocument("/dataset/[email protected]/ds/v1/file.csv")
def test_presigned_url_request_passes_request_timeout(self, monkeypatch):
doc = self._make_doc(monkeypatch)
diff --git
a/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala
b/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala
index 7dc5742a18..7cc549004d 100644
---
a/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala
+++
b/amber/src/test/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowExecutionsResourceSpec.scala
@@ -910,7 +910,7 @@ class WorkflowExecutionsResourceSpec
val content =
"""{
| "operators": [
- | {"operatorID": "scanA", "operatorProperties": {"fileName":
"/datasets/[email protected]/LockedDS/v1/data.csv"}},
+ | {"operatorID": "scanA", "operatorProperties": {"fileName":
"/dataset/[email protected]/LockedDS/v1/data.csv"}},
| {"operatorID": "downstreamB", "operatorProperties": {}}
| ],
| "links": [
@@ -969,7 +969,7 @@ class WorkflowExecutionsResourceSpec
val content =
"""{
| "operators": [
- | {"operatorID": "scan", "operatorProperties": {"fileName":
"/datasets/[email protected]/MyDS/v1/data.csv"}}
+ | {"operatorID": "scan", "operatorProperties": {"fileName":
"/dataset/[email protected]/MyDS/v1/data.csv"}}
| ],
| "links": []
|}""".stripMargin
@@ -1408,7 +1408,7 @@ class WorkflowExecutionsResourceSpec
private def scanOperator(operatorId: String, datasetName: String): String =
s"""{"operatorID": "$operatorId", "operatorProperties": """ +
- s"""{"fileName":
"/datasets/$foreignOwnerEmail/$datasetName/v1/data.csv"}}"""
+ s"""{"fileName":
"/dataset/$foreignOwnerEmail/$datasetName/v1/data.csv"}}"""
// Seeds three datasets owned by somebody other than testUser and wires the
workflow as
//
diff --git a/bin/single-node/examples/workflows/[Example] Data Exploration on
Movies Dataset.json b/bin/single-node/examples/workflows/[Example] Data
Exploration on Movies Dataset.json
index cdc9a0e870..5df44f5382 100644
--- a/bin/single-node/examples/workflows/[Example] Data Exploration on Movies
Dataset.json
+++ b/bin/single-node/examples/workflows/[Example] Data Exploration on Movies
Dataset.json
@@ -8,7 +8,7 @@
"fileEncoding": "UTF_8",
"customDelimiter": ",",
"hasHeader": true,
- "fileName":
"/datasets/texera/popular-movies-of-imdb/v1/TMDb_updated.csv",
+ "fileName":
"/dataset/texera/popular-movies-of-imdb/v1/TMDb_updated.csv",
"offset": 0,
"limit": 1000
},
diff --git a/bin/single-node/examples/workflows/[Example] Machine Learning on
Iris Dataset.json b/bin/single-node/examples/workflows/[Example] Machine
Learning on Iris Dataset.json
index 7e9a5d11fd..e4cdc3a8af 100644
--- a/bin/single-node/examples/workflows/[Example] Machine Learning on Iris
Dataset.json
+++ b/bin/single-node/examples/workflows/[Example] Machine Learning on Iris
Dataset.json
@@ -8,7 +8,7 @@
"fileEncoding": "UTF_8",
"customDelimiter": ",",
"hasHeader": true,
- "fileName": "/datasets/texera/iris-species/v1/Iris.csv"
+ "fileName": "/dataset/texera/iris-species/v1/Iris.csv"
},
"inputPorts": [],
"outputPorts": [
diff --git
a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala
b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala
index 8481906c9c..4915d57897 100644
---
a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala
+++
b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala
@@ -60,8 +60,8 @@ object DocumentFactory {
*/
def openReadonlyDocument(fileUri: URI): ReadonlyVirtualDocument[_] = {
fileUri.getScheme match {
- case DATASET_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri,
ResourceType.Datasets)
- case MODEL_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri,
ResourceType.Models)
+ case DATASET_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri,
ResourceType.Dataset)
+ case MODEL_FILE_URI_SCHEME => new LakeFSFileDocument(fileUri,
ResourceType.Model)
case "file" => new ReadonlyLocalFileDocument(fileUri)
case unsupportedScheme =>
throw new UnsupportedOperationException(
@@ -184,8 +184,8 @@ object DocumentFactory {
*/
def openDocument(uri: URI): (VirtualDocument[_], Option[Schema]) = {
uri.getScheme match {
- case DATASET_FILE_URI_SCHEME => (new LakeFSFileDocument(uri,
ResourceType.Datasets), None)
- case MODEL_FILE_URI_SCHEME => (new LakeFSFileDocument(uri,
ResourceType.Models), None)
+ case DATASET_FILE_URI_SCHEME => (new LakeFSFileDocument(uri,
ResourceType.Dataset), None)
+ case MODEL_FILE_URI_SCHEME => (new LakeFSFileDocument(uri,
ResourceType.Model), None)
case VFS_FILE_URI_SCHEME =>
val IcebergLocation(warehouse, namespace, storageKey) =
resolveIcebergLocation(uri)
diff --git
a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala
b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala
index 1a2e4fcf2a..95127e234d 100644
---
a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala
+++
b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/FileResolver.scala
@@ -93,7 +93,7 @@ object FileResolver {
* caller can dispatch to the right backing table.
* - Legacy unprefixed (datasets only, backward compat):
/ownerEmail/datasetName/versionName/
* fileRelativePath (>= 4 segments), resolved as a dataset. Models are
new and always require
- * the /models/ prefix.
+ * the /model/ prefix.
*
* @param fileName the file path to parse
* @return Some((resourceType, ownerEmail, resourceName, versionName,
fileRelativePath)) if valid,
@@ -115,11 +115,11 @@ object FileResolver {
)
case None =>
// Legacy unprefixed dataset path (backward compat):
/ownerEmail/datasetName/versionName/<file>.
- // TODO(datasets-prefix): require the prefix once all stored paths are
migrated (36.sql).
+ // TODO(dataset-prefix): require the prefix once all stored paths are
migrated (36.sql).
if (pathSegments.length >= 4)
Some(
(
- ResourceType.Datasets,
+ ResourceType.Dataset,
pathSegments(0),
pathSegments(1),
pathSegments(2),
@@ -139,8 +139,8 @@ object FileResolver {
*
* Input: /<prefix>/ownerEmail/resourceName/versionName/fileRelativePath
(or legacy unprefixed)
* Output: {scheme}:///{repositoryName}/{versionHash}/fileRelativePath
- * e.g. /datasets/[email protected]/twitter/v1/dir/f.csv ->
dataset:///dataset-15/adeq233td/dir/f.csv
- * /models/[email protected]/resnet/v1/weights/m.pt ->
model:///model-15/adeq233td/weights/m.pt
+ * e.g. /dataset/[email protected]/twitter/v1/dir/f.csv ->
dataset:///dataset-15/adeq233td/dir/f.csv
+ * /model/[email protected]/resnet/v1/weights/m.pt ->
model:///model-15/adeq233td/weights/m.pt
*
* @throws java.io.FileNotFoundException if the path is not a valid
versioned-resource path, the
* resource/version does not exist,
or the URI is malformed
@@ -152,10 +152,10 @@ object FileResolver {
)
val (scheme, repositoryName, versionHash) = resourceType match {
- case ResourceType.Datasets =>
+ case ResourceType.Dataset =>
val (repo, hash) = lookupDataset(ownerEmail, resourceName,
versionName, fileName)
(DATASET_FILE_URI_SCHEME, repo, hash)
- case ResourceType.Models =>
+ case ResourceType.Model =>
val (repo, hash) = lookupModel(ownerEmail, resourceName, versionName,
fileName)
(MODEL_FILE_URI_SCHEME, repo, hash)
case other =>
@@ -306,7 +306,7 @@ object FileResolver {
return None
}
parsePrefixedPath(path).collect {
- case (ResourceType.Datasets, ownerEmail, datasetName, _, _) =>
(ownerEmail, datasetName)
+ case (ResourceType.Dataset, ownerEmail, datasetName, _, _) =>
(ownerEmail, datasetName)
}
}
}
diff --git
a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala
b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala
index 087dc6ef87..47e90a1818 100644
---
a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala
+++
b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/ResourceType.scala
@@ -26,8 +26,8 @@ package org.apache.texera.amber.core.storage
* Path shape: /<prefix>/ownerEmail/resourceName/versionName/fileRelativePath
*/
object ResourceType extends Enumeration {
- val Datasets: Value = Value("datasets")
- val Models: Value = Value("models")
+ val Dataset: Value = Value("dataset")
+ val Model: Value = Value("model")
/**
* Returns the resource type named by the given path segment, or None if it
is not a known
diff --git
a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala
b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala
index b821ff72f5..55fab33ce2 100644
---
a/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala
+++
b/common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocument.scala
@@ -60,8 +60,8 @@ object LakeFSFileDocument {
*/
def presignEndpointOf(resourceType: ResourceType.Value): String =
resourceType match {
- case ResourceType.Datasets => datasetPresignEndpoint
- case ResourceType.Models => modelPresignEndpoint
+ case ResourceType.Dataset => datasetPresignEndpoint
+ case ResourceType.Model => modelPresignEndpoint
}
}
diff --git
a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala
b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala
index a957819b31..f219383f25 100644
---
a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala
+++
b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala
@@ -137,7 +137,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers
with BeforeAndAfterA
val datasetUri = new URI(s"dataset:///repo/$versionHash/file.txt")
val doc = DocumentFactory.openReadonlyDocument(datasetUri)
doc shouldBe a[LakeFSFileDocument]
- doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe
ResourceType.Datasets
+ doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe
ResourceType.Dataset
doc.getURI shouldBe datasetUri
}
@@ -145,7 +145,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers
with BeforeAndAfterA
val modelUri = new URI(s"model:///model-1/$versionHash/weights/model.pt")
val doc = DocumentFactory.openReadonlyDocument(modelUri)
doc shouldBe a[LakeFSFileDocument]
- doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe
ResourceType.Models
+ doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe
ResourceType.Model
doc.getURI shouldBe modelUri
val resource = doc.asInstanceOf[OnVersionedFileResource]
@@ -181,7 +181,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers
with BeforeAndAfterA
val datasetUri = new URI(s"dataset:///repo/$versionHash/file.txt")
val (doc, schemaOpt) = DocumentFactory.openDocument(datasetUri)
doc shouldBe a[LakeFSFileDocument]
- doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe
ResourceType.Datasets
+ doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe
ResourceType.Dataset
schemaOpt shouldBe None
}
@@ -189,7 +189,7 @@ class DocumentFactorySpec extends AnyFlatSpec with Matchers
with BeforeAndAfterA
val modelUri = new URI(s"model:///model-1/$versionHash/weights/model.pt")
val (doc, schemaOpt) = DocumentFactory.openDocument(modelUri)
doc shouldBe a[LakeFSFileDocument]
- doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe
ResourceType.Models
+ doc.asInstanceOf[LakeFSFileDocument].resourceType shouldBe
ResourceType.Model
schemaOpt shouldBe None
}
diff --git
a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
index 56d5cd36ce..fa569f97ca 100644
---
a/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
+++
b/common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/model/LakeFSFileDocumentSpec.scala
@@ -35,7 +35,7 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with
Matchers {
private val versionHash = "97fd4c2a755b69b7c66d322eab40b7e5c2ad5d10"
// URI parsing is shared by every resource type; exercise it through the
dataset type.
- private def datasetDoc(uri: URI) = new LakeFSFileDocument(uri,
ResourceType.Datasets)
+ private def datasetDoc(uri: URI) = new LakeFSFileDocument(uri,
ResourceType.Dataset)
"LakeFSFileDocument" should "parse a valid 3-segment dataset URI into its
components" in {
val uri = new URI(s"dataset:///test_dataset/$versionHash/1.txt")
@@ -160,7 +160,7 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with
Matchers {
"http://localhost:9092/api/dataset/presign-download";
)
.trim
- LakeFSFileDocument.presignEndpointOf(ResourceType.Datasets) shouldBe
expected
+ LakeFSFileDocument.presignEndpointOf(ResourceType.Dataset) shouldBe
expected
}
// Each resource type resolves its own endpoint: a dataset grant must not
authorize a model
@@ -173,9 +173,9 @@ class LakeFSFileDocumentSpec extends AnyFlatSpec with
Matchers {
"http://localhost:9092/api/model/presign-download";
)
.trim
- LakeFSFileDocument.presignEndpointOf(ResourceType.Models) shouldBe expected
- LakeFSFileDocument.presignEndpointOf(ResourceType.Models) should not be
- LakeFSFileDocument.presignEndpointOf(ResourceType.Datasets)
+ LakeFSFileDocument.presignEndpointOf(ResourceType.Model) shouldBe expected
+ LakeFSFileDocument.presignEndpointOf(ResourceType.Model) should not be
+ LakeFSFileDocument.presignEndpointOf(ResourceType.Dataset)
}
// The user JWT token is shared by every LakeFS-backed document, whatever
its resource type.
diff --git
a/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala
b/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala
index e414371222..2649a3d860 100644
---
a/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala
+++
b/common/workflow-core/src/test/scala/org/apache/texera/amber/storage/FileResolverSpec.scala
@@ -122,13 +122,13 @@ class FileResolverSpec
private val localCsvFilePath =
"common/workflow-core/src/test/resources/country_sales_small.csv"
- private val datasetACsvFilePath =
"/datasets/[email protected]/test_dataset/v2/directory/a.csv"
+ private val datasetACsvFilePath =
"/dataset/[email protected]/test_dataset/v2/directory/a.csv"
- private val dataset1TxtFilePath =
"/datasets/[email protected]/test_dataset/v1/1.txt"
+ private val dataset1TxtFilePath =
"/dataset/[email protected]/test_dataset/v1/1.txt"
- private val modelWeightsFilePath =
"/models/[email protected]/test_model/v2/weights/model.pt"
+ private val modelWeightsFilePath =
"/model/[email protected]/test_model/v2/weights/model.pt"
- private val modelReadmeFilePath =
"/models/[email protected]/test_model/v1/README.md"
+ private val modelReadmeFilePath =
"/model/[email protected]/test_model/v1/README.md"
// Legacy unprefixed form — still resolvable as a dataset (backward compat).
private val unprefixedDataset1TxtFilePath =
"/[email protected]/test_dataset/v1/1.txt"
@@ -137,11 +137,11 @@ class FileResolverSpec
private val unknownResourceTypeFilePath =
"/notAResourceType/[email protected]/test_dataset/v1/1.txt"
- // A model name presented under the datasets prefix must not resolve as a
dataset.
- private val modelNameUnderDatasetPrefix =
"/datasets/[email protected]/test_model/v1/README.md"
+ // A model name presented under the dataset prefix must not resolve as a
dataset.
+ private val modelNameUnderDatasetPrefix =
"/dataset/[email protected]/test_model/v1/README.md"
- // A dataset name presented under the models prefix must not resolve as a
model.
- private val datasetNameUnderModelPrefix =
"/models/[email protected]/test_dataset/v1/1.txt"
+ // A dataset name presented under the model prefix must not resolve as a
model.
+ private val datasetNameUnderModelPrefix =
"/model/[email protected]/test_dataset/v1/1.txt"
override protected def beforeAll(): Unit = {
initializeDBAndReplaceDSLContext()
@@ -225,7 +225,7 @@ class FileResolverSpec
"FileResolver" should "throw not found exception when a prefixed path has
too few segments" in {
assertThrows[FileNotFoundException] {
- FileResolver.resolve("/datasets/[email protected]/test_dataset")
+ FileResolver.resolve("/dataset/[email protected]/test_dataset")
}
}
@@ -254,12 +254,12 @@ class FileResolverSpec
"parseDatasetOwnerAndName" should "extract owner email and dataset name from
a valid path" in {
assert(
-
FileResolver.parseDatasetOwnerAndName("/datasets/[email protected]/test_dataset/v1/1.txt")
+
FileResolver.parseDatasetOwnerAndName("/dataset/[email protected]/test_dataset/v1/1.txt")
== Some(("[email protected]", "test_dataset"))
)
// extra segments beyond the file-relative path are ignored
assert(
-
FileResolver.parseDatasetOwnerAndName("/datasets/[email protected]/ds/v2/directory/nested/a.csv")
+
FileResolver.parseDatasetOwnerAndName("/dataset/[email protected]/ds/v2/directory/nested/a.csv")
== Some(("[email protected]", "ds"))
)
}
@@ -272,7 +272,7 @@ class FileResolverSpec
}
it should "return None when the prefixed path has too few segments" in {
-
assert(FileResolver.parseDatasetOwnerAndName("/datasets/[email protected]/ds").isEmpty)
+
assert(FileResolver.parseDatasetOwnerAndName("/dataset/[email protected]/ds").isEmpty)
assert(FileResolver.parseDatasetOwnerAndName("owner/dataset").isEmpty)
}
diff --git
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala
index fb27257865..26a6496a24 100644
---
a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala
+++
b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExec.scala
@@ -32,7 +32,7 @@ import org.apache.texera.dao.jooq.generated.tables.User.USER
object FileListerSourceOpExec {
/**
- * Parses a dataset version path
(/datasets/ownerEmail/datasetName/versionName) into its
+ * Parses a dataset version path
(/dataset/ownerEmail/datasetName/versionName) into its
* (resourceTypePrefix, ownerEmail, datasetName, versionName) components.
*
* @throws IllegalArgumentException if the path is not a well-formed
dataset version path
@@ -42,14 +42,14 @@ object FileListerSourceOpExec {
): (String, String, String, String) = {
val segments = datasetVersionPath.split("/").filter(_.nonEmpty)
val invalidPath = s"Invalid dataset version path '$datasetVersionPath'; " +
- "expected /datasets/ownerEmail/datasetName/versionName"
+ "expected /dataset/ownerEmail/datasetName/versionName"
if (segments.headOption.exists(ResourceType.isValidPrefix)) {
require(segments.length >= 4, invalidPath)
(segments(0), segments(1), segments(2), segments(3))
} else {
require(segments.length >= 3, invalidPath)
- (ResourceType.Datasets.toString, segments(0), segments(1), segments(2))
+ (ResourceType.Dataset.toString, segments(0), segments(1), segments(2))
}
}
diff --git
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala
index 19fb7513d5..ec0ffd9a29 100644
---
a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala
+++
b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/dataset/FileListerSourceOpExecSpec.scala
@@ -23,10 +23,10 @@ import org.scalatest.flatspec.AnyFlatSpec
class FileListerSourceOpExecSpec extends AnyFlatSpec {
- "parseDatasetVersionPath" should "extract components from a
datasets-prefixed path" in {
+ "parseDatasetVersionPath" should "extract components from a dataset-prefixed
path" in {
val (prefix, owner, name, version) =
-
FileListerSourceOpExec.parseDatasetVersionPath("/datasets/[email protected]/twitterDataset/v1")
- assert(prefix == "datasets")
+
FileListerSourceOpExec.parseDatasetVersionPath("/dataset/[email protected]/twitterDataset/v1")
+ assert(prefix == "dataset")
assert(owner == "[email protected]")
assert(name == "twitterDataset")
assert(version == "v1")
@@ -34,7 +34,7 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
it should "work when the owner segment is a username without an '@'" in {
val (prefix, owner, name, version) =
-
FileListerSourceOpExec.parseDatasetVersionPath("/datasets/texera/test-ds/v1")
+
FileListerSourceOpExec.parseDatasetVersionPath("/dataset/texera/test-ds/v1")
assert(owner == "texera")
assert(name == "test-ds")
assert(version == "v1")
@@ -42,8 +42,8 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
it should "ignore trailing slashes and extra segments" in {
val (prefix, owner, name, version) =
-
FileListerSourceOpExec.parseDatasetVersionPath("/datasets/alice/ds/v2/extra/")
- assert(prefix == "datasets")
+
FileListerSourceOpExec.parseDatasetVersionPath("/dataset/alice/ds/v2/extra/")
+ assert(prefix == "dataset")
assert(owner == "alice")
assert(name == "ds")
assert(version == "v2")
@@ -52,11 +52,11 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
"canonicalVersionPath" should "rebuild the prefixed version path from its
components" in {
assert(
FileListerSourceOpExec.canonicalVersionPath(
- "datasets",
+ "dataset",
"[email protected]",
"twitterDataset",
"v1"
- ) == "/datasets/[email protected]/twitterDataset/v1"
+ ) == "/dataset/[email protected]/twitterDataset/v1"
)
}
@@ -64,17 +64,17 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
// Emitted paths must be rooted at the parsed components, not the raw
configured path:
// a stray "extra" segment would otherwise leak into every emitted file
path.
val (prefix, owner, name, version) =
-
FileListerSourceOpExec.parseDatasetVersionPath("/datasets/alice/ds/v2/extra/")
+
FileListerSourceOpExec.parseDatasetVersionPath("/dataset/alice/ds/v2/extra/")
assert(
FileListerSourceOpExec.canonicalVersionPath(prefix, owner, name, version)
- == "/datasets/alice/ds/v2"
+ == "/dataset/alice/ds/v2"
)
}
it should "still accept a legacy unprefixed path" in {
val (prefix, owner, name, version) =
FileListerSourceOpExec.parseDatasetVersionPath("/alice/ds/v1")
- assert(prefix == "datasets")
+ assert(prefix == "dataset")
assert(owner == "alice")
assert(name == "ds")
assert(version == "v1")
@@ -85,7 +85,7 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
FileListerSourceOpExec.parseDatasetVersionPath("/alice/ds/v1")
assert(
FileListerSourceOpExec.canonicalVersionPath(prefix, owner, name, version)
- == "/datasets/alice/ds/v1"
+ == "/dataset/alice/ds/v1"
)
}
@@ -99,7 +99,7 @@ class FileListerSourceOpExecSpec extends AnyFlatSpec {
it should "reject a prefixed path with too few segments" in {
assertThrows[IllegalArgumentException] {
- FileListerSourceOpExec.parseDatasetVersionPath("/datasets/alice/ds")
+ FileListerSourceOpExec.parseDatasetVersionPath("/dataset/alice/ds")
}
}
diff --git
a/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
b/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
index 8bb1a70661..4c8243b850 100644
---
a/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
+++
b/file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala
@@ -353,7 +353,7 @@ class DatasetResource extends LazyLogging {
private val COVER_IMAGE_SIZE_LIMIT_BYTES: Long = 10 * 1024 * 1024 // 10 MB
private val ALLOWED_IMAGE_EXTENSIONS: Set[String] = Set(".jpg", ".jpeg",
".png", ".gif", ".webp")
- private val resourceType = ResourceType.Datasets
+ private val resourceType = ResourceType.Dataset
/**
* Helper function to get the dataset from DB with additional information
including user access privilege and owner email
diff --git
a/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala
b/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala
index 821d02045c..85709b8e5b 100644
---
a/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala
+++
b/file-service/src/main/scala/org/apache/texera/service/type/dataset/DatasetFileNode.scala
@@ -25,8 +25,8 @@ import org.apache.texera.amber.core.storage.ResourceType
import scala.collection.mutable
// DatasetFileNode represents a unique file in dataset, its full path is in
the format of:
-// /datasets/ownerEmail/datasetName/versionName/fileRelativePath
-// e.g. /datasets/[email protected]/twitterDataset/v1/california/irvine/tw1.csv
+// /dataset/ownerEmail/datasetName/versionName/fileRelativePath
+// e.g. /dataset/[email protected]/twitterDataset/v1/california/irvine/tw1.csv
class DatasetFileNode(
val name: String, // direct name of this node
val nodeType: String, // "file" or "directory"
@@ -79,10 +79,10 @@ object DatasetFileNode {
): List[DatasetFileNode] = {
val rootNode = new DatasetFileNode("/", "directory", null, "")
- // Root the tree at the datasets prefix node (a directory node named
"datasets").
- val datasetsNode =
- new DatasetFileNode(ResourceType.Datasets.toString, "directory",
rootNode, "")
- rootNode.children = Some(List(datasetsNode))
+ // Root the tree at the dataset prefix node (a directory node named
"dataset").
+ val prefixNode =
+ new DatasetFileNode(ResourceType.Dataset.toString, "directory",
rootNode, "")
+ rootNode.children = Some(List(prefixNode))
// Owner level nodes map
val ownerNodes = mutable.Map[String, DatasetFileNode]()
@@ -91,8 +91,8 @@ object DatasetFileNode {
case ((ownerEmail, datasetName, versionName), objects) =>
val ownerNode = ownerNodes.getOrElseUpdate(
ownerEmail, {
- val newNode = new DatasetFileNode(ownerEmail, "directory",
datasetsNode, ownerEmail)
- datasetsNode.children = Some(datasetsNode.getChildren :+ newNode)
+ val newNode = new DatasetFileNode(ownerEmail, "directory",
prefixNode, ownerEmail)
+ prefixNode.children = Some(prefixNode.getChildren :+ newNode)
newNode
}
)
diff --git
a/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala
b/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala
index 160412f8b9..12321d4180 100644
---
a/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala
+++
b/file-service/src/test/scala/org/apache/texera/service/type/dataset/DatasetFileNodeSpec.scala
@@ -99,13 +99,13 @@ class DatasetFileNodeSpec extends AnyFlatSpec with Matchers
{
Map(("[email protected]", "twitter", "v1") -> objects)
)
- // The tree is rooted at a single "datasets" prefix node; owners nest
under it.
+ // The tree is rooted at a single "dataset" prefix node; owners nest under
it.
roots should have size 1
- val datasetsNode = roots.head
- datasetsNode.getName shouldBe "datasets"
- datasetsNode.getNodeType shouldBe "directory"
+ val prefixNode = roots.head
+ prefixNode.getName shouldBe "dataset"
+ prefixNode.getNodeType shouldBe "directory"
- val ownerNode = datasetsNode.getChildren.find(_.getName ==
"[email protected]").get
+ val ownerNode = prefixNode.getChildren.find(_.getName ==
"[email protected]").get
ownerNode.getNodeType shouldBe "directory"
val datasetNode = ownerNode.getChildren.find(_.getName == "twitter").get
@@ -123,7 +123,7 @@ class DatasetFileNodeSpec extends AnyFlatSpec with Matchers
{
val file1 = bDir.getChildren.find(_.getName == "1.csv").get
file1.getNodeType shouldBe "file"
file1.getSize shouldBe Some(2L)
- file1.getFilePath shouldBe "/datasets/[email protected]/twitter/v1/b/1.csv"
+ file1.getFilePath shouldBe "/dataset/[email protected]/twitter/v1/b/1.csv"
// Total size equals the sum of the three files.
DatasetFileNode.calculateTotalSize(roots) shouldBe 6L
diff --git a/frontend/src/app/common/type/datasetVersionFileTree.spec.ts
b/frontend/src/app/common/type/datasetVersionFileTree.spec.ts
index 64231dd156..207182e5fd 100644
--- a/frontend/src/app/common/type/datasetVersionFileTree.spec.ts
+++ b/frontend/src/app/common/type/datasetVersionFileTree.spec.ts
@@ -41,7 +41,7 @@ describe("getRelativePathFromDatasetFileNode", () => {
const node: DatasetFileNode = {
name: "tw1.csv",
type: "file",
- parentDir:
"/datasets/[email protected]/twitterDataset/v1/california/irvine",
+ parentDir:
"/dataset/[email protected]/twitterDataset/v1/california/irvine",
};
expect(getRelativePathFromDatasetFileNode(node)).toBe("california/irvine/tw1.csv");
});
@@ -50,7 +50,7 @@ describe("getRelativePathFromDatasetFileNode", () => {
const node: DatasetFileNode = {
name: "readme.txt",
type: "file",
- parentDir: "/datasets/[email protected]/twitterDataset/v1",
+ parentDir: "/dataset/[email protected]/twitterDataset/v1",
};
expect(getRelativePathFromDatasetFileNode(node)).toBe("readme.txt");
});
@@ -59,7 +59,7 @@ describe("getRelativePathFromDatasetFileNode", () => {
const node: DatasetFileNode = {
name: "v1",
type: "directory",
- parentDir: "/datasets/[email protected]/twitterDataset",
+ parentDir: "/dataset/[email protected]/twitterDataset",
};
expect(getRelativePathFromDatasetFileNode(node)).toBe("");
});
@@ -68,7 +68,7 @@ describe("getRelativePathFromDatasetFileNode", () => {
const node: DatasetFileNode = {
name: "f.csv",
type: "file",
- parentDir: "/datasets/[email protected]/twitterDataset//v1/sub",
+ parentDir: "/dataset/[email protected]/twitterDataset//v1/sub",
};
expect(getRelativePathFromDatasetFileNode(node)).toBe("sub/f.csv");
});
diff --git a/frontend/src/app/common/type/datasetVersionFileTree.ts
b/frontend/src/app/common/type/datasetVersionFileTree.ts
index 98f898d443..efbb875128 100644
--- a/frontend/src/app/common/type/datasetVersionFileTree.ts
+++ b/frontend/src/app/common/type/datasetVersionFileTree.ts
@@ -32,7 +32,7 @@ export function getFullPathFromDatasetFileNode(node:
DatasetFileNode): string {
/**
* Returns the relative path of a DatasetFileNode by stripping the first four
segments
- * (datasets/ownerEmail/datasetName/versionName).
+ * (dataset/ownerEmail/datasetName/versionName).
* @param node The DatasetFileNode whose relative path is needed.
* @returns The relative path (without the first four segments and without a
leading slash).
*/
diff --git a/frontend/src/app/common/type/resource-type.ts
b/frontend/src/app/common/type/resource-type.ts
index a129c917e2..22c4a8592a 100644
--- a/frontend/src/app/common/type/resource-type.ts
+++ b/frontend/src/app/common/type/resource-type.ts
@@ -17,8 +17,8 @@
*/
/**
- * The leading segment of a logical file path, identifying the resource kind
(e.g. /datasets/...).
+ * The leading segment of a logical file path, identifying the resource kind
(e.g. /dataset/...).
*/
export enum ResourceType {
- Datasets = "datasets",
+ Dataset = "dataset",
}
diff --git
a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts
b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts
index 78c35761a4..7bb6af6e0d 100644
---
a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts
+++
b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/dataset-detail.component.spec.ts
@@ -2694,7 +2694,7 @@ describe("DatasetDetailComponent rendered template", ()
=> {
const leaf = (name: string): DatasetFileNode => ({
name,
type: "file",
- parentDir: `/datasets/${OWNER}/ds/v1/nested`,
+ parentDir: `/dataset/${OWNER}/ds/v1/nested`,
size: 2048,
});
@@ -2719,7 +2719,7 @@ describe("DatasetDetailComponent rendered template", ()
=> {
// The heading is the full path — the copy-path button beside it copies
// exactly this string — not the bare file name or the relative path.
expect(text(q<HTMLElement>(fixture.nativeElement,
".file-title-main"))).toBe(
- `/datasets/${OWNER}/ds/v1/nested/b.csv`
+ `/dataset/${OWNER}/ds/v1/nested/b.csv`
);
// 2048 bytes reaches the reader as a human-readable size, not as a raw
count.
expect(text(q<HTMLElement>(fixture.nativeElement,
".file-size"))).toBe("2.00 KB");
diff --git
a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts
b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts
index 806d2a6f25..ff9e82609a 100644
---
a/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts
+++
b/frontend/src/app/dashboard/component/user/user-dataset/user-dataset-explorer/user-dataset-version-filetree/user-dataset-version-filetree.component.spec.ts
@@ -183,7 +183,7 @@ describe("UserDatasetVersionFiletreeComponent", () => {
const file = (name: string): DatasetFileNode => ({
name,
type: "file",
- parentDir: "/datasets/owner/dataset/v1",
+ parentDir: "/dataset/owner/dataset/v1",
});
// Deliberately synchronous: the tree keeps a pending timer, so awaiting
whenStable()
@@ -262,9 +262,9 @@ describe("UserDatasetVersionFiletreeComponent", () => {
const emitted: string[] = [];
component.setCoverImage.subscribe((path: string) => emitted.push(path));
- // parentDir has exactly the four stripped segments
(datasets/owner/dataset/version),
+ // parentDir has exactly the four stripped segments
(dataset/owner/dataset/version),
// so the relative path is just the file name.
- component.onSetCover({ name: "photo.png", type: "file", parentDir:
"/datasets/owner/dataset/v1" });
+ component.onSetCover({ name: "photo.png", type: "file", parentDir:
"/dataset/owner/dataset/v1" });
expect(emitted).toEqual(["photo.png"]);
});
diff --git
a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts
b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts
index 604d86a9f9..f42ae00a81 100644
---
a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts
+++
b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.spec.ts
@@ -111,7 +111,7 @@ describe("DatasetSelectionModalComponent", () => {
it("ngOnInit initializes selectedDataset and selectedVersion from
data.selectedPath", () => {
modalData.fileMode = true;
- modalData.selectedPath = `/datasets/${OWNER}/myds/v1`;
+ modalData.selectedPath = `/dataset/${OWNER}/myds/v1`;
build();
@@ -154,7 +154,7 @@ describe("DatasetSelectionModalComponent", () => {
expect(datasetService.retrieveDatasetVersionFileTree).toHaveBeenCalledWith(10,
100);
expect(component.fileTree).toEqual([fileNode]);
- expect(component.selectedPath).toBe(`/datasets/${OWNER}/myds/v1`);
+ expect(component.selectedPath).toBe(`/dataset/${OWNER}/myds/v1`);
});
it("onFileSelected sets selectedPath to the node's full path in file mode",
() => {
@@ -303,12 +303,12 @@ describe("DatasetSelectionModalComponent", () => {
confirmButton().click();
-
expect(modalRef.close).toHaveBeenCalledWith(`/datasets/${OWNER}/myds/v1`);
+ expect(modalRef.close).toHaveBeenCalledWith(`/dataset/${OWNER}/myds/v1`);
});
it("takes the path from the file tree's selection in file mode", () => {
modalData.fileMode = true;
- modalData.selectedPath = `/datasets/${OWNER}/myds/v1`;
+ modalData.selectedPath = `/dataset/${OWNER}/myds/v1`;
build();
fixture.debugElement
diff --git
a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts
b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts
index 97ecf579ca..b5768aa685 100644
---
a/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts
+++
b/frontend/src/app/workspace/component/dataset-selection-modal/dataset-selection-modal.component.ts
@@ -85,7 +85,7 @@ export class DatasetSelectionModalComponent implements OnInit
{
const selectedPath = this.data.selectedPath;
if (selectedPath) {
const segments = selectedPath.split("/").filter(part => part.length
> 0);
- // TODO(datasets-prefix): require the prefix once all ml model
support PRs are done.
+ // TODO(dataset-prefix): require the prefix once all ml model
support PRs are done.
if ((Object.values(ResourceType) as string[]).includes(segments[0]))
{
segments.shift();
}
@@ -124,7 +124,7 @@ export class DatasetSelectionModalComponent implements
OnInit {
this.fileTree = data.fileNodes;
});
if (!this.data.fileMode) {
- this.selectedPath =
`/${ResourceType.Datasets}/${this.selectedDataset.ownerEmail}/${this.selectedDataset.dataset.name}/${this.selectedVersion.name}`;
+ this.selectedPath =
`/${ResourceType.Dataset}/${this.selectedDataset.ownerEmail}/${this.selectedDataset.dataset.name}/${this.selectedVersion.name}`;
}
}
}
diff --git
a/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts
b/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts
index 394e9773cc..f5ba4045e8 100644
---
a/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts
+++
b/frontend/src/app/workspace/component/dataset-version-selector/dataset-version-selector.component.spec.ts
@@ -101,11 +101,11 @@ describe("DatasetVersionSelectorComponent", () => {
});
it("shows the chosen path in a read-only box", () => {
- setFormControl("/datasets/[email protected]/myds/v1");
+ setFormControl("/dataset/[email protected]/myds/v1");
fixture.detectChanges();
const box = selectionBox()!;
- expect(box.value).toBe("/datasets/[email protected]/myds/v1");
+ expect(box.value).toBe("/dataset/[email protected]/myds/v1");
// The path is only editable through the picker, never by typing.
expect(box.readOnly).toBe(true);
expect(box.required).toBe(true);
@@ -114,13 +114,13 @@ describe("DatasetVersionSelectorComponent", () => {
it("opens the picker from the button and shows whatever it returns", () =>
{
setFormControl("");
fixture.detectChanges();
- modalServiceSpy.create.mockReturnValue({ afterClose:
of("/datasets/[email protected]/myds/v2") });
+ modalServiceSpy.create.mockReturnValue({ afterClose:
of("/dataset/[email protected]/myds/v2") });
openButton().click();
fixture.detectChanges();
expect(modalServiceSpy.create).toHaveBeenCalledTimes(1);
- expect(selectionBox()!.value).toBe("/datasets/[email protected]/myds/v2");
+ expect(selectionBox()!.value).toBe("/dataset/[email protected]/myds/v2");
});
});
});
diff --git a/sql/changelog.xml b/sql/changelog.xml
index c0f735a09c..b9c96431a5 100644
--- a/sql/changelog.xml
+++ b/sql/changelog.xml
@@ -104,6 +104,11 @@
<sqlFile path="sql/updates/38.sql"/>
</changeSet>
+ <!-- Singularize the resource-type prefix on stored dataset paths -->
+ <changeSet id="39" author="tanishqgandhi1908">
+ <sqlFile path="sql/updates/39.sql"/>
+ </changeSet>
+
<!-- example changeSet
<changeSet id="1" author="author">
<sqlFile path="sql/updates/1.sql"/>
diff --git a/sql/updates/39.sql b/sql/updates/39.sql
new file mode 100644
index 0000000000..4c48244bdd
--- /dev/null
+++ b/sql/updates/39.sql
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+\c texera_db
+
+SET search_path TO texera_db;
+
+BEGIN;
+
+-- The resource-type prefix on dataset logical paths is singular, matching the
table
+-- it names: /dataset/ownerEmail/datasetName/versionName/... This supersedes
36.sql,
+-- which introduced the prefix in its plural form; that changeSet is left as it
+-- shipped, so databases that already recorded it are brought forward here
instead.
+--
+-- Stored workflows carry such paths inside workflow.content and
+-- workflow_version.content, in two operator properties:
+-- * fileName (scan-source operators): /owner/name/version/file
+-- * datasetVersionPath (file-lister operator): /owner/name/version
+--
+-- Both are normalized to that form: a "/datasets/" prefix has its leading
segment
+-- rewritten, and a path with no prefix gets "dataset" prepended. The prefixed
case
+-- is tested first, since such a path looks unprefixed to the other branch and
must
+-- not be prefixed twice.
+--
+-- Either case applies only when the path's owner and name segments match an
+-- existing (user.email, dataset.name) pair -- unique, and read at parts 1 and
2
+-- unprefixed, 2 and 3 prefixed. That guard is what keeps the migration off the
+-- plain filesystem paths and URLs this column also holds: "/datasets" is an
+-- ordinary directory name, and FileResolver tries localResolveFunc first, so a
+-- fileName of /datasets/imdb/movies.csv can be a working local mount. It
matches
+-- no dataset, so it is left alone.
+--
+-- Values already in the target form match neither case, making this
idempotent.
+-- jsonb_set uses create_missing = false so absent properties are never added.
+
+DO $$
+DECLARE
+ wf_count INT := 0;
+ wv_count INT := 0;
+BEGIN
+ WITH affected AS (
+ SELECT w.wid
+ FROM workflow w,
+ jsonb_array_elements(
+ CASE
+ WHEN jsonb_typeof(w.content::jsonb -> 'operators') =
'array'
+ THEN w.content::jsonb -> 'operators'
+ ELSE '[]'::jsonb
+ END
+ ) AS op,
+ LATERAL (
+ SELECT op #>> '{operatorProperties,fileName}' AS fn,
+ op #>> '{operatorProperties,datasetVersionPath}' AS dvp
+ ) f
+ WHERE jsonb_typeof(w.content::jsonb -> 'operators') = 'array'
+ AND (
+ (f.fn ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/',
2)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/',
3)))
+ OR
+ (f.dvp ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'),
'/', 2)
+ AND d.name = split_part(ltrim(f.dvp, '/'),
'/', 3)))
+ OR
+ (f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/',
1)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/',
2)))
+ OR
+ (f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'),
'/', 1)
+ AND d.name = split_part(ltrim(f.dvp, '/'),
'/', 2)))
+ )
+ GROUP BY w.wid
+ ),
+ updated AS (
+ UPDATE workflow w
+ SET content = jsonb_set(w.content::jsonb, '{operators}', (
+ SELECT jsonb_agg(
+ jsonb_set(
+ jsonb_set(
+ op,
+ '{operatorProperties,fileName}',
+ CASE
+ WHEN f.fn ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u
ON d.owner_uid = u.uid
+ WHERE u.email =
split_part(ltrim(f.fn, '/'), '/', 2)
+ AND d.name =
split_part(ltrim(f.fn, '/'), '/', 3))
+ THEN to_jsonb(regexp_replace(f.fn, '^/datasets/',
'/dataset/'))
+ WHEN f.fn IS NOT NULL AND left(f.fn, 9) <>
'/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u
ON d.owner_uid = u.uid
+ WHERE u.email =
split_part(ltrim(f.fn, '/'), '/', 1)
+ AND d.name =
split_part(ltrim(f.fn, '/'), '/', 2))
+ THEN to_jsonb('/dataset/' || ltrim(f.fn, '/'))
+ ELSE COALESCE(op #>
'{operatorProperties,fileName}', 'null'::jsonb)
+ END,
+ false
+ ),
+ '{operatorProperties,datasetVersionPath}',
+ CASE
+ WHEN f.dvp ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp,
'/'), '/', 2)
+ AND d.name = split_part(ltrim(f.dvp,
'/'), '/', 3))
+ THEN to_jsonb(regexp_replace(f.dvp, '^/datasets/',
'/dataset/'))
+ WHEN f.dvp IS NOT NULL AND left(f.dvp, 9) <>
'/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp,
'/'), '/', 1)
+ AND d.name = split_part(ltrim(f.dvp,
'/'), '/', 2))
+ THEN to_jsonb('/dataset/' || ltrim(f.dvp, '/'))
+ ELSE COALESCE(op #>
'{operatorProperties,datasetVersionPath}', 'null'::jsonb)
+ END,
+ false
+ )
+ ORDER BY ord
+ )
+ FROM jsonb_array_elements(
+ CASE
+ WHEN jsonb_typeof(w.content::jsonb -> 'operators') =
'array'
+ THEN w.content::jsonb -> 'operators'
+ ELSE '[]'::jsonb
+ END
+ ) WITH ORDINALITY AS t(op, ord),
+ LATERAL (
+ SELECT op #>> '{operatorProperties,fileName}'
AS fn,
+ op #>> '{operatorProperties,datasetVersionPath}'
AS dvp
+ ) f
+ ))::text
+ FROM affected a
+ WHERE w.wid = a.wid
+ RETURNING 1
+ )
+ SELECT count(*) INTO wf_count FROM updated;
+
+ WITH affected AS (
+ SELECT wv.vid
+ FROM workflow_version wv,
+ jsonb_array_elements(
+ CASE
+ WHEN jsonb_typeof(wv.content::jsonb -> 'operators') =
'array'
+ THEN wv.content::jsonb -> 'operators'
+ ELSE '[]'::jsonb
+ END
+ ) AS op,
+ LATERAL (
+ SELECT op #>> '{operatorProperties,fileName}' AS fn,
+ op #>> '{operatorProperties,datasetVersionPath}' AS dvp
+ ) f
+ WHERE jsonb_typeof(wv.content::jsonb -> 'operators') = 'array'
+ AND (
+ (f.fn ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/',
2)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/',
3)))
+ OR
+ (f.dvp ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'),
'/', 2)
+ AND d.name = split_part(ltrim(f.dvp, '/'),
'/', 3)))
+ OR
+ (f.fn IS NOT NULL AND left(f.fn, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.fn, '/'), '/',
1)
+ AND d.name = split_part(ltrim(f.fn, '/'), '/',
2)))
+ OR
+ (f.dvp IS NOT NULL AND left(f.dvp, 9) <> '/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp, '/'),
'/', 1)
+ AND d.name = split_part(ltrim(f.dvp, '/'),
'/', 2)))
+ )
+ GROUP BY wv.vid
+ ),
+ updated AS (
+ UPDATE workflow_version wv
+ SET content = jsonb_set(wv.content::jsonb, '{operators}', (
+ SELECT jsonb_agg(
+ jsonb_set(
+ jsonb_set(
+ op,
+ '{operatorProperties,fileName}',
+ CASE
+ WHEN f.fn ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u
ON d.owner_uid = u.uid
+ WHERE u.email =
split_part(ltrim(f.fn, '/'), '/', 2)
+ AND d.name =
split_part(ltrim(f.fn, '/'), '/', 3))
+ THEN to_jsonb(regexp_replace(f.fn, '^/datasets/',
'/dataset/'))
+ WHEN f.fn IS NOT NULL AND left(f.fn, 9) <>
'/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u
ON d.owner_uid = u.uid
+ WHERE u.email =
split_part(ltrim(f.fn, '/'), '/', 1)
+ AND d.name =
split_part(ltrim(f.fn, '/'), '/', 2))
+ THEN to_jsonb('/dataset/' || ltrim(f.fn, '/'))
+ ELSE COALESCE(op #>
'{operatorProperties,fileName}', 'null'::jsonb)
+ END,
+ false
+ ),
+ '{operatorProperties,datasetVersionPath}',
+ CASE
+ WHEN f.dvp ~ '^/datasets/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp,
'/'), '/', 2)
+ AND d.name = split_part(ltrim(f.dvp,
'/'), '/', 3))
+ THEN to_jsonb(regexp_replace(f.dvp, '^/datasets/',
'/dataset/'))
+ WHEN f.dvp IS NOT NULL AND left(f.dvp, 9) <>
'/dataset/'
+ AND EXISTS (SELECT 1 FROM dataset d JOIN "user" u ON
d.owner_uid = u.uid
+ WHERE u.email = split_part(ltrim(f.dvp,
'/'), '/', 1)
+ AND d.name = split_part(ltrim(f.dvp,
'/'), '/', 2))
+ THEN to_jsonb('/dataset/' || ltrim(f.dvp, '/'))
+ ELSE COALESCE(op #>
'{operatorProperties,datasetVersionPath}', 'null'::jsonb)
+ END,
+ false
+ )
+ ORDER BY ord
+ )
+ FROM jsonb_array_elements(
+ CASE
+ WHEN jsonb_typeof(wv.content::jsonb -> 'operators') =
'array'
+ THEN wv.content::jsonb -> 'operators'
+ ELSE '[]'::jsonb
+ END
+ ) WITH ORDINALITY AS t(op, ord),
+ LATERAL (
+ SELECT op #>> '{operatorProperties,fileName}'
AS fn,
+ op #>> '{operatorProperties,datasetVersionPath}'
AS dvp
+ ) f
+ ))::text
+ FROM affected a
+ WHERE wv.vid = a.vid
+ RETURNING 1
+ )
+ SELECT count(*) INTO wv_count FROM updated;
+
+ RAISE NOTICE 'Normalized the resource-type path prefix in % workflow and %
workflow_version row(s).', wf_count, wv_count;
+END $$;
+
+COMMIT;
