eye-gu opened a new issue, #18600: URL: https://github.com/apache/dolphinscheduler/issues/18600
### Search before asking000a000a- [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.000a000a> Functional requirement origin: #18377 "[Improvement][datasource] Enable users to customize datasources". The requirement is taken from that issue; the design below is derived independently from the current codebase.000a> DSIP id note: at the time of submission the highest DSIP id in use is 109, so 110 is used here.000a000a### Motivation000a000aDolphinScheduler's datasource layer is *pluginized in name but not in fact*. Although datasource support is implemented as SPI plugins (`DataSourceProcessor` / `DataSourceChannelFactory`, 28 built-in plugins), the plugin contract is hard-wired to the compiled-in `DbType` enum (`dolphinscheduler-spi/.../spi/enums/DbType.java`):000a000a- `DataSourceProcessor#getDbType()` returns `DbType`, a compile-time enum shipped inside `dolphinscheduler-spi`. A plugin compiled outside the DolphinSch eduler tree can only return an existing constant — it can **override** a built-in type, but can never **introduce a new one**.000a- `BaseDataSourceParamDTO#getType()` returns `DbType` as well, so the parameter DTOs have the same constraint.000a- The `type` column of `t_ds_datasource` stores the enum's integer code (`tinyint`), which by definition cannot represent an unknown type.000a- Both frontend surfaces — the "create datasource" dialog (`dolphinscheduler-ui/src/views/datasource/list/use-form.ts`) and the task-node datasource selector (`.../node/fields/use-datasource.ts`) — enumerate types from hardcoded arrays; a type installed on the backend is invisible to users.000a000aAs a result, anyone whose database is not among the built-in types (proprietary engines, internal database proxies, newly released databases) has no supported extension path short of forking the project, editing the enum, the DAO layer and the frontend, and rebuilding everything.000a000a**Goal (functional requirement)** — a deployment administrator can install an externally compiled datasource plugin package into an existing DolphinScheduler installation, **without modifying or recompiling any DolphinScheduler code**, and the new datasource type becomes usable end-to-end:000a000a- **FR1 — Registerable**: the plugin can declare a datasource type identity that is unique and resolvable across api-server, master and worker.000a- **FR2 — Visible**: the new type shows up in the UI, both in the datasource creation entry and in the datasource selector of task nodes that consume datasources.000a- **FR3 — Full lifecycle**: for the new type, users can create a datasource instance, test connectivity, update it, list/browse it, grant it to other users, and (where the driver supports it) browse metadata (databases / tables / columns).000a- **FR4 — Executable**: tasks that reference datasource instances of the new type run correctly (minimum scope: SQL and Procedure tasks).000a- **FR5 � � Install-only rollout**: installation = drop the plugin package into the plugins directory and restart the servers; no source change, no rebuild of DolphinScheduler itself.000a000a**Non-goals (proposed for v1):**000a000a- No plugin hot-loading / installing plugins through the web UI. Plugin installation stays an administrator/deployment action (see Security alignment below).000a- No plugin-provided custom UI forms. Custom types get a generic JDBC-style form (see open decision D3).000a- No DataX / Sqoop support for custom types in v1 (their type→generator mappings are hardcoded; see Aspect 5).000a000a### Design Detail000a000aThe requirement decomposes into the following aspects. Each aspect states the current code fact that blocks the requirement, and what has to be true after the change.000a000a#### Aspect 0 — Current state summary (why each aspect exists)000a000a| # | Aspect | Blocking fact today |000a|---|--------|---------------------|000a| 1 | Type identity in the SPI contr act | `getDbType()` / `getType()` return compiled-in `DbType`; registry keyed by enum name |000a| 2 | Type discovery | No API exposes registered types; frontend hardcodes arrays |000a| 3 | Persistence | `t_ds_datasource.type` is `tinyint` storing the enum code; entity field is `DbType` |000a| 4 | api-server handling | Routing and dialect behavior switch on `DbType` (`DbType.ofName`, `valueOf`, switch statements) |000a| 5 | Task execution path | `SqlTask`/`ProcedureTask`/`DataxTask` do `DbType.valueOf(...)`; context objects carry `DbType` |000a| 6 | Frontend | Type lists, per-type forms and default ports are hardcoded in `use-form.ts` / `use-datasource.ts` / `types.ts` |000a| 7 | Packaging & driver delivery | Works today (shaded jars on the classpath via `plugins/datasource-plugins`), needs conventions for external plugins |000a| 8 | Security alignment | Plugin install must stay inside the existing trusted-deployment boundary |000a000a#### Aspect 1 — String-based type identity in t he plugin contract000a000aThe SPI must stop using a compile-time enum as the *identity* of a datasource type.000a000a- A datasource type is identified by a **unique, case-normalized string name** (e.g. `"MYSQL"`, `"DORIS"`, `"MY_INTERNAL_DB"`), declared by the plugin and used consistently by `DataSourceProcessor`, `DataSourceChannelFactory` and the parameter DTOs.000a- `DbType` degrades from "the closed set of all possible types" to "named constants for the built-in types" (see open decision D1). All identity lookups (`DataSourcePluginManager` maps, `DataSourceUtils` routing) become string-keyed; today they already key maps by `dbType.getName()`, so the direction of change is natural.000a- Conflict rules: two installed plugins declaring the same type name must fail fast at startup with a clear message. The existing `PrioritySPI` mechanism (same name resolved by priority, ambiguity raises) already provides the semantics for *deliberately overriding* a built-in type; new names simply must not collide.000a- `BaseDataSourceParamDTO` carries the type as a string field, replacing `getType(): DbType`, so JSON round-tripping works for types unknown to the core.000a000a#### Aspect 2 — Type registry and a discovery API000a000aThe backend must be able to enumerate what is installed, and the frontend must learn the type list from the backend instead of hardcoding it.000a000a- `DataSourcePluginManager` (or a thin façade over it) exposes the registered types together with minimal metadata a UI needs: unique name, display label, and simple capability flags (e.g. `supportsConnectivityTest`, `supportsMetadata`, `jdbcCompatible`).000a- A new REST endpoint, e.g. `GET /datasources/types`, returns that list. This is the single contract between backend plugins and both frontend surfaces (Aspect 6).000a- Create/update APIs validate the submitted type against the registry and reject unknown types with an explicit error message ("datasource type X is not installed") instead of fail ing deep inside enum parsing (`DbType.ofName` currently throws `NoSuchElementException` for unknown names).000a- Capability flags replace enum switches where behavior varies per type (see Aspect 4), so a custom type gets sane defaults without core code knowing it.000a000a#### Aspect 3 — Persistence of the datasource type000a000a`t_ds_datasource.type` must be able to store identities that do not exist at compile time.000a000a- Change the column from `tinyint` (integer enum code) to `varchar` storing the type name string, in all three fresh-install schemas (MySQL, PostgreSQL, H2: `dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_{mysql,postgresql,h2}.sql`) and keep the `UNIQUE (name, type)` constraint semantics.000a- The `DataSource` entity (`dolphinscheduler-dao/.../entity/DataSource.java`) changes its `type` field from `DbType` to `String`; mappers and repositories follow.000a- Upgrade scripts under `dolphinscheduler-dao/src/main/resources/sql/upgrade/<version>_schema/ {mysql,postgresql}/` convert existing integer codes to name strings (a code→name mapping over the current 29 enum values; see Migration Plan). The stale column comment ("0:mysql,1:postgresql,...") is corrected at the same time.000a- Graceful degradation: a datasource row whose type plugin is *not installed* must still be listable (the UI shows the raw type name and marks it unavailable); connect/run attempts return a clear "plugin not found" error rather than an NPE or enum parse failure. The message `"datasource plugin '%s' is not found"` already exists in `DataSourceClientProvider` and should become the uniform behavior.000a000a#### Aspect 4 — api-server adaptations000a000a- **Routing**: `DataSourceUtils.buildDatasourceParam` routes the submitted JSON to the right processor by string type name (today: `DbType.ofName(type.toUpperCase())`).000a- **API surface**: `DataSourceController#queryDataSourceList` binds `type` as `DbType` (Spring enum-name conversion). Binding it as `Stri ng` keeps the wire format ("MYSQL") unchanged — the frontend already sends enum *names*, so this is not a breaking API change.000a- **Dialect behavior**: type-specific switches such as `DataSourceServiceImpl#getDbSchemaPattern` (HIVE/ORACLE/SQLSERVER/CLICKHOUSE/DATABEND/PRESTO) and the `AbstractDataSourceProcessor` REDSHIFT special case must become capability/metadata-driven (Aspect 2) with safe defaults, so unknown types get working default behavior for metadata browsing instead of being silently wrong.000a- Unchanged by design: password encryption (`PasswordUtils`), Kerberos handling, and datasource authorization are already type-agnostic (permissions bind to datasource ids).000a000a#### Aspect 5 — Task execution path (master + worker)000a000aGood news first: task definitions already store the datasource type as a **string** in `task_params` JSON (`SqlParameters.type`, `DataxParameters.dsType/dtType` are `String`), and datasource instances are referenced by integer id. The wor k is removing enum round-trips:000a000a- **master**: `TaskExecutionContextFactory#assembleDataSourceParameters` puts a `DbType` into the execution context; context objects (`DataxTaskExecutionContext`, `SqoopTaskExecutionContext`, ...) carry `DbType` fields — all become strings.000a- **worker task plugins**: `SqlTask` (`DbType.valueOf(sqlParameters.getType())`), `ProcedureTask`, and the DataX/Sqoop call sites resolve the processor by string type name via the plugin manager. Unknown type → clear task failure message naming the missing plugin.000a- **DataX / Sqoop scope**: `DataxUtils#getReaderPluginName/getWriterPluginName` and the Sqoop source/target generators hardcode type→external-tool mappings that only make sense for built-in types. In v1 these tasks declare themselves limited to built-in types (the frontend already supports a per-task-type whitelist via `supportedDatasourceType`), so custom types are selectable only for SQL / Procedure tasks. Lifting this restriction is a follow-up, not part of this DSIP.000a000a#### Aspect 6 — Frontend: dynamic type list + generic fallback form000a000a- **Type list source**: both hardcoded arrays — `datasourceTypeList` in `views/datasource/list/use-form.ts` (28 entries with per-type labels/default ports) and `datasourceTypes` in `node/fields/use-datasource.ts` (an inconsistent subset, some entries disabled) — are replaced by data from `GET /datasources/types` (Aspect 2). Per-task-type whitelisting (`supportedDatasourceType`) continues to filter the dynamic list.000a- **Forms for custom types**: built-in types keep their dedicated forms (`changeType` in `use-form.ts`). A type the frontend has no dedicated form for renders a **generic JDBC form**: host, port, user, password, database, extra JDBC parameters (`other`), optional principal — the fields the common `BaseConnectionParam` already supports. This is sufficient for the overwhelming majority of JDBC drivers; the display label and default port come from the type metadata.000a- **Type definitions**: the `IDataBase` string-literal union in `service/modules/data-source/types.ts` becomes a plain `string` validated at runtime against the types API.000a- **i18n / labels**: labels for built-in types keep current texts; custom types display the label from the API metadata (no per-type i18n keys needed for custom types).000a000a#### Aspect 7 — Packaging, delivery and driver management000a000aThe deployment path already works mechanically (shaded plugin jars in `plugins/datasource-plugins/` are appended to the classpath by the server start scripts, discovered via `ServiceLoader`); what this DSIP adds is convention and guarantees:000a000a- A custom plugin is a single shaded jar bundling: the `DataSourceProcessor` + `DataSourceChannelFactory` implementations, `META-INF/services` registrations (e.g. via `@AutoService`), and **the JDBC driver it needs** (same pattern as built-in plugin poms, documented in `docs/docs/en/contribute/backend/spi/d atasource.md`).000a- The plugin must be installed on **every** api-server, master and worker node that will touch the type (api for UI/lifecycle, master+worker for execution); the developer/operations documentation states this explicitly.000a- Driver version conflicts between plugins remain the known limitation of the flat-classpath model; document it as a constraint (one driver major version per classpath), same as today's behavior for built-in plugins. Classloader isolation is explicitly out of scope.000a- **Reference plugin**: `DbType` contains `H2(9)` but no H2 plugin module exists. Implementing the H2 datasource plugin as the first "externally developed" plugin gives the proposal a living example that is testable in CI with zero external services (in-memory JDBC), and doubles as the test vehicle for the whole path (registry → UI → SQL task execution).000a000a#### Aspect 8 — Security alignment000a000aThis feature must stay within the trust boundaries documented in the proj ect's [security model](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/security-model.md):000a000a- Installing a plugin jar is an **administrator/deployment action** (filesystem access to the server), identical in trust level to installing task plugins or storage plugins today. Ordinary platform users cannot install plugins; they can only *use* an installed type within existing datasource authorization. The feature adds no new privilege escalation path.000a- Parameters a user enters into a custom datasource (including JDBC `other` parameters) fall squarely under the model's existing "user-configured plugin parameters are trusted user behavior" clause.000a- Registry conflict handling (Aspect 1) must fail fast and loudly, so a malicious/accidental same-name plugin cannot silently override a built-in type without priority rules catching it.000a000a#### Open design decisions (for the mail-thread discussion)000a000a- **D1 — Fate of `DbType`**: (a) *reco mmended*: keep the enum as a deprecated constants holder for migration/compat, remove its integer code from persistence, all runtime paths use strings — mirrors how task plugins use `String taskType`; (b) keep full enum semantics for built-ins and add a parallel string path — rejected as dual-maintenance.000a- **D2 — Storage**: (a) *recommended*: in-place `type` column conversion `tinyint → varchar(64)` with data migration; (b) side table `type_code → type_name` mapping — rejected (extra join, worse ergonomics, still can't express custom types in the main column).000a- **D3 — UI for custom types**: (a) *recommended for v1*: generic JDBC fallback form; (b) plugin-declared form schema returned by the types API (field list rendered dynamically) — more powerful, deferred to a follow-up DSIP.000a- **D4 — Task coverage in v1**: SQL + Procedure only; DataX/Sqoop restricted to built-ins via existing whitelist mechanism.000a- **D5 — Types API shape**: `GET /datasources/ty pes` returning `[{ type, label, defaultPort?, capabilities... }]` — exact field list to be settled in review.000a000a#### Suggested sub-task breakdown000a000aConsistent with the DSIP process, this is large enough to be split:000a000a1. SPI contract: string type identity in `DataSourceProcessor` / DTOs / `DataSourcePluginManager` (+ conflict rules).000a2. DAO: `DataSource` entity, mappers, fresh-install schemas for MySQL/PG/H2.000a3. Upgrade scripts (code→name migration, MySQL + PostgreSQL) and tools support.000a4. api-server: string routing, `GET /datasources/types`, unknown-type validation, capability-driven dialect defaults.000a5. master/worker: execution context and SQL/Procedure/DataX/Sqoop call-site adaptation.000a6. Frontend: dynamic type lists, generic fallback form, whitelist filtering.000a7. Reference H2 plugin + developer guide (`contribute/backend/spi/datasource.md` en/zh) + ops doc (install on all nodes, driver conflicts).000a8. Tests & e2e (see Test Plan).000a000a## # Compatibility, Deprecation, and Migration Plan000a000a#### Data migration000a000a- `t_ds_datasource.type`: `tinyint`/`int` → `varchar`, one upgrade script per dialect (MySQL, PostgreSQL) in the upgrade directory of the release carrying this change, following the existing `t_ds_version`-driven `UpgradeDao` flow. The DML maps each of the 29 live integer codes to its name constant (0→`MYSQL`, 1→`POSTGRESQL`, ..., 28→`DOLPHINDB`). Rows referencing codes with no enum value (corrupted data) are reported rather than silently converted.000a- `task_params` in `t_ds_task_definition` / `t_ds_task_instance` already stores type names as strings and is **not touched**.000a- `connection_params` JSON is plugin-owned and unchanged.000a000a#### API compatibility000a000a- Wire formats that already use enum *names* (e.g. `GET /datasources/list?type=MYSQL`) remain byte-compatible; only server-side binding types change.000a- Java-level breakage is confined to internal modules (`dolphinscheduler -spi`, `dolphinscheduler-datasource-api`, `dolphinscheduler-dao`) — this is a dev-branch feature release, but downstream plugin authors are affected and must be informed via migration notes: existing third-party plugins compiled against `getDbType()` need a trivial recompile against the new contract.000a000a#### Deployment / rolling upgrade000a000a- All servers (api, master, worker) must be upgraded together in a cluster that uses custom types; a node missing a plugin degrades gracefully (Aspect 3) — datasource lists still render, connect/run fail with an explicit "plugin not installed" error.000a- Built-in-only deployments keep working identically after migration; no user action required.000a000a#### Deprecations000a000a- `DbType` integer codes and `@EnumValue` persistence deprecated; enum retained (D1) as constants for one removal cycle.000a000a### Test Plan000a000a#### Unit tests000a000a- `DataSourcePluginManager`: string-keyed registration, duplicate-name fail-fast, priority override, missing-plugin lookup error.000a- `DataSourceUtils` / processor routing by string type; unknown type rejected with explicit error.000a- Migration converter: every one of the 29 codes maps to the expected name; unknown code surfaces an error.000a- Controller binding: `type` accepts any registered name; unregistered names produce a 4xx with a clear message.000a000a#### Integration / DAO tests000a000a- Upgrade test from the previous schema with rows covering (at least) MYSQL, POSTGRESQL, HIVE, ORACLE, SSH, DOLPHINDB — assert post-migration names and uniqueness.000a- api-server: create / update / connect-test / list / authorize a datasource of a **custom test processor**; datasource rows with an uninstalled type list correctly and fail connect with the documented error.000a000a#### Worker / execution tests000a000a- `SqlTask` and `ProcedureTask` against the **reference H2 plugin** (in-memory): full create → bind → execute path with a custom type.000a- Failure-path test: task referencing an uninstalled type fails with the "plugin not found" message.000a000a#### Frontend / e2e000a000a- `dolphinscheduler-e2e`: create an H2 datasource through the generic fallback form, bind it to an SQL task node, run the workflow successfully.000a- Regression: existing datasource e2e suites (mysql, postgresql, hive, clickhouse, sqlserver, dolphindb docker fixtures) pass unchanged.000a000a#### Regression scope000a000a- All 28 built-in plugins' existing processor/channel unit tests pass unmodified beyond the contract recompile.000a- Built-in UI forms render exactly as before (no visual/behavioral change for known types).000a000a### Code of Conduct000a000a- [x] I agree to follow this project's [Code of Conduct](https://apache.org/foundation/policies/conduct)000a000a---000a000a**Next steps (per the DSIP process)**: a `[DISCUSS][DSIP-110]` mail will be sent to `[email protected]` linking this issue; sub-issues will be created from the *Suggested sub-task brea kdown* once the design direction is agreed.000a -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
