morningman opened a new pull request, #68037:
URL: https://github.com/apache/doris/pull/68037
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #67916 (the same SQL Server type-mapping fix had to be made
twice, once per copy)
Problem Summary:
fe-core still carried a full copy of the JDBC data-source implementation
next to the
fe-connector-jdbc plugin: 13 dialect clients with their own type mappings
(`datasource/jdbc/client/*`, ~3000 lines), the driver-jar policy and url
normalization
inside `JdbcResource`, and an engine hook (`ConnectorValidationContext
.validateAndResolveDriverPath/computeDriverChecksum`) that lent that policy
to plugins.
The only production user of the fe-core clients was the streaming/CDC
framework, and the
two copies had drifted: three streaming-driven fixes (#64740 composite PK
order, #63402
PostgreSQL neighbour-table leak, #61433 PostgreSQL array types) never
reached the
connector, and #67916 had to patch both.
After this PR fe-core has no JDBC implementation; what remains is generic
SPI use plus
two `@Deprecated` persistence shells (`JdbcResource`, `JdbcTable`).
1. **SPI (major 8.0)** — `ConnectorTableMetadataOps.getPrimaryKeys`,
`ConnectorPassthroughSqlOps.executeQuery(session, sql, params)` +
`ConnectorQueryResult`, `ConnectorContext.fetchPluginFile` (a cloud
deployment's
object-store copy of a missing plugin file); `ConnectorValidationContext`
loses its two
driver methods. `DriverUrlPolicy` (fe-connector-spi) is the one
driver-jar policy —
url grammar, `jdbc_driver_url_white_list`, structural
`jdbc_driver_secure_path`
matching, bare-name resolution across the current and pre-2.1 drivers
directories,
cloud fetch, checksum — applied by the jdbc, iceberg and paimon
connectors.
2. **fe-connector-jdbc** — implements the two new methods; applies
`DriverUrlPolicy`
at CREATE and at load; ports the three fixes above (KEY_SEQ ordering in
the base
client, PostgreSQL exact schema/table filter, `bpchar[]` elements as
STRING).
3. **Streaming** — `StreamingSourceClient` (fe-core) opens a temporary
connector through
`ConnectorFactory` with the job's source properties (same names as a JDBC
catalog),
goes through the `PluginDrivenMetadata` funnel and
`ConnectorColumnConverter`, and
runs the framework's probe queries with bound parameters. The four call
sites keep
their logic. The fe-core clients and their tests are deleted.
4. **`JdbcResource`** — `@Deprecated` shell: Gson tag, `configs`, property
list and
defaults, `SHOW RESOURCES` rows unchanged; validation and the driver
checksum come
from the jdbc plugin. Dead `JdbcTransactionManager` and `*_JDBC_*`
plan/rule types
removed; the `"jdbc"` gate around the schema-load debug point removed.
5. **Gate** — `build-support/check-fe-core-jdbc-free.sh` (validate phase,
self-tested)
forbids `org.apache.doris.datasource.jdbc.*`, `com.zaxxer.hikari.*` and
`java.sql`
connection/statement imports in fe-core main sources (`httpv2/`, JDBC to
the FE
itself, exempt). fe-core drops its HikariCP dependency.
### Release note
- A JDBC catalog now applies `jdbc_driver_secure_path` /
`jdbc_driver_url_white_list`
when it loads its driver, not only at CREATE; a catalog whose driver sits
outside a
since-tightened allow-list fails to connect instead of loading the jar.
- `CREATE RESOURCE ... type=jdbc` is deprecated in favour of
`CREATE CATALOG ... "type"="jdbc"`; it keeps working and `SHOW RESOURCES`
shows
`jdbc_url` as written instead of a normalized form.
- Streaming jobs need the jdbc connector plugin (bundled by default). The
FE-side
metadata connection is built by the connector (its url normalization forces
`tinyInt1isBit=false` / `yearIsDateType=false`); the persisted job url and
the BE CDC
reader are unchanged. PostgreSQL array columns whose element type the old
client
rejected now map to `ARRAY<STRING>` (as the CDC client already did), empty
tables no
longer fail array-dimension detection, and `char(n)[]` maps to
`ARRAY<STRING>` in
JDBC catalogs again.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Unit tests run locally: fe-connector-spi (164, incl. 23 new
`DriverUrlPolicyTest`),
fe-connector-jdbc (240; new PostgreSQL/OceanBase/base-client tests,
pre-create validation
and load-time policy tests), fe-connector-paimon (574),
fe-connector-iceberg (1431, one
failure pre-existing on master: `IcebergWritePlanProviderTest
.planMergePreservesExplicitlyEmptyReadAcrossConcurrentFirstAppend`), the
full fe-core
suite (10993 tests; the 12 failures in 10 classes are unrelated to this
change and fail
identically on the base commit on the same machine),
`build-support/tests/run.sh`.
Regression suites to watch in CI: `external_table_p0/jdbc`,
`job_p0/streaming_job/cdc`,
`query_p0/show/test_nereids_show_resources`,
`manager/test_manager_interface_3`,
`test_cloud_plugin_auto_download` (cloud).
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change --> See the release note.
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. --> Deprecate `CREATE
RESOURCE type=jdbc`;
note that the driver allow-lists are enforced when a JDBC catalog
loads its driver.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
