merlau opened a new pull request, #12258:
URL: https://github.com/apache/seatunnel/pull/12258
### Purpose of this pull request
This pull request introduces native **MariaDB** support to Apache SeaTunnel,
adding both a native **JDBC dialect & catalog** and a dedicated **MariaDB CDC
Connector**:
1. **MariaDB JDBC Extension (`seatunnel-connectors-v2/connector-jdbc`)**:
- Dedicated `MariaDBDialect` and `MariaDBCatalog` with comprehensive data
type mapping (45+ SQL types, `TINYINT(1)` boolean handling, unsigned integers,
bit types, spatial geometry, and NTZ/LTZ timestamp handling).
- Native upsert SQL generation (`INSERT ... ON DUPLICATE KEY UPDATE`).
2. **Dedicated MariaDB CDC Connector
(`seatunnel-connectors-v2/connector-cdc/connector-cdc-mariadb`)**:
- Independent CDC module supporting incremental snapshot reading, binlog
streaming, watermark alignment, dynamic in-flight schema evolution, and
checkpoint restore.
- Native MariaDB 3-part integer GTID sequence parsing
(`<domain_id>-<server_id>-<sequence_no>`) and state restoration.
---
### Root Causes & Historical Context
This implementation directly resolves recurring community issues where users
attempted to run MariaDB workloads against SeaTunnel:
1. **Unresolved Bug [#9148](https://github.com/apache/seatunnel/issues/9148)
(`MySQL CDC Unsupported MySQL version: 11.4.2-MariaDB-log`)**:
- Users attempting to use `MySQL-CDC` against MariaDB 11.x encountered
fatal startup crashes:
```
java.lang.UnsupportedOperationException: Unsupported MySQL version:
11.4.2-MariaDB-log
at
org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.mysql.MySqlVersion.parse
```
- *Status*: Automatically closed by stale-bot with zero fix, leaving
MariaDB CDC completely unsupported.
2. **Unresolved Bug
[#10030](https://github.com/apache/seatunnel/issues/10030) (`Unsupported MySQL
version: 11.4.7-4-MariaDB-enterprise-log`)**:
- Enterprise users attempting to connect to MariaDB with standard MySQL
JDBC configurations crashed during catalog resolution.
3. **Limitations of Prior Stopgap PR
[#11604](https://github.com/apache/seatunnel/pull/11604)**:
- PR #11604 attempted to mitigate #10030 via documentation by warning
users not to use MySQL JDBC drivers, directing them to
`org.mariadb.jdbc.Driver`.
- *Why this was insufficient*: Instructing users to use
`org.mariadb.jdbc.Driver` forced SeaTunnel into the fallback `GenericDialect`.
This resulted in:
- No catalog/schema metadata introspection (`GenericCatalog`).
- Inability to perform high-performance primary-key upserts (`INSERT
... ON DUPLICATE KEY UPDATE`).
- Loss of 30+ MariaDB-specific type conversions.
- **Zero support for CDC** (PR #11604 only applied to batch JDBC).
4. **Fulfills Tracking Issue
[#3018](https://github.com/apache/seatunnel/issues/3018)**:
- Fulfills item #141 on the SeaTunnel Connector-V2 master tracking list.
---
### Architectural Rationale: Why a Dedicated Connector is Required
Rather than attempting to retrofit MariaDB workarounds into the existing
MySQL modules, isolating MariaDB into dedicated modules was strictly necessary
due to fundamental engine divergence:
* **Incompatible GTID Formats**: MySQL uses 128-bit UUID interval sets
(`uuid:1-100`). MariaDB uses integer sequence triples
(`domain_id-server_id-sequence`). GTID purge comparisons, offset algebra, and
restore mechanics cannot parse across engine formats.
* **Binlog Event Protocol Divergence**: MariaDB binlogs introduce specific
event types (`MARIADB_GTID_EVENT` [162], `MARIADB_GTID_LIST_EVENT` [163],
`ANNOTATE_ROWS_EVENT` [160]) and require specific slave capability handshakes
(`SET @mariadb_slave_capability = 4`).
* **Driver Streaming Differences**: MySQL Connector/J requires
`statement.setFetchSize(Integer.MIN_VALUE)` to avoid client memory exhaustion.
The MariaDB JDBC driver respects positive user-defined fetch sizes.
* **Future-Proof Debezium Roadmaps**: Forking the connector architecture
isolates MariaDB's future Debezium parser and binlog client upgrades,
eliminating any risk of regressions to the production-critical
`connector-cdc-mysql`.
---
### Does this PR introduce _any_ user-facing change?
Yes:
* Users can now configure `source { MariaDB-CDC { ... } }` for streaming
binlog ingestion from MariaDB 10.x and 11.x servers.
* Users can now use `source { Jdbc { ... } }` and `sink { Jdbc { ... } }`
with `url = "jdbc:mariadb://..."` and `driver = "org.mariadb.jdbc.Driver"`,
enjoying full native dialect features (catalog discovery, upsert, schema
evolution).
---
### How was this patch tested?
Full 1:1 test parity was ported and validated against the MySQL test suites:
#### 1. Unit Tests (100% Parity)
- Added `MariaDbBinlogOffsetTest`, `MariaDbGtidSetTest`,
`MariaDbGtidUtilsTest`, `MariaDbConnectionConfigurationTest`,
`MariaDbDatabaseFilterTest`, `MariaDbDateTypeConvertTest`,
`MariaDbDialectFactoryTest`, `MariaDbDialectTest`,
`MariaDbGeometryTypeConvertTest`, `MariaDbNumericTypeConvertTest`,
`MariaDbSchemaTest`, `MariaDbTableFilterTest`, `MariaDbTimeTypeConvertTest`.
#### 2. E2E Integration Tests (100% Parity)
- **JDBC E2E**:
- `JdbcMariaDBIT`: Expanded from 8 types to 45+ data types, testing
`TINYINT(1)` boolean handling, unsigned integers, and spatial data.
- `JdbcMariaDBMultipleTablesIT`: Multi-table routing and schema mismatch
recovery.
- `JdbcMariaDBTimestampIT`: Session-aware LTZ vs NTZ timestamp mappings.
- **CDC E2E**:
- `MariaDbCDCSingleTableIT`, `MariaDbCDCMultiTableIT`,
`MariaDbCDCCheckpointIT`, `MariaDbCDCStopIT`, `MariaDbCDCBinlogDeleteIT`.
- `MariaDbCDCWithSchemaChangeIT` & `MariaDbCDCWithFlinkSchemaChangeIT`.
- *Note on deactivated test*:
`testMysqlCdcWithSchemaEvolutionCaseExactlyOnce` is marked `@Disabled` citing
upstream MariaDB engine constraint
[MDEV-14732](https://jira.mariadb.org/browse/MDEV-14732) (`XAER_PROTO` error
when executing `XA START` on connections with transactions in `PREPARED` state,
preventing single-connection 2PC XA sinks). All non-XA in-flight schema
evolution DDL tests pass 100%.
#### Verification Commands Executed:
- `./mvnw spotless:apply` (Enforced Google Java Format across all modified
and new files)
- `./mvnw clean test-compile -DskipTests` (`BUILD SUCCESS` across all
affected modules)
---
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. (N/A - Purely additive)
* [x] If you are contributing the connector code, please check that the
following files are updated:
1. [x] Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. [x] Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. [x] Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
(covered by existing `seatunnel-connectors-v2/connector-cdc/**`)
4. [x] Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. [x] Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
