MartijnVisser commented on code in PR #231: URL: https://github.com/apache/flink-connector-jdbc/pull/231#discussion_r4071529202
########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). Review Comment: `push_pr.yml` is `jdk: [ '17' ]`, so PR CI is 17 only, and #228 keeps it that way. Weekly JDK 8 and 11 is the `v3.4` row alone, `v4.1` is 17. And `maven.compiler.source` evaluates to 1.8, not 11, since nothing overrides `target.java.version` from the parent. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. Review Comment: FLINK-40724 landed after you branched and pins the wrapper to 3.9.16. On a rebase this line inverts, and every `mvn` in the file, line 202 included, should become `./mvnw`, the way flink and the Kafka connector do it. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) Review Comment: There is no `*E2eTest` class any more, `JdbcExactlyOnceSinkE2eTest` went with FLINK-36659. Core ITCases also run on Derby without Docker, which line 46 says. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) +- Unix-like environment (Linux, macOS, WSL) +- The connector builds against `flink.version` in the root `pom.xml`. A change must also work against every Flink version in `.github/workflows/push_pr.yml` and the `main` row of `.github/workflows/weekly.yml`. + +## Commands + +### Build + +- Build without tests: `mvn clean package -DskipTests` +- Full build with tests: `mvn clean verify` +- Build against another Flink version (what CI does): `mvn clean install -DskipTests -Dflink.version=<version>` +- Single module: `mvn clean install -DskipTests -pl flink-connector-jdbc-core` +- A database module depends on `flink-connector-jdbc-core`; build it with its dependency using `-pl flink-connector-jdbc-mysql -am`. +- Dependency convergence (CI runs this on PRs): `mvn clean install -DskipTests -Pcheck-convergence -Dflink.convergence.phase=install` Review Comment: Could you add the `-Dfast` line the Kafka file has? Here it skips rat, checkstyle, spotless, enforcer, javadoc and japicmp, unlike Kafka where japicmp still runs. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) +- Unix-like environment (Linux, macOS, WSL) +- The connector builds against `flink.version` in the root `pom.xml`. A change must also work against every Flink version in `.github/workflows/push_pr.yml` and the `main` row of `.github/workflows/weekly.yml`. + +## Commands + +### Build + +- Build without tests: `mvn clean package -DskipTests` +- Full build with tests: `mvn clean verify` +- Build against another Flink version (what CI does): `mvn clean install -DskipTests -Dflink.version=<version>` +- Single module: `mvn clean install -DskipTests -pl flink-connector-jdbc-core` +- A database module depends on `flink-connector-jdbc-core`; build it with its dependency using `-pl flink-connector-jdbc-mysql -am`. +- Dependency convergence (CI runs this on PRs): `mvn clean install -DskipTests -Pcheck-convergence -Dflink.convergence.phase=install` + +### Testing + +- `*Test` classes are unit tests; `*ITCase` classes are integration tests that start a real database via Testcontainers (Docker required). `flink-connector-jdbc-core` also runs against in-memory Derby and H2, so its unit tests need no Docker. +- Single unit test class: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest` +- Single test method: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest#testX` +- Single ITCase in a database module: `mvn test -pl flink-connector-jdbc-postgres -am -Dtest=PostgresDialectITCase` +- ArchUnit rules live in `flink-connector-jdbc-architecture`; violation stores are frozen (see Testing Standards). +- The standalone `flink-connector-jdbc-backward-compatibility` module is not part of the root reactor; CI builds it separately (`.github/workflows/backwards_compatibility.yml`) to check that current code reads savepoints written by older connector versions. +- CI for PRs and the weekly build is defined by `apache/flink-connector-shared-utils` (`.github/workflows/ci.yml@ci_utils`); its Maven command line, including the license check, is the reference when a local run differs from CI. + +### Code Quality + +- Format code: `mvn spotless:apply` (the `java21` profile sets `spotless.skip=true`, so run it on JDK 11 or 17; google-java-format does not run on JDK 21) +- Check formatting: `mvn spotless:check` +- Checkstyle: `mvn checkstyle:check` (config in `tools/maven/checkstyle.xml`, suppressions in `tools/maven/suppressions.xml`) +- License headers: `mvn apache-rat:check` +- japicmp compares against `japicmp.referenceVersion` from the root `pom.xml` and only checks stable API. Review Comment: It checks `@Public` only, the effective pom excludes `@PublicEvolving`, `@Experimental` and `@Internal`. Worth being precise since most of this connector is `@PublicEvolving`. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) +- Unix-like environment (Linux, macOS, WSL) +- The connector builds against `flink.version` in the root `pom.xml`. A change must also work against every Flink version in `.github/workflows/push_pr.yml` and the `main` row of `.github/workflows/weekly.yml`. + +## Commands + +### Build + +- Build without tests: `mvn clean package -DskipTests` +- Full build with tests: `mvn clean verify` +- Build against another Flink version (what CI does): `mvn clean install -DskipTests -Dflink.version=<version>` +- Single module: `mvn clean install -DskipTests -pl flink-connector-jdbc-core` +- A database module depends on `flink-connector-jdbc-core`; build it with its dependency using `-pl flink-connector-jdbc-mysql -am`. +- Dependency convergence (CI runs this on PRs): `mvn clean install -DskipTests -Pcheck-convergence -Dflink.convergence.phase=install` + +### Testing + +- `*Test` classes are unit tests; `*ITCase` classes are integration tests that start a real database via Testcontainers (Docker required). `flink-connector-jdbc-core` also runs against in-memory Derby and H2, so its unit tests need no Docker. +- Single unit test class: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest` +- Single test method: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest#testX` +- Single ITCase in a database module: `mvn test -pl flink-connector-jdbc-postgres -am -Dtest=PostgresDialectITCase` Review Comment: `PostgresDialectITCase` doesn't exist, `PostgresDynamicTableSourceITCase` would work. The command shape is fine, I ran it against `DerbyDynamicTableSourceITCase` and 15 tests passed. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) +- Unix-like environment (Linux, macOS, WSL) +- The connector builds against `flink.version` in the root `pom.xml`. A change must also work against every Flink version in `.github/workflows/push_pr.yml` and the `main` row of `.github/workflows/weekly.yml`. + +## Commands + +### Build + +- Build without tests: `mvn clean package -DskipTests` +- Full build with tests: `mvn clean verify` +- Build against another Flink version (what CI does): `mvn clean install -DskipTests -Dflink.version=<version>` +- Single module: `mvn clean install -DskipTests -pl flink-connector-jdbc-core` +- A database module depends on `flink-connector-jdbc-core`; build it with its dependency using `-pl flink-connector-jdbc-mysql -am`. +- Dependency convergence (CI runs this on PRs): `mvn clean install -DskipTests -Pcheck-convergence -Dflink.convergence.phase=install` + +### Testing + +- `*Test` classes are unit tests; `*ITCase` classes are integration tests that start a real database via Testcontainers (Docker required). `flink-connector-jdbc-core` also runs against in-memory Derby and H2, so its unit tests need no Docker. +- Single unit test class: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest` +- Single test method: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest#testX` +- Single ITCase in a database module: `mvn test -pl flink-connector-jdbc-postgres -am -Dtest=PostgresDialectITCase` +- ArchUnit rules live in `flink-connector-jdbc-architecture`; violation stores are frozen (see Testing Standards). +- The standalone `flink-connector-jdbc-backward-compatibility` module is not part of the root reactor; CI builds it separately (`.github/workflows/backwards_compatibility.yml`) to check that current code reads savepoints written by older connector versions. +- CI for PRs and the weekly build is defined by `apache/flink-connector-shared-utils` (`.github/workflows/ci.yml@ci_utils`); its Maven command line, including the license check, is the reference when a local run differs from CI. + +### Code Quality + +- Format code: `mvn spotless:apply` (the `java21` profile sets `spotless.skip=true`, so run it on JDK 11 or 17; google-java-format does not run on JDK 21) +- Check formatting: `mvn spotless:check` +- Checkstyle: `mvn checkstyle:check` (config in `tools/maven/checkstyle.xml`, suppressions in `tools/maven/suppressions.xml`) +- License headers: `mvn apache-rat:check` +- japicmp compares against `japicmp.referenceVersion` from the root `pom.xml` and only checks stable API. + +### Documentation + +- There is no docs build in this repository. The Flink docs build in `apache/flink` clones the release branch of this repository and renders `docs/content` and `docs/content.zh`. +- Documentation exists in English (`docs/content`) and Chinese (`docs/content.zh`); the two carry the same file set. + +## Repository Structure + +### Modules + +The root `pom.xml` reactor contains: + +- `flink-connector-jdbc-architecture` — ArchUnit tests and their frozen violation stores (`archunit-violations/`). +- `flink-connector-jdbc-core` — The connector: the FLIP-27 `JdbcSource`, the Sink V2 `JdbcSink`, the legacy `JdbcInputFormat` / `JdbcRowOutputFormat`, the Table/SQL factory (identifier `jdbc`), the dialect SPI, the XA/exactly-once machinery, and the Derby + H2 in-memory test support. Everything database-agnostic lives here. +- `flink-connector-jdbc-cratedb`, `-db2`, `-mysql`, `-oceanbase`, `-oracle`, `-postgres`, `-sqlserver`, `-trino` — one module per database. Each contributes a `JdbcDialect`, a `JdbcFactory` (dialect SPI), usually a `JdbcCatalog` + type mapper, and a lineage location extractor, discovered via `META-INF/services`. + +Not in the reactor: + +- `flink-connector-jdbc-backward-compatibility` — standalone module, built by its own CI workflow, that verifies savepoint/state compatibility with older connector versions. + +### Supporting directories + +- `docs/content/docs/connectors/` and `docs/content.zh/docs/connectors/` — DataStream and Table docs, English and Chinese, same file set. +- `tools/maven/` — checkstyle config and suppressions. Review Comment: `tools/releasing/shared` is a submodule and worth listing here, otherwise an agent will wonder why the directory is empty. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) +- Unix-like environment (Linux, macOS, WSL) +- The connector builds against `flink.version` in the root `pom.xml`. A change must also work against every Flink version in `.github/workflows/push_pr.yml` and the `main` row of `.github/workflows/weekly.yml`. + +## Commands + +### Build + +- Build without tests: `mvn clean package -DskipTests` +- Full build with tests: `mvn clean verify` +- Build against another Flink version (what CI does): `mvn clean install -DskipTests -Dflink.version=<version>` +- Single module: `mvn clean install -DskipTests -pl flink-connector-jdbc-core` +- A database module depends on `flink-connector-jdbc-core`; build it with its dependency using `-pl flink-connector-jdbc-mysql -am`. +- Dependency convergence (CI runs this on PRs): `mvn clean install -DskipTests -Pcheck-convergence -Dflink.convergence.phase=install` + +### Testing + +- `*Test` classes are unit tests; `*ITCase` classes are integration tests that start a real database via Testcontainers (Docker required). `flink-connector-jdbc-core` also runs against in-memory Derby and H2, so its unit tests need no Docker. +- Single unit test class: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest` +- Single test method: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest#testX` +- Single ITCase in a database module: `mvn test -pl flink-connector-jdbc-postgres -am -Dtest=PostgresDialectITCase` +- ArchUnit rules live in `flink-connector-jdbc-architecture`; violation stores are frozen (see Testing Standards). +- The standalone `flink-connector-jdbc-backward-compatibility` module is not part of the root reactor; CI builds it separately (`.github/workflows/backwards_compatibility.yml`) to check that current code reads savepoints written by older connector versions. +- CI for PRs and the weekly build is defined by `apache/flink-connector-shared-utils` (`.github/workflows/ci.yml@ci_utils`); its Maven command line, including the license check, is the reference when a local run differs from CI. + +### Code Quality + +- Format code: `mvn spotless:apply` (the `java21` profile sets `spotless.skip=true`, so run it on JDK 11 or 17; google-java-format does not run on JDK 21) +- Check formatting: `mvn spotless:check` +- Checkstyle: `mvn checkstyle:check` (config in `tools/maven/checkstyle.xml`, suppressions in `tools/maven/suppressions.xml`) +- License headers: `mvn apache-rat:check` +- japicmp compares against `japicmp.referenceVersion` from the root `pom.xml` and only checks stable API. + +### Documentation + +- There is no docs build in this repository. The Flink docs build in `apache/flink` clones the release branch of this repository and renders `docs/content` and `docs/content.zh`. +- Documentation exists in English (`docs/content`) and Chinese (`docs/content.zh`); the two carry the same file set. + +## Repository Structure + +### Modules + +The root `pom.xml` reactor contains: + +- `flink-connector-jdbc-architecture` — ArchUnit tests and their frozen violation stores (`archunit-violations/`). +- `flink-connector-jdbc-core` — The connector: the FLIP-27 `JdbcSource`, the Sink V2 `JdbcSink`, the legacy `JdbcInputFormat` / `JdbcRowOutputFormat`, the Table/SQL factory (identifier `jdbc`), the dialect SPI, the XA/exactly-once machinery, and the Derby + H2 in-memory test support. Everything database-agnostic lives here. +- `flink-connector-jdbc-cratedb`, `-db2`, `-mysql`, `-oceanbase`, `-oracle`, `-postgres`, `-sqlserver`, `-trino` — one module per database. Each contributes a `JdbcDialect`, a `JdbcFactory` (dialect SPI), usually a `JdbcCatalog` + type mapper, and a lineage location extractor, discovered via `META-INF/services`. + +Not in the reactor: + +- `flink-connector-jdbc-backward-compatibility` — standalone module, built by its own CI workflow, that verifies savepoint/state compatibility with older connector versions. + +### Supporting directories + +- `docs/content/docs/connectors/` and `docs/content.zh/docs/connectors/` — DataStream and Table docs, English and Chinese, same file set. +- `tools/maven/` — checkstyle config and suppressions. +- `tools/ci/` — the log4j config CI passes to Maven. +- `.github/workflows/` — `push_pr.yml` (PR CI), `weekly.yml` (release branches and Flink snapshots), `backwards_compatibility.yml` (the standalone module). + +### Key packages in `flink-connector-jdbc-core/src/main/java` + +- `org.apache.flink.connector.jdbc` — top-level user-facing option/builder types: `JdbcConnectionOptions`, `JdbcExecutionOptions`, `JdbcExactlyOnceOptions`, `JdbcStatementBuilder`, and the legacy `JdbcInputFormat` / `JdbcRowOutputFormat`. +- `org.apache.flink.connector.jdbc.core.datastream.source` — FLIP-27 source. `@PublicEvolving`: `JdbcSource`, `JdbcSourceBuilder`, `JdbcSourceOptions`. `@Internal`: `enumerator/`, `reader/`, `split/`. +- `org.apache.flink.connector.jdbc.core.datastream.sink` — Sink V2 sink. `@PublicEvolving`: `JdbcSink`, `JdbcSinkBuilder`. `writer/` and `committer/` are `@Internal`. +- `org.apache.flink.connector.jdbc.core.table` — Table/SQL layer. `JdbcDynamicTableFactory` (identifier `jdbc`, registered in `META-INF/services/org.apache.flink.table.factories.Factory`), `JdbcConnectorOptions` (`@PublicEvolving`), and the `@Internal` `JdbcDynamicTableSource` / `JdbcDynamicTableSink`. +- `org.apache.flink.connector.jdbc.core.database` — the pluggable-database SPI: `JdbcFactory` and `JdbcDialect` (both `@PublicEvolving`), `dialect/` (statement building, converters), and `catalog/` (`JdbcCatalog`, the catalog factory). Review Comment: `JdbcDialect` is in `core.database.dialect`, not next to `JdbcFactory`. This reads as if both sit in the same package. ########## .github/PULL_REQUEST_TEMPLATE.md: ########## @@ -0,0 +1,92 @@ +<!-- +*Thank you very much for contributing to the Apache Flink JDBC connector - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.* + +*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.* + +## Contribution Checklist + + - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue. + + - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component. + Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`. + + - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review. + + - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. GitHub Actions runs the same build for every push and pull request against the Flink versions and JDKs listed in `.github/workflows/push_pr.yml`. Review Comment: Kafka's template uses `./mvnw` here, flink's still says `mvn`. Now that FLINK-40724 has landed I would follow Kafka. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) +- Unix-like environment (Linux, macOS, WSL) +- The connector builds against `flink.version` in the root `pom.xml`. A change must also work against every Flink version in `.github/workflows/push_pr.yml` and the `main` row of `.github/workflows/weekly.yml`. + +## Commands + +### Build + +- Build without tests: `mvn clean package -DskipTests` +- Full build with tests: `mvn clean verify` +- Build against another Flink version (what CI does): `mvn clean install -DskipTests -Dflink.version=<version>` +- Single module: `mvn clean install -DskipTests -pl flink-connector-jdbc-core` +- A database module depends on `flink-connector-jdbc-core`; build it with its dependency using `-pl flink-connector-jdbc-mysql -am`. +- Dependency convergence (CI runs this on PRs): `mvn clean install -DskipTests -Pcheck-convergence -Dflink.convergence.phase=install` + +### Testing + +- `*Test` classes are unit tests; `*ITCase` classes are integration tests that start a real database via Testcontainers (Docker required). `flink-connector-jdbc-core` also runs against in-memory Derby and H2, so its unit tests need no Docker. +- Single unit test class: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest` +- Single test method: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest#testX` +- Single ITCase in a database module: `mvn test -pl flink-connector-jdbc-postgres -am -Dtest=PostgresDialectITCase` +- ArchUnit rules live in `flink-connector-jdbc-architecture`; violation stores are frozen (see Testing Standards). +- The standalone `flink-connector-jdbc-backward-compatibility` module is not part of the root reactor; CI builds it separately (`.github/workflows/backwards_compatibility.yml`) to check that current code reads savepoints written by older connector versions. +- CI for PRs and the weekly build is defined by `apache/flink-connector-shared-utils` (`.github/workflows/ci.yml@ci_utils`); its Maven command line, including the license check, is the reference when a local run differs from CI. + +### Code Quality + +- Format code: `mvn spotless:apply` (the `java21` profile sets `spotless.skip=true`, so run it on JDK 11 or 17; google-java-format does not run on JDK 21) +- Check formatting: `mvn spotless:check` +- Checkstyle: `mvn checkstyle:check` (config in `tools/maven/checkstyle.xml`, suppressions in `tools/maven/suppressions.xml`) +- License headers: `mvn apache-rat:check` +- japicmp compares against `japicmp.referenceVersion` from the root `pom.xml` and only checks stable API. + +### Documentation + +- There is no docs build in this repository. The Flink docs build in `apache/flink` clones the release branch of this repository and renders `docs/content` and `docs/content.zh`. +- Documentation exists in English (`docs/content`) and Chinese (`docs/content.zh`); the two carry the same file set. + +## Repository Structure + +### Modules + +The root `pom.xml` reactor contains: + +- `flink-connector-jdbc-architecture` — ArchUnit tests and their frozen violation stores (`archunit-violations/`). +- `flink-connector-jdbc-core` — The connector: the FLIP-27 `JdbcSource`, the Sink V2 `JdbcSink`, the legacy `JdbcInputFormat` / `JdbcRowOutputFormat`, the Table/SQL factory (identifier `jdbc`), the dialect SPI, the XA/exactly-once machinery, and the Derby + H2 in-memory test support. Everything database-agnostic lives here. +- `flink-connector-jdbc-cratedb`, `-db2`, `-mysql`, `-oceanbase`, `-oracle`, `-postgres`, `-sqlserver`, `-trino` — one module per database. Each contributes a `JdbcDialect`, a `JdbcFactory` (dialect SPI), usually a `JdbcCatalog` + type mapper, and a lineage location extractor, discovered via `META-INF/services`. + +Not in the reactor: + +- `flink-connector-jdbc-backward-compatibility` — standalone module, built by its own CI workflow, that verifies savepoint/state compatibility with older connector versions. + +### Supporting directories + +- `docs/content/docs/connectors/` and `docs/content.zh/docs/connectors/` — DataStream and Table docs, English and Chinese, same file set. +- `tools/maven/` — checkstyle config and suppressions. +- `tools/ci/` — the log4j config CI passes to Maven. +- `.github/workflows/` — `push_pr.yml` (PR CI), `weekly.yml` (release branches and Flink snapshots), `backwards_compatibility.yml` (the standalone module). + +### Key packages in `flink-connector-jdbc-core/src/main/java` + +- `org.apache.flink.connector.jdbc` — top-level user-facing option/builder types: `JdbcConnectionOptions`, `JdbcExecutionOptions`, `JdbcExactlyOnceOptions`, `JdbcStatementBuilder`, and the legacy `JdbcInputFormat` / `JdbcRowOutputFormat`. +- `org.apache.flink.connector.jdbc.core.datastream.source` — FLIP-27 source. `@PublicEvolving`: `JdbcSource`, `JdbcSourceBuilder`, `JdbcSourceOptions`. `@Internal`: `enumerator/`, `reader/`, `split/`. +- `org.apache.flink.connector.jdbc.core.datastream.sink` — Sink V2 sink. `@PublicEvolving`: `JdbcSink`, `JdbcSinkBuilder`. `writer/` and `committer/` are `@Internal`. +- `org.apache.flink.connector.jdbc.core.table` — Table/SQL layer. `JdbcDynamicTableFactory` (identifier `jdbc`, registered in `META-INF/services/org.apache.flink.table.factories.Factory`), `JdbcConnectorOptions` (`@PublicEvolving`), and the `@Internal` `JdbcDynamicTableSource` / `JdbcDynamicTableSink`. +- `org.apache.flink.connector.jdbc.core.database` — the pluggable-database SPI: `JdbcFactory` and `JdbcDialect` (both `@PublicEvolving`), `dialect/` (statement building, converters), and `catalog/` (`JdbcCatalog`, the catalog factory). +- `org.apache.flink.connector.jdbc.datasource` — connection and transaction management. `connections/` holds `JdbcConnectionProvider` / `SimpleJdbcConnectionProvider`; `connections/xa/` and `transactions/xa/` hold the XA two-phase-commit path used for exactly-once. +- `org.apache.flink.connector.jdbc.internal` — `JdbcOutputFormat`, the DML/insert/read option holders (`internal/options/`), and the connection/statement execution glue. `@Internal`. +- `org.apache.flink.connector.jdbc.lineage` — OpenLineage facet extraction and the `JdbcLocationExtractorFactory` SPI. +- `org.apache.flink.connector.jdbc.split` — input-split parameter providers for the legacy input format. + +## Architecture Boundaries + +1. **Source (FLIP-27).** `JdbcSourceEnumerator` runs on the coordinator thread and hands `JdbcSourceSplit`s to `JdbcSourceReader`; `JdbcSourceSplitReader` runs on the fetcher thread and owns the JDBC `Connection`/`ResultSet`. A JDBC connection is not thread-safe — access it only from the reader thread. +2. **Sink (Sink V2).** `JdbcSink` builds on `JdbcOutputFormat`, which batches rows and flushes on a size/time trigger and on checkpoint. The at-least-once path uses a plain `JdbcConnectionProvider`; per-record failures surface on the task thread at flush time. +3. **Exactly-once (XA).** The exactly-once sink uses distributed (XA) transactions via `connections/xa` (`XaConnectionProvider`, `PoolingXaConnectionProvider`) and `transactions/xa` (Xid generation, per-checkpoint transaction lifecycle). XA driver support varies by database; this is the most driver-sensitive path — re-verify it on any driver bump. Not every dialect supports it. +4. **Dialect SPI.** A database plugs in through `JdbcFactory` (`acceptsURL` decides which JDBC URLs it claims; `createDialect` / `createDialect(compatibleMode)` builds the `JdbcDialect`), discovered via `META-INF/services/org.apache.flink.connector.jdbc.core.database.JdbcFactory`. The `JdbcDialect` owns quoting, LIMIT, upsert/UPSERT-or-MERGE statement construction, and the `JdbcDialectConverter` type mapping. Keep database-specific behaviour inside the dialect, not in core. +5. **Table layer.** `JdbcDynamicTableSource` / `JdbcDynamicTableSink` wrap the DataStream connectors; all Table options live in `JdbcConnectorOptions`. +6. **Connector vs Flink.** Production code may depend only on stable (`@Public` / `@PublicEvolving`) Flink API outside connector and util packages (enforced by ArchUnit). Every Flink API used must exist with the same annotation in `flink.version`, because the connector is released for several Flink minor versions. + +## Common Change Patterns + +### Adding a new database + +1. Create a `flink-connector-jdbc-<db>` module mirroring an existing one (e.g. `flink-connector-jdbc-postgres`), with the DB module as a sibling of `flink-connector-jdbc-core` and added to the root `pom.xml` `<modules>`. +2. Implement `JdbcDialect` + `JdbcDialectConverter`, and a `JdbcFactory` whose `acceptsURL` matches the driver's JDBC sub-protocol; register the factory in `META-INF/services/org.apache.flink.connector.jdbc.core.database.JdbcFactory`. +3. If the catalog is supported, add a `JdbcCatalog` + type mapper; add a `JdbcLocationExtractorFactory` for lineage. +4. Pin the JDBC driver version as a `<db>.version` property in the module `pom.xml` and add the driver dependency (test or provided scope, as the sibling modules do). +5. Add a Testcontainers `*Database` helper under `testutils/`, a `*TestBase`, and dialect/catalog `*ITCase`s. +6. Document the dialect in `docs/content` and `docs/content.zh`. + +### Adding a Table/SQL option + +1. Define the `ConfigOption<T>` in `JdbcConnectorOptions` +2. Register and validate it in `JdbcDynamicTableFactory` (`optionalOptions()` / validation) +3. Add a factory test and, when behaviour changes, an `ITCase` +4. Add the option row to `docs/content/docs/connectors/table/*.md` and the same file under `docs/content.zh/` +5. Fill in the Release Notes field on the JIRA ticket + +### Adding a DataStream builder option + +1. Add it to `JdbcSourceBuilder` or `JdbcSinkBuilder` with validation in `build()` +2. Add a builder unit test and, when behaviour changes, an `ITCase` +3. Document it under `docs/content/docs/connectors/datastream/` and the `.zh` copy + +### Changing checkpointed state + +State is written by `SimpleVersionedSerializer` implementations. Current versions: `JdbcSourceSplitSerializer` 0, `JdbcSourceEnumStateSerializer` 0, `JdbcWriterStateSerializer` 2, `JdbcCommitableSerializer` 1. + +1. Bump the version and keep a read path for every older version +2. Add a serializer test that decodes bytes of the previous version +3. Cover the change with the `flink-connector-jdbc-backward-compatibility` module + +### Bumping a database driver version + +1. Change the `<db>.version` property in the module `pom.xml` +2. If the change affects exactly-once, re-verify the XA path against that database +3. Verify: `dependency:tree` for new transitive dependencies, and the module's ITCase suite + +### Bumping `flink.version` or changing the CI matrix + +1. Get consensus on the JIRA ticket first; this changes which Flink versions the branch supports +2. Update `push_pr.yml` and `weekly.yml`; keep one JDK per Flink version and stay under the ASF limit of 20 concurrent jobs +3. Verify: the build against every Flink version in the matrix + +### Fixing a flaky test + +1. Name the race or ordering that fails, with the CI log excerpt +2. Wait on the condition (`CommonTestUtils.waitUtil`), never on time; for a database, wait for readiness through the Testcontainers wait strategy +3. Do not add `Thread.sleep`, larger timeouts, retries or `@Disabled` +4. Verify: run the test repeatedly and state the number of runs in the PR + +## Coding Standards + +- **Format Java files with Spotless after editing:** `mvn spotless:apply` (google-java-format, AOSP style). It is auto-skipped on JDK 21, so format on 11 or 17. +- **Checkstyle:** `tools/maven/checkstyle.xml`. Fix the code rather than suppressing rules. +- **Apache License 2.0 header** required on all new files (enforced by Apache Rat). Use an HTML comment for markdown files. +- **API stability annotations:** Every user-facing API class and method must have one. `@Public` (stable across minor releases), `@PublicEvolving` (may change in minor releases), `@Experimental` (may change at any time), `@Internal` (no guarantees; do not depend on). +- **JDBC resources:** close `Connection`, `Statement`, and `ResultSet` deterministically (try-with-resources). A `Connection` is single-threaded; do not share it across threads. +- **Logging:** parameterized SLF4J statements (`{}` placeholders), never string concatenation. +- **No Java serialization** for new state; use `SimpleVersionedSerializer`. +- **Use `final`** for variables and fields where applicable. +- **Comments:** do not restate what the code does; explain the "why" where it is non-obvious. +- **Keep database-specific logic in the dialect,** not in `flink-connector-jdbc-core`. +- Full code style guide: https://flink.apache.org/how-to-contribute/code-style-and-quality-preamble/ + +## Testing Standards + +- Add tests for new behavior, covering success, failure, and edge cases. +- Use **JUnit 5** + **AssertJ** assertions. +- **Integration tests:** name classes with the `ITCase` suffix; they start their database through Testcontainers. Reuse the module's `testutils/*Database` helper and `*TestBase`; do not hand-roll container setup. +- **Core tests** use in-memory Derby and H2 (`flink-connector-jdbc-core`), so they run without Docker; prefer these for database-agnostic behaviour. +- **Red-green verification:** for bug fixes, confirm the new test fails without the fix before it passes with it. +- **ArchUnit:** violation stores under `flink-connector-jdbc-architecture/archunit-violations/` (and `archunit-violations/` in modules that have them) are frozen. A local run removes lines when violations disappear; commit removed lines with the change, never add lines. +- **Backward compatibility:** when a serializer version changes, cover it in `flink-connector-jdbc-backward-compatibility`. +- Follow the testing conventions at https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing + +## Commits and PRs + +### Commit message format + +- `[FLINK-XXXX][component] Description` where FLINK-XXXX is the JIRA issue number +- `[hotfix][component] Description` for typo/doc/CI-config fixes without JIRA +- Each commit must have a meaningful message including the JIRA ID. If you don't know the ticket number, ask. +- Separate cleanup/refactoring from functional changes into distinct commits +- When AI tools were used: add a `Generated-by: <Tool Name and Version>` trailer per [ASF generative tooling guidance](https://www.apache.org/legal/generative-tooling.html) + +### Pull request conventions + +- Title format: `[FLINK-XXXX][component] Title of the pull request` Review Comment: The Kafka file has the backport form here and this repo uses it, see #238 and #239. Worth keeping. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) +- Unix-like environment (Linux, macOS, WSL) +- The connector builds against `flink.version` in the root `pom.xml`. A change must also work against every Flink version in `.github/workflows/push_pr.yml` and the `main` row of `.github/workflows/weekly.yml`. + +## Commands + +### Build + +- Build without tests: `mvn clean package -DskipTests` +- Full build with tests: `mvn clean verify` +- Build against another Flink version (what CI does): `mvn clean install -DskipTests -Dflink.version=<version>` +- Single module: `mvn clean install -DskipTests -pl flink-connector-jdbc-core` +- A database module depends on `flink-connector-jdbc-core`; build it with its dependency using `-pl flink-connector-jdbc-mysql -am`. +- Dependency convergence (CI runs this on PRs): `mvn clean install -DskipTests -Pcheck-convergence -Dflink.convergence.phase=install` + +### Testing + +- `*Test` classes are unit tests; `*ITCase` classes are integration tests that start a real database via Testcontainers (Docker required). `flink-connector-jdbc-core` also runs against in-memory Derby and H2, so its unit tests need no Docker. +- Single unit test class: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest` +- Single test method: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest#testX` +- Single ITCase in a database module: `mvn test -pl flink-connector-jdbc-postgres -am -Dtest=PostgresDialectITCase` +- ArchUnit rules live in `flink-connector-jdbc-architecture`; violation stores are frozen (see Testing Standards). +- The standalone `flink-connector-jdbc-backward-compatibility` module is not part of the root reactor; CI builds it separately (`.github/workflows/backwards_compatibility.yml`) to check that current code reads savepoints written by older connector versions. +- CI for PRs and the weekly build is defined by `apache/flink-connector-shared-utils` (`.github/workflows/ci.yml@ci_utils`); its Maven command line, including the license check, is the reference when a local run differs from CI. + +### Code Quality + +- Format code: `mvn spotless:apply` (the `java21` profile sets `spotless.skip=true`, so run it on JDK 11 or 17; google-java-format does not run on JDK 21) +- Check formatting: `mvn spotless:check` +- Checkstyle: `mvn checkstyle:check` (config in `tools/maven/checkstyle.xml`, suppressions in `tools/maven/suppressions.xml`) +- License headers: `mvn apache-rat:check` +- japicmp compares against `japicmp.referenceVersion` from the root `pom.xml` and only checks stable API. + +### Documentation + +- There is no docs build in this repository. The Flink docs build in `apache/flink` clones the release branch of this repository and renders `docs/content` and `docs/content.zh`. +- Documentation exists in English (`docs/content`) and Chinese (`docs/content.zh`); the two carry the same file set. + +## Repository Structure + +### Modules + +The root `pom.xml` reactor contains: + +- `flink-connector-jdbc-architecture` — ArchUnit tests and their frozen violation stores (`archunit-violations/`). +- `flink-connector-jdbc-core` — The connector: the FLIP-27 `JdbcSource`, the Sink V2 `JdbcSink`, the legacy `JdbcInputFormat` / `JdbcRowOutputFormat`, the Table/SQL factory (identifier `jdbc`), the dialect SPI, the XA/exactly-once machinery, and the Derby + H2 in-memory test support. Everything database-agnostic lives here. +- `flink-connector-jdbc-cratedb`, `-db2`, `-mysql`, `-oceanbase`, `-oracle`, `-postgres`, `-sqlserver`, `-trino` — one module per database. Each contributes a `JdbcDialect`, a `JdbcFactory` (dialect SPI), usually a `JdbcCatalog` + type mapper, and a lineage location extractor, discovered via `META-INF/services`. + +Not in the reactor: + +- `flink-connector-jdbc-backward-compatibility` — standalone module, built by its own CI workflow, that verifies savepoint/state compatibility with older connector versions. + +### Supporting directories + +- `docs/content/docs/connectors/` and `docs/content.zh/docs/connectors/` — DataStream and Table docs, English and Chinese, same file set. +- `tools/maven/` — checkstyle config and suppressions. +- `tools/ci/` — the log4j config CI passes to Maven. +- `.github/workflows/` — `push_pr.yml` (PR CI), `weekly.yml` (release branches and Flink snapshots), `backwards_compatibility.yml` (the standalone module). + +### Key packages in `flink-connector-jdbc-core/src/main/java` + +- `org.apache.flink.connector.jdbc` — top-level user-facing option/builder types: `JdbcConnectionOptions`, `JdbcExecutionOptions`, `JdbcExactlyOnceOptions`, `JdbcStatementBuilder`, and the legacy `JdbcInputFormat` / `JdbcRowOutputFormat`. +- `org.apache.flink.connector.jdbc.core.datastream.source` — FLIP-27 source. `@PublicEvolving`: `JdbcSource`, `JdbcSourceBuilder`, `JdbcSourceOptions`. `@Internal`: `enumerator/`, `reader/`, `split/`. +- `org.apache.flink.connector.jdbc.core.datastream.sink` — Sink V2 sink. `@PublicEvolving`: `JdbcSink`, `JdbcSinkBuilder`. `writer/` and `committer/` are `@Internal`. +- `org.apache.flink.connector.jdbc.core.table` — Table/SQL layer. `JdbcDynamicTableFactory` (identifier `jdbc`, registered in `META-INF/services/org.apache.flink.table.factories.Factory`), `JdbcConnectorOptions` (`@PublicEvolving`), and the `@Internal` `JdbcDynamicTableSource` / `JdbcDynamicTableSink`. +- `org.apache.flink.connector.jdbc.core.database` — the pluggable-database SPI: `JdbcFactory` and `JdbcDialect` (both `@PublicEvolving`), `dialect/` (statement building, converters), and `catalog/` (`JdbcCatalog`, the catalog factory). +- `org.apache.flink.connector.jdbc.datasource` — connection and transaction management. `connections/` holds `JdbcConnectionProvider` / `SimpleJdbcConnectionProvider`; `connections/xa/` and `transactions/xa/` hold the XA two-phase-commit path used for exactly-once. +- `org.apache.flink.connector.jdbc.internal` — `JdbcOutputFormat`, the DML/insert/read option holders (`internal/options/`), and the connection/statement execution glue. `@Internal`. +- `org.apache.flink.connector.jdbc.lineage` — OpenLineage facet extraction and the `JdbcLocationExtractorFactory` SPI. +- `org.apache.flink.connector.jdbc.split` — input-split parameter providers for the legacy input format. + +## Architecture Boundaries + +1. **Source (FLIP-27).** `JdbcSourceEnumerator` runs on the coordinator thread and hands `JdbcSourceSplit`s to `JdbcSourceReader`; `JdbcSourceSplitReader` runs on the fetcher thread and owns the JDBC `Connection`/`ResultSet`. A JDBC connection is not thread-safe — access it only from the reader thread. +2. **Sink (Sink V2).** `JdbcSink` builds on `JdbcOutputFormat`, which batches rows and flushes on a size/time trigger and on checkpoint. The at-least-once path uses a plain `JdbcConnectionProvider`; per-record failures surface on the task thread at flush time. +3. **Exactly-once (XA).** The exactly-once sink uses distributed (XA) transactions via `connections/xa` (`XaConnectionProvider`, `PoolingXaConnectionProvider`) and `transactions/xa` (Xid generation, per-checkpoint transaction lifecycle). XA driver support varies by database; this is the most driver-sensitive path — re-verify it on any driver bump. Not every dialect supports it. +4. **Dialect SPI.** A database plugs in through `JdbcFactory` (`acceptsURL` decides which JDBC URLs it claims; `createDialect` / `createDialect(compatibleMode)` builds the `JdbcDialect`), discovered via `META-INF/services/org.apache.flink.connector.jdbc.core.database.JdbcFactory`. The `JdbcDialect` owns quoting, LIMIT, upsert/UPSERT-or-MERGE statement construction, and the `JdbcDialectConverter` type mapping. Keep database-specific behaviour inside the dialect, not in core. +5. **Table layer.** `JdbcDynamicTableSource` / `JdbcDynamicTableSink` wrap the DataStream connectors; all Table options live in `JdbcConnectorOptions`. +6. **Connector vs Flink.** Production code may depend only on stable (`@Public` / `@PublicEvolving`) Flink API outside connector and util packages (enforced by ArchUnit). Every Flink API used must exist with the same annotation in `flink.version`, because the connector is released for several Flink minor versions. + +## Common Change Patterns + +### Adding a new database + +1. Create a `flink-connector-jdbc-<db>` module mirroring an existing one (e.g. `flink-connector-jdbc-postgres`), with the DB module as a sibling of `flink-connector-jdbc-core` and added to the root `pom.xml` `<modules>`. +2. Implement `JdbcDialect` + `JdbcDialectConverter`, and a `JdbcFactory` whose `acceptsURL` matches the driver's JDBC sub-protocol; register the factory in `META-INF/services/org.apache.flink.connector.jdbc.core.database.JdbcFactory`. +3. If the catalog is supported, add a `JdbcCatalog` + type mapper; add a `JdbcLocationExtractorFactory` for lineage. +4. Pin the JDBC driver version as a `<db>.version` property in the module `pom.xml` and add the driver dependency (test or provided scope, as the sibling modules do). +5. Add a Testcontainers `*Database` helper under `testutils/`, a `*TestBase`, and dialect/catalog `*ITCase`s. +6. Document the dialect in `docs/content` and `docs/content.zh`. + +### Adding a Table/SQL option + +1. Define the `ConfigOption<T>` in `JdbcConnectorOptions` +2. Register and validate it in `JdbcDynamicTableFactory` (`optionalOptions()` / validation) +3. Add a factory test and, when behaviour changes, an `ITCase` +4. Add the option row to `docs/content/docs/connectors/table/*.md` and the same file under `docs/content.zh/` +5. Fill in the Release Notes field on the JIRA ticket + +### Adding a DataStream builder option + +1. Add it to `JdbcSourceBuilder` or `JdbcSinkBuilder` with validation in `build()` +2. Add a builder unit test and, when behaviour changes, an `ITCase` +3. Document it under `docs/content/docs/connectors/datastream/` and the `.zh` copy + +### Changing checkpointed state + +State is written by `SimpleVersionedSerializer` implementations. Current versions: `JdbcSourceSplitSerializer` 0, `JdbcSourceEnumStateSerializer` 0, `JdbcWriterStateSerializer` 2, `JdbcCommitableSerializer` 1. + +1. Bump the version and keep a read path for every older version +2. Add a serializer test that decodes bytes of the previous version +3. Cover the change with the `flink-connector-jdbc-backward-compatibility` module + +### Bumping a database driver version + +1. Change the `<db>.version` property in the module `pom.xml` +2. If the change affects exactly-once, re-verify the XA path against that database +3. Verify: `dependency:tree` for new transitive dependencies, and the module's ITCase suite + +### Bumping `flink.version` or changing the CI matrix + +1. Get consensus on the JIRA ticket first; this changes which Flink versions the branch supports +2. Update `push_pr.yml` and `weekly.yml`; keep one JDK per Flink version and stay under the ASF limit of 20 concurrent jobs +3. Verify: the build against every Flink version in the matrix + +### Fixing a flaky test + +1. Name the race or ordering that fails, with the CI log excerpt +2. Wait on the condition (`CommonTestUtils.waitUtil`), never on time; for a database, wait for readiness through the Testcontainers wait strategy +3. Do not add `Thread.sleep`, larger timeouts, retries or `@Disabled` +4. Verify: run the test repeatedly and state the number of runs in the PR + +## Coding Standards + +- **Format Java files with Spotless after editing:** `mvn spotless:apply` (google-java-format, AOSP style). It is auto-skipped on JDK 21, so format on 11 or 17. +- **Checkstyle:** `tools/maven/checkstyle.xml`. Fix the code rather than suppressing rules. +- **Apache License 2.0 header** required on all new files (enforced by Apache Rat). Use an HTML comment for markdown files. +- **API stability annotations:** Every user-facing API class and method must have one. `@Public` (stable across minor releases), `@PublicEvolving` (may change in minor releases), `@Experimental` (may change at any time), `@Internal` (no guarantees; do not depend on). +- **JDBC resources:** close `Connection`, `Statement`, and `ResultSet` deterministically (try-with-resources). A `Connection` is single-threaded; do not share it across threads. +- **Logging:** parameterized SLF4J statements (`{}` placeholders), never string concatenation. +- **No Java serialization** for new state; use `SimpleVersionedSerializer`. +- **Use `final`** for variables and fields where applicable. +- **Comments:** do not restate what the code does; explain the "why" where it is non-obvious. +- **Keep database-specific logic in the dialect,** not in `flink-connector-jdbc-core`. +- Full code style guide: https://flink.apache.org/how-to-contribute/code-style-and-quality-preamble/ + +## Testing Standards + +- Add tests for new behavior, covering success, failure, and edge cases. +- Use **JUnit 5** + **AssertJ** assertions. +- **Integration tests:** name classes with the `ITCase` suffix; they start their database through Testcontainers. Reuse the module's `testutils/*Database` helper and `*TestBase`; do not hand-roll container setup. +- **Core tests** use in-memory Derby and H2 (`flink-connector-jdbc-core`), so they run without Docker; prefer these for database-agnostic behaviour. +- **Red-green verification:** for bug fixes, confirm the new test fails without the fix before it passes with it. +- **ArchUnit:** violation stores under `flink-connector-jdbc-architecture/archunit-violations/` (and `archunit-violations/` in modules that have them) are frozen. A local run removes lines when violations disappear; commit removed lines with the change, never add lines. +- **Backward compatibility:** when a serializer version changes, cover it in `flink-connector-jdbc-backward-compatibility`. +- Follow the testing conventions at https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing + +## Commits and PRs + +### Commit message format + +- `[FLINK-XXXX][component] Description` where FLINK-XXXX is the JIRA issue number +- `[hotfix][component] Description` for typo/doc/CI-config fixes without JIRA +- Each commit must have a meaningful message including the JIRA ID. If you don't know the ticket number, ask. +- Separate cleanup/refactoring from functional changes into distinct commits +- When AI tools were used: add a `Generated-by: <Tool Name and Version>` trailer per [ASF generative tooling guidance](https://www.apache.org/legal/generative-tooling.html) + +### Pull request conventions + +- Title format: `[FLINK-XXXX][component] Title of the pull request` +- A corresponding JIRA issue is required (except hotfixes for typos, docs, or CI config) +- Fill out the PR template completely but concisely: purpose, change log, testing approach, impact assessment +- Each PR should address exactly one issue +- Ensure `mvn clean verify` passes before opening a PR +- Always push to your fork, not directly to `apache/flink-connector-jdbc` +- Rebase onto the latest target branch before submitting Review Comment: `.asf.yaml` disables the merge button here too, so the Kafka line about squash and rebase applies verbatim. ########## .github/PULL_REQUEST_TEMPLATE.md: ########## @@ -0,0 +1,92 @@ +<!-- +*Thank you very much for contributing to the Apache Flink JDBC connector - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.* + +*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.* + +## Contribution Checklist + + - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue. + + - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component. + Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`. + + - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review. + + - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. GitHub Actions runs the same build for every push and pull request against the Flink versions and JDKs listed in `.github/workflows/push_pr.yml`. + + - Each pull request should address only one issue, not mix up code from multiple issues. + + - Each commit in the pull request has a meaningful commit message (including the JIRA id) + + - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below. + + +**(The sections below can be removed for hotfixes of typos)** +--> + +## What is the purpose of the change + +*(For example: This pull request adds retry-on-transient-failure to the JDBC sink, so a dropped connection during a checkpoint no longer fails the job.)* + + +## Brief change log + +*(for example:)* + - *`JdbcOutputFormat` retries a batch on a `SQLTransientConnectionException`* + - *A new `JdbcExecutionOptions` field bounds the retry count* + - *A metric counts retried batches* + + +## Verifying this change + +Please make sure both new and modified tests in this PR follow [the conventions for tests defined in our code quality guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing). + +*(Please pick either of the following options)* + +This change is a trivial rework / code cleanup without any test coverage. + +*(or)* + +This change is already covered by existing tests, such as *(please describe tests)*. + +*(or)* + +This change added tests and can be verified as follows: + +*(example:)* + - *Added a unit test in `JdbcOutputFormatTest` for the retry bound* + - *Extended `JdbcExactlyOnceSinkE2eTest` with a connection drop mid-checkpoint* Review Comment: `JdbcExactlyOnceSinkE2eTest` and `*DialectITCase` don't exist. `*DialectTest` does. ########## .github/PULL_REQUEST_TEMPLATE.md: ########## @@ -0,0 +1,92 @@ +<!-- +*Thank you very much for contributing to the Apache Flink JDBC connector - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.* + +*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.* + +## Contribution Checklist + + - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue. + + - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component. + Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`. + + - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review. + + - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. GitHub Actions runs the same build for every push and pull request against the Flink versions and JDKs listed in `.github/workflows/push_pr.yml`. + + - Each pull request should address only one issue, not mix up code from multiple issues. + + - Each commit in the pull request has a meaningful commit message (including the JIRA id) + + - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below. + + +**(The sections below can be removed for hotfixes of typos)** +--> + +## What is the purpose of the change + +*(For example: This pull request adds retry-on-transient-failure to the JDBC sink, so a dropped connection during a checkpoint no longer fails the job.)* + + +## Brief change log + +*(for example:)* + - *`JdbcOutputFormat` retries a batch on a `SQLTransientConnectionException`* + - *A new `JdbcExecutionOptions` field bounds the retry count* + - *A metric counts retried batches* + + +## Verifying this change + +Please make sure both new and modified tests in this PR follow [the conventions for tests defined in our code quality guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing). + +*(Please pick either of the following options)* + +This change is a trivial rework / code cleanup without any test coverage. + +*(or)* + +This change is already covered by existing tests, such as *(please describe tests)*. + +*(or)* + +This change added tests and can be verified as follows: + +*(example:)* + - *Added a unit test in `JdbcOutputFormatTest` for the retry bound* + - *Extended `JdbcExactlyOnceSinkE2eTest` with a connection drop mid-checkpoint* + - *Added dialect coverage in the relevant `*DialectTest` / `*DialectITCase`, exercised against the database via Testcontainers* + +## Does this pull request potentially affect one of the following parts: + + - Dependencies (does it add or upgrade a dependency, including `flink.version` or a database driver version such as `mysql.version` / `postgres.version`): (yes / no) + - The public API, i.e., is any changed class annotated with `@Public(Evolving)` or `@Experimental`, or are the Table options or the JDBC dialect SPI (`JdbcDialect` / `JdbcFactory`) changed: (yes / no) + - Checkpointed state, its serializers, or exactly-once delivery (source splits, enumerator state, writer state, committables, XA transactions): (yes / no / don't know) Review Comment: `backwards_compatibility.yml` runs on every PR and is the gate that catches this, but nothing in the template points at it. Worth naming the module here, neither flink nor Kafka has an equivalent to copy. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) +- Unix-like environment (Linux, macOS, WSL) +- The connector builds against `flink.version` in the root `pom.xml`. A change must also work against every Flink version in `.github/workflows/push_pr.yml` and the `main` row of `.github/workflows/weekly.yml`. + +## Commands + +### Build + +- Build without tests: `mvn clean package -DskipTests` +- Full build with tests: `mvn clean verify` +- Build against another Flink version (what CI does): `mvn clean install -DskipTests -Dflink.version=<version>` +- Single module: `mvn clean install -DskipTests -pl flink-connector-jdbc-core` +- A database module depends on `flink-connector-jdbc-core`; build it with its dependency using `-pl flink-connector-jdbc-mysql -am`. +- Dependency convergence (CI runs this on PRs): `mvn clean install -DskipTests -Pcheck-convergence -Dflink.convergence.phase=install` + +### Testing + +- `*Test` classes are unit tests; `*ITCase` classes are integration tests that start a real database via Testcontainers (Docker required). `flink-connector-jdbc-core` also runs against in-memory Derby and H2, so its unit tests need no Docker. +- Single unit test class: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest` +- Single test method: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest#testX` +- Single ITCase in a database module: `mvn test -pl flink-connector-jdbc-postgres -am -Dtest=PostgresDialectITCase` +- ArchUnit rules live in `flink-connector-jdbc-architecture`; violation stores are frozen (see Testing Standards). +- The standalone `flink-connector-jdbc-backward-compatibility` module is not part of the root reactor; CI builds it separately (`.github/workflows/backwards_compatibility.yml`) to check that current code reads savepoints written by older connector versions. +- CI for PRs and the weekly build is defined by `apache/flink-connector-shared-utils` (`.github/workflows/ci.yml@ci_utils`); its Maven command line, including the license check, is the reference when a local run differs from CI. + +### Code Quality + +- Format code: `mvn spotless:apply` (the `java21` profile sets `spotless.skip=true`, so run it on JDK 11 or 17; google-java-format does not run on JDK 21) +- Check formatting: `mvn spotless:check` +- Checkstyle: `mvn checkstyle:check` (config in `tools/maven/checkstyle.xml`, suppressions in `tools/maven/suppressions.xml`) +- License headers: `mvn apache-rat:check` +- japicmp compares against `japicmp.referenceVersion` from the root `pom.xml` and only checks stable API. + +### Documentation + +- There is no docs build in this repository. The Flink docs build in `apache/flink` clones the release branch of this repository and renders `docs/content` and `docs/content.zh`. +- Documentation exists in English (`docs/content`) and Chinese (`docs/content.zh`); the two carry the same file set. + +## Repository Structure + +### Modules + +The root `pom.xml` reactor contains: + +- `flink-connector-jdbc-architecture` — ArchUnit tests and their frozen violation stores (`archunit-violations/`). +- `flink-connector-jdbc-core` — The connector: the FLIP-27 `JdbcSource`, the Sink V2 `JdbcSink`, the legacy `JdbcInputFormat` / `JdbcRowOutputFormat`, the Table/SQL factory (identifier `jdbc`), the dialect SPI, the XA/exactly-once machinery, and the Derby + H2 in-memory test support. Everything database-agnostic lives here. +- `flink-connector-jdbc-cratedb`, `-db2`, `-mysql`, `-oceanbase`, `-oracle`, `-postgres`, `-sqlserver`, `-trino` — one module per database. Each contributes a `JdbcDialect`, a `JdbcFactory` (dialect SPI), usually a `JdbcCatalog` + type mapper, and a lineage location extractor, discovered via `META-INF/services`. + +Not in the reactor: + +- `flink-connector-jdbc-backward-compatibility` — standalone module, built by its own CI workflow, that verifies savepoint/state compatibility with older connector versions. + +### Supporting directories + +- `docs/content/docs/connectors/` and `docs/content.zh/docs/connectors/` — DataStream and Table docs, English and Chinese, same file set. +- `tools/maven/` — checkstyle config and suppressions. +- `tools/ci/` — the log4j config CI passes to Maven. +- `.github/workflows/` — `push_pr.yml` (PR CI), `weekly.yml` (release branches and Flink snapshots), `backwards_compatibility.yml` (the standalone module). + +### Key packages in `flink-connector-jdbc-core/src/main/java` + +- `org.apache.flink.connector.jdbc` — top-level user-facing option/builder types: `JdbcConnectionOptions`, `JdbcExecutionOptions`, `JdbcExactlyOnceOptions`, `JdbcStatementBuilder`, and the legacy `JdbcInputFormat` / `JdbcRowOutputFormat`. +- `org.apache.flink.connector.jdbc.core.datastream.source` — FLIP-27 source. `@PublicEvolving`: `JdbcSource`, `JdbcSourceBuilder`, `JdbcSourceOptions`. `@Internal`: `enumerator/`, `reader/`, `split/`. Review Comment: These three packages are not `@Internal`. `JdbcSourceSplit`, `CheckpointedOffset`, `ResultExtractor`, `SplitterEnumerator`, `JdbcSqlSplitEnumeratorBase` and `JdbcSourceEnumeratorState` are all `@PublicEvolving`, so an agent reading this would change them freely. `JdbcSourceOptions` has no annotation at all. The path is also `core/datastream/source/split`. ########## AGENTS.md: ########## @@ -0,0 +1,246 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink JDBC Connector AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink JDBC connector codebase. + +## Prerequisites + +- Java 11, 17, or 21. Java 11 is the source baseline: the build compiles with source level 11, so Java 11 syntax must be used everywhere. PR CI runs on JDK 17 and 21 (`.github/workflows/push_pr.yml`); the weekly build additionally covers JDK 8 and 11 across the release branches (`.github/workflows/weekly.yml`). +- Maven 3.8.6. There is no Maven wrapper in this repository; use a system `mvn`. +- Git +- Docker (every `*ITCase` and `*E2eTest` starts its database through Testcontainers) +- Unix-like environment (Linux, macOS, WSL) +- The connector builds against `flink.version` in the root `pom.xml`. A change must also work against every Flink version in `.github/workflows/push_pr.yml` and the `main` row of `.github/workflows/weekly.yml`. + +## Commands + +### Build + +- Build without tests: `mvn clean package -DskipTests` +- Full build with tests: `mvn clean verify` +- Build against another Flink version (what CI does): `mvn clean install -DskipTests -Dflink.version=<version>` +- Single module: `mvn clean install -DskipTests -pl flink-connector-jdbc-core` +- A database module depends on `flink-connector-jdbc-core`; build it with its dependency using `-pl flink-connector-jdbc-mysql -am`. +- Dependency convergence (CI runs this on PRs): `mvn clean install -DskipTests -Pcheck-convergence -Dflink.convergence.phase=install` + +### Testing + +- `*Test` classes are unit tests; `*ITCase` classes are integration tests that start a real database via Testcontainers (Docker required). `flink-connector-jdbc-core` also runs against in-memory Derby and H2, so its unit tests need no Docker. +- Single unit test class: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest` +- Single test method: `mvn test -pl flink-connector-jdbc-core -Dtest=JdbcOutputFormatTest#testX` +- Single ITCase in a database module: `mvn test -pl flink-connector-jdbc-postgres -am -Dtest=PostgresDialectITCase` +- ArchUnit rules live in `flink-connector-jdbc-architecture`; violation stores are frozen (see Testing Standards). +- The standalone `flink-connector-jdbc-backward-compatibility` module is not part of the root reactor; CI builds it separately (`.github/workflows/backwards_compatibility.yml`) to check that current code reads savepoints written by older connector versions. +- CI for PRs and the weekly build is defined by `apache/flink-connector-shared-utils` (`.github/workflows/ci.yml@ci_utils`); its Maven command line, including the license check, is the reference when a local run differs from CI. + +### Code Quality + +- Format code: `mvn spotless:apply` (the `java21` profile sets `spotless.skip=true`, so run it on JDK 11 or 17; google-java-format does not run on JDK 21) +- Check formatting: `mvn spotless:check` +- Checkstyle: `mvn checkstyle:check` (config in `tools/maven/checkstyle.xml`, suppressions in `tools/maven/suppressions.xml`) +- License headers: `mvn apache-rat:check` Review Comment: This fails on a clean main with four unapproved files, so it sends people after pre-existing noise. CI's licence gate is `exec:java@check-license`, a different check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
