jiangxt2 opened a new pull request, #12902: URL: https://github.com/apache/gravitino/pull/12902
### What changes were proposed in this pull request? This pull request adds an opt-in governed Apache Doris batch read and write adapter for `jdbc-doris` catalogs in the Spark 3.5 connector on Scala 2.12. The adapter preserves Gravitino authorization and credential vending, validates the Gravitino logical schema against Doris FE and JDBC physical metadata, and selects either the official Doris tablet reader or Spark JDBC V2 according to the query semantics. Predicates over normalized Doris columns remain Spark residuals with Spark-visible String semantics, and unsupported schema or planner combinations fail closed. Governed writes support batch append and a separately enabled full-table truncate overwrite. They require `MODIFY_TABLE` authorization before physical Doris access, validate the exact writable schema, delegate to the official Doris Stream Load writer, force 2PC and strict safety options, and reject streaming, predicate overwrite, dynamic overwrite, row-level operations, and Spark catalog DDL. The change also adds the Doris Server Catalog property contract, strict vended-credential handling for specialized mode, a Spark version gate, a documented Doris release boundary, compile-only external connector dependencies, documentation, unit tests, and real Doris integration tests. ### Why are the changes needed? The existing generic JDBC fallback does not provide a Doris-specific contract for tablet reads, JDBC-lossy Doris types, physical-schema drift, mixed native/JDBC pushdown semantics, or governed Stream Load writes. This change provides that contract without changing the default `jdbc-doris` behavior or the behavior of MySQL, PostgreSQL, and other JDBC providers. Fix: #12846 ### Does this PR introduce _any_ user-facing change? Yes. * `spark.sql.gravitino.enableDorisSupport=true` opts into the specialized adapter; the default remains `false`, which preserves generic JDBC behavior. * Specialized mode supports Spark 3.5.3 or later in the 3.5 line with Scala 2.12. The certified matrix is documented for Spark 3.5.3/Doris 3.0.6.2 and Spark 3.5.9/Doris 4.0.6, plus Spark 3.5.9/Doris 3.0.6.2 standalone deployment coverage. Other Doris releases are not certified and do not automatically fall back; users must leave the flag disabled to select generic JDBC for them. * Specialized mode requires catalog-managed `jdbc-url`, `jdbc-driver`, `doris-fenodes`, and `doris-query-port`, plus a vended `JdbcCredential`; configured `jdbc-user` and `jdbc-password` are not accepted as specialized Spark connection credentials. * `doris-write-mode=batch` enables governed append, and `doris-write-overwrite-mode=truncate` separately enables non-atomic full-table truncate overwrite. Both remain disabled by default. * The Server Catalog registers the optional SQL-lane properties `doris-jdbc-partition-column`, `doris-jdbc-lower-bound`, `doris-jdbc-upper-bound`, `doris-jdbc-num-partitions`, and `doris-jdbc-fetch-size` and validates the positive-integer fields. Specialized Spark catalog initialization additionally requires all four partition options to be present together and revalidates positive-integer values. * The official Doris Spark Connector 26.0.0 and MySQL Connector/J remain external runtime dependencies and must be available to the Spark driver and executors. ### How was this patch tested? Post-rebase checks on the submitted head: * `./gradlew :catalogs:catalog-jdbc-doris:test --tests org.apache.gravitino.catalog.doris.TestDorisCatalogPropertiesMetadata -PskipITs` — passed. * `./gradlew :spark-connector:spark-3.5:test --tests 'org.apache.gravitino.spark.connector.jdbc.doris.Test*' --tests org.apache.gravitino.spark.connector.plugin.TestDorisSparkVersionGate35 --tests org.apache.gravitino.spark.connector.plugin.TestGravitinoDriverPlugin --tests org.apache.gravitino.spark.connector.catalog.TestBaseCatalogAuthorization --tests org.apache.gravitino.spark.connector.version.TestCatalogNameAdaptor -PskipITs` — passed. * `./gradlew spotlessCheck rat` — passed. * `git diff --check origin/main...HEAD` — passed. Real Doris evidence collected before the final rebase: * Spark 3.5.3 + Doris 3.0.6.2 embedded: `SparkJdbcDorisCatalogIT35` 23 tests and `SparkJdbcDorisLoadPrivilegeIT35` 3 tests passed, with two environment-mode skips and no failures or errors. * Spark 3.5.9 + Doris 4.0.6 embedded: `SparkJdbcDorisCatalogIT35` 23 tests and `SparkJdbcDorisLoadPrivilegeIT35` 3 tests passed, with two environment-mode skips and no failures or errors. * Spark 3.5.3 + Doris 3.0.6.2 and Spark 3.5.9 + Doris 4.0.6: `SparkJdbcDorisAuthorizationIT35` passed once on each combination with no failures, errors, or skips. * Spark 3.5.9 + Doris 3.0.6.2 standalone: `SparkJdbcDorisCatalogIT35` passed 23 tests against two active one-core workers with the external Gravitino, Doris, and MySQL JARs, with one deploy-only skip and no failures or errors. The final rebase integrated upstream changes in shared Spark test configuration and documentation without changing the Doris production behavior. The non-Docker checks above were rerun on the submitted head; the real Doris matrix was not rerun after that rebase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
