Zerolzj opened a new pull request, #10662: URL: https://github.com/apache/paimon/pull/10662
### Purpose Related to #5436 and the identifier-based extension discussion in https://github.com/apache/paimon/issues/5436#issuecomment-5550987251. This is a **draft design/implementation proposal**, not an assertion that a provider SPI has already been agreed upon. Please advise whether a PIP is needed before considering this API. Some existing fixed-bucket layouts use deterministic bucket algorithms other than `default`, `mod`, or `hive`. Adopting such files without reshuffling also requires subsequent writes and bucket-aware reads to agree on the same algorithm. This patch keeps the algorithm implementation outside Core while preserving the existing built-in behavior. The Java proposal: - Add an immutable `bucket-function.provider` semantic identifier, discovered through the existing `FactoryUtil`/`Factory` mechanism. No implementation class name is stored in table metadata. - Give the provider precedence over `bucket-function.type`, for fixed positive bucket counts only. Dynamic, unaware and postpone bucket modes are deliberately excluded. - Route schema-based writer extraction, entry-level pruning and the current manifest bucket-range pruning through the configured function. - Fail rather than silently use the default algorithm when a bucket-computing operation cannot resolve its provider. - Prevent Spark provider tables from taking default-algorithm-only V2/SQL write paths or advertising default bucket partitioning/ordering. Preserve the existing default optimization and Spark's explicit data-grouping policy. - Retain the enum-based constructor overloads for compatibility. - Document identifier stability and the function's key/range/lifecycle contract. Keep implementation fixtures generic. Example table options: ``` bucket = 4 bucket-function.provider = example-custom-bucket ``` The provider JAR must be installed in each participating Java runtime. Existing identifiers must retain the same semantics; changing an algorithm requires a new identifier. #### Questions for maintainers 1. Is an identifier-based third-party factory appropriate, or should additional algorithms remain built-in only? 2. Is a separate immutable provider option appropriate, or would an extensible `bucket-function.type` be preferable? 3. Should the first implementation require matching providers everywhere, or define an explicit full-scan-only fallback for runtimes without one? 4. What cross-runtime compatibility contract should precede support in other languages? #### Scope and limitations This proposal implements the Java/Core/Spark side. It **does not add Python, Rust or C++ providers**, and does not claim compatibility with runtimes that do not recognize the option. Such runtimes must not compute buckets for these tables; deployment/version gating remains a requirement for adopters. It does not enable custom-provider shuffle-free joins, change file formats, import existing files, or implement a new bucket algorithm. ### Tests Based on community master `53b77402c`, with JDK 17: - Core: `BucketFunctionProviderTest`, `BucketSelectorTest`, `FixedBucketRowKeyExtractorTest`, `SchemaValidationTest`: 96 tests passed. - A missing-provider regression was first run against the unchanged implementation and failed because it silently fell back to the built-in function; it passes with the SPI. - Provider pruning tests cover both individual buckets and manifest bucket ranges. - Spark 3.5.8: `CustomBucketFunctionTest`, `V2WriteRequireDistributionTest`, `BucketedTableQueryTest`: 17 tests passed. - Spark 3.3.4: `CustomBucketFunction33Test`: 8 tests passed. - Spark scan tests explicitly enable the current `scan.preserve-data-grouping` policy and keep a positive built-in/default control. Commands: ```bash mvn -pl paimon-core -am -DskipTests package mvn -pl paimon-core -am \ -Dtest=BucketFunctionProviderTest,BucketSelectorTest,FixedBucketRowKeyExtractorTest,SchemaValidationTest \ -DfailIfNoTests=false -DwildcardSuites=none test mvn -Pspark3 -pl paimon-spark/paimon-spark-3.3 -am \ -Dtest=none -DfailIfNoTests=false \ -DwildcardSuites=org.apache.paimon.spark.sql.CustomBucketFunctionTest,org.apache.paimon.spark.sql.CustomBucketFunction33Test,org.apache.paimon.spark.sql.V2WriteRequireDistributionTest,org.apache.paimon.spark.sql.BucketedTableQueryTest test ``` Normal formatting/checkstyle checks are enabled. This does not claim the full project test suite, other engine matrices, or cross-language conformance have been validated. One commit, 21 files, 501 insertions and 16 deletions. No release/version/deployment changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
