pzhdfy opened a new pull request, #69493:
URL: https://github.com/apache/doris/pull/69493
## What problem does this PR solve?
Follow-up to #66227, which introduced the paimon-rust reader. This PR
upgrades the Paimon SDK to the community **2.1.0** release and ports the
primary-key vector (ANN) search path onto the community stack: `ORDER BY
l2_distance_approximate|inner_product_approximate(col, vector) LIMIT n` is
rewritten into a pushdown onto the paimon scan and served by the paimon-rust
vector reader on BE.
Related PR: #68988 (for branch-4.1 ， current closed)
**Problem Summary:**
**1. Upgrade Paimon to the community 2.1.0**
- Port FE off the paimon privilege package removed in 2.1.
- The 2.1 `S3FileIO` reads S3 credentials **only** from
`CatalogContext.options()` (verified at bytecode level: `S3FileIO#configure ->
loadHadoopConfigFromContext` never touches the Hadoop `Configuration`, unlike
1.4; unchanged in the community 2.1.0). Doris keeps a catalog's storage
credentials in the Hadoop conf only, so every s3-warehouse paimon catalog
initialized the SDK with no credentials and failed with `Unable to load
credentials`. `mirrorHadoopS3OptionsToCatalogOptions` now mirrors the
`fs.s3a.*` entries onto the catalog options for the FileSystem/HMS/JDBC
metastores, keeping user extras (path style, connection limits) flowing as well.
- The community 2.1.0 **restores reading manifests whose `_FILE` field is
the legacy nullable-union type** (written by paimon versions older than 1.4),
so the pre-provisioned paimon1 CI regression baseline and old production tables
stay readable; the internal 2.1 BETA build rejected them with `expected RECORD
but found UNION`.
- The aggregate-merge overflow behavior **matches 1.4 again** in the
community release (silent wrap, e.g. a TINYINT sum of 127 and 1 reads back as
-128); an intermediate fail-loud behavior only existed in the internal BETA
build — the branch history contains the suite adaptation for it (`1bac394`)
followed by its revert (`1871ef6`), a net zero for reviewers.
- One upstream 2.1.0 behavior change the tests now pin: the persisted
timestamp-to-date conversion for pre-epoch values takes the preceding calendar
date (floor, matching Arrow) instead of truncating toward zero — a TIMESTAMP(3)
of -1ms evolved to DATE reads back as day -1 (UT expectation updated).
**2. PK-vector (ANN) search**
- New rewrite rule `PushDownVectorTopNIntoPaimonScan` (registered in
`Rewriter`) matches a vector distascan and pushes the query vector, the indexed
vector column, the metric and the limit down to the scan.A pre-flight check
refuses incremental scans (`t@incr`), so an incremental query can never
silently resolve to a snapshot read.
- `AnnTopNInfo` carries the pushed-down info from `LogicalFileScan` to
`PhysicalFileScan` and onto `PaimonScanNode` (`PhysicalPlanTranslator` sets it
only inside the `PAIMON_EXTERNAL_TABLE` branch); the `PlanNodes.thrift`
additions are additive fields.
- BE: `paimon_rust_table_reader` grows the vector-search path, reading
through the rust PK-vector build-op on this path (the rust vector readers do
not consume it) — noted in the reader.
- The pushdown reuses the existing `enable_paimon_rust_reader` and
`enable_file_scanner_v2` switches; with either off, the query brute-forces
through the normal scan.
**3. Unify the rust eligibility gates**
- `PaimonRustEligibility` evaluates the 12 table-level/storage/BE checks
once and is shared by the batch and vector paths. The vector gate is now
computed independently of the batch gate (it previously inherited
`fallbackRead` / `scan.mode` restrictions from the batch path) and both
branches share a common gate chain with a batch-only split-suitability suffix.
- Compaction-merge splits that are raw-convertible skip the merge-reader
gates.
- JNI fallback reasons are logged so a silent downgrade to the JNI reader is
diagnosable.
**Tests**
- FE UT: new `PushDownVectorTopNIntoPaimonScanTest` (positive /
incremental-refused / gate-negative coverage), `PaimonScanNodeTest` gate
coverage, planner `PaimonPredicateConverter` tests, `PaimonUtilTest`; 485
paimon-related FE UTs pass against the 2.1.0 stack.
- New end-to-end regression `test_paimon_pk_vector_ann`: the suite builds
its own fixture through the Spark ThriftServer — an ivf-rq PK-vector index with
`nlist = nprobe = 4` on 512 deterministic 16-dim vectors, i.e. **exhaustive
recall, so ANN must equal brute force by construction**. It compares against
the brute-force answer comsabled, asserts the ANN leg really executed on the
paimon-rust vector reader (profile check against asilent brute-force pass), and
covers both `l2` and `inner_product` metrics, residual filters and non-index
projections.
- Fixed two regression fixtures broken by the 2.1 Spark stack: the
creation-time-millis helper was shifted by the session UTC offset (every
`scan.creation-time-millis` leg silently resolved to the latest snapshot;
Doris-side resolution verified correct by hand-reproducing the queries), and
`assertSparkDorisResultEquals` now decodes the `0x`-hex tokens Doris renders
for VARBINARY nested in arrays/maps/structs so identical nested binary values
compare equal across engines.
- Adapted two write suites to observable 2.1 changes (oversized-row
rejection message gained a qualifier; unset table-property defaults are no
longer materialized by `SHOW TBLPROPERTIES`) — the underlying behaviors are
unchanged.
- Swept the locally runnable `external_table_p0/paimon` regression suites
against the 2.1.0 stack: read-path 34/37 and write-path 26/32 green (the rest
need a Spark 4.0 ThriftServer for VARIANT DDL or pre-provisioned fixtures that
exist only in CI), plus the HDFS-based suites against the original
pre-provisioned paimon1 OSS baseline — no Doris-side regressions.
## Release note
The Paimon catalog now runs on the community Paimon 2.1.0 release. Queries
of the form `ORDER BY l2_distance_approximate|inner_product_approximate(<vector
col>, <query vector>) LIMIT n` over paimon tables with a PK-vector index are
pushed down into the paimon-rust vector reader (requires
`enable_paimon_rust_reader` and `enable_file_sy paimon versions older than 1.4
(legacy union manifests) remain readable.
## Check List (For Author)
- Test
- [x] Unit test
- [x] Regression test
- [x] Manual test (add the script if necessary)
- [ ] No need to test
- [ ] Behavior changed: yes (the new switch-gated ANN pushdown; paimon-side:
pre-epoch timestamp-to-dat
- [x] Does this need documentation?
## Check List (For Reviewer who merge this PR)
- [ ] Does the PR have a corresponding release note?
- [ ] Is the release note filled in correctly?
- [ ] Are the test cases added or updated correctly?
- [ ] Is the document updated (if necessary)?
- [ ] Does this PR need to be picked to other branches? (branch-4.2 only:
**yes**)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
