xylaaaaa opened a new pull request, #70536: URL: https://github.com/apache/doris/pull/70536
### Background Apache Doris PR #69904 adds the Delta Lake reader for the latest table state. Delta records each committed change as a numbered version of its transaction log. Reading an older version requires using that version for column binding, file selection and deletion vectors. ### What problem does this PR solve? Issue Number: #69884 Related PR: #69904 (required base, merge first) PR Type: Feature Problem Summary: Allow an ordinary Delta table reference to select a numeric version using `FOR VERSION AS OF`. This is a stacked draft based on `e321ede256` from #69904. GitHub's Files changed tab includes the base PR until it merges. [Review only the changes for version reads](https://github.com/xylaaaaa/doris/compare/e321ede256aef248a3541a9a9d2302f978ac87b0...codex/delta-version-as-of). ### Details - **Change map:** The Delta FE connector accounts for 65 added and 8 removed production lines. Unit tests and fixtures add 136 lines and remove 2. The Spark comparison adds 172 test lines and the framework generated 91 output lines. - **Scope:** Numeric versions, including zero, through Delta filesystem, Delta HMS and the HMS gateway. - **Design:** Load the requested Kernel snapshot, verify the target table UUID and reader features, and retain its schema. Reuse the checked handle for that table and version within the query. The file scan, projected columns and exact COUNT operate on the same version. - **Usage:** `SELECT * FROM catalog.db.t FOR VERSION AS OF 3`. - **Default behavior:** Ordinary reads continue to use the latest snapshot and the configured latest cache. - **Compatibility:** The table's current protocol must be readable when opening its handle. An unavailable log version fails explicitly. Timestamp selection and the history system table remain open in #69884. The HMS gateway still uses its existing statistics delegation for CBO estimates. Kernel reads a version while its required log and data files are available. Spark Delta 4.4.1 adds a separate, default-on age guard based on deleted-file retention, which can reject readable data. [Spark implementation](https://github.com/delta-io/delta/blob/v4.4.1/spark/src/main/scala/org/apache/spark/sql/delta/SnapshotManagement.scala#L1656-L1689) ### Testing - Full local macOS arm64 ASAN build with `FE_MAVEN_THREADS=4 ./build.sh --fe --be -j10`. Started the resulting FE and BE and verified a distributed query. - `bash run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test'` passed. The Delta connector and Delta metastore report 106 tests executed with zero failures, including all six new history tests. Two opt-in diagnostics were skipped. - Generated the golden output for `test_deltalake_version_as_of` via `run-regression-test.sh` and reran the unchanged suite without `-genOut`. Both runs passed with Spark 4.0.0, Delta 4.4.1, Hive Metastore and MinIO. All three catalog routes matched Spark on historical columns, types, ordered rows, counts, column mapping by name/ID, a physical deletion vector and multi-version joins. The new test exercises synchronous and streaming scheduling with the existing default threshold of 1. - The existing `test_deltalake_hive` passed. Two existing suites (`test_deltalake_basic` and `test_deltalake_parquet_deletion_vector`) expect an exact split count in EXPLAIN. At the default streaming threshold their assertions fail on the estimate inherited from #69904. Both original suites pass with batch scheduling disabled on their own regression connection. ### Release note Delta Lake tables can be queried at a numeric version with `FOR VERSION AS OF`, including the schema and deletion vectors recorded at that version. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes, to accompany the Delta reader website documentation tracked in #69884. - Pre-submission status: Locally verified for the new feature. Draft pending #69904. Two pre-existing streaming EXPLAIN expectations are detailed in Testing. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Implementation, tests and PR description with AI assistance. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
