github-actions[bot] commented on code in PR #66065: URL: https://github.com/apache/doris/pull/66065#discussion_r3652327623
########## regression-test/suites/external_table_p0/PAIMON_ICEBERG_READ_WRITE_P0_COVERAGE.md: ########## @@ -0,0 +1,102 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Paimon / Iceberg Read and Write P0 Coverage + +## Scope + +This matrix compares the supported Doris surface with the format capabilities documented by +[Apache Paimon](https://paimon.apache.org/docs/1.0/), its +[ecosystem matrix](https://paimon.apache.org/docs/master/ecosystem/), and Iceberg's +[write](https://iceberg.apache.org/docs/latest/spark-writes/) and +[evolution](https://iceberg.apache.org/docs/latest/evolution/) documentation. A format feature is a Doris P0 contract +only when Doris supports it. Unsupported write or format boundaries require deterministic negative +coverage and no metadata or data mutation. + +Doris currently documents Paimon data access as read-only. Master also supports Paimon catalog, +database and table metadata creation/deletion, but it has no Paimon data sink. Iceberg supports read, +DDL, INSERT, INSERT OVERWRITE, CTAS and, for compatible V2/V3 tables, DELETE, UPDATE and MERGE INTO. + +## Audit result + +The master inventory before this change contained 41 Paimon suites and 145 Iceberg suites, including +35 Iceberg write suites and four dedicated Iceberg DML suites. Iceberg P0 is complete for the Doris- +supported read/write surface: its existing matrices cover file-format versions, delete encodings, +evolution, time travel, DDL, distributed writes, row-level DML, failure atomicity and unsupported +boundaries. + +Paimon P0 was incomplete. No suite selected a `merge-engine`, and write rejection was not checked +across all DML shapes with pre/post data and snapshot invariants. PM01-PM03 below close those gaps. +PM04 exposes one remaining product defect as an opt-in negative regression: failed Paimon CTAS leaves +target metadata. Streaming writes, overwrite, delete/update and merge listed as unsupported by the +Paimon ecosystem matrix are boundary tests rather than positive Doris P0 contracts. + +## Risks + +| ID | Risk | Source | Impact | Priority | +| --- | --- | --- | --- | --- | +| R01 | Append and primary-key tables are planned with the wrong merge semantics | Black box: Paimon table models | Silent wrong results | P0 | +| R02 | Deduplicate, partial-update, aggregation and first-row produce the same result for duplicate keys | White box: LSM sorted-run merge | Silent wrong results | P0 | +| R03 | Fixed and dynamic bucket tables expose duplicate or stale cross-partition keys | Paimon data distribution | Silent wrong results | P0 | +| R04 | Automatic and forced-JNI routing disagree on Paimon merged rows, or native raw-file reads disagree where conversion is supported | Doris split routing | Query correctness | P0 | +| R05 | A rejected Paimon write creates a snapshot, changes data or leaves a CTAS table | Doris read-only boundary | Data or metadata mutation | P0 | +| R06 | Iceberg schema or partition evolution binds old field/spec IDs during current or historical reads | Iceberg evolution | Silent wrong results | P0 | +| R07 | Position/equality deletes or V3 deletion vectors are applied to the wrong file or snapshot | Iceberg row-level deletes | Deleted data visible or live data lost | P0 | +| R08 | Iceberg writes lose rows, route them to the wrong transform, or publish partial failed commits | Doris distributed Iceberg sink | Data loss or corruption | P0 | +| R09 | Snapshot, tag or branch reads/writes leak schema or data across references | Both formats | Historical data corruption | P0 | +| R10 | Unsupported format, DML mode or CTAS mutates state before rejection | Capability boundary | Partial commits or orphan metadata | P0 | + +## Feature matrix + +| Format | Capability | P0 status | Main suites | +| --- | --- | --- | --- | +| Paimon | Append table, partitioned table, primitive and nested types | Covered | `test_paimon_catalog`, `test_paimon_partition_table`, `test_paimon_full_schema_change` | +| Paimon | Primary-key deduplicate, partial-update, aggregation and first-row | Covered | `test_paimon_merge_engine_matrix` | +| Paimon | Fixed bucket, dynamic bucket and cross-partition update | Covered | `test_paimon_merge_engine_matrix`, `test_paimon_partition_pk_delete_refs` | +| Paimon | Parquet/ORC and mixed-format reads; JNI/native parity | Covered | `test_paimon_merge_engine_matrix`, `paimon_tb_mix_format`, `test_paimon_cpp_reader` | Review Comment: [P2] Cite the suite that actually verifies JNI/native routing `test_paimon_cpp_reader` does not establish this row's parity on the current FE. Logical splits are explicitly tagged `PAIMON_JNI`, so BE ignores `enable_paimon_cpp_reader`; raw splits auto-route native in both halves. The false/true toggle therefore compares the same per-split routing and cannot prove a second reader. `test_paimon_partition_pk_delete_refs` already asserts JNI-only versus positive-native plans (including `SplitStat` types) for both Parquet and ORC. Please cite that explicit proof here, or add equivalent route assertions to the listed matrix/mixed-format suites instead of relying on the stale CPP test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
