morningman opened a new pull request, #69227:
URL: https://github.com/apache/doris/pull/69227
### What problem does this PR solve?
Issue Number: None
Related PR: #66466, #68242
Problem Summary:
**In short.** Ports the primary-key Variant coverage of branch-4.1 #66466 to
master: a Paimon deduplicate table whose second snapshot updates two keys, and
three blocks of `test_paimon_catalog_variant` that read it through JNI. The
rest of #66466 and the WAL fix #68242 are not ported, because master handles
those cases another way; see below.
**Background.** #66466 did three things on branch-4.x:
1. In BE, a generic gate in `TableReader::validate_file_mapping` rejects a
physical Variant read from any file format but Parquet. #68242 then exempted
WAL, which stores Doris blocks and supports Variant, because the gate broke
group commit WAL replay.
2. In FE, `PaimonScanNode` rejects a query that projects a Variant column
while Variant V2 is disabled.
3. In the tests, `run13.sql` gets `variant_primary_key_smoke`, and
`test_paimon_catalog_variant` reads it.
**The problem, and what it cost.** Every Variant table master's
`test_paimon_catalog_variant` reads is append-only, so the regression does not
check that a Paimon merge read keeps only the latest row of each key while it
materializes Variant values.
**How this PR fixes it.**
- `run13.sql` creates `variant_primary_key_smoke` as on branch-4.x: a
deduplicate primary-key table in Parquet, written in two snapshots. The second
updates ids 1 and 2 and inserts id 4; id 3 holds a root array.
- The JNI section of `test_paimon_catalog_variant` gets the three branch-4.x
blocks: the whole Variant value, Variant paths cast to scalars, and a predicate
on a Variant path. All three expect the latest value of ids 1 and 2.
The other parts of #66466 and #68242 do not apply to master:
| Branch-4.x change | Master |
|---|---|
| FE rejects a Variant projection while Variant V2 is disabled | #66858 made
Variant V2 the default and removed the switch, so there is nothing to check |
| BE rejects a physical Variant read from any format but Parquet | Would
reject the native Variant reads of the ADBC reader (#68883). Master keeps the
Iceberg-specific gate in `IcebergTableReader`, admits Arrow Variant reads in
`FileScannerV2` only for ADBC, and plans a Paimon split that projects a Variant
natively only when every file is Parquet or an ORC file whose schema predates
the Variant column. Paimon 1.4.2 cannot create an ORC table with a VARIANT
column (`Unsupported type: VARIANT`) |
| WAL is exempted from that gate | WAL is not gated on master |
**Results.** On a local cluster against the Iceberg docker environment,
Spark reads `variant_primary_key_smoke` as four rows, with the second-snapshot
values for ids 1 and 2. Doris returns the same rows in the three new blocks,
and every other result block of the suite is unchanged.
Original author: @zhangstar333 (#66466)
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- `test_paimon_catalog_variant` on a local cluster. The local BE
predates #69016, so the suite's profile check fails there; every result block
passes, including the new ones. The `.out` was generated by the script from a
copy of the suite without that check, which writes no output; the only change
is the three new blocks.
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
