morningman opened a new pull request, #69224:
URL: https://github.com/apache/doris/pull/69224
### What problem does this PR solve?
Issue Number: None
Related PR: #66346, #66536, #67023
Problem Summary:
**In short.** Ports the docker part of branch-4.1 #66346 and #66536 to
master. The Spark of the Iceberg docker environment, which creates the Paimon
fixtures and reads back what Doris writes to Paimon, moves from Paimon 1.3.1 to
1.4.2, the version Doris itself has used since #67395. The Paimon fixture
script gets the change Paimon 1.4.2 needs, `test_paimon_write_external_paths`
gets back its weight-robin case, and the file sizes in
`test_paimon_partition_table.out` follow the new fixture.
**Background.** #66346 upgraded Paimon to 1.4.2 on both sides of the
regression tests: the Doris FE/BE dependency, and the `paimon-spark` /
`paimon-s3` jars the docker Spark loads, shipped in the fixture package
`iceberg_data_spark40_paimon142.zip`. Master took the dependency part with
#67395 but kept the docker environment on `iceberg_data_spark40.zip` and Paimon
1.3.1. #66536 followed the upgrade on branch-4.x: Paimon 1.4.2 parses a present
Variant shredding schema property as JSON, so the fixture can no longer turn
shredding off by setting the property to an empty string.
**The problem, and what it cost.**
- The Paimon suites check Doris against fixtures and read-backs from an
older Paimon than the one Doris runs, so a difference between the two versions
does not show up in CI.
- `test_paimon_write_external_paths` dropped its weight-robin case when it
was ported (#68858). Paimon 1.3.1 does not know that strategy, which Paimon 1.4
introduced: its `ExternalPathStrategy` accepts only `none`, `specific-fs` and
`round-robin`, and Spark fails on such a table with `Could not parse value
'weight-robin' for key 'data-file.external-paths.strategy'`. Doris writes to
weight-robin tables are not tested.
- The jars cannot be upgraded on their own. Under Paimon 1.4.2, `run13.sql`
sets `parquet.variant.shreddingSchema` to `''` and then inserts, and that
insert fails with `MismatchedInputException: No content to map due to
end-of-input`. `variant_mixed_su` keeps only its shredded row, and the Spark
entrypoint, which runs under `set -e`, stops there.
**How this PR fixes it.**
- `run-thirdparties-docker.sh` downloads
`iceberg_data_spark40_paimon142.zip`, the package branch-4.x uses. Compared
with `iceberg_data_spark40.zip`, it replaces `paimon-s3-1.3.1.jar` and
`paimon-spark-4.0-1.3.1.jar` with `paimon-s3-1.4.2.jar` and
`paimon-spark-4.0_2.13-1.4.2.jar`; the other 3881 entries have the same size
and CRC. The Hong Kong and Beijing buckets both serve it, with the same ETag.
- `iceberg.yaml.tpl` mounts the two 1.4.2 jars into the Spark container.
- `run13.sql` removes the property with `UNSET TBLPROPERTIES` before writing
the unshredded file of `variant_mixed_su` (#66536), so the table still has one
shredded and one unshredded file. The other rows branch-4.x added to
`run13.sql`, a primary-key Variant table, come with a separate change and are
not part of this PR.
- `test_paimon_write_external_paths` creates the weight-robin table again,
writes six rows from Doris in separate statements, checks that every data file
is under one of the two weighted paths, and compares the rows Doris and Spark
read, as in branch-4.x (#67023).
- `test_paimon_partition_table.out` is regenerated. Paimon 1.4.2 writes
smaller Parquet files for the `run07.sql` tables, so the file size column of
`SHOW PARTITIONS` changes, for example 2216 to 2100 bytes; partitions, row
counts and file counts do not. The 19 new sizes equal those in the branch-4.x
copy of the file, which was generated with the same package (#66346 made the
same change there).
**Results.** On a local single-BE cluster, with the Iceberg docker
environment rebuilt from scratch by the script (it downloaded and unpacked the
new package):
| | Before (Paimon 1.3.1 in docker Spark) | After (Paimon 1.4.2) |
|---|---|---|
| Spark container start | Succeeds | Succeeds; every fixture script runs,
`variant_mixed_su` has both rows |
| Same start with only the jars upgraded | — | Fails at the `run13.sql`
insert after the empty shredding property |
| Weight-robin table | Spark fails: `Could not parse value 'weight-robin'` |
Created and read by Spark; Doris writes six rows to the weighted paths |
| `SHOW PARTITIONS` on the `run07.sql` tables | Sizes as in master's `.out`
| Sizes as in branch-4.x's `.out` |
Of the 94 suites under `external_table_p0/paimon`, every suite that can run
on this machine passes with the new package: the 72 that passed at once, plus
`test_paimon_write_external_paths` and `test_paimon_partition_table` with the
regenerated `.out` files, and `paimon_time_travel`, which converts commit times
with the JVM default time zone and needs `TZ=Asia/Shanghai` locally. The others
need what this machine does not run: the Hive docker environment (HDFS or HMS,
12 suites), cloud credentials (3), or Linux `/proc` (1). Three suites failed
because the local FE/BE was not a master build. For two of them the checks that
touch the new Spark pass: `test_paimon_catalog_variant`, including its
`variant_mixed_su` blocks, without the profile check that needs counters this
BE lacks, and the version of `test_paimon_write_variant_errors` that matches
this FE, which reads back Doris writes with Spark. The third,
`test_paimon_alter_properties`, does not use Spark.
Original author: @suxiaogang223 (#66346, #67023), @Gabriel39 (#66536)
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- The suites under `external_table_p0/paimon` on a local cluster against
the rebuilt Iceberg docker environment, as above. The suites that need the Hive
docker environment, cloud credentials or Linux were not run locally.
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
