morningman opened a new pull request, #66469:
URL: https://github.com/apache/doris/pull/66469
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #66468 (the branch-4.0 twin of this PR), #64945 / #65209
(existing master plugin wait behavior this builds on)
Problem Summary:
Forward-port of the fixes from a chronically red branch-4.0 P2 regression
pipeline triage
(7 failures; no BE crash/OOM — all case-level). Every root cause below also
exists on master
(1 FE product bug + 4 test/CI fixes). The branch-4.0 twin is #66468; the
#64822 backport included
there is not needed here.
1. **[load] Broker load pending-task retry self-conflicts on its own label**
(FE product bug).
`BrokerLoadPendingTask.executeTask()` runs `getAllFileStatus()` then
`beginTxn()`, and
`LoadTask.exec` retries the whole task on failure (retryTime=3). If the
first attempt
registers the txn but throws before `transactionId` is assigned (observed:
`beginTransaction` failing on the edit-log write when the FE meta disk
hit ENOSPC), every
retry hits `LabelAlreadyUsedException` against the job's OWN PREPARE txn,
burns all retries
within seconds and cancels the job with a misleading
`Label [...] has already been used, relate to txn [...], status
[PREPARE]`.
Fix: make `beginTxn()` retry-idempotent — reuse an already-assigned
`transactionId`, and on
`LabelAlreadyUsedException` adopt the label's txn iff it is ours
(`callbackId == job id` and
status PREPARE). Foreign conflicts and lookup failures still rethrow the
original exception.
Adds 3 Mockito unit tests.
2. **plugin_compaction.groovy**: treat base compaction's `E-808`
(`BE_NO_SUITABLE_VERSION`, the
by-design "nothing to base-compact" result from
`BaseCompaction::pick_rowsets_to_compact`)
as benign, exactly like cumulative's E-2000/E-2010. A lagging publish can
legitimately make
one replica's cumulative trigger a no-op (E-2000, already ignored),
leaving its cumulative
point behind so the later base trigger deterministically has nothing to
merge on that
replica — observed killing `test_base_compaction_no_value` 7/11 recent
branch-4.0 P2 runs.
Suites still verify the compaction effect via their own
rowset/segment-count asserts.
3. **compaction_width_array_column**: with `BUCKETS 2` the fixture loads a
~56GB 197-segment
overlapping rowset into a single tablet; compaction writes the full
output on the same
mount before deleting the input, which structurally ENOSPCs on 100GB CI
data disks
(11/11 recent branch-4.0 runs red; ASAN runs hit MEM_LIMIT for the same
oversize). Bump to
`BUCKETS 16` (~7GB/tablet) and raise the load to 16G exec_mem_limit /
load_parallelism 1
(the load phase was exceeding the default 8G with a ~12G peak).
4. **inverted_index_p2/test_show_data**: all four sub-suites exact-compare
physical index
sizes produced by different writer paths (inline-at-load vs ALTER+BUILD
INDEX vs
index-compaction merge vs rebuild). Settle-sequence analysis shows the
values are stable
per run yet differ by a fixed per-index-file overhead (~7KB per replica,
12-15% relative on
the httplogs fixture), so exact equality flakes chronically (6/11, 6/11,
3/11, 4/11 recent
branch-4.0 runs across the four sub-suites). Replace the five exact
compares with 20%
tolerance — a missing index still shows as a ~37% deficit and real bloat
as 2x+.
Pure-data (`no_index_size`) exact compares are kept.
5. **cold_heat_separation_p2/table_modify_resouce_and_policy(+_by_hdfs)**:
both phases waited
only for `tablets[0]` to finish cooldown, then asserted `remote_data_size
> 0` for EVERY
replica row. Replicated cooldown is
leader-uploads/followers-follow-async, so a lagging
follower failed the assert with zero grace (7/11 recent branch-4.0 runs).
Make the wait
cover all replica rows (local==0 && remote>0) with the same 100x10s
budget.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. Broker load jobs whose pending task is retried after the
job's txn was
already begun/registered now reuse (or adopt) that txn instead of
being cancelled
with a misleading "Label has already been used" error. No behavior
change outside
that failure path.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
Verification: `mvn test -pl fe-core -Dtest=BrokerLoadJobTest` (build cache
disabled) green
including the 3 new tests, 0 checkstyle violations; all changed groovy files
pass offline
`FileSystemCompiler` syntax check; test_show_data.groovy and
compaction_width_array_column.groovy
end up byte-identical to their fixed branch-4.0 (#66468) versions.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
