OIiveirra opened a new pull request, #68726:
URL: https://github.com/apache/doris/pull/68726
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Repeated queries against a single remotely accessible file split can
repeatedly select the same backend when external file caching and
consistent-hash scheduling are disabled. The round-robin scheduler reconstructs
its candidate list from a host map, losing the query-local candidate order
chosen during initialization.
This change preserves that order for round-robin assignment. It also keeps
local scheduling and split redistribution within the active backend candidates
when load selection retains a preferred subset. Replacing the order is
restricted to round-robin policies.
For unfiltered, ungrouped `COUNT(*)` and `COUNT(1)` over File, HDFS, HTTP,
Local, and S3 TVFs, this change enables the existing file COUNT pushdown under
the existing session switches. DISTINCT, grouped aggregates, filtered plans,
merge stages, and stream/ingest TVFs are excluded. A worker with exactly one
assigned COUNT scan range uses one instance instead of creating empty
local-shuffle instances. Workers with multiple ranges retain the existing
parallelism rules.
Previously saved performance measurements on master baseline `0dcd2a31a7b`,
using three BEs on one host, a single-split CSV containing 1,000,000 rows,
concurrency 64, a 10-second warmup, and a 30-second measurement window,
recorded median QPS of 179.900 before and 561.000 after (3.118x). All six
rounds reported zero errors. These measurements predate the final review fixes
and are workload-specific; the final PR revision has not been benchmarked
again. This result does not establish the same improvement for other queries or
customer deployments.
### Release note
Improve round-robin scheduling for external file scans and reduce execution
overhead for unfiltered file TVF row-count queries with a single assigned scan
range.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Validation performed:
- 19 focused JUnit tests passed across `FederationBackendPolicyTest`,
`FileTvfCountPushDownTest`, and `UnassignedScanSingleRemoteTableJobTest`. They
were executed using a standalone JUnit Launcher with an explicit Byte Buddy
agent and available FE compiled dependencies. The modified scheduling class and
its test were compiled from the final sources. This is not a successful
standard `run-fe-ut.sh` reactor run; that run remains pending because the
shared FE build artifacts were inconsistent.
- FE Checkstyle passed with `mvn -o -pl fe-core -am -DskipTests validate`.
- `git diff --check` passed.
- The S3 compression regression includes new COUNT cases, but execution
and framework-generated `.out` entries are still pending. This PR is a draft
until those results and final-version validation are available.
- Behavior changed:
- [ ] No.
- [x] Yes. Round-robin assignment preserves initialized candidate order;
preferred load subsets are honored by local scheduling and redistribution;
eligible file TVF COUNT scans use pushdown and one instance per worker when
that worker has one assigned range.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
