Jeff-Shen-00 opened a new pull request, #70334:
URL: https://github.com/apache/doris/pull/70334
### Background
`tools/tpcds-tools/bin/load-tpcds-data.sh` streams a generated TPC-DS data
set into Doris, issuing one `curl` Stream Load per `.dat` file. `-c N` bounds
how many of them run at once with a FIFO on fd 3 that holds N tokens. Until
now a token was taken once per table, before a background subshell that
walked that table's files one by one.
### What problem does this PR solve?
Issue Number: close #70315
Related PR: None
PR Type: Enhancement
Problem Summary:
A table generated with many chunks is loaded strictly one file at a time
while tokens held for other tables sit idle, so the tail of a load is serial
no matter how large `-c` is. Each file now takes its own token, so `-c N`
means up to N files in flight across all tables. Result lines still name each
file, and `-x` labels stay unique per file.
### Details
- **Before and after:** before, `-c N` allowed up to N tables to be in
flight, with each table's files loaded sequentially inside its subshell,
so the files of one table could never overlap with each other. After, every
file is its own background subshell and `-c N` allows up to N files in
flight across all tables. The token FIFO, its setup, and the final `wait`
are unchanged.
- **Configs and defaults:** `-c` keeps its name and default of 5; only its
unit changes from tables to files, and the help text now says so. `-x` is
unchanged: the label is still `${TXN_ID}_${FILE_ID}`, which is unique per
file.
- **Impact on existing users:** the same Stream Load requests carry the same
columns, separator, and label; only the concurrency granularity, and
therefore the completion order and wall time, change. Two robustness fixes
come with the move. The token is now returned from an `EXIT` trap: with
file-level tokens a `curl` that exits non-zero terminates the subshell
under `set -e` before it can return its token, and once `PARALLEL` tokens
leak the parent blocks on the read forever (the old script hung the same
way once one failed file took out a whole table). And a `curl` that exits
non-zero is now reported as a failed file instead of aborting the subshell
silently.
### Testing
The script was copied to an isolated tree with stub `curl`, `jq`, and `mysql`
executables; the stub `curl` records the in-flight count and can be told to
fail one file. No Doris cluster was started, so Stream Load responses were
stubbed, but the script's own token protocol, globbing, label construction,
and reporting ran as written.
- `-c 3` over 14 files in three tables (4, 4, and 6 files): all 14 loaded,
peak in-flight 3.
- `-c 2` with `-x txnA`: all 14 loaded, peak in-flight 2.
- `-c 100`: peak in-flight 14, all loaded.
- One file made to fail with `-c 1`: the failing file printed the red FAIL
line, the other 13 loaded, exit status 0.
- The same failing-file run against the pre-change script: killed by
`timeout` at 20s (exit 124) after one file, i.e. it hangs.
`bash -n` passes on the result.
### Notes for Reviewers
- `trap 'echo >&3' EXIT` replaces the trailing `echo >&3`, so the token is
returned exactly once on the success path and on the `set -e` abort path.
- The loop body reads `table_name` and `file` directly: a background subshell
inherits their values at fork time, so no copy is needed.
- The `|| ret="${CURL_FAIL}"` arms stop `set -e` from aborting on a transport
error, which is what lets the existing FAIL branch print it.
### Release note
`load-tpcds-data.sh -c N` now loads up to N data files concurrently across
all tables instead of N tables with serialized files, which shortens loads
dominated by one large table. A file whose `curl` fails is now reported and
no longer hangs the whole load.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: exercised with stubbed
`curl`/`jq`/`mysql`; no live Doris cluster was involved.
### AI Disclosure
- Tool: Claude Code
- Generated: the script changes, the stub harness used for testing, and this
description.
- Human review: pending; the author has not yet reviewed the generated
parts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
