HappenLee opened a new pull request, #68278:
URL: https://github.com/apache/doris/pull/68278
### What problem does this PR solve?
The SSB tools currently require generating local `.tbl` files and running
separate table-creation, Stream Load, flat-table preparation, and benchmark
commands. This PR adds a Trino SSB generator plugin and `run-ssb.sh` to execute
the whole workflow with one command:
```bash
cd tools/ssb-tools
./bin/run-ssb.sh -s 1 -d ssb_sf1
```
The connector streams the existing `ssb-dbgen` output into Doris through
`INSERT INTO ... SELECT ...`, without intermediate data files or a separate
Trino server. It preserves the old tools' ten `lineorder` partitions, enables
dbgen's existing 64-bit path, and fixes an out-of-bounds usage-counter update
for SSB random streams. Each scan owns its generator process and terminates it
on close or early completion.
The script reuses the existing DDL and query suites, prepares SSB and SSB
Flat, collects statistics, and records three executions per query with SQL/BE
query result caching disabled. It requires a new database to prevent duplicate
loads and stops on failed or not-yet-visible inserts. `--queries-only` reruns
the benchmark against existing data.
### Release note
Add an optional streaming SSB generator and one-command benchmark workflow
for SF1, SF100, and SF1000. The plugin must first be installed on every FE and
BE, followed by a restart. Existing file-based scripts remain available.
### Check List (For Author)
- Test
- [x] Unit Test: 8 plugin tests and 6 script orchestration tests passed.
Plugin coverage includes full SF1 checksums of all five tables, handle
serialization, projection, SF1000 large keys, producer failures, and early
closure.
- [x] Manual test: ASAN BE+FE build passed with `./build.sh --be --fe
-j48`. Installed the plugin on an isolated local cluster and ran the complete
SF1 workflow. `lineorder` and `lineorder_flat` each contain 6,001,215 rows.
Repeated all 26 queries three times with result caching disabled; outputs are
stable. Verified rejection of an existing database and no residual generator
processes after an SF1000 `LIMIT` query.
- Checkstyle, Bash syntax checks, ShellCheck, shfmt, and `git diff
--check` passed.
- SF100 and SF1000 were not loaded in full; SF1000 validation covers
large keys and early scan completion.
- Behavior changed:
- [x] Yes. Adds an optional generator plugin and benchmark entry point;
existing scripts and query SQL are preserved.
- Does this need documentation?
- [x] Yes. Usage, installation, generator consistency, and timing
semantics are documented in `tools/ssb-tools/README.md` in this PR.
The existing Q1.3 queries use different week definitions: regular SSB
filters `d_weeknuminyear`, while SSB Flat uses `weekofyear()`. Their aggregate
results differ for the same generated data; the other 12 query pairs match.
This pre-existing difference is documented without changing the benchmark SQL.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
