SEZ9 opened a new pull request, #11553:
URL: https://github.com/apache/seatunnel/pull/11553
### Purpose of this pull request
Implements #11549 — adds a standalone accuracy benchmark for the AI CLI
under `seatunnel-cli/benchmark/`. All new files; no product code changes.
The benchmark answers a question the project previously could not answer
quantitatively: **when the CLI's prompts, connector metadata, or repair logic
change, does config-generation accuracy actually improve — and by how much, for
which models?**
**What's included** (19 files, ~4,000 lines):
| Component | Files | Role |
|-----------|-------|------|
| Task suite | `tasks/tier{1,2,3}_*.json` | 100 declarative tasks: 20 simple
/ 45 medium / 35 complex-DAG, 12 ETL categories, 10 Chinese prompts, 18 rule
probes targeting known LLM failure modes |
| L1 static gate | `scoring.py` | pyhocon parse + connector metadata +
per-task assertions (offline) |
| L2/L3 gates | `execution.py` | engine `--dry-run static`; real job
execution on the official `apache/seatunnel` image (batch: exit code;
streaming: 60s health) |
| Data environment | `docker/` | docker-compose with
MySQL/PostgreSQL/Kafka/ClickHouse/Elasticsearch/MinIO + optional
Doris/StarRocks profile, seeded schemas/topics |
| Runner | `runner.py` | model matrix (`models.json` or
`--provider/--model/--api-key`), repair loop driven by the failing gate's real
errors, `--trials` for pass^k, CLI version + commit stamping |
| Reports | `report.py` | layer funnel, repair curve (pass@1 / pass@≤k),
per-tier & per-category matrices → console + summary.md/csv + results.json |
| Entry point | `run_benchmark.sh` | one command: dependency install →
environment preflight → run → report |
| Tests | `tests/test_benchmark_scoring.py` | 12 tests over the L1 scorer
and task-suite integrity |
**Design guarantees**:
- Gates mirror the CLI's own check → dry-run → run pipeline and degrade
gracefully (missing engine/Docker disables deeper gates per task, never fails
the run).
- All verdicts are deterministic (exit codes, liveness, regex assertions) —
no LLM-as-judge.
- Credentials are environment variables only, never written to files (ASF
compliance); `results/` and `models.json` are git-ignored.
- Reports are stamped with the CLI version + git commit, so any CLI change
can be A/B measured on identical tasks.
**Track record**: runs of this benchmark found and isolated #11508 (fixed in
#11509), #11510 (image-level opengauss-jdbc driver conflict — workaround
documented inline and applied identically for every model), and #11511.
Measured results for 7 models are documented on the website (AI CLI → Model
Benchmark, #11551).
### Does this PR introduce _any_ user-facing change?
No changes to shipped behavior. Adds an optional developer tool under
`seatunnel-cli/benchmark/` with its own README; two `.gitignore` entries for
runtime artifacts.
### How was this patch tested?
- `tests/test_benchmark_scoring.py`: 12 tests covering the L1 scorer
(structure checks, connector matching, assertion evaluation,
credential-placeholder handling) and task-suite integrity (100 tasks load,
unique ids, compilable regexes, service references match the compose file).
Full seatunnel-cli suite: 54 passed.
- The extracted PR tree was verified self-contained: tests pass and all 100
tasks load from a clean checkout with no repo-local dependencies.
- End-to-end: the harness has executed ~1,150 task evaluations across 7
Bedrock models (L1) and 3 models through the full L1→L3 pipeline with real job
execution and repair loops — the published numbers in #11551 were produced by
exactly this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
