goutamadwant opened a new pull request, #12241: URL: https://github.com/apache/seatunnel/pull/12241
<!-- PR title: [Test][CLI] Add opt-in paraphrase benchmark suite --> ### Purpose of this pull request Closes #12240. Related to #11616. Add a separate, opt-in paraphrase suite to the existing CLI benchmark. The default benchmark has 100 tasks with fixed wording. The new suite contains 12 alternative prompts covering routing, CDC prerequisites and connector options. Each variant inherits its original task's assertions and execution fixtures and pins the parent contract by fingerprint. ### Does this PR introduce _any_ user-facing change? Yes. `--suite paraphrase` selects only the 12 variants. The default remains `--suite baseline` with the same 100 tasks, task definitions and report formats. Variants have distinct IDs and saved parent provenance. Invalid selections and changed parent contracts fail before provider setup. The existing generation, scoring, repair and comparison paths are reused. This is a public regression corpus, not an unseen holdout. Running the benchmark still calls the configured model; offline tests do not establish model accuracy or generalization. ### How was this patch tested? Before implementation, the runner rejected `--suite paraphrase` as an unknown argument. Added 52 offline cases covering selection, inherited fixtures, parent drift, malformed definitions, scoring parity, skipped gates, saved provenance and revision comparisons. All 12 prompts were separately checked against their original task requirements. From `seatunnel-cli`: ```bash python -m pytest tests -q python -m black --check benchmark/paraphrases.py tests/test_benchmark_paraphrases.py python -m ruff check --isolated --select E4,E7,E9,F benchmark/paraphrases.py benchmark/runner.py tests/test_benchmark_paraphrases.py ``` The complete CLI suite passed on Python 3.10 and 3.11: 181 tests and 3 subtests on each runtime. The original 100-task definitions remain identical. No model calls or engine jobs were run for these tests. Root formatting and the complete no-tests build passed on Java 11: ```bash ./mvnw spotless:apply ./mvnw -q -DskipTests verify ``` The unchanged upstream baseline also passed the same no-tests build. This checks compilation and packaging, not execution of the full Java test suite. The [fork Build](https://github.com/goutamadwant/seatunnel/actions/runs/34320639314) passed for the submitted commit. ### Check list * [x] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md). No new dependencies or JARs. * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs (EN/ZH documentation and benchmark README updated.) * [x] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR. Existing default behavior is unchanged. * [x] If you are contributing the connector code, please check that the following files are updated: Not a connector change. 1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) 3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml) 4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/) 5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
