SEZ9 commented on PR #11553: URL: https://github.com/apache/seatunnel/pull/11553#issuecomment-5119976863
@yzeng1618 @davidzollo Both of you called the same core defect and you were right â my previous round only *warned* about skipped gates without excluding them from the metrics. Fixed in head `8cef011d5`: 1. **Skipped gates excluded from all success metrics** (both #1): `evaluate_layers` now computes `full_gate_passed` = all requested gates *executed and* passed, exactly the split davidzollo proposed (`all_passed` survives only as repair-loop control â a skipped gate isn't repairable, but it's no longer a success either). `first_pass_round` is set solely from `full_gate_passed`, so pass@1 / pass@âĪk / pass^k, per-tier/category rates, and the CSV `passed` column all inherit the strict semantics. The new regression (`test_skipped_gate_excluded_from_pass_metrics`) runs the full `run_task_with_repairs` path with a skipped L3 and asserts `first_pass_round` stays `None`, `full_gate_passed` is false, and no repair round is spent on it. README metric definitions updated to state the exclusion explicitly. 2. **Per-task fault isolation + env hygiene** (both #2): an unexpected harness exception (scoring crash, setup-file I/O, subprocess construction) now becomes one structured failed trial carrying the traceback in `generation_error` instead of aborting the model run, and `restore_env` moved into a `finally` so `AI_PROVIDER`/model variables are restored even when a run dies mid-model. 3. **`git diff --check`** (davidzollo #3): trailing whitespace removed. Suite: 50 passed. Scope kept to the three findings â no metric-pipeline re-architecture beyond the pass-predicate change you asked for. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
