eugenegujing opened a new pull request, #7649:
URL: https://github.com/apache/texera/pull/7649
### What changes were proposed in this PR?
`WorkflowCompiler` scans every Python-based physical operator's generated
code for the `#EXCEPTION DURING CODE GENERATION:` marker that
`PythonOperatorDescriptor` embeds when an operator's `generatePythonCode`
throws. On a hit it either appends a `RuntimeException("Operator is not
configured properly: ...")` to the caller's error list (the editing-time,
lenient path) or throws it immediately when no error list was given (the
pre-execution, strict path). `WorkflowCompilerSpec`'s 15 tests never exercised
either branch; the only existing test of the marker is the producer-side
`PythonOperatorDescriptorSpec`, which asserts the marker is written but not
that the compiler reacts to it.
This adds 7 tests to `WorkflowCompilerSpec` (15 → 22). Test-only: no
production file is touched, and none of the 15 existing tests is modified — the
diff removes exactly three lines, all of them import lines being widened.
Lenient path:
1. `should accumulate a per-operator error when a Python operator's code
generation fails` — the error is attributed to the right logical operator,
carries the expected message, and compilation **continues** (the rest of the
plan still lands in the physical plan and the storage set).
2. `should attribute each Python code-generation failure to its own logical
operator` — two failing Python operators with two distinct messages, each keyed
to its own id.
3. `should trim the marker's message and report it as a plain
RuntimeException`
4. `should report no code-generation error for a well-formed Python
operator` — asserts the operator really is still Python-based, so it cannot
pass by quietly ceasing to be one.
5. `should not subject non-Python operators to the code-generation check`
Strict path:
6. `in strict mode should throw immediately when a Python operator's code
generation failed`
7. `in strict mode should not throw for a well-formed Python operator`
Five of the seven drive the **real shipped `SortOpDesc`**, whose
`generatePythonCode` opens with `require(attributes.nonEmpty, ...)` and a
per-key `require` (`SortOpDesc.scala:34-38`). A Sort dropped on the canvas and
left unconfigured is therefore a genuine, user-reachable route into the marker
state, and it conveniently yields two distinct messages — which is what makes
test 2's per-operator attribution meaningful. A configured Sort gives the
negative control in tests 4 and 7: same operator, same code path, codegen
simply succeeds.
Test 3 needs a small test-only fixture (`PaddedFailurePyOp`, ~15 lines,
modelled on `PythonOperatorDescriptorSpec`'s `ThrowingPyOp`): no shipped
operator raises a whitespace-padded message, and without padding the `.trim` is
unobservable. It is also the spec's only source-operator Python case.
Two notes:
- Four assertions hard-code `SortOpDesc`'s exact `require` wording, so
rewording those messages in `workflow-operator` will fail these
`workflow-compiler` tests. That cross-module coupling is deliberate — it pins
the end-to-end string a user actually sees — but it is worth knowing.
- Test 3 also asserts the reported message starts with
`java.lang.RuntimeException: `. That prefix comes from `err.toString` at
`WorkflowCompiler.scala:66` and is part of what the UI renders today; the
assertion carries a comment saying so, so if that pre-existing wart is ever
fixed there is one self-explaining test to update.
### Any related issues, documentation, discussions?
Closes #7647
Builds on the specs added by #5019 / #5022 and the module unification in
#6143.
### How was this PR tested?
`sbt "WorkflowCompiler/testOnly *WorkflowCompilerSpec"` — 22 tests, all
passing (15 pre-existing + 7 new). Running the whole module
(`WorkflowCompiler/test`) is green too: 3 suites / 57 tests / 0 failures.
`WorkflowCompiler/scalafmtCheck` and `WorkflowCompiler/Test/scalafmtCheck` are
clean.
Every new test was mutation-checked, twice and independently: the production
check was temporarily broken, the suite re-run, and the file reverted (verified
byte-identical afterwards). Highlights, with the tests that are the *sole*
killer of a mutant:
| Mutant in `WorkflowCompiler.scala` | Caught by |
|---|---|
| lenient arm appends **and then** throws (abort instead of continue) | 1
only |
| record only the first codegen failure per compile | 2 only |
| drop `.trim` on the captured message | 3 only |
| skip the check for source operators (`&& !isSourceOperator`) | 3 only |
| strict arm swallows instead of throwing (`case None => ()`) | 6 only |
| flip the `errorList` arms (`Some` → throw, `None` → skip) | 1, 6 |
| change the message prefix | 1, 2, 3, 6 |
| `group(1)` → `group(0)` | 1, 2, 3, 6 |
| marker regex changed so it never matches | 1, 2, 3, 6 |
| regex relaxed to `(.*)` (matches every operator) | 1, 2, 3, 4, 6, 7 |
| attribute the error to the wrong logical operator id | 1, 2, 3 |
| build the error but never record it | 1, 2, 3, 6 |
| `RuntimeException` → `IllegalArgumentException` | 3 |
Two mutants survive, both semantically equivalent rather than gaps:
reporting the *last* marker match instead of the first (a codegen-failure body
is exactly the one-line marker, so first == last), and anchoring the regex with
`^` — the latter is in fact the fix for a bug found along the way (below), and
all 22 tests stay green under it, so the suite does not over-fit to the current
unanchored check.
One more mutant is worth calling out honestly: removing the `isPythonBased`
guard entirely does not fail test 5 — it aborts the whole suite, because
`PhysicalOp.getCode` throws an `IllegalAccessError` (a `LinkageError`, which is
not `NonFatal`) for non-code operators. Test 5 therefore documents the guard's
intent rather than detecting its removal. It is kept for that reason, with a
self-check that stops it silently becoming a no-op.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
