aglinxinyuan opened a new pull request, #8750:
URL: https://github.com/apache/texera/pull/8750
### What changes were proposed in this PR?
Root cause: inside a loop block, a loop-variable reference (#8639) is
written into the operator's setting when the iteration's state arrives. The
compiler, however, builds the operator's output schema and its partitioning
earlier, when the workflow compiles, from the reference's placeholder. A
reference in a property the plan is built from therefore never reaches the
schema or the partitioning.
```
Before: Loop Start (a = "x") -> Projection (alias $a) -> Loop End
output column named "$a", null in every row
After: compile error: Projection cannot refer to loop variables
(/attributes/0/alias -> $a):
its output schema or partitioning is built from that property when
the workflow is
compiled, before the loop runs
```
| Piece | What it does |
|---|---|
| `@FixedAtCompileTime` | Marks a descriptor property that the compiled plan
is built from: an output column's name or type, or what an input is partitioned
on. |
| `StateReferencing.fixedAtCompileTime` | Finds the references under a
marked property, at any depth. It goes through Jackson's bean introspection, so
renamed JSON names such as `"result attribute"` resolve. |
| `WorkflowCompiler` | Inside a loop block, a reference under a marked
property is a compile error, reported before the physical plan is built so it
is the operator's first error. Outside every block nothing changes: a `$a`
alias is still the literal column name. |
| `ExecutionReconfigurationService` | Modifying an operator that refers to
loop variables while the workflow runs is refused, since its rebuilt executor
would never have the variables written into it. |
Marked properties:
| Operator | Properties |
|---|---|
| Projection | drop option, attributes |
| Dictionary Matcher, Unnest String | result attribute |
| Aggregate | aggregations, group-by keys |
| File Scan From Input | attribute name, file-name column option |
| Type Casting | cast units |
| Hash Join, Interval Join | the join attributes |
| Sort Partitions | sort attribute, domain min and max |
**Dependencies.** This builds on #8639, whose commits are in this branch
until it lands on `main`. The change in this PR is the top commit, b421fcec76.
### Any related issues, documentation, discussions?
Closes #8749
Related to #8635 and #8639.
### How was this PR tested?
New tests, each run red against the pre-change code first:
| Spec | Covers |
|---|---|
| `StateReferenceModuleSpec` | `fixedAtCompileTime` picks the references
under a marked property whether it is renamed, escaped, nested or declared by a
trait, and nothing from a descriptor with no marked property or no reference. |
| `WorkflowCompilerSpec` | Inside a loop block, a reference in each marked
property is a compile error, and it comes before the operator's other errors. A
reference in a property the executor reads, on an operator whose plan is built
from other properties, still binds. Outside every block, a `$a` Projection
alias stays the literal column name. |
| `ExecutionReconfigurationServiceSpec` | An edit that makes an operator
inside a loop block refer to a loop variable is refused. An operator inside a
block that refers to no loop variable, and a `$x` literal outside every block,
can still be modified. |
```
sbt "WorkflowCore/testOnly org.apache.texera.amber.core.state.*
org.apache.texera.amber.core.executor.*" # 82 passed
sbt "WorkflowCompiler/test"
# 86 passed
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.metadata.*
...projection.* ...aggregate.* \
...sortPartitions.* ...dictionary.* ...unneststring.*"
# 367 passed
sbt "WorkflowExecutionService/testOnly
org.apache.texera.web.service.ExecutionReconfigurationServiceSpec" # 18 passed
sbt "scalafmtCheckAll"
"<WorkflowCore|WorkflowOperator|WorkflowCompiler|WorkflowExecutionService>/scalafixAll
--check" # clean
```
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
