aglinxinyuan opened a new issue, #8749:
URL: https://github.com/apache/texera/issues/8749
### What happened?
Inside a loop block, a loop-variable reference (#8635, #8639) in a property
that the compiler builds an operator's output schema or partitioning from gives
wrong output or a misleading error.
Root cause: the compiler builds the schema and the partitioning when the
workflow compiles, from the reference's placeholder. The loop state writes the
variable's value into the operator's setting only later, when the loop runs, so
the value never reaches them.
```
Before: Loop Start (a = "x") -> Projection (alias $a) -> Loop End
output column named "$a", null in every row
After: compile error naming the property: its value decides the plan
before the loop runs
```
| Operator and property | What happens today |
|---|---|
| Projection alias | a column literally named `$a`, null in every row |
| Projection drop option | the keep-mode schema, while the operator drops |
| Dictionary Matcher, Unnest String, Aggregate result attribute; File Scan
From Input attribute | the output column stays literally `$r` |
| Sort Partitions domain min / max | the range partition stays 0 to 0 |
| Hash Join and Interval Join keys, Type Casting attribute, Aggregate
group-by keys, Sort Partitions attribute | a misleading error later in the run |
The same holds for a running workflow's operator that is modified while it
runs: its rebuilt executor never has the loop variables written into it.
### How to reproduce?
On the #8639 branch: build `Text Input -> Loop Start (a = "x") -> Projection
(alias of a column set to $a) -> Loop End` and run it. Loop End's result has a
column named `$a` whose values are all null.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
64c8160c10 (the #8639 branch)
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
