aglinxinyuan opened a new issue, #6660:
URL: https://github.com/apache/texera/issues/6660
### What happened?
A loop cannot run when a Java/Scala built-in operator (e.g. `Limit`) is
inside the loop body. The Loop End worker crashes on the first iteration:
```
RuntimeError: no loop-back state URI configured for LoopStart ''
(have: ['LoopStart-operator-4d423fa9-...'])
```
The worker enters an error pause and the loop never iterates. A Python-only
loop body works fine.
### How to reproduce?
Run any loop with a Scala operator between Loop Start and Loop End, e.g.:
```
TextInput ──▶ LoopStart ──▶ Limit ──▶ LoopEnd
▲ (Scala) │
└──── back-edge ───────┘
```
### Root cause
Per-iteration loop state rides a `StateFrame` envelope — `loop_counter`
(nesting depth countdown) and `loop_start_id` (the back-jump target) —
materialized as their own columns next to `content` (`State` schema, #5900).
The **Python** worker carries the envelope on every hop. The **Scala** side
strips it at every seam: `StateFrame` has no envelope fields, so a JVM hop
rebuilds states with the defaults `(0, "")`.
| Seam (Scala) | What it does today |
|---|---|
| `StateFrame` (`DataPayload.scala`) | no `loopCounter` / `loopStartId`
fields |
| `InputPortMaterializationReaderThread` | rebuilds via
`State.fromTuple(row)` — columns dropped |
| `DataProcessor.processInputState` | forwards `processState` output without
the incoming envelope |
| `OutputManager.saveStateToStorageIfNeeded` | writes `state.toTuple()` —
defaults `(0, "")` |
| `NetworkOutputBuffer.sendState` | builds `StateFrame(state)` — defaults |
| `PythonProxyClient` / `PythonProxyServer` | Arrow flight bridge drops the
columns in both directions |
So `LoopStart → Limit → LoopEnd`: LoopStart stamps `(0, LoopStart-id)` → the
`Limit` hop re-emits `(0, "")` → LoopEnd captures `loop_start_id = ""` → the
back-jump write-address lookup (`InitializeExecutorRequest.loopStartStateUris`)
fails with the error above.
In a **nested** loop the zeroed `loop_counter` is just as harmful: an
enclosing loop's state crossing a Scala hop inside the inner body loses its
depth, so the inner Loop End mis-consumes it instead of passing it through.
### Expected behavior
| | before | after |
|---|---|---|
| `LoopStart → Limit → LoopEnd` (3 iters) | worker error, 0 iterations | 3
iterations, LoopEnd materializes 3 rows |
| nested 3×3 with `Limit` in inner body | inner LoopEnd mis-consumes outer
state | 9 outer rows / 3 inner rows |
The envelope columns already exist in the materialized State schema and on
the Arrow flight wire — a JVM operator only ever needs to **carry them through
unchanged** (loop operators are Python-only; the +1/−1 bookkeeping lives in the
Python runtime).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
