The GitHub Actions job "Required Checks" on texera.git/backport/6661-carry-the-loop-stateframe-envelope-throu-v1.2 has failed. Run started by GitHub user Yicong-Huang (triggered by Yicong-Huang).
Head commit for run: d2e091d3cca5015a4d8559fc69164587d7d32e79 / Xinyuan Lin <[email protected]> fix(amber): carry the loop StateFrame envelope through JVM operators (#6661) ### What changes were proposed in this PR? A loop with a Java/Scala built-in operator in its body could never iterate — the Loop End worker crashed on the first iteration: ``` RuntimeError: no loop-back state URI configured for LoopStart '' ``` **Root cause.** Per-iteration loop state rides a `StateFrame` envelope — `loop_counter` (nesting-depth countdown) and `loop_start_id` (the back-jump target) — materialized as their own columns next to `content` (#5900). The Python worker carries the envelope on every hop; the Scala side stripped it at every seam, so any JVM hop re-emitted states with the "no loop" defaults `(0, "")`: ``` LoopStart ──(0, LS-id)──▶ Limit(Scala) ──(0, "")──▶ LoopEnd ▲ envelope stripped │ ▼ back-jump lookup by "" fails ``` In a nested loop the zeroed `loop_counter` additionally makes the inner Loop End mis-consume the enclosing loop's state instead of passing it through. **Fix.** Loop operators are Python-only, so a JVM operator only ever needs to *carry the envelope through unchanged* (the +1/−1 bookkeeping lives in the Python runtime). The columns already exist in the materialized State schema and on the Arrow flight wire — no format change, purely Scala-side plumbing: | Seam | Change | |---|---| | `StateFrame` (`DataPayload.scala`) | gains `loopCounter` / `loopStartId` fields (defaults `0` / `""`, mirroring Python's `StateFrame`) | | `State` (`State.scala`) | `loopCounterFrom` / `loopStartIdFrom` extractors read the envelope columns back off a row | | `InputPortMaterializationReaderThread` | rebuilds the envelope when replaying materialized states | | `DataProcessor` → `OutputManager` | the incoming envelope is threaded through `processInputState` to `emitState` / `saveStateToStorageIfNeeded` / `sendState` | | `PythonProxyClient` / `PythonProxyServer` | envelope preserved across the Arrow flight bridge in both directions | Scala-originated states (start/end-channel handlers) keep the defaults — correct, since a JVM operator can never be a Loop Start. **Size:** the fix itself is **+38 / −17 lines of code** (excluding comments/blanks; +78 / −20 gross) across the 8 source files above; the remaining **+255 lines** are test cases (4 unit specs + 4 e2e workflows). ### Any related issues, documentation, discussions? Closes #6660 ### How was this PR tested? - **Unit** — each seam is pinned locally: `StateSpec` (envelope column round-trip + no-loop defaults), `DataProcessorSpec` (a state pass-through emits the exact incoming envelope, not just *a* `StateFrame`), `OutputManagerSpec` (`emitState` stamps the envelope onto every buffer), `NetworkOutputBufferSpec` (`sendState` stamps the frame). - **E2E** — four new `LoopIntegrationSpec` cases run real workflows with JVM operators in the loop body: 1. single loop with `Limit` (`TextInput → LoopStart → Limit → LoopEnd`, asserts 3 accumulated iterations — reproduces the reported failure verbatim, fails against the pre-fix code); 2. nested 3×3 loop with `Limit` in the inner body (asserts 9 outer / 3 inner rows — pins that the counter *magnitude* survives the JVM hop, not just the id); 3. single loop with a **chain of two JVM operators** (`Limit → Sleep(0)`) — pins the JVM→JVM state handoff (one Scala worker writes the envelope columns, another reads them back), which the single-op cases never exercise; 4. nested 3×3 loop with the **chain inside the inner body** — the strongest combination: nested (counter magnitude) × multi-hop × both operator kinds. Manually tested with the this workflow: [loop.json](https://github.com/user-attachments/files/30251886/loop.json) <img width="570" height="180" alt="Screenshot 2026-07-21 at 20 56 23" src="https://github.com/user-attachments/assets/0c002e9e-694e-445b-9ca7-b95a2cb405ae"; /> ### Was this PR authored or co-authored using generative AI tooling? (backported from commit 91a701b7c1dc16f3a896971ddef4c1859a50e51f) Generated-by: Claude Code (Fable 5) Report URL: https://github.com/apache/texera/actions/runs/30512272046 With regards, GitHub Actions via GitBox
