The GitHub Actions job "Comment commands" on texera.git/main has succeeded. Run started by GitHub user gupta-sahil01 (triggered by gupta-sahil01).
Head commit for run: f5217504c3cfd8ac1d56f6617d7849895af17f59 / Yicong Huang <[email protected]> fix(amber): order worker state by version, not timestamp (#6011) ### What changes were proposed in this PR? A fast **source** operator stayed **orange (RUNNING)** in the editor after the run finished (issue #6010). **Why it's a bug.** The controller reconstructs each worker's state from several unordered channels (source `RUNNING` via the `startWorker` response, non-source `RUNNING` via `workerStateUpdated`, `COMPLETED`/`PAUSED` via `queryStatistics`/`pauseWorker` responses) and reconciled them with **last-`System.nanoTime()`-wins** in `WorkerExecution`. Worker state, however, is single-writer and strictly ordered causally. For a tiny source the run finishes almost instantly, so the `startWorker` response — carrying the stale `RUNNING` it sampled at launch — can reach the controller *after* `COMPLETED` was recorded; its later receipt timestamp wins and clobbers `COMPLETED`. Results render fine (separate path), so only the border is stuck. This PR orders worker state **causally** instead of by wall clock: | Before | After | | --- | --- | | `WorkerExecution.update(nanoTime, state)` — last-write-wins by receipt time | `updateState(version, state)` — newest **logical version** wins | | stale late RUNNING clobbers COMPLETED | RUNNING (v2) < COMPLETED (v3) ⇒ COMPLETED kept | | — | terminal states (COMPLETED/TERMINATED) are absorbing | - `WorkerStateManager` bumps a monotonic `stateVersion` on every `transitTo` (its state-machine logical clock). - The version rides on every state report: `WorkerStateResponse`, `WorkerStateUpdatedRequest`, `WorkerMetrics` (3 new proto fields). - The controller applies a state only when its version is newer. Stats stay timestamp-ordered (monotonic snapshots can share a state version). - A single per-worker counter ⇒ no cross-process clock-sync assumptions (why not worker timestamps). - **pyamber parity:** the Python worker also reports state (start/pause/resume + query-statistics). Its `StateManager` now bumps the same monotonic version and the four handlers include it; otherwise version-0 reports would be dropped after the first and reconfiguration (RUNNING→PAUSED→RUNNING) would hang. ### Any related issues, documentation, discussions? Closes #6010. The timestamp-based `update` was introduced in #3557. ### How was this PR tested? JDK 17. Scala unit + Scala/Python integration + Python unit: ``` sbt "WorkflowExecutionService/testOnly \ *WorkerExecutionSpec *WorkerStateManagerSpec *OperatorExecutionSpec \ *RegionExecutionCoordinatorSpec *WorkflowExecutionCoordinatorSpec \ *RegionExecutionSpec *WorkflowExecutionSpec" # Scala-Python reconfiguration end-to-end (spawns Python UDF workers): sbt "WorkflowExecutionService/testOnly *ReconfigurationIntegrationSpec" # 3 passed # Python unit: cd amber && pytest src/test/python/core/architecture/managers/test_state_manager.py # 9 passed ``` - `WorkerExecutionSpec`: version ordering (positive + stale/equal-version negatives), terminal-state absorption, `forceTerminate`, independent stats-vs-state ordering, and a named regression for #6010 (`COMPLETED` survives a late `RUNNING`). Verified the regression goes **red** when `updateState` is reverted to last-write-wins. - `WorkerStateManagerSpec` / `test_state_manager.py`: version starts at 0, bumps per successful transition, no bump on no-op self-transition or rejected transition. - `ReconfigurationIntegrationSpec` reproduced the failure (timeout) before the pyamber fix and passes after. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 (1M context), via Claude Code --------- Co-authored-by: Claude Fable 5 <[email protected]> Report URL: https://github.com/apache/texera/actions/runs/30223407575 With regards, GitHub Actions via GitBox
