aglinxinyuan opened a new issue, #8937:
URL: https://github.com/apache/texera/issues/8937
### What happened?
Two `LoopIntegrationSpec` cases added in #8639 time out intermittently on
`main` and on unrelated PRs (macOS and Ubuntu `amber-integration` jobs):
- "bind the inner loop's variable where it shadows the outer loop's of the
same name"
- "bind a $i reference after two branches of the loop body meet again"
Each failure is `com.twitter.util.TimeoutException: 1.minutes+30.seconds` at
about 180 s: the 90 s deadline, once per `withRetry` attempt. Codecov test
analytics lists about 10 such failures on `main` between 2026-10-05 and
2026-10-07. The older loop cases have failed the same way now and then, for
example "run a nested loop whose inner body chains multiple JVM operators" on
`main` on 2026-08-25.
These runs are not just slow. Locally the two cases take 55 s and 31 s, and
a timed-out run is idle: the JVM's CPU is flat and every dispatcher is waiting.
One worker of the 2-worker Filter (usually the one that got no tuples) handles
the materialized input's END_CHANNEL and reports its input port completed.
After that it never finalizes its output and never reports that it finished, so
the region never completes. Nothing is logged.
The cause is a race between `InputPortMaterializationReaderThread` and the
DP thread:
- The reader thread enqueues END_CHANNEL, and only after that sets its
`finished` flag.
- `EndChannelHandler` finalizes the worker's output only when
`InputManager.isPortCompleted` holds for every input port. For a port read from
materialization, that method returns the reader thread's `finished` flag.
- A DP thread that is waiting on the queue can take END and run
`EndChannelHandler` before the reader thread sets the flag. It sees the port as
not completed and skips `finalizeOutput`, and nothing calls it again.
```
Before: reader enqueues END -> DP processes END, flag not set yet -> output
never finalized -> region hangs
After: reader enqueues END -> DP processes END, port completed -> output
finalized -> region completes
```
The same flag also goes wrong in the other direction. It can already be set
while END is still queued, so a worker with a second materialized input port
could finalize before that port's END has been processed.
The race dates from #3460 (June 2025), which added the flag and this check.
#8639's cases hit it most often because they re-run a 2-worker JVM operator fed
from a materialized input on every loop iteration. Expected behavior: a
materialized input port counts as completed when its END has been processed,
the same as a port fed by a link.
### How to reproduce?
On `main`, pin the sbt JVM to 3 CPUs (its Python workers inherit the
affinity), then run the branching case a few times:
```
sbt "WorkflowExecutionService/testOnly *LoopIntegrationSpec -- -z \"after
two branches of the loop body meet again\""
```
With `UDF_PYTHON_PATH`, `STORAGE_ICEBERG_CATALOG_TYPE=postgres` and
`TEXERA_HOME` set, and `AMBER_TEST_FILTER` unset so that `-z` selects the case.
On a 32-core Windows machine, 3 of 5 pinned runs hung on the first attempt. One
of them hung on the retry too and failed exactly as on CI. A healthy pinned run
takes 40 s. Without pinning, all 8 cases pass, in 26–55 s each.
For a deterministic repro, call `EndChannelHandler.endChannel` on a worker
whose only input port is read from materialization, before the reader thread
sets its flag. The worker emits only `FinalizePort(input)`, with no output
`FinalizePort` and no `FinalizeExecutor`.
### Version/Branch
1.4.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
be65bf8564
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
com.twitter.util.TimeoutException: 1.minutes+30.seconds
at com.twitter.util.Promise.ready(Promise.scala:680)
at com.twitter.util.Await$.result(Awaitable.scala:155)
at
org.apache.texera.amber.engine.e2e.TestUtils$.runWorkflowAndReadResults(TestUtils.scala:187)
at
org.apache.texera.amber.engine.e2e.LoopIntegrationSpec.runAndGetMaterializedRowCounts(LoopIntegrationSpec.scala:163)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
