aglinxinyuan opened a new pull request, #8699:
URL: https://github.com/apache/texera/pull/8699
### What changes were proposed in this PR?
`OneToOnePartitioner.flush(to, ecm)` ignored `to`. `MainLoop` sends an ECM
with one `OutputManager.emit_ecm` call per output channel, and `emit_ecm`
flushes every output link each time. So on a Python worker with a one-to-one
link next to another output link, the first channel flushed took the one-to-one
link's pending tuples, and each channel that was not the link's receiver got an
extra copy of its ECM. With a round-robin link to R1 and R2 added before a
one-to-one link to C, and tuples 1, 2, 3 pending at EndChannel:
```
Before After
R1 [1, 3] END [1, 2, 3] END [1, 3] END
R2 [2] END END [2] END
C END [1, 2, 3] END
```
`flush` now yields nothing unless `to` is the link's receiver, and leaves
the pending batch for that receiver's own flush, as the round-robin, hash and
range partitioners and Scala's per-receiver `NetworkOutputBuffer` do.
The input-port materialization reader finished a replay shorter than one
batch only because `flush` never read the receiver. It built its one-to-one
partitioner with the worker's `ActorVirtualIdentity`, but the partitioner
matches on the sender's name, and every channel the coordinator gives a reader
is sent by the reader's virtual actor, not by the worker. So the partitioner
had no receiver, and a replay of one batch or more (400 tuples by default) died
with `AttributeError` before any tuple or EndChannel went out. The reader now
passes its virtual actor's name, which is how `ResourceAllocator` names the
sender (`getFromActorIdForInputPortStorage`).
### Any related issues, documentation, discussions?
Closes #8698.
### How was this PR tested?
The new tests fail on main and pass with the fix:
- `TestOneToOnePartitioner`: a flush to a worker that is not the receiver
yields nothing, with or without a pending batch, and keeps the batch for the
receiver's own flush. Each case runs against a worker that receives another
link and against one that receives another sender's channel of the same link.
- `TestEmitChain`: calls `emit_ecm` once per output channel in
`get_output_channel_ids()` order, as `MainLoop._send_ecm_to_data_channels`
does, for a one-to-one link beside a round-robin link (in both link orders) and
for two one-to-one links. Each receiver gets only its own tuples, then one ECM.
- `TestRunTupleLoop`: a reader built through `__init__` with the channel the
coordinator sends queues StartChannel, `[1, 2]`, `[3]`, EndChannel for tuples
1, 2, 3 at batch size 2.
`test_state_materialization_e2e.py` now gives its reader that same channel,
instead of one sent by an unrelated producer worker; with the old channel, the
fixed reader finds no receiver.
```
cd amber
pytest src/test/python/core/architecture/sendsemantics/test_partitioners.py \
src/test/python/core/architecture/packaging/test_output_manager.py \
src/test/python/core/storage/runnables/test_input_port_materialization_reader_runnable.py
\
src/test/python/core/architecture/packaging/test_state_materialization_e2e.py
ruff check src/main/python src/test/python
ruff format --check src/main/python src/test/python
```
The full `pytest src/test/python` run has no failures that main does not
also have.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
