aglinxinyuan opened a new pull request, #7488: URL: https://github.com/apache/texera/pull/7488
### What changes were proposed in this PR? #7447 proposed deleting `PythonWorkflowWorkerSpec.scala`, which has been fully commented out for years and no longer compiles against today's APIs. Following the review feedback there (https://github.com/apache/texera/pull/7447#issuecomment-5234671357), this PR replaces the dead file with real unit tests instead of only deleting it. The commented-out spec drove `PythonWorkflowWorker` end-to-end, which needs a live Python process — that path is covered by the e2e tests. What can be unit-tested without Python is the JVM side of the JVM↔Python Arrow Flight bridge, which had no coverage until now: ``` JVM Python ┌───────────────────────────────────────┐ │ PythonWorkflowWorker (actor) │ e2e-tested only (needs Python) │ ├─ PythonProxyClient ──── Flight ────┼────▶ network_receiver.py │ │ PythonProxyClientSpec: fake │ │ │ Python Flight server in Scala │ │ └─ PythonProxyServer ◀─── Flight ────┼───── network_sender.py │ PythonProxyServerSpec: test │ │ plays the Python Flight client │ └───────────────────────────────────────┘ ``` | New spec | Subject | Behavior pinned down | |---|---|---| | `PythonProxyServerSpec` | `PythonProxyServer` / `AmberProducer` | `handshake` completes the port promise and replies `ok`; `control` actions route `ControlInvocation` / `ReturnInvocation` to the output gateway on the control channel and ack with a little-endian credit value; `Data` / `State` / `ECM` puts are reassembled into `DataFrame` / `StateFrame` (loop envelope preserved) / `EmbeddedControlMessage` and acked with credits | | `PythonProxyClientSpec` | `PythonProxyClient` | heartbeat handshake happens before the queue is drained; queued `ControlInvocation` / `ReturnInvocation` / actor commands arrive as `control` / `actor` actions with intact protobuf payloads; `DataFrame` / `StateFrame` / ECM puts arrive under the right `PythonDataHeader` with tuples, loop envelope, and bytes intact; queue-size acks update `getQueuedCredit`; connection retries abort with `WorkflowRuntimeException` (no server listening, non-`ack` heartbeat); `close()` before any connection does not throw | Both specs stand in for the Python worker with plain Arrow Flight components (`FlightClient` / `NoOpFlightProducer`), so no Python process is involved. Together with the existing `WorkerBatchInternalQueueSpec` and `PythonWorkflowWorkerStartupConfigSpec`, every class in the `pythonworker` package except the actor itself now has unit coverage. The commented-out `PythonWorkflowWorkerSpec.scala` is removed, superseded by these tests. ### Any related issues, documentation, discussions? Supersedes #7447. ### How was this PR tested? This PR is tests-only. Both new specs pass locally: ``` sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.architecture.pythonworker.PythonProxyServerSpec org.apache.texera.amber.engine.architecture.pythonworker.PythonProxyClientSpec" ``` ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
