HyukjinKwon opened a new pull request, #57024:
URL: https://github.com/apache/spark/pull/57024
### What changes were proposed in this pull request?
This makes the pipelined Python UDF read path work under Unix Domain Socket
(UDS) mode.
`BasePythonRunner.createPipelinedDataIn` (added in SPARK-56642 for pipelined
JVM↔Python UDF
data transfer) sets up the read side via
`worker.channel.socket().setSoTimeout(...)` and
`worker.channel.socket().getInputStream`. When Spark runs with Unix Domain
Sockets
(`spark.python.unix.domain.socket.enabled=true`), `worker.channel` is an
`AF_UNIX`
`SocketChannel`, whose `socket()` throws
`java.lang.UnsupportedOperationException: Not supported`
(a Unix-domain channel has no `java.net.Socket` adapter). Every pipelined
UDF task then fails:
```
java.lang.UnsupportedOperationException: Not supported
at
java.base/sun.nio.ch.SocketChannelImpl.socket(SocketChannelImpl.java:228)
at
org.apache.spark.api.python.BasePythonRunner.createPipelinedDataIn(PythonRunner.scala:447)
```
This adds a UDS branch that reads straight from the channel via
`Channels.newInputStream(worker.channel)` (with no `SO_TIMEOUT`-based
idle-timeout detection, since
Unix-domain channels do not support `SO_TIMEOUT`). This mirrors how the
existing sync-mode server
loop already guards its `setSoTimeout` calls behind `!isUnixDomainSock` and
reads via
`Channels.newInputStream`. The default TCP path is unchanged (the new code
only runs when UDS is
enabled).
### Why are the changes needed?
`build_uds.yml` (the "Build / Unix Domain Socket" scheduled workflow, one of
the README CI badges)
is red: `pyspark.sql.tests.pandas.test_pipelined_udf` fails with `FAILED
(failures=1, errors=12)`
because pipelined UDFs cannot run at all under UDS mode.
Note: this is distinct from #56995 (SPARK-57931), which restores the
channel's blocking mode after
pipelined execution — a different bug in the same method. This PR fixes the
UDS `socket()` crash on
the read path.
### Does this PR introduce _any_ user-facing change?
No. It makes an existing feature (pipelined Python UDF) work under UDS mode;
behavior in the default
TCP mode is unchanged.
### How was this patch tested?
`pyspark.sql.tests.pandas.test_pipelined_udf` under UDS mode.
**Before (red — apache/spark `master`):**
- `build_uds` → `pyspark-sql` FAILED (`test_pipelined_udf`: `FAILED
(failures=1, errors=12)`):
https://github.com/apache/spark/actions/runs/28488010762
**After (green — with this change, fork verification):**
- `build_uds` (UDS mode) → `pyspark-sql` module PASSED:
https://github.com/HyukjinKwon/spark/actions/runs/28758332630
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Co-authored-by: Isaac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
