HyukjinKwon commented on code in PR #55201:
URL: https://github.com/apache/spark/pull/55201#discussion_r3039092386
##########
python/pyspark/worker.py:
##########
@@ -3420,4 +3420,16 @@ def process():
# TODO: Remove the following two lines and use `Process.pid()` when we
drop JDK 8.
write_int(os.getpid(), sock_file)
sock_file.flush()
- main(sock_file, sock_file)
+ try:
+ main(sock_file, sock_file)
+ finally:
+ # SPARK-53759: Flush before close to ensure all buffered data reaches
+ # the socket. On Python 3.12+, changed GC finalization ordering
+ # (https://github.com/python/cpython/issues/97922) can cause the
+ # underlying socket to close before BufferedRWPair flushes its write
+ # buffer, resulting in data loss and EOFException on the JVM side.
+ # This mirrors the explicit flush in daemon.py's worker() finally
block.
+ try:
+ sock_file.flush()
Review Comment:
Is this the only code path to fix? If not, I think we should better backport
the original fix to reduce the diff between branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
