anblanco commented on code in PR #55201:
URL: https://github.com/apache/spark/pull/55201#discussion_r3040407302
##########
python/pyspark/worker.py:
##########
@@ -3420,4 +3420,16 @@ def process():
# TODO: Remove the following two lines and use `Process.pid()` when we
drop JDK 8.
write_int(os.getpid(), sock_file)
sock_file.flush()
- main(sock_file, sock_file)
+ try:
+ main(sock_file, sock_file)
+ finally:
+ # SPARK-53759: Flush before close to ensure all buffered data reaches
+ # the socket. On Python 3.12+, changed GC finalization ordering
+ # (https://github.com/python/cpython/issues/97922) can cause the
+ # underlying socket to close before BufferedRWPair flushes its write
+ # buffer, resulting in data loss and EOFException on the JVM side.
+ # This mirrors the explicit flush in daemon.py's worker() finally
block.
+ try:
+ sock_file.flush()
Review Comment:
#54458 is the full change, which was a much larger refactor.
I figured that the try-finally would cherry-pick much more cleanly into the
older backport branches, but I can also assess the feasibility of attempting to
cherry-pick
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
