Yicong-Huang opened a new pull request, #58780: URL: https://github.com/apache/spark/pull/58780
### What changes were proposed in this pull request? Wrap the log read-and-assert step of the three worker-logging tests in `test_python_datasource.py` (`test_data_source_reader_with_logging`, `test_data_source_reader_pushdown_with_logging`, and `test_data_source_writer_with_logging`) in `eventually` so it polls until the asynchronously captured worker logs are visible. The query that produces the logs stays outside the poll so it runs once. This applies the same stabilization approach as SPARK-59007 (#58289), which fixed the analogous Arrow grouped map worker-logging tests but did not cover these data source tests. ### Why are the changes needed? These tests are flaky. Observed on master nightly CI with `[DIFFERENT_ROWS]` where `spark.tvf.python_worker_logs()` returned fewer rows than expected (e.g. the trailing `TestJsonReader.read: InputPartition(value=None)` row was missing) and re-ran green afterwards. Root cause: worker logs are captured asynchronously. Python workers emit log records on stdout; on the JVM side a per-worker `RedirectThread` (`PythonWorkerLogCapture`) drains stdout and only saves the log block to the `BlockManager` once it reads the trailing marker line. That drain runs independently of the query result, which returns over a separate socket channel, so `python_worker_logs()` invoked right after the query can observe missing or zero blocks. Polling the read side waits out the race. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Built with `build/sbt -Phive package` and ran `python/run-tests --testnames 'pyspark.sql.tests.test_python_datasource PythonDataSourceTestsWithSimpleWorker.test_data_source_reader_with_logging,...'` for all three logging tests; all pass. ### Was this patch authored or co-authored using generative AI tooling? No. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
