Yicong-Huang opened a new pull request, #58289: URL: https://github.com/apache/spark/pull/58289
### What changes were proposed in this pull request? This PR fixes flakiness in the two worker-logging tests of `ApplyInArrowTests`, `test_apply_in_arrow_with_logging` and `test_apply_in_arrow_iter_with_logging`, by wrapping the log read-and-assert step in `@eventually` so it polls until the asynchronously captured logs become visible. The `applyInArrow` call that produces the logs stays outside the poll so it runs exactly once. ### Why are the changes needed? `spark.tvf.python_worker_logs()` reads log records that Python workers emit on their stdout. On the JVM side those records are drained by a per-worker `RedirectThread` (`PythonWorkerLogCapture`), and the corresponding log block is only saved to the `BlockManager` once that thread reads the trailing marker line and closes the writer. This stdout drain is asynchronous with respect to the query result, which is returned over a separate socket channel. As a result, `python_worker_logs()` invoked immediately after the query completes can observe zero blocks, and the assertion fails with `[DIFFERENT_ROWS]` (actual empty vs. the expected WARNING rows). Polling the read side waits out this race without changing the feature, which captures worker logs on a best-effort basis. The producing UDF is intentionally kept outside the poll: re-running it would append more log rows and never match the expected count. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Built with `build/sbt -Phive package` and ran the affected tests locally: ``` python/run-tests --testnames 'pyspark.sql.tests.arrow.test_arrow_grouped_map' ``` Both logging tests and the full `test_arrow_grouped_map` module pass. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
