dongjoon-hyun opened a new pull request, #562: URL: https://github.com/apache/spark-connect-swift/pull/562
### What changes were proposed in this pull request? This PR aims to fix `DataFrame` actions to use a per-call result instead of the actor-shared `batches` buffer, so that concurrent actions on the same `DataFrame` instance return correct results. - `DataFrame.execute()` now accumulates `RecordBatch`es into a local `Mutex` and returns them. - `collect()` and `collect(as:)` iterate the returned batches. - The unused `DataFrame.batches` and `addBatches` are removed. `head`, `take`, `first`, `tail`, and `show` create a new `DataFrame` and call `collect()`, so they are fixed together. `count()` already uses a local `Atomic` counter and is not affected. ### Why are the changes needed? `execute()` cleared the actor-shared `batches` and then refilled it across `await` suspension points. Due to actor reentrancy, concurrent actions on the same `DataFrame` interleave and append into the same buffer, so they silently return wrong results. ```swift let df = try await spark.range(300_000) async let a = df.collect() async let b = df.collect() // Before this PR, each returns more than 300000 rows. ``` ### Does this PR introduce _any_ user-facing change? Yes. Concurrent `collect()` and `collect(as:)` calls on the same `DataFrame` return correct results now. There is no API change. ### How was this patch tested? Pass the CIs with the newly added test case. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
