vishalmore90 commented on PR #40283: URL: https://github.com/apache/beam/pull/40283#issuecomment-6031027947
Hi @tvalentyn, thanks for the review and raising these valid concerns. The AI feedback is spot on. Calling `.compute()` inside a Dask worker is indeed a known anti-pattern that can lead to deadlocks (Tasks Launching Tasks), and my initial change unfortunately destroys parallelism for the side input evaluation. However, it's worth noting that the original `list(self.bag)` implementation *also* implicitly calls `.compute()` inside the worker, so the deadlock risk was already present. My change just traded the OOM crash for worse performance (serial execution and recomputation). Since we don't currently have a robust storage layer for side inputs in the Dask Runner, any solution without one will be a tradeoff between memory (OOMs) and performance (deadlocks/serial execution). As suggested, it is too risky to enable this lazy streaming behavior on the default path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
