luoyuliuyin commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2134270804
> Is this trying to save the initialisation overhead, or the shutdown overhead after each send? Reducing the shutdown overhead is most more dangerous as discussed above, but it is not obvious to me from the above discussion that’s what we’re looking for. The former can be easily avoided without consequences if lines 341-344/345 are changed to > > ```python > with self.send_pool as pool: > key_and_async_results = list(pool.map(...)) > ``` In this way, self.send_pool will be closed after processing the first batch of data, and then an error `RuntimeError: cannot schedule new futures after shutdown` will be reported when processing the second batch of data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
