potiuk commented on PR #30259: URL: https://github.com/apache/airflow/pull/30259#issuecomment-1481704851
> Yes. I am using Mutliprocessing manager. > > It works well when I test it in breeze, but indeed, that's a specific setup, and I'd need to test this in a more prod-like environment, where maybe multiprocessing could show its limits. Yeah - I would love to see (actually) if we can make it works at least to some extent. I do **see** the problem and I understnad the need to solve it (taming user's wild ideas is not easy, I am aware of it). Especially in Celery (where we have limited impact on what's going on - also you should be aware that Celery uses `billiard` as direct `multiprocessing` replacement. This is a forked multiprocessing library that has a number of under-the-hoods things "corrected" and there were multiple reports of our users where their multoprocessing code crashed in the Celery-run tasks (there are 33 issues with `billiard` in our issues https://github.com/apache/airflow/issues?q=is%3Aissue+billiard+is%3Aclosed ) > > About the users not following best practices, I think it's an easy stance to take, but not very helpful. We want users to fall in the "pit of success". If we know that they are doing things that are not recommended, we should either make those things work well anyway, or make it a lot harder to do the not recommended stuff. I was actually thinking of completely blocking using DB calls in dag parsing. That would be more helpful. I think caching and TTL might have very unexpected behaviours and hit our users much more than slow parsing, so if we do it, this has to be done extremely carefully. I can see multiple scenarios where things might go wrong. > Also, about what you say above, users using variables as XCOM replacement (skipping over the fact that this is probably not a recommended practice as well ?), that would still work because I invalidate the cache on Variable.set, so the changes are propagated instantly for those. Unfortunately not. They will only work if you have single machine setup working and all processes somehow sharing the same memory. It stops working when two tasks are run on two different machines, because (unlike the db, non-cached variable) the second machine will see the new change only after the TTL passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
