jscheffl commented on PR #65943: URL: https://github.com/apache/airflow/pull/65943#issuecomment-4444389148
@wjddn279 @diogosilva30 Thanks for all valid discussions. Really cool. So improvements in general welcome. @wjddn279 The optimization that was added to LocalExecutor is still on my (long) bucket list. I even left a comment in https://github.com/apache/airflow/blob/main/providers/edge3/src/airflow/providers/edge3/cli/worker.py#L460 to remind me whenever I see the code again. Feel free to raise a PR (to be faster than me)! In general I am mostly wondering why at @diogosilva30 this error is happeing often while in our environment we do not see this at all. Must be some environmental side effect. So for the moment I'd accept it as an option but as of memory overhead and time only optional. Main line should be kept as is until more errors reported. I am not sure if for many environments a pre-fork makes sense. It might be another (tuning) option. I assume the typical pattern for celery is different than for Edge... at least I think. I#d in general favor simplicity over complexity. And a pre-fork pool might add a level of complexity making it harder to maintain or adding bugs. My actual "dream" would be that we could make supervisor bing mostly async such that we do not need to fork() or spawn another process at all just to run another process... but just have a set of in-process supervisor instances which themself just fork the execution for the task. Because today we fork worker -> supervisor -> task execution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
