Taragolis commented on issue #34339: URL: https://github.com/apache/airflow/issues/34339#issuecomment-1719126077
> There are a couple of other reasons why we build our own, but that may change in the future. If not yet, you should consider to use any process daemon/supervisor behind your shell entrypoint. As I mention before it could be tiny or dumb-init both of them presented in debian repo. `dumb-init` uses in official Airflow image, `tiny` uses in official Prefect image, as user I can't find any differences between this two init managers both of them do great job - no more zombie process. > highly dynamic graphs of small lightweight tasks. It also could be a reason of fantom failures, if DAG graph could be change during execution, e.g.: task deleted, dependency between tasks changed and etc, then a lot of funny side effects could happen. > Processing time is really variable. Sometimes it's 4 seconds, sometimes it's 25, sometimes it's 50+. And before you ask, no the box CPU is pretty constant. However, I'm doing my best to eliminate exogenous factors before I start claiming the problem is 100% in airflow. - DAG generated thought the DB? - DAG placed on some network share, e.g. NFS? --- The more I look at this issue the more filling that is not an issue rather then discussion and troubleshooting 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
