argibbs commented on issue #34339: URL: https://github.com/apache/airflow/issues/34339#issuecomment-1719040510
> About timeouts, it seems a bit suspicious when 200 dags can't be processed in reasonable time, so I guess this some dynamic DAGs, which created from single python file. > > The might be a good reason why some bad things happen, just because Airflow could decide that DAG might be deleted. I'm not sure in this case it is a bug or it's "by design". Some links which might improve your DAG parsing time Processing time is really variable. Sometimes it's 4 seconds, sometimes it's 25, sometimes it's 50+. And before you ask, no the box CPU is pretty constant. However, I'm doing my best to eliminate exogenous factors before I start claiming the problem is 100% in airflow. I'm not really sure I believe that a dag flickering in and out of existence could ever be a good thing. OK, yes, it led me to discover the dag processor timeout, _but only once I'd dropped down to a single scheduler_. There must be better ways. Thank you for the links, there are a couple of new features in there since 2.3.3 that I wasn't aware of, especially the parsing context. It would be good to get parse time down (rather than just upping the timeout) because that'll have other benefits. That said, I think this is an orthogonal problem (i.e. off-topic) for the original issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
