Hello there, Please advise if you can. In our environment, we have noticed that dynamic tasks place a lot of stress on scheduler, webserver and DB (connections wise) We are running about 1000 Dynamic Tasks every 30 min and parsing time increases from 5 to 65 sec with Runtime from 2sec to 350+ . This happens at execution time then it drops to normal while still executing tasks. Webserver hangs for few minutes. Running 1.10.1.
Dynamic Tasks: Number of DAGs: 44 Total task number: 950 DagBag parsing time: 65.879642000000001 Static Tasks: Number of DAGs: 73 Total task number: 1351 DagBag parsing time: 1.731088 Is this something you aware of? Any advises on Dynamic tasks optimization/best practices? Thank you in advance, Eugene
