Using the SequentialExecutor is very strongly discouraged for anything other than "playing around with my first DAG".
LocalExecutor will run tasks in a pool of processes allowing more than one to run at the same time and is what I would recommend. Switching to this would let you very easily see if it behaves the same. I've not noticed this behaviour with LocalExec. -ash > On 17 Dec 2019, at 00:20, Reed Villanueva <[email protected]> wrote: > > Have multiple dags and airflow set to run with SequentialExecutor. Fine with > the fact that only one task will run at a given time with this execution > mode, but want dags that are scheduled at the same time to have tasks > scheduled in a balanced way (currently appears that dag A (which starts 5min. > earlier than dag B) gets more of its tasks scheduled while B often just > waiting around, even after both have been running for some time). > > Is there a way to have airflow tasks explicitly scheduled in balanced way > across dags? Any other workaround to this situation (wouldn't want to > implement any concurrency > <https://airflow.apache.org/docs/stable/faq.html#how-can-my-airflow-dag-run-faster> > just to see the same balancing problem magnified where one dag still gets > all the scheduling, just more of it)? > > > This electronic message is intended only for the named > recipient, and may contain information that is confidential or > privileged. If you are not the intended recipient, you are > hereby notified that any disclosure, copying, distribution or > use of the contents of this message is strictly prohibited. If > you have received this message in error or are not the named > recipient, please notify us immediately by contacting the > sender at the electronic mail address noted above, and delete > and destroy all copies of this message. Thank you.
