>From the earlier discussions with David - this is also (and mainly) about optimisation. Those operators do very little, and when you add total overhead that Airflow adds for scheduling and running every task, then it turns out that looping such operator's execute in a single interpreter is many, many times faster (several orders of magnitude) than running them sequentially as tasks.
On Fri, Oct 4, 2024 at 10:16 AM Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > Well, it looks like we do have concurrency control for mapped tasks after > all. > > See max_active_tis_per_dagrun which was added in > https://github.com/apache/airflow/pull/29094. > > So this would allow you to map over your 3000 users in a single run, but > process only one at a time (or 5 or 10 at a time). Does that help your use > case? >
