DannyMeyer opened a new issue, #54830:
URL: https://github.com/apache/airflow/issues/54830
### Apache Airflow version
3.0.5
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When running a DAG with multiple simultaneous tasks, only half of the
configured LocalExecutors are started.
For example, executing a DAG with 20 tasks spawns only ~10 LocalExecutors.
As a result, the remaining tasks are left in the queue instead of executing
in parallel.
### What you think should happen instead?
The system should spawn executor instances only to the extent required by
the DAG, but not exceeding the limit defined in the configuration.
### How to reproduce
1. Run brand new installation with virtualenv
2. use configs (additional to standard config)
```
AIRFLOW__CORE__PARALLELISM=64
AIRFLOW__CORE__DAG_CONCURRENCY=32
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG=32
```
3. run dag to test concurrency; around half of the tasks will be queued up
```python3
from airflow import DAG
from airflow.providers.standard.operators.bash import BashOperator
from datetime import datetime
with DAG(
dag_id="test_concurrency",
start_date=datetime(2024,1,1),
schedule=None,
catchup=False,
max_active_tasks=20,
max_active_runs=1,
) as dag:
for i in range(20):
BashOperator(task_id=f"s{i:02d}", bash_command="sleep 5")
```
4. Check amount of running LocalExecutor; Will match with amount of running
tasks at once
```shell
ps aux | grep LocalExecutor | wc -l
```
5. Run warmup dag as an workaround
```python3
from airflow import DAG
from airflow.providers.standard.operators.bash import BashOperator
from datetime import datetime
with DAG(
dag_id="warmup",
start_date=datetime(2024,1,1),
schedule=None,
catchup=False,
max_active_tasks=64,
max_active_runs=1,
tags={"warmup", "system"}
) as dag:
for i in range(64):
BashOperator(task_id=f"warmup_{i:02d}", bash_command="echo
'spooled'")
```
6. Run test_concurrency again, this time no taks will be queued up
### Operating System
Debian 12.11
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
