AardJan opened a new issue, #55144:
URL: https://github.com/apache/airflow/issues/55144
### Apache Airflow version
3.0.6
### If "Other Airflow 2 version" selected, which one?
2.10.3 and 2.11.0
### What happened?
We’re dynamically generating datasets using dataset alias. With a few or
even a few hundred datasets, no problem.
But when a task generates thousands, the runtime becomes huge. The thing is:
the task itself runs fast — it’s actually spending most of the time outside the
task, processing and writing outlets/metadata.
Two main issues here:
- Performance – A task that should take seconds ends up running for minutes
because of outlets handling.
- UI confusion – The Airflow UI doesn’t reflect what’s really happening:
- Task durations show as 0s, 0.16s, 50s, 0s.
- But the full DAG takes ~2.47 minutes
- While the task is running, the UI shows 1+ minutes elapsed… then when it
finishes, it’s magically reported as ~51s. Users get confused by this mismatch.
There are cases where the discrepancy is more than 4 minutes for a single
task.
During task execution:
<img width="3872" height="1220" alt="Image"
src="https://github.com/user-attachments/assets/41074d20-1fb6-4167-9aaf-9245f7bd4b65";
/>
After DAG finished:
<img width="4138" height="1162" alt="Image"
src="https://github.com/user-attachments/assets/2a78b3c5-3950-4714-bd05-8ab31d703fe9";
/>
### What you think should happen instead?
Optimization how outlets/datasetalias writes are handled at scale,
especially for big dataset packs.
Show this “post-task processing time” somewhere in the UI so users
understand where the time is going.
### How to reproduce
DAG outlet producer:
```
import os
from datetime import datetime
from airflow.sdk import DAG, Asset, AssetAlias, get_current_context, Param
from airflow.decorators import task
from airflow.providers.standard.operators.empty import EmptyOperator
DAG_ID = os.path.splitext(os.path.basename(__file__))[0]
asset_alias = AssetAlias("performance-test-1")
with DAG(
dag_id=DAG_ID,
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
params={
"dataset_qty": Param(1, type="integer", minimum=1),
}
) as dag:
start = EmptyOperator(task_id="start")
@task
def build_dataset_list():
ctx = get_current_context()
qty = int(ctx["params"].get("dataset_qty", 1))
return [f"example_dataset_{i}" for i in range(1, qty + 1)]
@task(outlets=[asset_alias])
def emit(datasets: list[str], outlet_events):
# Airflow will register an event for the AssetAlias because it is in
outlets.
for name in datasets:
print(f"Emitting logical asset event for: {name} via alias
{asset_alias.name}")
outlet_events[asset_alias].add(Asset(name))
return datasets
end = EmptyOperator(task_id="end")
ds_list = build_dataset_list()
start >> ds_list >> emit(ds_list) >> end
```
DAG outlet consumer:
```
import os
from datetime import datetime
from airflow.sdk import DAG, Asset, AssetAlias, get_current_context, Param
from airflow.decorators import task
from airflow.providers.standard.operators.empty import EmptyOperator
DAG_ID = os.path.splitext(os.path.basename(__file__))[0]
asset_alias = AssetAlias("performance-test-1")
with DAG(
dag_id=DAG_ID,
start_date=datetime(2024, 1, 1),
schedule=[asset_alias],
catchup=False,
) as dag:
@task(inlets=[asset_alias])
def consume_dataset_event_from_dataset_alias(*, inlet_events=None):
for event in inlet_events[asset_alias]:
print(event)
consume_dataset_event_from_dataset_alias()
```
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Amazon (AWS) MWAA
### Deployment details
Tested on:
- MWAA 2.10.3
- Local environment with Astro CLI:
- 2.11.0
- 3.0.6
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
