echoleein opened a new issue, #77148:
URL: https://github.com/apache/airflow/issues/77148
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
We are experiencing an intermittent issue with PartitionedAssetTimetable in
Apache Airflow 3.3.1.
Most scheduled DAG runs work as expected. However, approximately once a
week, the downstream DAG fails to trigger even though all 7 required asset
events have been successfully emitted with the same partition key.
File 1 — repro_asset_checker.py
```
from datetime import timedelta
import time
from airflow import DAG
from airflow.sdk import Asset, CronPartitionTimetable
from airflow.operators.python import PythonOperator
# Shared asset definitions
ASSETS = [
Asset(f"repro://asset_{i}")
for i in range(1, 8)
]
def emit_asset(delay_seconds=0, **context):
if delay_seconds:
time.sleep(delay_seconds)
print(
f"Asset producer completed. "
f"Partition: {context['partition_key']}"
)
with DAG(
dag_id="repro_asset_checker",
schedule=CronPartitionTimetable(
"0 18 * * *",
timezone="UTC",
run_offset=0,
key_format="%Y%m%d",
),
catchup=False,
max_active_runs=1,
tags=["asset-repro"],
) as dag:
for i, asset in enumerate(ASSETS):
PythonOperator(
task_id=f"emit_asset_{i + 1}",
python_callable=emit_asset,
op_kwargs={
# First event arrives earlier.
# Remaining 6 arrive approximately
# 2 seconds later.
"delay_seconds": 0 if i == 0 else 2,
},
outlets=[asset],
)
```
File 2 — repro_asset_consumer.py
```
from airflow import DAG
from airflow.sdk import Asset, PartitionedAssetTimetable
from airflow.operators.python import PythonOperator
ASSETS = [
Asset(f"repro://asset_{i}")
for i in range(1, 8)
]
def process_partition(**context):
print(
"Consumer triggered for partition: "
f"{context['partition_key']}"
)
asset_expression = ASSETS[0]
for asset in ASSETS[1:]:
asset_expression = asset_expression & asset
with DAG(
dag_id="repro_asset_consumer",
schedule=PartitionedAssetTimetable(
assets=asset_expression,
),
catchup=False,
max_active_runs=1,
tags=["asset-repro"],
) as dag:
PythonOperator(
task_id="process_partition",
python_callable=process_partition,
)
```
Since the 7 entries each have a different partitioned_dag_runs ID, it
appears we received 6 and 1 entries, respectively.
Observed Result
```
{
"partitioned_dag_runs": [
{
"id": 288,
"partition_key": "20261115",
"total_received": 6,
"total_required": 7,
"state": "pending",
"created_dag_run_id": null
},
{
"id": 287,
"partition_key": "20261115",
"total_received": 1,
"total_required": 7,
"state": "pending",
"created_dag_run_id": null
}
]
}
```
### What you think should happen instead?
Expected Result:
- Partition: 20261115
- Asset Events: 7 / 7
- Pending Dag Runs: 0
- Created Dag Runs: 1
- Consumer state: success
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]