Lee-W commented on code in PR #58543:
URL: https://github.com/apache/airflow/pull/58543#discussion_r2979672314
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -2227,6 +2228,50 @@ def _update_state(dag: SerializedDAG, dag_run: DagRun):
dag_run.run_id,
)
continue
+ # For AssetAndTimeSchedule, defer starting until all required
assets are queued.
+ if isinstance(dag.timetable, AssetAndTimeSchedule):
+ # Count required assets for this DAG's schedule
+ required_count = (
Review Comment:
Instead of doing this, we probably should do something similar to
`dag_ready` in `dag.py`
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -2227,6 +2228,50 @@ def _update_state(dag: SerializedDAG, dag_run: DagRun):
dag_run.run_id,
)
continue
+ # For AssetAndTimeSchedule, defer starting until all required
assets are queued.
+ if isinstance(dag.timetable, AssetAndTimeSchedule):
+ # Count required assets for this DAG's schedule
Review Comment:
```suggestion
# Count required assets for this Dag's schedule
```
##########
airflow-core/src/airflow/timetables/assets.py:
##########
@@ -90,3 +94,75 @@ def generate_run_id(self, *, run_type: DagRunType, **kwargs:
typing.Any) -> str:
if run_type != DagRunType.ASSET_TRIGGERED:
return self.timetable.generate_run_id(run_type=run_type, **kwargs)
return super().generate_run_id(run_type=run_type, **kwargs)
+
+
+class AssetAndTimeSchedule(Timetable):
Review Comment:
I think we'll also need a task sdk counter part
##########
airflow-core/src/airflow/models/dag.py:
##########
@@ -679,6 +679,12 @@ def dag_ready(dag_id: str, cond: SerializedAssetBase,
statuses: dict[UKey, bool]
for ser_dag in ser_dags:
dag_id = ser_dag.dag_id
statuses = dag_statuses[dag_id]
+ timetable = ser_dag.dag.timetable
Review Comment:
What is this part for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
