GitHub user ldacey added a comment to the discussion: Add the ability to
backfill a DAG based on past Asset Events
I moved to all asset-based schedules and I use incremental windows (track max
timestamp for extracts and then use that next schedule) instead of data
intervals which backfill supports.
Since my assets are Iceberg or Delta Tables I agree that backfilling is
possible since basically all I need is a way to fill in partitions. The data
interval would be the partition. My assets pass the `partition_filter` though
extra events between each layer which is used to read the data to process it
for the next layer.
For now, I have variations of this which handles backfilling outside of Airflow
and I change it to month / year / day depending on the table partitioning.
```
_start = date(2016, 6, 1)
_end = date(2026, 2, 1)
_current = _start
while _current < _end:
_next_month = _current + relativedelta(months=1)
process_bronze_to_silver(
partition_filters=[
("date", ">=", _current),
("date", "<", _next_month),
]
)
_current = _next_month
```
GitHub link:
https://github.com/apache/airflow/discussions/59886#discussioncomment-15561193
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
