Hello everyone,

I'd like to start a discussion about a new AIP:
AIP-129 Backfill for Asset Consumers with Dated Partitions
https://cwiki.apache.org/confluence/x/HQiWGw
A draft proof of concept is available at
https://github.com/apache/airflow/pull/77152.

This is the first in a series of AIPs that I plan in order to strengthen
asset-based scheduling in Airflow, starting with backfill. Without proper
backfill support, users have to rerun producers or trigger runs by hand,
which can be expensive.

Today, a Dag scheduled on partitioned assets cannot be backfilled, even
when the input partitions already exist. For example, a new weekly report
consumer should be able to build past reports from existing daily sales
partitions without rerunning the sales producer.

The proposal extends the existing backfill mechanism to supported
RollupMapper and IdentityMapper consumers:
- Select consumer partitions by date range or explicit keys.
- Use the newest retained asset event for each required input, rather than
replaying historical events.
- Create at most one consumer run per key per request, and report missing
inputs for incomplete partitions.
- Reuse the existing backfill controls, without rerunning producers or
changing live scheduling.

Identity mapping is included for one-to-one partitioned pipelines, so that
moving a daily consumer from cron to an asset dependency does not mean
losing
backfill support. The AIP describes its date-resolution and historical-data
limitations.

The first version is deliberately narrow. It refuses, among others, hourly
rollups, mixed rollup and identity inputs, OR conditions, and any consumer
when multi-team is enabled.

I'd particularly welcome feedback on a specific design choice:
The proposal leaves live scheduling independent. A pending live row can
therefore produce another run for a key that a backfill has already processed,
even without a new event arriving afterward. Is that trade-off acceptable
for the first version, or should skipping pending keys or coordinated
handling be required?

Feedback on the scope and the other proposed defaults is welcome too.


Thanks,
Shahar

Reply via email to