GitHub user PrithwishDas101 added a comment to the discussion: Coverage-based completeness check for partitioned assets (streaming producers)
I like the idea of starting with the task-side sensor before changing the scheduler's partition model. That would let the team validate the use case with a smaller implementation before committing to schema changes and new rollup semantics. One thing I'd clarify early is the interval contract: - Treat intervals as half-open `[start, end)` and normalize timestamps consistently, including timezone handling. - Make duplicate and replayed events idempotent so retries don't distort coverage. - Define overlap handling explicitly. Unioning intervals seems like a reasonable default for completeness, while “latest event wins” probably needs a clearer definition of what makes an event supersede another. For the initial implementation, an `extra`-based convention seems useful for experimentation, provided the required fields and validation rules are documented. If the approach proves useful at scale, the need for indexed range queries could help justify a more formal representation later. I'd be interested in seeing the sensor approach validated against irregular batch sizes, gaps, overlapping events, and reprocessing before deciding whether scheduler-level coverage belongs in the core partition model. GitHub link: https://github.com/apache/airflow/discussions/74595#discussioncomment-18946230 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
