eladkal commented on issue #30974:
URL: https://github.com/apache/airflow/issues/30974#issuecomment-2088039805
> Should these rules be defined as Dataset args?
It's a property of a specific DAG that uses the dataset. It's not a property
of the dataset itself.
> A common pattern
uranusjr commented on issue #30974:
URL: https://github.com/apache/airflow/issues/30974#issuecomment-2081854430
I’ve been thinking about this. Yeah the above `wait_no_longer_than`
interface seems to be the way to go (maybe with a different name, say
`DatasetTimeout`).
I initially con
cmarteepants commented on issue #30974:
URL: https://github.com/apache/airflow/issues/30974#issuecomment-2052540712
@uranusjr you're closer to this than I am. What is your opinion?
I think there's a lot of value in supporting these scenarios - but there's
something about the approach
cmarteepants commented on issue #30974:
URL: https://github.com/apache/airflow/issues/30974#issuecomment-1927291755
Gotcha, makes sense.
Paraphrasing:
DAG `C`: DAG that contains Task that outputs to dataset `C`
The crux of the problem is that DAG `C` cannot leverage data
dakshin-k commented on issue #30974:
URL: https://github.com/apache/airflow/issues/30974#issuecomment-1926517418
@cmarteepants Yes that would solve the problem at hand, however it's not
ideal as re-computing `C` multiple times could cause an issue as there's no
guarantee that operation is i
cmarteepants commented on issue #30974:
URL: https://github.com/apache/airflow/issues/30974#issuecomment-1921943496
> Hi, just to provide a bit more info on one of the use cases that inspired
this issue:
>
> * We have two datasets, say A and B, both of which are provided by third
par
