Re: [I] extra flag to make catchup=False mean "first run is next scheduled" [airflow]
potiuk commented on issue #45691: URL: https://github.com/apache/airflow/issues/45691#issuecomment-2599659501 For now - I convert it to a discussion as this certainly not an issue or a simple feature that is clear whether or if we should follow it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] extra flag to make catchup=False mean "first run is next scheduled" [airflow]
potiuk closed issue #45691: extra flag to make catchup=False mean "first run is next scheduled" URL: https://github.com/apache/airflow/issues/45691 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] extra flag to make catchup=False mean "first run is next scheduled" [airflow]
potiuk commented on issue #45691: URL: https://github.com/apache/airflow/issues/45691#issuecomment-2599659352 > "eh, I don't want to have to look at another argument in the config" Yes. that's a very good reason in fact. Adding more confusion and options is not desireable "product" property. Sometimes even at the expense not handling all cases. You can have a product with million configurable parameters that is useless and far too generic. So "I do not want to have yet another knob to turn" is quite a good reason for not accepting it - from product point of view, even if individual cases are not happy. Generally it's impossible to make everyone happy, some people will still be somewhat unhappy. This looks like an important change in behaviour that also might impact some of the other discussions we have about Airlfow 3 - namely a lot of discussions about https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-83+amendment+to+support+classic+Airflow+authoring+style and resulting in https://cwiki.apache.org/confluence/display/AIRFLOW/Option+2+clarification+doc+WIP . While not 100% related, this PR and the backfill change mentioned by Elad are very much related to the catchup behaviour and show how you should approach such discussions. My suggestion is @seth-olsen if you feel very strongly about this one, start a discussion on devlist and put forth your arguments. Analyse all the past dicussions that @eladkal so helpfully provided, read them in detail, anylyse why thigns were rejected - try to understand other's arguments (even if you do not agree with them, trying to understand what others are saying is a good idea), and come up with a concrete proposal how you think your case should be addressed and justify it. Eventually everything we do is consensus driven (i.e. we want to get to consensus where generally we agree to a direction) but if we cannot reach consensus, the last resort is voting https://www.apache.org/foundation/voting.html Note that - similarly to what Daniel did, when presenting your proposal you shoudl consider all the cases and combinations in your proposal, what it means what consequences it has when introduced, what it means for backwards compatibility etc.. Just thorough thinking followed by discussion, reaching consensus and if not possible, defining the outcome and calling for a vote. Your case is way simpler than what Daniel discussed, but the mechanism is very similar. So I propose you start a `[DISCUSS]` thread on our devlist, where you explain rationale and refer to the past dicussions - but make sure it covers all cases and earlier arguments , let it run for a while and have people discuss it, drive it to consensus if possible and then call for a vote when you think general direction is set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] extra flag to make catchup=False mean "first run is next scheduled" [airflow]
seth-olsen commented on issue #45691: URL: https://github.com/apache/airflow/issues/45691#issuecomment-2599539169 As the thread and stale/closed PRs mention, there are many cases where the workaround with start_date is undesirable or even impossible. I'm not sure why neither of those PRs was accepted, seemed like a lot of the opposition was along the lines of "eh, I don't want to have to look at another argument in the config". Very glad I didn't waste time writing up a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] extra flag to make catchup=False mean "first run is next scheduled" [airflow]
eladkal commented on issue #45691: URL: https://github.com/apache/airflow/issues/45691#issuecomment-2595356010 Related: https://lists.apache.org/thread/j2n8h308ffq46sx0vcfnl61snh7tyjlo and https://github.com/apache/airflow/pull/38168, https://github.com/apache/airflow/pull/35392 The current behavior is consistent with how data pipeline works. Today you are processing yesterday data. Thus, if you set catchup=False and you'd like to avoid creating the first run then your `start_date` needs to be adjusted with +1 day. Generally speaking, this behavior is not so suitable for other patterns. The backfill pain will be resolved with [AIP-78 Scheduler-managed backfill](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-78+Scheduler-managed+backfill) the rest will be handled in Airflow 3.1+ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] extra flag to make catchup=False mean "first run is next scheduled" [airflow]
boring-cyborg[bot] commented on issue #45691: URL: https://github.com/apache/airflow/issues/45691#issuecomment-2593714123 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] extra flag to make catchup=False mean "first run is next scheduled" [airflow]
seth-olsen opened a new issue, #45691: URL: https://github.com/apache/airflow/issues/45691 ### Description Currently, `catchup=False` means "first run after turning on will be the latest scheduled run" and there is no way to produce the behavior where turning on a DAG means "make the first run after turning on occur at the next schedule time instead of the previous one". Proposal: make a new flag (call it something to the effect of `no_catchup_means_no_past_runs`) which when set to `True` means that turning on a DAG that has the setting `catchup=False` will result in the first run being the one on the schedule that occurs next in time (rather than the one that occurs in the most recent past). ### Use case/motivation The use case is whenever you want to turn on a DAG without it running right away. ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
