+1 for having a centralized repository for these kind of resources.
Thank you, Robert Sanders for sharing it!
Aizhamal
On Wed, Aug 21, 2019 at 11:59 PM Chao-Han Tsai wrote:
> This is great. I think we can create a repo (or a just under airflow) for
> these common DAGs that will be used by
Re
> What are people's feelings on changing the default execution to schedule
> interval start
and
> I'm in favor of doing that, but then exposing new variables of
> "interval_start" and "interval_end", etc. so that people write
> clearer-looking at-a-glance DAGs
While I am def on board with
I'm in favor of doing that, but then exposing new variables of
"interval_start" and "interval_end", etc. so that people write
clearer-looking at-a-glance DAGs
On Fri, Aug 23, 2019 at 10:39 AM Dan Davydov
wrote:
> What are people's feelings on changing the default execution to schedule
>
What are people's feelings on changing the default execution to schedule
interval start and communicating this to existing users in the Updating
notes so that they can preserve the old behavior? Could potentially cause
headaches for users who don't read the notes but I think it might make
sense to
I am for this change, since I feel like in general the start of the
interval is more intuitive (I have been working on Airflow for 3 years and
this still trips me up). That being said I'm not sure how I feel about
allowing customization at DAG level instead of cluster level as it makes it
harder
GitHub recently introduced the idea of "Draft" PRs:
https://github.blog/2019-02-14-introducing-draft-pull-requests/
Could we do something similar either with that system or something else?
Run a minimal set until it's marked as "ready for testing", and then run a
larger suite.
On Fri, Aug 23,
Maybe your 4th point covers this but there are frequent Doc only changes.
In this case we should not run "real-test" but only static checks - mypy,
pylint. flake8, doc generation.
On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk
wrote:
> Hello everyone,
>
> On top of moving out from Travis (I will
Hello everyone,
On top of moving out from Travis (I will resume working on it next week) I
thought about some ways to improve the feedback cycle time we have with CI
(super long now).
Maybe we should consider being smarter with running tests only when really
needed.
After introducing pre-commit
DST: I recall problems with DST especially when the hour goes back and the
daily schedule time technically occurs twice the same day or does not occur
at all. We have some code that chooses arbitrary the first occurence in the
latter case (there was a problem that it worked differently python 3.6
This is a change to one of Airflow's core concepts, and it would require a
lot of work for existing DAGs to cut over to it. Given that, my personal
preference would be to allow arbitrary customization rather than just a bit
toggle. Such as allowing passing in a mapping function: given an
Changing mid-flight is always a massive edge case already for many parts of the
scheduler. Can we easily test this sort of behaviour in unit tests?
I don't think DST needs extra tests as it uses the existing functions that are
already well tested, no?
-a
> On 23 Aug 2019, at 13:24, Jarek
Happy for it as well. There are a number of cases where scheduling at start
makes more sense and as we see Airflow is used now in multiple cases where
there is no need to process data from an interval and wait until that data
is ready.
But indeed some more tests would be great - especially for
Happy for this feature to merged
On Fri, Aug 23, 2019, 11:49 Ash Berlin-Taylor wrote:
> This has come up a few times before, someone has now opened a PR that
> makes this a global+per-dag setting:
> https://github.com/apache/airflow/pull/5787 and it also includes docs
> that I think does a good
Looks good. Two things im a bit concerned about is:
1. What happens if you changed the choice for the dag mid air?
2. Tests seem a bit light
Cheers
Bolke
Verstuurd vanaf mijn iPad
> Op 23 aug. 2019 om 12:49 heeft Ash Berlin-Taylor het
> volgende geschreven:
>
> This has come up a few times
This has come up a few times before, someone has now opened a PR that makes
this a global+per-dag setting: https://github.com/apache/airflow/pull/5787 and
it also includes docs that I think does a good job of illustrating the two
modes.
Does anyone object to this being merged? If no one says
15 matches
Mail list logo
