The vote has passed with 6 binding +1 votes and no -1 votes.
I will proceed with a PR to implement the proposal.
Votes:
Jarek Potiuk +1 (binding)
Arthur Wiedmer +1 (binding)
Tomasz Urbaszek +1 (binding)
Daniel Standish +1 (binding)
Dennis Akpenyi +1 (non-binding)
Drew Hubl +1 (non-binding)
Josh F
Hi Jarek,
Appreciate the background of this move. That makes sense to keep it as
warning.
Thanks,
Ping
On Thu, Feb 10, 2022 at 1:40 PM Jarek Potiuk wrote:
> This has been changed BACK to warning in this change
> https://github.com/apache/airflow/pull/21074 after this issue reported:
> https:
This has been changed BACK to warning in this change
https://github.com/apache/airflow/pull/21074 after this issue reported:
https://github.com/apache/airflow/issues/20709
The change introduces a new type of error that a provider can raise if
there is an "optional feature" that depends on not inst
Hi all,
I am thinking if we can make the warning level to debug level or make it
configurable to ignore those warnings from:
```
except ImportError as e:
# When there is an ImportError we turn it into debug warnings as
this is
# an expected case when only some providers are ins
One more thing, callbacks (daniel) - also bad thing is that we really need
it to set the "dag status" not run anything. The "watcher" is mainly there
to propagate the "failure" status to Dag to get the Dag "fail" when any of
the tasks fail - if we have callbacks for all tasks, The group
approach al
@dstandish.- Yeah - callback is a bit too "invasive". I thought about it
but at least some examples use default_args and adding "callback" there
would mix the concerns and make it difficult to extract parts of the
examples in our documentation.
@ash - yeah. I realized that when you posted it that
The other thing that comes to mind is you can add your "normal" tasks to a
task group and then do `my_group >> watcher`
Also I noticed that dag can take success / failure callbacks. Maybe the
"watcher" task makes sense as a callback.
The one caveat to this is you have to do it "last" (obviously) and the
direction matters.
For instance `watcher() << dag.tasks` doesn't work as it tries to set
watcher to a dependency of itself which falls foul of the cycle
detector (the Acyclic property of DAG is enforced).
But given list >
Ash! you are my hero :)
On Thu, Feb 10, 2022 at 7:58 PM Ash Berlin-Taylor wrote:
> dag.tasks >> watcher()
>
> No new syntax nor pre-commit needed :)
>
> On Thu, Feb 10 2022 at 15:35:58 +0100, Jarek Potiuk
> wrote:
>
> Hey everyone,
>
> I have a small proposal about adding a new overloaded oper
dag.tasks >> watcher()
No new syntax nor pre-commit needed :)
On Thu, Feb 10 2022 at 15:35:58 +0100, Jarek Potiuk
wrote:
Hey everyone,
I have a small proposal about adding a new overloaded operator -
initial proposal '>>=` (and a method), resulting from our discussion
on AIP-47 - system te
I like the concept, but don't have strong opinions on implementation. One
thought would be that a similar operator may be handy if we're making that
change. "run task ONLY IF an upstream task failed" might compliment your
proposed "run task EVEN IF an upstream task failed". For example, noti
Hi all,
As part of AIP-42 we need to split out the task instance log files to
include the "map index". The simple way of doing this would be just to
add another sub-folder, so for instance, we'd have
{{ dag_id }}/{{ task_id }}/{{ logical_date }}/{{ map_index }}/{{
try_number }}.log
ma
Hey everyone,
I have a small proposal about adding a new overloaded operator - initial
proposal '>>=` (and a method), resulting from our discussion on AIP-47 -
system tests refactoring (
https://lists.apache.org/thread/htd4013yn483qfhwv11vc26jpf2yvjph).
The ">>=" operator is not really 100% neces
Any more comments here or in the PR?
On Sun, Feb 6, 2022 at 1:32 PM Jarek Potiuk wrote:
> I do not think we reached consensus yet on those but I thought maybe just
> attempting it and showing how much more consistent and better it would
> look,
> I have attempted to describe the policies I had i
>
>
> I think both these sticking points are really a trade-off of simplicity vs
> consistency/reliability. And to be clear I'm not arguing for things to be
> more complex just for the heck of it, I agree that simplicity is great! But
> just that there needs to be a balance and we can't get caught
On Wed, Feb 9, 2022 at 10:35 PM Giorgio Zoppi
wrote:
> Hello,
> we had some experiments last year about using Airflow to run tests
> directly from the CI, it was good but at the end we had no way to complete
> the solution, also
> it was a bit too much.
>
Yeah. Having "full" airflow to run the t
16 matches
Mail list logo
