I think AIP is borderline, but would probably err on the side of a tiny AIP
since it's a fairly large change in a part of Airflow that is touching the
user interface.
I do not think we should support RunTimeParams to modify the topology (at
> least at the beginning).
I strognly agree and think we
How should we go about this? Is an AIP needed? GitHub issues?
Given most implementation for the backend seems to be done, it may be just
needed to do a few issues on GitHub and work on them.
Gerard Casas Saez
Twitter | Cortex | @casassaez
On Jun 16, 2020, 2:07 AM -0600, Tomasz Urbaszek , wrote:
+1 for the idea
Tomek
On Tue, Jun 16, 2020 at 1:39 AM Kaxil Naik wrote:
> Oh yes that sounds good, +1 to the idea as long as it can return a JSON
> serializable object I am fine with it.
>
> On Tue, Jun 16, 2020 at 12:29 AM Gerard Casas Saez
> wrote:
>
> > By XCom support before XComArg I mean
Oh yes that sounds good, +1 to the idea as long as it can return a JSON
serializable object I am fine with it.
On Tue, Jun 16, 2020 at 12:29 AM Gerard Casas Saez
wrote:
> By XCom support before XComArg I meant as XCom parameters for operators.
> You needed to use {{contex[‘ti’].xcom_pull(…)}} in
By XCom support before XComArg I meant as XCom parameters for operators. You
needed to use {{contex[‘ti’].xcom_pull(…)}} instead of using XComArg objects
like you can do as latest master.
Gerard Casas Saez
Twitter | Cortex | @casassaez
On Jun 15, 2020, 5:02 PM -0600, Kaxil Naik , wrote:
> Isn't
It is possible, but this is similar to XCom support before XComArg, only allow
string templatized fields + difficult to specify from the DAG.
My proposal is more focused on making it more functionally defined for the
functional DAG approach.
dag = DAG(
dag_id='airflow_tutori
Isn't it already possible using params (
https://github.com/apache/airflow/blob/master/airflow/models/dag.py#L138-L141
)?
Sample Usage:
https://gist.github.com/kaxil/335d90da8821a4e515046ff0f470fc97#file-airflow_params_usage_2-py
Currently, we allowing passing params in the DAG and overriding the
I do not think we should support RunTimeParams to modify the topology (at least
at the beginning).
Modify the topology involves quite a bit more of deeper changes. Even though it
may be useful, I believe the value/time tradeoff, is high, so focusing on
enabling parametrization on fixed topology
I think this is a great idea! One thing that I think we should figure out
before implementing is how to do so alongside DAG serialization, i.e.
letting these params modify DAG topology might make it hard to store
serialized representations for the Airflow services to consume and render,
though that
As we wrap the work on AIP-31 (functional definition), I wanted to bring
another idea here for discussion.
The concept is to parametrize pipelines using a similar class than XComArg that
we introduced recently. As of 1.10.10, we can use the UI to set the DagRun
configuration on the trigger DAG
10 matches
Mail list logo
