A bit of an update: discussing some design aspects w Tomasz in the doc.
My plan is to transition this into a Confluence AIP next week (Thursday) and
send a separate note to the mailing list. This should allow people to chip in
with their thoughts before is more established and iterate on it mean
Wrote this small proposal
https://docs.google.com/document/d/1ammZ2iGmGuoXNCtbpLdnj2D23nEYrCaLMbETjim3C0Q/edit?usp=sharing
I can add it to Confluence as well if pointed to where should it live (sorry
not really familiar enough w Airflow Confluence).
Working to add some more context on existing
Ok let's try it, don't know if we're violating some Apache process here but
I guess we'll find out :).
On Wed, Feb 5, 2020 at 4:22 PM Tomasz Urbaszek
wrote:
> Google docs is good to work out final version that can be published on
> confluence. But that’s only my opinion.
>
> T.
>
> On Wed, 5 Feb
Google docs is good to work out final version that can be published on
confluence. But that’s only my opinion.
T.
On Wed, 5 Feb 2020 at 22:12, Dan Davydov
wrote:
> Traditionally we've done this in confluence within the AIP although I think
> I would prefer google docs at some point in the futur
Traditionally we've done this in confluence within the AIP although I think
I would prefer google docs at some point in the future maybe :). I would
use confluence though for this.
On Wed, Feb 5, 2020 at 3:52 PM Gerard Casas Saez
wrote:
> Happy to drive this. What would be a good place to put th
Happy to drive this. What would be a good place to put this design doc?
Guessing confluence, not sure under what directory though.
Gerard Casas Saez
Twitter | Cortex | @casassaez
On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk , wrote:
> +1 short design doc would be cool.
>
> wt., 4 lut 2020, 21:16 u
+1 short design doc would be cool.
wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek <
tomasz.urbas...@polidea.com> napisał:
> Do you think we should start with some design doc for that? In this
> way, we can work out the best solution and allow other to add 2 cents?
>
> T.
>
>
> On Tue, Feb 4, 2
Do you think we should start with some design doc for that? In this
way, we can work out the best solution and allow other to add 2 cents?
T.
On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman
wrote:
>
> I think if we’re not breaking any other operators (which I doubt we are) it’s
> a great 2.0 fe
I think if we’re not breaking any other operators (which I doubt we are) it’s a
great 2.0 feature. It would also look great in a “What’s New in Airflow 2.0”
announcement ;).
Docs are always a challenge, but we could set up a google doc and hack it out
in a day or two.
+1
via Newton Mail
[htt
I like the idea, especially the backwards compatibility.
I would love to understand more about whether it will work (it looks like
it will) without modifying the 100s of operators we already have. If so,
this looks like a nice addition to the current way how we define Dags and
even allows for incr
+1 one for this idea. Something similar popped in my mind when I saw
Kubeflow approach some time ago.
T.
On Mon, Feb 3, 2020 at 8:05 PM Daniel Imberman
wrote:
>
> I like this idea a lot.
>
> We could create something similar to an “executor_config” so people can
> pre-populate most of the para
I like this idea a lot.
We could create something similar to an “executor_config” so people can
pre-populate most of the parameters necessary for a python_operator and pass it
in
e.g.
Config = AirflowPythonConfig(…)
@airflow.make_python_operator(config) def my_func():
On Mon, Feb 3, 2020 a
I like it : ). I think the difficulty in creating operators and chaining
them together is one of the most common complaints about Airflow compared
to other frameworks. Would be curious to see a comparison to other
interfaces e.g. Dagster as well. I would be curious to see what other
committers like
Hi everyone!
Starting a conversation here about extending Airflow for supporting a more
functional way to define DAGs including better data dependency/lineage clarity
on the DAG itself. I believe adding this functional extension would allow to
support more Data pipelines use cases and extend Ai
14 matches
Mail list logo
