Yeah I am solely focusing on the DAG authors' API - I trust Daniel and team (in
not working on the impl of this AIP) to do whatever is right there.
> I personally think we should encourage the context manager use case
> for regular use, but leave the direct manipulation of dependencies for
> powe
Yep I think we are all converging.
I **think** the context manager is good (and I saw it initially in the
doc from Daniel) and I tend to agree this (or similar) syntactic sugar
will be the way people will interact with setup/teardown.
I personally believe there are two slightly independent stream
The "update-provider-dependencies" is the right pre-commit name BTW,
not "generate...".
On Fri, Mar 24, 2023 at 6:47 PM Jarek Potiuk wrote:
>
> The provider.yaml is the source of information, but when you install
> application via setup.py locally after changing it, you need to
> generate the "ge
The provider.yaml is the source of information, but when you install
application via setup.py locally after changing it, you need to
generate the "generated/provider_dependencies.json" - this is the file
that is used by setup.py when you install it via ".[extras]". This
file will get automatically
Just want to thank you Ash for seriously engaging with the proposal and
trying to find a solution to your concerns. I am optimistic that we can
find common ground and get this feature out there.
OK so with that dag example we looked at very similar examples. I'd love
it if we could do this (and
While waiting on an adjacent PR
(https://github.com/apache/airflow/pull/30067) to holistically update
Google Cloud provider deps, I'm attempting to update the provider.yaml
for Google Cloud that upgrades google-cloud-dataproc from 5.0.0 to
5.4.0 in order to test an enhancement to Dataproc operators
Okay, after chatting with TP a bit this morning (due to being easy for me to
grab on Slack in an overlapping timezone) I think I've realised our disconnect.
We both want explicit, but what we see as explicit is different!
To me, the explicit was "you've entered a task group" (or a DAG, because a
Two points:
> I like Daniel's idea: implement this _in_ the provider itself (means you
> can't use in memory cache, but can use the DB or redis etc) then it doesn't
> even need user to upgrade to 2.6.
> Also it means that a problem with a single provider doesn't make all of core
> significantly
I like Daniel's idea: implement this _in_ the provider itself (means you can't
use in memory cache, but can use the DB or redis etc) then it doesn't even need
user to upgrade to 2.6.
Also it means that a problem with a single provider doesn't make all of core
significantly more complex.
And it
Jarek I can argue that by doing that the problem is not gone but just
reduced to a level where it's even harder to spot and adresses by the
users. This is the worst kind of problems to tackle!
In my prespective if we don't address the issue directly (general solution
to top level code issues) any
Added :)
On Fri, Mar 24, 2023 at 4:30 AM Bowrna Prabhakaran wrote:
>
> Can I get added to the invitation as well? (mailbow...@gmail.com)
> Thanks
>
> On Fri, Mar 24, 2023 at 2:37 AM Jarek Potiuk wrote:
>
> > did
> >
> > On Thu, Mar 23, 2023 at 9:22 PM c c wrote:
> > >
> > > Can I be added to th
TL;DR; I am in favour of the change (limiting it to DagFileProcessor
only) and turning using Variables at top-level as good practice
(2.6+).
Here is why.
I partially agree with Elad /Ash, but I am not 0/1 on that subject any
more. I think there is a more nuanced approach that we could take
here.
12 matches
Mail list logo
