Hallo,
Please unsubscribe me from your lists.
Thank you.
Best Regards,
Konstantinos Daniilidis
>This change would bring parity between the `output` property and the
classic `xcom_pull()` method. The obvious drawback is this would be a
slight authoring change for existing DAGs that use the `output` property.
Perhaps if the change could be automated in migration tooling the behavior
change wou
I've also witnessed the zipped DAGs feature to be used quite a bit - in
scenarios similar to what Jarek & Constance described, and also to e.g.
avoid downloading a multitude of files from blob storage (less effective
cost & performance wise).
On Wed, Feb 5, 2025 at 6:08 PM Constance Martineau
wro
Mock and Connection don’t make sense together. A Connection is effectively a
dataclass that is loaded from the DB. There’s nothing to mock.
Also are you aware that you can set `ARIFLOW_CONN_*` environment variables and
those will be looked at before the DB. That plus
[monkeypatch.setenv](https:
I am also +1 for dropping 3.9 support early. That can save a lot of
boilerplate for __future__ for type hints as well.
As well as Ash sais we would most probably anyway only support 3.0 and
3.1 with 3.9
And I can say, migrating Python from 3.8 straight into 3.12 was only
generating one bug in ou
Hey All,
Here are the minutes and recording from our February 2025 Airflow Town
Hall! <
https://docs.google.com/document/d/1TSr6Kxb338aTFglAkTpwyJ9xcqCxxLUrvV54NjuR5PM/edit?usp=sharing
>
A big thanks to our speakers Ash-Berlin Taylor, Amogh Desai, Buğra Öztürk, Evie
Crutchley, and Jens Scheffler!
> Pulling a connection from the DB itself shouldn’t/can’t be slow - It’s a
> single row. I think I’m just confused or misdirected about your comment about
> database here. Can you give a concrete example of the change you would make,
> and how this will speed things up?
It's not only pulling, I
+1 binding
On Fri, Feb 7, 2025 at 4:36 PM Shahar Epstein wrote:
> +1 non-binding
>
> On Wed, Feb 5, 2025 at 12:04 AM Jarek Potiuk wrote:
>
> > Hey all,
> >
> > I have just cut the new wave Airflow Providers packages. This email is
> > calling a vote on the release,
> > which will last for 72 ho
+1 non-binding
On Wed, Feb 5, 2025 at 12:04 AM Jarek Potiuk wrote:
> Hey all,
>
> I have just cut the new wave Airflow Providers packages. This email is
> calling a vote on the release,
> which will last for 72 hours - which means that it will end on February 07,
> 2025 21:58 PM UTC and until 3
As clearly explained in the above example - this is not a problem with
databricsk and azure. This is aproblem in Azure BECAUSE we move Databricks.
So clearly a side effect of one group of tests on another. As such - we
have no idea how many other cases we have like that and who might be
causing it.
> How is it flakey though? This is the part I’m asking for explaining
about. Surely it’s messed up and _ALWAYS_ messed up?
See above example posted in the thread. It explains everything. I am not
sure if you missed it, but it's the third time I am explaining it and you
keep on asking. The example
How does caplog introduce flakiness though? That still isn’t
> On 7 Feb 2025, at 11:07, Jarek Potiuk wrote:
>
> Ash - take a close look at the example I sent. Maybe you missed it.
>
> * Caplog gets logs from loggers - If your loggers are messed 1000 tests
> before and not restored - you will
https://endoflife.date/python for a visual of the versions and their support
status for anyone following along.
We do have a published get-out-of-jail card:
> This policy is best-effort which means there may be situations where we might
> terminate support earlier if circumstances require it.
I would be for it - but this should be accompanied with a clear proposal of
the policy we are going to use forward for Airflow 3. We cannot make such
"ad-hoc" decisions based on "I want to use that package". We need to have
solid reasoning and clear indication for our users what kind of support
the
Hi all,
I have a proposal that we increase the minimum required python version to 3.10,
in a slight departure from our published python version req
https://github.com/apache/airflow?tab=readme-ov-file#support-for-python-and-kubernetes-versions
As a reminder, Python 3.9 is already in security on
And just to clarify what I personally think of a social / community effect
here and be very blunt about it - sorry for those who might find it harsh
I personally think it is good that there is more (personal) friction to
work on 3.1 related features than working on making Airflow 3.0 a success.
De
Agree with Ash. 100% focus on what we want to do for Airflow 3 first. If
anyone wants to work on Airflow 3.1 features, they could work in their own
branches/forks and keep rebasing it.
J.
On Fri, Feb 7, 2025 at 10:22 AM Ash Berlin-Taylor wrote:
> We “could", but creating a branch for 3.1 will
> The providers tests will soon (but possibly not before 3.0 at this point)
need to be converted to use the TaskSDK properly which won’t/can't actually
use the DB, so we will need to do something soon.
Just to clarify - my goal is absolutely to have all providers use Task SDK
before Airflow 3.0. A
Awesome!!
Good work from the community, features are building like lego blocks :D!
Thanks & Regards,
Amogh Desai
On Fri, Feb 7, 2025 at 3:49 PM Rahul Vats wrote:
> Awesome! We have started testing mapped tasks with Alpha2.
>
> Regards,
> Rahul Vats
>
> On Fri, 7 Feb 2025 at 15:01, Ash Berlin-
Aleksander - feel free. If you want to fix it - feel free to do so. So far
it failed mine, Jens' and about 5 other people attempts to find and fix the
Caplog problems and we all failed miserably - always reverting to log
mocking.
But I am sure it. An be fix, only I think we have more important th
Ash - take a close look at the example I sent. Maybe you missed it.
* Caplog gets logs from loggers - If your loggers are messed 1000 tests
before and not restored - you will have failing tests. Side effects
* Mocking logs is mocking lag method call. It always work and is
side-effect resistant
Awesome! We have started testing mapped tasks with Alpha2.
Regards,
Rahul Vats
On Fri, 7 Feb 2025 at 15:01, Ash Berlin-Taylor wrote:
> The big new part of AIP-72 (Task Execution Interface) that was included in
> this Alpha was mapped tasks — so you should now be able to use and test
> mapped ta
The big new part of AIP-72 (Task Execution Interface) that was included in this
Alpha was mapped tasks — so you should now be able to use and test mapped tasks
(and task groups) in Celery and Local executor.
-ash
> On 6 Feb 2025, at 21:03, Vikram Koka wrote:
>
> Awesome!
> Great to see the co
The providers tests will soon (but possibly not before 3.0 at this point) need
to be converted to use the TaskSDK properly which won’t/can't actually use the
DB, so we will need to do something soon.
> Hence that’s why when I do refactorings in provider unit tests, I’ve already
> replaced those
Great idea, I support :)
On Fri, Feb 7, 2025 at 9:35 AM Blain David wrote:
> Hello,
>
>
>
> The caplog vote triggered me to launch this proposal as it’s also related
> to unit testing, and as I think we want our unit tests as clean and as
> simple and as fast as possible.
>
> I think it would be
We “could", but creating a branch for 3.1 will be an absolute nightmare to keep
it up to date with main and it will conflict almost every day. I’m not touching
that with someone else’s bargepole.
In short: best not to work on anything for 3.1 yet as there’s nowhere we can
merge it to that doesn
I'd like to second this. If there is something wrong with caplog then maybe
it is better to fix it rather than move to a self maintained alternative?
Newcomers still will be prefered to use common capsys before they faced
some local specifics and wonder why it still exists.
--
,,,^..^,,,
On Fri,
+100 to what Jarek and Michal said. Changing DAG code for this will seriously
impact Airflow 3 adoption.
We can move the code to a provider without having to change the DAG Author
interface can’t we? For instance we can change the DAG parser to convert these
options in to an SMTP notifier can’t
How does replacing caplog with mocking help anything, assuming that you test
what is logged against the mock?
Please explain it to me like I’m 5, cos I don’t see how it makes the blindest
bit of difference. Testing what is on the caplog output vs what is recorded on
the mock log object is funct
+10 on that. My next step after finishing Provider's move, was to make
essentially all unit tests in Providers non-DB tests and removing "real
connection" usage is part of it.
This is essentially stage 3 of
https://github.com/apache/airflow/issues/42632 that is planned and I want
to make POC and i
I will need one more PMC member binding vote to release it before we
release 2.10.5 (Ideally today so that I release tomorrow morning and update
constraints for 2.10.5).
On Fri, Feb 7, 2025 at 8:10 AM Blain David wrote:
> +1 non-binding.
>
> Kind regards,
> David
>
> -Original Message-
>
31 matches
Mail list logo
