Re: [DISCUSS] indexes for API calls

be worth to > think about it? Especially because that would open some doors such as using > NoSQL DB engines to run Airflow. > > (Sorry if I deviated the conversation a bit). > > On 2024/05/31 12:10:11 Andrey Anshin wrote: > > IMHO, blindy adding new indexes into the `dag_run

Re: [DISCUSS] indexes for API calls

IMHO, blindy adding new indexes into the `dag_run` and `task_instance` tables will cause additional maintenance costs. There are 8 indexes already exists per each of this tables SELECT pi.schemaname schema_name, pi.tablename table_name, count(*) num FROM pg_indexes pi WHERE

Re: [DISCUSS] Restore the SQL server backend

There was a proposal to keep it in the past [1] with a short explanation why the maintainers did not want to keep it. > many Microsoft customers who are using Airflow Microsoft also supports and participates in the development of PostgreSQL, there is one Core Team member and couple of Major

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

IMHO, In case if we decide to keep only Postgres support we need to have really powerful arguments to provide an interface which helps integrate with other DBs. In this case, we must clearly understand what the community is responsible for in this case and how it can be sure that nothing is

Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

incompatibilities (and prevent adding future ones). We can > see how much complexity we are dealing with when we attempt to enable the > tests for 2.8 and then 2.7 and decide if it's worth it. The change I added > makes it easy to just "enable" tests for 2.8 and 2.7 separately. &

Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

BTW, forget to mention that we should also check Pytest: Good Integration Practices from https://docs.pytest.org/en/stable/explanation/goodpractices.html On Fri, 10 May 2024 at 13:07, Andrey Anshin wrote: > I think the current solution with run tests against installed packages > migh

Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

this discussion - my proposal is to use what we have > > now and simply get back-compatibility working without changing the > > structure (yet), but as part of Airflow 2 vs. Airflow 3 we should make > sure > > this topic is fully covered and we get to consensus on the answers.

Re: [PROPOSAL] Improved compatiblity checks for Providers (running unit tests for multiple airflow versions)

Great job, Jarek! I would have some proposals, which should be considered as a long term We should rework our test structure to fully run provider tests without touching the Core tests. The main problem here is that we configure a lot of things into the root conftest.py which might be a problem

Re: [VOTE] Proposal for adding Telemetry via Scarf

+1 binding On Thu, 9 May 2024 at 13:25, Wei Lee wrote: > Got it. Thanks Jarek for pointing out! > > Best, > Wei > > > On May 9, 2024, at 3:59 PM, Ankit Chaurasia wrote: > > > > +1 non-binding > > > > *Ankit Chaurasia* > > > > > > > > > > > > > > On Thu, May 9, 2024 at 11:16 AM Aritra Basu

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

rflow 5 In the user perspective, they have at least bug fix support for a while, if someone want to use legacy version it their choice, however no new features, no new version of providers (after one year) Best Wishes *Andrey Anshin* On Sat, 4 May 2024 at 19:17, Bolke de Bruin wrote: &

Re: [VOTE] Airflow Providers prepared on May 01, 2024

+1 (binding) Check licences, signatures, checksums, and changes On Fri, 3 May 2024 at 15:58, Pankaj Koti wrote: > +1 (non-binding) > > Tested my set of changes. > > Best regards, > > *Pankaj Koti* > Senior Software Engineer (Airflow OSS Engineering team) > Location: Pune, Maharashtra, India >

Re: [PROPOSAL] Keep >= LATEST MINOR for all providers using common.sql provider

There are some drawbacks I saw here, it would force to upgrade other providers to the latest version. Some scenarios from the my head: End users use amazon provider and google provider, and do not use common.sql and both of them have mandatory common.sql as dependency. if everything works fine

Re: [DISCUSS] Asynchronous SQLAlchemy

If I do not miss something, usage of DB is not covered by Airflow Public Interface, in this case we could easily replace one-by-one sync methods by async. There is some places exists where it might be mixin, as mentioned before Secrets Backend, but it could be done by wrapping it into the: -

Re: [VOTE] March 2024 PR of the Month

+1 for #36755 Python3.12 On Mon, 25 Mar 2024 at 23:54, Briana Okyere wrote: > Hey All, > > It’s once again time to vote for the PR of the Month! > > With the help of the `get_important_pr_candidates` script in dev/stats, > we've identified the following candidates: > > PR #37937: refactor:

Re: [VOTE] Release Airflow 2.8.4 from 2.8.4rc1

+1 binding Checked files, licences, signatures and also my changes. One small nit that I've found is that the key which is used for sign releases is expired, I'm not sure if it should be considered as a showstopper. ❯ gpg --verify apache_airflow-2.8.4-py3-none-any.whl.asc

Re: [VOTE] Remove experimental API

Imberman, Vikram Koka, Xiaodong Den +1: 3 votes: Andrey Anshin, Jarek Potiuk, Hussein Awala +0: 1 vote: Elad Kalif On Fri, 22 Mar 2024 at 15:43, Andrey Anshin wrote: > When I prepared the voting results I figured out that I do not vote by > myself. > > So here my +1 binding > >

Re: [VOTE] Remove experimental API

When I prepared the voting results I figured out that I do not vote by myself. So here my +1 binding On Wed, 20 Mar 2024 at 22:40, Andrey Anshin wrote: > There is no specific rush, in case it is considered as > experimental feature, this vote shows that it is not, it might be r

Re: [VOTE] Remove experimental API

hould do > >> so > >>> as > >>>> part of a major breaking release. Perhaps if we haven’t already we > >> should > >>>> at least add deprecation warnings? > >>>> > >>>> -1 but very down to add deprec

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

> It handles everything. Now if you want to send a Slack message from a > PythonOperator you need to initialize a hook, find the right function to invoke etc. > And to Elad point " "I know there is an operator that does X, so I will just use it inside the python function I invoke from the

Re: [DISCUSS] Applying D105 rule for our codebase ("undocumented magic methods") ?

±0 from my side Maybe we have to review all current methods which do not follow this rule to find a really useful meaning, and do not enforce (disable it). So for avoid unnecessary changes we might close https://github.com/apache/airflow/issues/37523 and remove/mark completed into the

Re: [VOTE] Remove experimental API

we > > should > > > > at least add deprecation warnings? > > > > > > > > -1 but very down to add deprecation warnings > > > > > > > > On Sat, Mar 16, 2024 at 4:19 PM Bas Harenslak > > > > > > > > > wrote: >

Re: [VOTE] Remove experimental API

Mar 2024 at 00:02, Andrey Anshin wrote: > I just wonder if `Experimental` is covered by > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features > ? > Or is it just another meaning of Experimental ? > > > > > On Sat, 16 Mar

Re: [VOTE] Remove experimental API

I just wonder if `Experimental` is covered by https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features ? Or is it just another meaning of Experimental ? On Sat, 16 Mar 2024 at 23:39, Jarek Potiuk wrote: > Would you still vote `-1` of course was the

[VOTE] Remove experimental API

Greetings everyone! I would like to start a vote proces about removal of Experimental API support into the next minor Airflow version, presumably 2.9, but it could be postponed to 2.10. By default experimental REST API turned off, and we recommend to use stable REST API:

Re: [DISCUSS] Deprecate cli options in Airflow Configurations and airflow.api.client

in > > problem that it did not exist back then). > > > > I'd say it calls for removal and significant note ("If you were still > using > > it, please use the REST API and Python Client instead"). > > > > On Tue, Feb 20, 2024 at 11:19 AM Andrey Anshin

Re: [VOTE] January 2024 PR of the Month

+1 for #34225 That was long journey for complete the PR On Wed, 28 Feb 2024 at 01:36, Hussein Awala wrote: > +1 for #37058 > > On Tue, Feb 27, 2024 at 9:06 PM Scheffler Jens (XC-AS/EAE-ADA-T) > wrote: > > > Oh I totally missed #36797 - I was

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

ve it for "internal" task_id > etc. etc. > > I think that would be super-powerful, as TP wrote - operator would mostly > fade away - but not disappear entirely > > J. > > > > On Tue, Feb 27, 2024 at 10:06 AM Andrey Anshin > wrote: > > > I think

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

I think manually using *execute*, *poke*, *choose_branch* and etc. should be considered as an antipattern, these methods only should be invoked by Airflow Worker. You never know what should be done before, especially if it about deferrable operator or sensors in reschedule mode On Tue, 27 Feb

Re: [VOTE] Airflow Providers prepared on February 23, 2024

+1 (binding) Verify signatures, checksums, files On Fri, 23 Feb 2024 at 20:40, Jarek Potiuk wrote: > Yep. I saw they tested it in > https://github.com/apache/airflow/issues/37648 . > Cool :) > > On Fri, Feb 23, 2024 at 5:36 PM Jarek Potiuk wrote: > > > +1 (binding): checked reproducibility,

Re: [VOTE] Airflow Providers prepared on February 19, 2024

+1 (binding) Verify signatures, checksums, files On Tue, 20 Feb 2024 at 15:53, Jarek Potiuk wrote: > +1 (binding). Tested signatures, licences, checksums, reproducibility. All > looks good. > > On Mon, Feb 19, 2024 at 1:51 PM Wei Lee wrote: > > > +1 (non-binding) > > > > Best, > > Wei > > >

Re: [DISCUSS] Deprecate cli options in Airflow Configurations and airflow.api.client

/jdz9l7bsnsw5c3t27dxfrx5pd4wvjlxt On Tue, 20 Feb 2024 at 00:14, Andrey Anshin wrote: > Just for clarification, the proposal is about depreciation or even removal > of something obsolete. If we could do some improvement at the same moment > it would be nice, if not also nice, that mean no additional work > >

Re: [DISCUSS] Considering trying out uv for our CI workflows

> > > other more important things. It is quite ok to use other tooling in > > places > > > like our CI, where they do some parts of the installation better. > > > > > > For me `pip` is going more into the direction of `usable reference > &g

Re: [DISCUSS] Considering trying out uv for our CI workflows

Yesterday my friend shared with me that tool and I've been told that more presumably it would be a niche tool. I've been told "who needs yet another installer which stands to resolve all your problems' '. I guess I was wrong? On Tue, 20 Feb 2024 at 00:53, Jarek Potiuk wrote: > Hey everyone, > >

Re: [DISCUSS] Deprecate cli options in Airflow Configurations and airflow.api.client

reply-to-all thred to define > the scope of cleanup. Also and especially for API cöient we need to check > for impact if breaking changes appear. > > Jens > > Sent from Outlook for iOS<https://aka.ms/o0ukef> > > From: Andrey Anshin > Sen

Re: [DISCUSS] Deprecate cli options in Airflow Configurations and airflow.api.client

terface of Airflow > > https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html > - and since the very beginning we did not have an intention for it to be > public API. > > So I would be very much for removing that whole part - even in Airflow 2.9. > > J. &

Re: [DISCUSS] Deprecate cli options in Airflow Configurations and airflow.api.client

mething we want to keep or do we want a different > design? Just dropping the implementation seems a bit 'lazy' to me, while I > understand that it hasn't been touched in a while. > > B. > > On Sun, 18 Feb 2024 at 12:17, Andrey Anshin > wrote: > > > Greetings guy

[DISCUSS] Deprecate cli options in Airflow Configurations and airflow.api.client

Greetings guys! I want to start a discussion about deprecation cli configuration options https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#cli as well as `airflow/api/client` in upcoming minor/feature release of Airflow (2.9 or 2.10 depends on). This options control

Re: [VOTE] Airflow Providers prepared on February 17, 2024

+1 (binding) Verify signatures, checksums, files On Sun, 18 Feb 2024 at 11:09, Jarek Potiuk wrote: > +1 (binding). Checked my changes, Verified reproducibility, signatures, > licences, checksums, all looks good. > > On Sun, Feb 18, 2024 at 6:10 AM Amogh Desai > wrote: > > > +1 non binding >

Re: [VOTE] Airflow Providers prepared on January 30, 2024

+1 binding > On 3 Feb 2024, at 03:39, Hussein Awala wrote: > > +1 (binding) > >> On Fri, Feb 2, 2024 at 1:28 PM Ephraim Anierobi >> wrote: >> >> +1 (binding) >> >>> On 2024/01/30 16:42:07 Elad Kalif wrote: >>> Hey all, >>> >>> I have just cut an ad-hoc release for the microsoft.azure

Re: [VOTE] AIP 61 - Hybrid Executors

+1 binding > On 1 Feb 2024, at 09:12, Oliveira, Niko wrote: > > Hey folks, > > > The AIP for Hybrid Executors has been out for a few weeks now. Some great > feedback came in and some challenges to scope which I think have all been > addressed, and the AIP document has been updated where

Re: [VOTE] Add the ability to report slack messages that don't meet code of conduct

I’m also for some bot/application, after quick search I’ve found this one https://github.com/slackapi/slack-reporting-tool I’m not sure is it suits our requirements and it required instance where this code would be deployed but it worthwhile to check it, especially if someone familiar with

Re: [VOTE] January 2024 PR of the Month

My vote goes to #36537. But I think we should mention PR #22253 as a PR which finally merged - "keep calm and add changes" On Tue, 23 Jan 2024 at 17:36, Wei Lee wrote: > My vote is for #36537. Love this packaging improvement! > > Best, > Wei > > > On Jan 23, 2024, at 10:30 PM, Ryan Hatter > >

Re: [VOTE] Release Airflow 2.8.1 from 2.8.1rc1

+1 binding Checked signatures, files, licences and also my changes One small nit which I've noticed when executing check_files.py. After we switch to hatch the source distribution name include non-normalized name: use _ (underscore) instead of - (hyphen): apache_airflow-2.8.1.tar.gz instead of

Re: [DISCUSS] Future of Pendulum in Airflow

gt; > > need to go as before. > > > > > > > > Honestly, I'm a bit scared (I do see the lack of responsiveness). I > > hope > > > > someone can take that away. > > > > > > > > Bolke > > > > > > > > > &

Re: [DISCUSS] "Require conversation resolution" in our PRs before merge?

We could try and if found it slows down for some reason we always might revert it back. Just one suggestion, sometimes discussion contains some useful information, e.g. "What the reason of finally decision", "Useful information why it should works by suggested way, or should not work", which

Re: [PROPOSAL] NEED URGENT FEEDBACK! MySQL Badly signed repo breaks Airflow images

I would rather stick to the Option 2, with small modifications: By default use MariaDB libraries which compatible with MySQL, so change everywhere INSTALL_MYSQL_CLIENT_TYPE=mariadb - For x86_64 keep option to switch to Oracle MySQL libraries - For ARM it would always forced (or until Oracle

Re: [LAZY CONSENSUS] Remove 'plexus' provider

Lazy consensus has been reached. On Tue, 5 Dec 2023 at 13:08, Andrey Anshin wrote: > Greetings everyone! > > I would like to raise a Lazy consensus about removing Plexus provider. > > In short - we want to remove this provider because Plexus was acquired by > AMD, all host

Re: [LAZY CONSENSUS] Remove 'apache.scoop' provider

Lazy consensus has been reached. On Tue, 5 Dec 2023 at 13:06, Andrey Anshin wrote: > Greetings everyone! > > I would like to raise a Lazy consensus about removing Apache Scoop > provider. > > In short - we want to remove the provider because the Apache Scoop PM

[LAZY CONSENSUS] Remove 'plexus' provider

Greetings everyone! I would like to raise a Lazy consensus about removing Plexus provider. In short - we want to remove this provider because Plexus was acquired by AMD, all host to API unavailable and there is no information about the current state of Plexus. For more detail you could have a

[LAZY CONSENSUS] Remove 'apache.scoop' provider

Greetings everyone! I would like to raise a Lazy consensus about removing Apache Scoop provider. In short - we want to remove the provider because the Apache Scoop PMC was terminated and the project moved into the Apache Attic. For more detail you could have a look following proposal:

Re: [ANNOUNCE] New committer: Utkarsh Sharma

Congrats!  On Mon, 4 Dec 2023 at 22:45, Amogh Desai wrote: > This is fantastic! Congratulations Utkarsh. > > Well deserved! > > Thanks & Best Regards, > Amogh Desai > > > On Mon, 4 Dec 2023 at 10:56 PM, Pankaj Singh > wrote: > > > Congratulations Utkarsh  > > > > On Mon, Dec 4, 2023 at

Re: [DISCUSS] Allowlist in serializatin [was: Serialization in Apache Airflow]

> Pickle and the likes can execute arbitrary code that is inside the > serialized object. > > Yep. This is super dangerous indeed. My fifty cents. This sounds scarier than it actually is, it mostly covered by this simple things: - Do not open attachment from unknown senders / untrusted sources

Re: [DISCUSS] Future of Pendulum in Airflow

pat > layer > > where we could optionally use pendulum instead would be the most "easy" > > approach for our users. Most of them would not even notice if we do a > good > > job, but those who somehow depend on pendulum objects would get a > > possibility to

Re: [DISCUSS] Future of Pendulum in Airflow

ake part and speed up > releases > >> and > >> > > > merging of issues that are blocking us (or would be blocking us in > >> the > >> > > > future). > >> > > > > >> > > > I tried to be very friendl

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

> Recently, a method was added `to_json_dict` which gives the json repr. This method is internal method which use in 1. Connection Serialisation, and it is a replacement *to_dict* method which was used before https://github.com/apache/airflow/pull/35723 2. Use in public method *as_json*

Re: [VOTE] Airflow Providers prepared on November 24, 2023

Tested my changes +1 (non binding) Best Wishes *Andrey Anshin* On Mon, 27 Nov 2023 at 10:30, utkarsh sharma wrote: > +1 (non binding) > > Thanks, > Utkarsh Sharma > > On Mon, Nov 27, 2023 at 11:59 AM Phani Kumar > wrote: > > > +1 non binding > >

Re: [DISCUSS] Suspend/Remove Apache Scoop provider

z aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > +1 > > Le jeu. 23 nov. 2023 à 17:52, Hussein Awala a écrit : > > > +1 > > > > On Thu, Nov 23, 2

[DISCUSS] Suspend/Remove Plexus provider

Greetings everyone! Not much time has passed since the last discussion about suspension/removal providers 藍 It is time to discuss Plexus Provider. During check providers links, I've found that link [1] from the Plexus provider description [2] can not be resolved. The same thing happened with

[DISCUSS] Suspend/Remove Apache Scoop provider

Greetings everyone! Since we began to actively use the mechanism to suspend/remove providers I want to start the discussion about suspend and potential remove Apache Scoop [1] provider. Apache Scoop moved into the Attic in July 2021 [2] due to inactive development [3] and Apache Scoop PMS was

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

honest I think it is hardly possible to implement this one in Airflow 2.x Best Wishes *Andrey Anshin* On Sun, 19 Nov 2023 at 02:25, utkarsh sharma wrote: > I do think the URI format for airflow connections also serves the purpose > of conveying the intent, similar to how we

[PROPOSAL] Deprecate URI Connection representation in favor of JSON

and we should keep Airflow Connection as URI in the future Airflow's major versions and support both ways to provide connections as JSON and URI. Best Wishes *Andrey Anshin*

Re: [DISCUSS] Future of Pendulum in Airflow

Just to clarify I'd like us to consider the possibility that no new pendulum would be released or released at the end of 2024, like a pessimistic scenario: - What should we do in this case? - Work out a backup plan. Best Wishes *Andrey Anshin* On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk

Re: [PrOPOSAL] Change default docker image to point to "latest supported"

of python, because some packages stop the build wheel for almost the EOL Python version. - Ctrl+C; Ctrl+V Personally I could not see any benefit of these changes but if someone wants to see this change I would just say 'Why not?". Best Wishes *Andrey Anshin* On Fri, 17 Nov 2023 at

Re: [DISCUSS] Future of Pendulum in Airflow

to resolve this without breaking changes? Because for me it would be a little weird if removal pendulum would be a main driver for release Airflow 3 Best Wishes *Andrey Anshin* On Thu, 28 Sept 2023 at 13:01, Andrey Anshin wrote: > This discussion is more about the known problem of pendul

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

), there is no process of removal at that moment. * 1 provider suspended and removed (Qubole) Best Wishes *Andrey Anshin* On Fri, 17 Nov 2023 at 12:17, Amogh Desai wrote: > Theres very little incentive in maintaining this if theres no one actively > maintaining it. > > I am totally for th

Re: [PrOPOSAL] Change default docker image to point to "latest supported"

version changed and it would change in any cases Best Wishes *Andrey Anshin* On Fri, 17 Nov 2023 at 10:10, Wei Lee wrote: > Agreed, as long as users can still use different versions through tags, > there are no drawbacks or incompatibilities with this great idea! >

Re: [ANNOUNCE] New committer: Jens Scheffler

Congrats Jens!   Best Wishes *Andrey Anshin* On Tue, 7 Nov 2023 at 23:33, Utkarsh Sharma wrote: > Congratulations Jens! :) > > Thanks, > Utkarsh Sharma > > On Wed, Nov 8, 2023 at 1:01 AM Vincent Beck wrote: > > > > Welcome onboard Jens! > > >

Re: [Reminder] How to reproducibly install Airflow

Maybe we could improve situations it by pin third meta issue into the Github Issues which are described best practices include: - Reproducible install - What we expect of good bug/feature request - Information about third-party Managed Airflow Best Wishes *Andrey Anshin* On Sat, 4 Nov 202

Re: [DISCUSS] Removing Qubole provider (and adding removal process)

I think in the case of Qubole it is pretty easy to remove it from the provider codebase. I'm pretty sure that almost no one even noticed this removal. Best Wishes *Andrey Anshin* On Thu, 26 Oct 2023 at 23:06, Bolke de Bruin wrote: > I suggest also removing it from pypi for secur

Re: Keep Mssql support

DB might stop us to implement other stuff around DB schema, e.g.: https://github.com/apache/airflow/pull/34112 So my assumption that MsSQL will have support until Airflow 2.7.x Best Wishes *Andrey Anshin* On Wed, 25 Oct 2023 at 21:00, Scheffler Jens (XC-DX/ETV5) wrote: > In response, off

Re: [Discussion] Requesting Feedback on Airflow Survey

4.x) something different rather than Postgres and SQLite (for testing) Best Wishes *Andrey Anshin* On Wed, 25 Oct 2023 at 19:08, Briana Okyere wrote: > Hey All, > > For the last few years, we've sent out surveys to get a sense of the state > of this Airflow community, and this

Re: [VOTE] Add providers for Pinecone, OpenAI & Cohere to enable first-class LLMOps

+1 binding Best Wishes *Andrey Anshin* On Wed, 25 Oct 2023 at 14:25, Kaxil Naik wrote: > Hello everyone, > > Following the discussion about adding five new providers, I am calling for > an official vote on adding providers for Pinecone, OpenAI & Cohere to th

Re: Keep Mssql support

ne exception with Google Composer v1 which seems use MySQL, even on Azure Data Factory Managed Airflow use Postgres as DB backend, see: https://learn.microsoft.com/en-us/azure/data-factory/concept-managed-airflow#architecture Best Wishes *Andrey Anshin* On Mon, 23 Oct 2023 at 23:01, agateaaa

Re: [DISCUSSION] Add 5 new Providers to enable first-class LLMOps

pull/34891#discussion_r1362910782 Add OpenAI Provider: https://github.com/apache/airflow/pull/35023#discussion_r1365235167 Add Weaviate Provider: https://github.com/apache/airflow/pull/35060/files#r1365765741 Best Wishes *Andrey Anshin* On Tue, 17 Oct 2023 at 22:42, Kaxil Naik wrote: >

Re: [DISCUSS] Future of Pendulum in Airflow

, and as expected it work now fine I do not check on Airflow codebase yet Best Wishes *Andrey Anshin* On Thu, 28 Sept 2023 at 18:03, Bolke de Bruin wrote: > FYI: > > I've just added: > > https://github.com/apache/airflow/pull/34667 > > which documents how to use newer

Re: [DISCUSS] Future of Pendulum in Airflow

mentations, and it looks like chaos right now. If you interested you could check this comment https://github.com/apache/airflow/pull/34492#issuecomment-1728301574 In additional pendulum.tz.timezone.Timezone in pendulum 3.0 will based on ZoneInfo Best Wishes *Andrey Anshin* On Thu, 2

Re: [DISCUSS] Future of Pendulum in Airflow

ted regular one 2. We would have fully qualified name backports.zoneinfo.ZoneInfo in Python 3.8 and zoneinfo.ZoneInfo in Python 3.9+. I think with some simple workaround it also could be solved Best Wishes *Andrey Anshin* On Thu, 28 Sept 2023 at 17:12, Bolke de Bruin wr

Re: [DISCUSS] Future of Pendulum in Airflow

; defacto standard for a long time). Pendulum has proven us in the past, > maybe we indeed should help the project if possible and if that isn't > possible verify formal correctness of any other library. > > Bolke > > [1] https://pendulum.eustace.io/faq/ > [2] https://gith

[DISCUSS] Future of Pendulum in Airflow

solution. I can't imagine how we could do that, because a lot of stuff depends on the pendulum and removing it would be a breaking change. Best Wishes *Andrey Anshin*

[DISCUSS] What we should do with DbApiHook.get_uri?

this methods might not work correctly without re-implementation 2. Implements `get_uri` without Connection.get_uri for Group 3 3. Raise NotImplemented error in `get_sqlalchemy_engine` for Group 4 My hope for collective wisdom/mind and help in solving this puzzle by proposing other ideas. Best Wishes *Andrey Anshin*

Re: [ANNOUNCE] Pankaj Koti and Amogh Desai as committers

Congrats Pankaj! Congrats Amogh! Best Wishes *Andrey Anshin* On Tue, 19 Sept 2023 at 18:41, Avi wrote: > Awesome! Congratulations Pankaj and Amog! > > > On Tuesday, Sep 19, 2023 at 14:35, Vikram Koka > mailto:vik...@astronomer.io.invalid)> > wrote: > >

Re: [VOTE] Drop MsSQL as supported backend

+1 (non-binding) RIP MS SQL as a backend Best Wishes *Andrey Anshin* On Thu, 31 Aug 2023 at 09:04, Daniel Imberman wrote: > +1 binding > > On Wed, Aug 30, 2023 at 10:01 PM Ryan Hatter > wrote: > > > +1 non-binding > > > > On Mon, Aug 28, 2023

Re: [DISCUSS] Preventing users from misusing _PIP_ADDITIONAL_REQUIREMENTS ?

as it is :D In addition we have `airflow standalone`, maybe we should make work DEBUG on top of it? No idea how to resolve this into docker endpoint :-( Best Wishes *Andrey Anshin* On Thu, 31 Aug 2023 at 09:38, Jarek Potiuk wrote: > Yep. I am also for the idea described by Pierre. In sh

Re: [DISCUSS] Preventing users from misusing _PIP_ADDITIONAL_REQUIREMENTS ?

ore, and there is no appearance of _PIP_ADDITIONAL_REQUIREMENTS in Airflow Core, this only part of docker image. For implements this warning we need to make Airflow Core know about this environment variable, even if it os.environ.get("_PIP_ADDITIONAL_REQUIREMENTS", "") ---- Best Wishes *A

Re: [DISCUSS] Preventing users from misusing _PIP_ADDITIONAL_REQUIREMENTS ?

y to get a quick answer for an issue/discussion like "My Airflow deployment does not start with exit code 42", rather than "My Airflow deployment constantly restarted". But I'm not sure if it is applicable and user-friendly. Best Wishes *Andrey Anshin* On Tue, 29 Aug 2

Re: [Discussion] DB backend versions policy

major version of Postgres 李 Best Wishes *Andrey Anshin* On Thu, 16 Feb 2023 at 23:39, Jarek Potiuk wrote: > Yeah. I think we only made "cloud" exception for K8S version because we > thought it is really something of a deployment environment. And I am still > not 100% co

Re: [Discussion] DB backend versions policy

*Andrey Anshin* On Mon, 13 Feb 2023 at 23:51, Ferruzzi, Dennis wrote: > Very detailed, thanks.I think I want to lean towards whatever the > official support for the package is and not measure ourselves by what the > various SaaS options are doing. I think there will always be s

Re: [Discussion] Deprecate auto cleanup RenderedTaskInstanceFields and decouple k8s_pod_yaml

into the TOAST table (facepalm) Best Wishes *Andrey Anshin* On Thu, 2 Feb 2023 at 05:58, Kaxil Naik wrote: > Maybe the “auto-maintenance” is true. Hopefully for not long though :) > > On Thu, 2 Feb 2023 at 01:41, Kaxil Naik wrote: > >> >> >Users often asked

Re: [Discussion] Deprecate auto cleanup RenderedTaskInstanceFields and decouple k8s_pod_yaml

h not deleting the rows > automatically. > But only if we limit that and handle the "worst" case properly. > > That would be my proposal how we can handle it to get both views taken > into account. > > What do you Andrey (and others) think? Does it make sense? Or do we &

[Discussion] DB backend versions policy

I do not include information about MS SQL Server, because it has experimental support. And also I do not think that we require any policy for SQLite, since it is only for development purposes. Best Wishes *Andrey Anshin*

Re: [Discussion] Deprecate auto cleanup RenderedTaskInstanceFields and decouple k8s_pod_yaml

bles" rather than the current default implementation? Because I get the result which is opposite what you said. And rendered_task_instance_fields don't grow faster than other tables that what I got. I would like to compare it with other findings and some reproducible metrics rather than with hypo

Re: [Discussion] Deprecate auto cleanup RenderedTaskInstanceFields and decouple k8s_pod_yaml

Airflow usage than for Airflow 2.3+ AIRFLOW__CORE__MAX_NUM_RENDERED_TI_FIELDS_PER_TASK=0 it is a valid option, especially if quite a few dynamic tasks uses, because right now it conflicts to each other Best Wishes *Andrey Anshin* On Tue, 31 Jan 2023 at 11:12, Jarek Potiuk wrote

Re: [Discussion] Deprecate auto cleanup RenderedTaskInstanceFields and decouple k8s_pod_yaml

ers the ability to clean up rendered templates tables, there could be another option: - Do not delete records on every task instance run. - Delete once per defined period (hourly, daily, weekly, monthly). In this case you really could not care about locks. WDYT? Best Wishes *Andrey Anshin* On

Re: [Discussion] Deprecate auto cleanup RenderedTaskInstanceFields and decouple k8s_pod_yaml

that immediately deleted), the same is valid for cleanup old tasks. Best Wishes *Andrey Anshin* On Sun, 29 Jan 2023 at 14:16, Jarek Potiuk wrote: > Yep. Agree this is not an efficient query and dynamic task mapping > makes the effect much worse. Generally speaking, selecting "wh

Re: [DISCUSSION] Move K8S and Celery Executors (and related) to respective providers?

Should we also create apache-airflow-providers-dask for DaskExecutor? Best Wishes *Andrey Anshin* On Sun, 29 Jan 2023 at 13:21, Jarek Potiuk wrote: > Hello Everyone, > > As a follow-up to AIP-51 - when it is completed (with few more quirks > like the one describ

Re: [Discussion] Deprecate auto cleanup RenderedTaskInstanceFields and decouple k8s_pod_yaml

ld cleanup only rendered fields, so it would be user's choice cleanup or not before migration, do periodical maintenance or not ---- Best Wishes *Andrey Anshin* On Sat, 28 Jan 2023 at 23:41, Kaxil Naik wrote: > Potentially it is a good idea to deprecate this option and recommend for >&

[Discussion] Deprecate auto cleanup RenderedTaskInstanceFields and decouple k8s_pod_yaml

as already tried or investigated this? *Skeleton #2:* We have a k8s_pod_yaml field which is exclusively used by K8S executors. Should we also decouple this field as part of AIP-51? Best Wishes *Andrey Anshin*

Re: [VOTE] Airflow Providers prepared on January 23, 2023

+1 (non-binding) Best Wishes *Andrey Anshin* On Mon, 23 Jan 2023 at 18:01, Elad Kalif wrote: > Hey all,I have just cut ad hoc release for Airflow Providers packages. This > email is calling a vote on the release,which will last for 72 hours - which > means that it will end o

Re: [ANNOUNCE] New committer Niko Oliveira (o-nikolas)

Congrats Niko!   Best Wishes *Andrey Anshin* On Tue, 20 Dec 2022 at 19:01, Beck, Vincent wrote: > Congrats Niko!!  > > > > *From: *Pankaj Singh > *Reply-To: *"dev@airflow.apache.org" > *Date: *Monday, December 19, 2022 at 9:13 PM > *To: *"d

Re: [PROPOSAL] (Internal) move of provider packages to isolated "providers" sub-folders

. In case of utilities we need to change imports from from tests.test_utils import awesome_stuff to from some_awesome_name.utils import awesome_stuff Best Wishes *Andrey Anshin* On Wed, 14 Dec 2022 at 15:46, Jarek Potiuk wrote: > Hey Denis, > > > tests/providers/airbyte/hoo

  1   2   >