Re: [ANNOUNCE] New committer: Ryan Hatter

gt; > -Original Message- > > > > From: Jed Cunningham > > > > Sent: Friday, 28 June 2024 19:58 > > > > To: dev@airflow.apache.org > > > > Subject: [ANNOUNCE] New committer: Ryan Hatter > > > > > > > > The Project Management Com

Refactor Scheduler Timed Events to be Async?

This might be a dumb question as I don't have experience with asyncio, but should the EventScheduler in the Airflow scheduler be rewritten to be asynchronous? The so called "timed events" (e.g. zombie reaping,

Re: [CALL FOR HELP] Help on Connexion 3 migration needed

Does the scope of this PR warrant an AIP? On Tue, Apr 16, 2024 at 6:40 AM Jarek Potiuk wrote: > Hello here, > > I have a kind request for help from maintainers (and other contributors who > are not maintainers) - on the Connexion 3 migration for Airflow. PR here > (unfortunately - it's one big

Re: Subject: Expressing Interest in Contributing to Apache Airflow

It's also worth checking out the community page , and specifically joining the Airflow community Slack . On Thu, Feb 15, 2024 at 10:24 AM Jarek Potiuk wrote: > Hello Rahul, > > There is nothing more than the

Re: [LAZY CONSENSUS] Rename slack channels

Ah! Great idea! On Tue, Feb 13, 2024 at 12:22 PM Akash Sharma <2akash111...@gmail.com> wrote: > +1 > > On Tue, 13 Feb 2024, 22:50 Briana Okyere, > wrote: > > > +1 > > > > On Mon, Feb 12, 2024 at 11:42 AM Jarek Potiuk wrote: > > > > > Hey here, > > > > > > Following the earlier discussion, I am

Re: [VOTE] AIP 61 - Hybrid Executors

+1 non-binding. This will be a great feature. On Thu, Feb 1, 2024 at 1:27 PM Ferruzzi, Dennis wrote: > +1 binding > > > - ferruzzi > > > > From: Igor Kholopov > Sent: Thursday, February 1, 2024 5:31 AM > To: dev@airflow.apache.org > Subject: RE: [EXTERNAL]

Re: [ANNOUNCE] Starting experimenting with "Require conversation resolution" setting

In my experience outside of Airflow, the benefit of not missing a review comment outweighs the friction of being required to resolve each conversation. On Mon, Jan 29, 2024 at 8:47 PM Wei Lee wrote: > I didn't notice much of a difference as a contributor. +1 vote > > Best, > Wei > > > On Jan

Re: [VOTE] January 2024 PR of the Month

Gotta agree with Constance and go with 22253 -- how cool that the author stuck with it all this time! On Tue, Jan 23, 2024 at 12:25 AM Aritra Basu wrote: > My vote is for #36537 it's been a huge effort and it makes huge > improvements in our packaging. Great to see it make it into airflow. > >

Re: [DISCUSSION] Enhanced Multi-Tenant Dataset Management in Airflow: Potential First Steps

I don't think it makes sense to include the create endpoint without also including dataset update and delete endpoints and updating the Datasets view in the UI to be able to manage externally created Datasets. With that said, I don't think the fact that Datasets are tightly coupled with DAGs is a

Re: AIP-61 - Hybrid Executors

ore. And > note that this is how Airflow behaves today already. > > Does that clear things up? Let me know if it doesn't and I'll have a third > go at it :) > > Cheers, > Niko > > > From: Ryan Hatter > Sent: Thursday, January 18,

Re: [DISCUSSION] Enabling `pre-commit.ci` application for Airflow

I'm in favor of this. I love making docs changes directly in GitHub, but I often make a tiny mistake like a trailing space and the tests fail. I think things like this discourage new contributors, as contributing to docs is the easiest way to start getting involved. On Thu, Jan 4, 2024 at 12:16 

Re: AIP-61 - Hybrid Executors

> > *IMPORTANT NOTE*: task instances that run on the default/environment > executor (i.e. with no specific override provided) will not persist the > executor in the same way so that they can be re-run/retried on any executor. Does this mean that any task that doesn't have the `executor`

Re: The "no_status" state

if scheduler passes along a task and decides that it is not > > ready > > > > to schedule to have an additional state calling e.g. “not_ready” in > the > > > > state model between “none” and “scheduled”. > > > > > > > > Finally on the other h

Re: [PROPOSE] Airflow Monthly Town-Hall

I'd like to be involved! On Tue, Nov 28, 2023 at 4:16 PM Viraj Parekh wrote: > I think this is a great idea -- I've heard from a lot of folks in the > community that it can be hard to keep up with everything going on with > Airflow. I think the community is really good at communicating these

Re: [VOTE] November PR of the Month

Another +1 for 32646... this will make DAG owners' lives much easier :) On Tue, Nov 28, 2023 at 4:21 PM Hussein Awala wrote: > +1 for #32646 > > On Tue, Nov 28, 2023 at 6:32 AM Rahul Vats wrote: > > > +1 for #32646 > > > > Regards, > > Rahul Vats > > > > On Tue, 28 Nov, 2023, 09:55 Aritra

Re: [VOTE] Add providers for Pinecone, OpenAI & Cohere to enable first-class LLMOps

+1 (non-binding) On Thu, Oct 26, 2023 at 9:32 AM Oliveira, Niko wrote: > +1 (binding) > > looking forward to having more native LLM capabilities in Airflow! > > > From: Aritra Basu > Sent: Wednesday, October 25, 2023 12:10:00 PM > To: dev@airflow.apache.org >

Re: Airflow Docs Development Issues

t, > > > tactical > > > > > > > >> solution > > > > > > > >> > > (for > > > > > > > >> > > > > the > > > > > > > >> > > > > > most

Airflow Docs Development Issues

*tl;dr* 1. The GitHub Action for building docs is running out of space. I think we should archive really old documentation for large packages to cloud storage. 2. Contributing to and building Airflow docs is hard. We should migrate to a framework, preferably one that uses markdown

The "no_status" state

Over the last couple weeks I've come across a rather tricky problem a few times. One DAG run gets "stuck" in the queued state, while subsequent DAG runs will be stuck running (screenshot below). One of these issues was caused by `max_active_runs` being met when a task instance from a previously

Re: Airflow projects

You can find some Airflow community resources here: https://airflow.apache.org/community/ Also feel free to join the Airflow community Slack: https://apache-airflow-slack.herokuapp.com/ On Mon, Sep 18, 2023 at 7:59 AM Avitabayan Sarmah wrote: > Thank you, I will do that. > > Regards, > Avitab

Re: [VOTE] September 2023 PR of the Month

Gotta go with #28900 -- what a huge scope! On Tue, Sep 26, 2023 at 1:45 PM Michael Robinson wrote: > Hi folks, > > It’s once again time to vote for the PR of the Month. > > With the help of the `get_important_pr_candidates` script in dev/stats, > I’ve identified the following candidates: > > *

Re: [VOTE] Airflow Providers prepared on September 08, 2023

+1 (non-binding). My change works *mostly* as expected, and the unexpected behavior isn't really a problem On Mon, Sep 11, 2023 at 1:40 PM Josh Fell wrote: > +1 (non-binding) > > Tested my changes (and another related

Re: Lazy Consensus - Removing the Experimental tag for Pluggy

+1 (non-binding) I've seen this used as a workaround for implementing a cluster policy when (for whatever reason) modifying airflow_local_settings.py is not possible. On Fri, Sep 8, 2023

Re: [DISCUSS] move from semver to a more "rolling" release cycle for core

What about more frequently using news fragments + using configuration settings as a way to introduce breaking changes that users can revert? Disabling "trigger dag with config" without Params by default

Re: [VOTE] Drop MsSQL as supported backend

+1 non-binding On Mon, Aug 28, 2023 at 3:29 PM Aritra Basu wrote: > +1 (non-binding) > Based on reading the previous mails, looks like a good idea to drop along > with the migration support > > -- > Regards, > Aritra Basu > > On Mon, Aug 28, 2023, 11:33 PM Oliveira, Niko > > wrote: > > > +1

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

I’d also like to be added please :) > On Apr 13, 2021, at 21:27, Xinbin Huang wrote: > >  > Hi Daniel & Ian, > > I am also interested in the idea of a serialization representation that can > be executed by workers directly. Can you also add me to the call? > > Thanks > Bin > >> On Tue,

Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor

it then as > an option. > > BTW. We already have a number of CeleryExecutor tests that use the > integrations, so Breeze has all what's needed: > > > https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109 > > J. > > On Mon, Mar 2

Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor

ecutors/test_celery_executor.py#L109 > > J. > > On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter wrote: > >> Hmm, maybe I was just getting twisted around with docker then. I’ll have >> a look at what you shared. >> >> Thanks Bin :) >> >> On Mar

Re: [DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor

the flag --skip-mounting-local-sources. You can find > more details here > https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze > > Best > Bin > > >> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter wrote: >> I recen

[DISCUSS] Add Breeze Support for CeleryExecutor and KubernetesExecutor

I recently had some trouble trying to fix a bug in the CeleryExecutor . The code change was small, but it was really difficult to set up a development environment using the CeleryExecutor. I ultimately had to muck around with the test case that covers

Re: dbt Provider

ow-operator/ >>> >>> I'd much rather we work with dbt to do what ever is needed to make it a >>> full provider than pull it in tree when it already exists as a third party >>> package. >>> >>> -ash >>> >>>> On 3 March 2021 18:0

Re: dbt Provider

xists as a third party package. > > -ash > >> On 3 March 2021 18:04:01 GMT, Ryan Hatter wrote: >> Hey all, >> >> dbt seems to continue to gain momentum. There's already an airflow-dbt >> project that is essentially a provider package. Would it make se

dbt Provider

Hey all, dbt seems to continue to gain momentum. There's already an airflow-dbt project that is essentially a provider package. Would it make sense to fold the dbt_hook

Re: [VOTE] Airflow Providers - release candidates from 2021-02-27

ding that you tested it before >> with real GSuite account is for me enough of a confirmation ;). >> >> J. >> >> On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer >> wrote: >>> Salutes for having a GSuite account just for the functionality 

Re: New Committers: James Timmins, Elad Kalif & Daniel Standish

Awesome! Congrats all! > On Mar 1, 2021, at 11:20, Tomasz Urbaszek wrote: > >  > Congrats Elad, Daniel and James! Well deserved indeed! > > T. > >> On Mon, 1 Mar 2021 at 19:12, Jarek Potiuk wrote: >> Woohoo! >> >>> On Mon, Mar 1, 2021 at 6:04 PM Sumit Maheshwari >>> wrote: >>>

Re: [VOTE] Airflow Providers - release candidates from 2021-02-27

esting.: >> >> * amazon : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica Kolenkaš, >> JavierLopezT >> * apache.druid: Xinbin Huang >> * apache.spark: Igor Khrol >> * cncf.kubernetes: jpyen, Ash Berlin-Taylor, Daniel Imberman >> * google: Vivek Bhoj

Re: Create official Apache Airflow publication on Medium.com

I’d be interested in helping edit/proofread articles :) > On Feb 20, 2021, at 11:40, Deng Xiaodong wrote: > >  > It's a good idea to me! > > Maybe worthwhile discussing how the Publication will be managed to ensure > high quality, like: criteria of stories/articles to include,