Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

[Scheffler Jens (XC-AS/EAE-ADA-T)]

+1 binding and many thanks for the lively discussion and alignment! Get it 
going!

Sent from Outlook for iOS

From: Pierre Jeambrun 
Sent: Thursday, April 18, 2024 7:14:39 PM
To: dev@airflow.apache.org 
Subject: Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

+1 (binding)

Le jeu. 18 avr. 2024 à 18:52, Ferruzzi, Dennis 
a écrit :

> This is exciting.  +1 binding.
>
>
>  - ferruzzi
>
>
> 
> From: Oliveira, Niko 
> Sent: Thursday, April 18, 2024 9:16 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [VOTE] AIP-67 Multi-team
> deployment of Airflow components
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 binding
>
> Excited for this one!
>
> 
> From: Aritra Basu 
> Sent: Thursday, April 18, 2024 7:37:08 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [VOTE] AIP-67 Multi-team
> deployment of Airflow components
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 (non-binding)
>
> --
> Regards,
> Aritra Basu
>
> On Thu, Apr 18, 2024, 7:42 PM Bishundeo, Rajeshwar
>  wrote:
>
> > +1 (non-binding)
> >
> > -- Rajesh
> >
> >
> >
> >
> >
> >
> > On 2024-04-18, 10:11 AM, "Vincent Beck"  > vincb...@apache.org>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > +1 binding
> >
> >
> > On 2024/04/18 11:10:32 Jarek Potiuk wrote:
> > > Hello here.
> > >
> > > I have not not heard a lot of feedback after my last update, so let me
> > > start a vote, hoping that the last changes proposed addressed most of
> the
> > > concerns.
> > >
> > > Just to recap. the proposal is here:
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents=05%7C02%7CJens.Scheffler%40de.bosch.com%7C075ccc6635394270d26f08dc5fcb2b9d%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638490573406864861%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=LEVIb6BvpfVoLciWboTSQGyWJFC1RAqIntlqYihOWrc%3D=0
> > <
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents=05%7C02%7CJens.Scheffler%40de.bosch.com%7C075ccc6635394270d26f08dc5fcb2b9d%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638490573406875661%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=ImDoy%2BncJYqXOK3DA9UtfjrHFwWWIvgq6EV5sWGp5Eo%3D=0
> > >
> > >
> > > Summarizing the most recent changes - responding to comments and doubts
> > > raised during the discussion:
> > >
> > > * renamed it to be multi-team to clarify that this is the only case it
> > > addresses
> > > * splitting it into two phases: without and with internal API AIP-44
> > (also
> > > named as GRPC server)
> > > * implifying the approach for Variables and Connections, where no
> changes
> > > in Airflow will be needed to handle the DB updates.
> > >
> > > This makes phase 1 simpler and not depending on AIP-44.
> > >
> > > The vote will last till next Friday 26th of April 2024 Noon CEST.
> > >
> > > J.
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org  > dev-unsubscr...@airflow.apache.org>
> > For 

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

[Jed Cunningham]

+1, looking forward to having better data!

Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

[Pierre Jeambrun]

+1 (binding)

Le jeu. 18 avr. 2024 à 18:52, Ferruzzi, Dennis 
a écrit :

> This is exciting.  +1 binding.
>
>
>  - ferruzzi
>
>
> 
> From: Oliveira, Niko 
> Sent: Thursday, April 18, 2024 9:16 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [VOTE] AIP-67 Multi-team
> deployment of Airflow components
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 binding
>
> Excited for this one!
>
> 
> From: Aritra Basu 
> Sent: Thursday, April 18, 2024 7:37:08 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [VOTE] AIP-67 Multi-team
> deployment of Airflow components
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 (non-binding)
>
> --
> Regards,
> Aritra Basu
>
> On Thu, Apr 18, 2024, 7:42 PM Bishundeo, Rajeshwar
>  wrote:
>
> > +1 (non-binding)
> >
> > -- Rajesh
> >
> >
> >
> >
> >
> >
> > On 2024-04-18, 10:11 AM, "Vincent Beck"  > vincb...@apache.org>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > +1 binding
> >
> >
> > On 2024/04/18 11:10:32 Jarek Potiuk wrote:
> > > Hello here.
> > >
> > > I have not not heard a lot of feedback after my last update, so let me
> > > start a vote, hoping that the last changes proposed addressed most of
> the
> > > concerns.
> > >
> > > Just to recap. the proposal is here:
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > >
> > >
> > > Summarizing the most recent changes - responding to comments and doubts
> > > raised during the discussion:
> > >
> > > * renamed it to be multi-team to clarify that this is the only case it
> > > addresses
> > > * splitting it into two phases: without and with internal API AIP-44
> > (also
> > > named as GRPC server)
> > > * implifying the approach for Variables and Connections, where no
> changes
> > > in Airflow will be needed to handle the DB updates.
> > >
> > > This makes phase 1 simpler and not depending on AIP-44.
> > >
> > > The vote will last till next Friday 26th of April 2024 Noon CEST.
> > >
> > > J.
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org  > dev-unsubscr...@airflow.apache.org>
> > For additional commands, e-mail: dev-h...@airflow.apache.org  > dev-h...@airflow.apache.org>
> >
> >
> >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > For additional commands, e-mail: dev-h...@airflow.apache.org
> >
>

Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

[Ferruzzi, Dennis]

This is exciting.  +1 binding.


 - ferruzzi



From: Oliveira, Niko 
Sent: Thursday, April 18, 2024 9:16 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [VOTE] AIP-67 Multi-team deployment 
of Airflow components

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.



AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
contenu ne présente aucun risque.



+1 binding

Excited for this one!


From: Aritra Basu 
Sent: Thursday, April 18, 2024 7:37:08 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [VOTE] AIP-67 Multi-team deployment 
of Airflow components

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.



AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
contenu ne présente aucun risque.



+1 (non-binding)

--
Regards,
Aritra Basu

On Thu, Apr 18, 2024, 7:42 PM Bishundeo, Rajeshwar
 wrote:

> +1 (non-binding)
>
> -- Rajesh
>
>
>
>
>
>
> On 2024-04-18, 10:11 AM, "Vincent Beck"  vincb...@apache.org>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
> +1 binding
>
>
> On 2024/04/18 11:10:32 Jarek Potiuk wrote:
> > Hello here.
> >
> > I have not not heard a lot of feedback after my last update, so let me
> > start a vote, hoping that the last changes proposed addressed most of the
> > concerns.
> >
> > Just to recap. the proposal is here:
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> >
> > Summarizing the most recent changes - responding to comments and doubts
> > raised during the discussion:
> >
> > * renamed it to be multi-team to clarify that this is the only case it
> > addresses
> > * splitting it into two phases: without and with internal API AIP-44
> (also
> > named as GRPC server)
> > * implifying the approach for Variables and Connections, where no changes
> > in Airflow will be needed to handle the DB updates.
> >
> > This makes phase 1 simpler and not depending on AIP-44.
> >
> > The vote will last till next Friday 26th of April 2024 Noon CEST.
> >
> > J.
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org  dev-unsubscr...@airflow.apache.org>
> For additional commands, e-mail: dev-h...@airflow.apache.org  dev-h...@airflow.apache.org>
>
>
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>

Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

[Oliveira, Niko]

+1 binding

Excited for this one!


From: Aritra Basu 
Sent: Thursday, April 18, 2024 7:37:08 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [VOTE] AIP-67 Multi-team deployment 
of Airflow components

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.



AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
contenu ne présente aucun risque.



+1 (non-binding)

--
Regards,
Aritra Basu

On Thu, Apr 18, 2024, 7:42 PM Bishundeo, Rajeshwar
 wrote:

> +1 (non-binding)
>
> -- Rajesh
>
>
>
>
>
>
> On 2024-04-18, 10:11 AM, "Vincent Beck"  vincb...@apache.org>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
> +1 binding
>
>
> On 2024/04/18 11:10:32 Jarek Potiuk wrote:
> > Hello here.
> >
> > I have not not heard a lot of feedback after my last update, so let me
> > start a vote, hoping that the last changes proposed addressed most of the
> > concerns.
> >
> > Just to recap. the proposal is here:
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> >
> > Summarizing the most recent changes - responding to comments and doubts
> > raised during the discussion:
> >
> > * renamed it to be multi-team to clarify that this is the only case it
> > addresses
> > * splitting it into two phases: without and with internal API AIP-44
> (also
> > named as GRPC server)
> > * implifying the approach for Variables and Connections, where no changes
> > in Airflow will be needed to handle the DB updates.
> >
> > This makes phase 1 simpler and not depending on AIP-44.
> >
> > The vote will last till next Friday 26th of April 2024 Noon CEST.
> >
> > J.
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org  dev-unsubscr...@airflow.apache.org>
> For additional commands, e-mail: dev-h...@airflow.apache.org  dev-h...@airflow.apache.org>
>
>
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>

Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

[Aritra Basu]

+1 (non-binding)

--
Regards,
Aritra Basu

On Thu, Apr 18, 2024, 7:42 PM Bishundeo, Rajeshwar
 wrote:

> +1 (non-binding)
>
> -- Rajesh
>
>
>
>
>
>
> On 2024-04-18, 10:11 AM, "Vincent Beck"  vincb...@apache.org>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
> +1 binding
>
>
> On 2024/04/18 11:10:32 Jarek Potiuk wrote:
> > Hello here.
> >
> > I have not not heard a lot of feedback after my last update, so let me
> > start a vote, hoping that the last changes proposed addressed most of the
> > concerns.
> >
> > Just to recap. the proposal is here:
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> >
> > Summarizing the most recent changes - responding to comments and doubts
> > raised during the discussion:
> >
> > * renamed it to be multi-team to clarify that this is the only case it
> > addresses
> > * splitting it into two phases: without and with internal API AIP-44
> (also
> > named as GRPC server)
> > * implifying the approach for Variables and Connections, where no changes
> > in Airflow will be needed to handle the DB updates.
> >
> > This makes phase 1 simpler and not depending on AIP-44.
> >
> > The vote will last till next Friday 26th of April 2024 Noon CEST.
> >
> > J.
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org  dev-unsubscr...@airflow.apache.org>
> For additional commands, e-mail: dev-h...@airflow.apache.org  dev-h...@airflow.apache.org>
>
>
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>

Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

[Bishundeo, Rajeshwar]

+1 (non-binding)

-- Rajesh 






On 2024-04-18, 10:11 AM, "Vincent Beck" mailto:vincb...@apache.org>> wrote:


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.






AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
contenu ne présente aucun risque.






+1 binding


On 2024/04/18 11:10:32 Jarek Potiuk wrote:
> Hello here.
>
> I have not not heard a lot of feedback after my last update, so let me
> start a vote, hoping that the last changes proposed addressed most of the
> concerns.
>
> Just to recap. the proposal is here:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>  
> 
>
> Summarizing the most recent changes - responding to comments and doubts
> raised during the discussion:
>
> * renamed it to be multi-team to clarify that this is the only case it
> addresses
> * splitting it into two phases: without and with internal API AIP-44 (also
> named as GRPC server)
> * implifying the approach for Variables and Connections, where no changes
> in Airflow will be needed to handle the DB updates.
>
> This makes phase 1 simpler and not depending on AIP-44.
>
> The vote will last till next Friday 26th of April 2024 Noon CEST.
>
> J.
>


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org 

For additional commands, e-mail: dev-h...@airflow.apache.org 







-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

[Vincent Beck]

+1 binding

On 2024/04/18 11:10:32 Jarek Potiuk wrote:
> Hello here.
> 
> I have not not heard a lot of feedback after my last update, so let me
> start a vote, hoping that the last changes proposed addressed most of the
> concerns.
> 
> Just to recap. the proposal is here:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> 
> Summarizing the most recent changes - responding to comments and doubts
> raised during the discussion:
> 
> * renamed it to be multi-team to clarify that this is the only case it
> addresses
> * splitting it into two phases: without and with internal API AIP-44 (also
> named as GRPC server)
> * implifying the approach for Variables and Connections, where no changes
> in Airflow will be needed to handle the DB updates.
> 
> This makes phase 1 simpler and not depending on AIP-44.
> 
> The vote will last till next Friday 26th of April 2024 Noon CEST.
> 
> J.
> 

-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [DISCUSS] Consider disabling self-hosted runners for commiter PRs

[Jarek Potiuk]

The change is merged, rebasing should trigger maintainers PRs using public
runners. they should be able to switch to "self-hosted" by "use self hosted
runners" label. The `main` and `v2-9-test` runs should still be run using
self-hosted runners.

I would love to hear back from the maintainers if that helps with their
experience.

On Thu, Apr 18, 2024 at 10:59 AM Jarek Potiuk  wrote:

> PR switching it here: https://github.com/apache/airflow/pull/39106 -
> sorry for the delay in following up on that one.
>
> J.
>
> On Fri, Apr 5, 2024 at 6:08 PM Wei Lee  wrote:
>
>> +1 for this. I do not yet have enough chance to experience many job
>> failures, but it won’t harm us to test them out. Plus, it saves some of the
>> cost.
>>
>> Best,
>> Wei
>>
>> > On Apr 5, 2024, at 11:36 PM, Jarek Potiuk  wrote:
>> >
>> > Seeing no big "no's" - I will prepare and run the experiment - starting
>> > some time next week, after we get 2.9.0 out - I do not want to break
>> > anything there. In the meantime, preparatory PR to add "use self-hosted
>> > runners" label is out https://github.com/apache/airflow/pull/38779
>> >
>> > On Fri, Apr 5, 2024 at 4:21 PM Bishundeo, Rajeshwar
>> >  wrote:
>> >
>> >> +1 with trying this out. I agree with keeping the canary builds
>> >> self-hosted in order to validate the usage for the PRs.
>> >>
>> >> -- Rajesh
>> >>
>> >>
>> >> From: Jarek Potiuk 
>> >> Reply-To: "dev@airflow.apache.org" 
>> >> Date: Friday, April 5, 2024 at 8:36 AM
>> >> To: "dev@airflow.apache.org" 
>> >> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Consider disabling
>> >> self-hosted runners for commiter PRs
>> >>
>> >>
>> >> CAUTION: This email originated from outside of the organization. Do not
>> >> click links or open attachments unless you can confirm the sender and
>> know
>> >> the content is safe.
>> >>
>> >>
>> >>
>> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>> externe.
>> >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
>> pouvez
>> >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
>> que
>> >> le contenu ne présente aucun risque.
>> >>
>> >>
>> >> Yeah. Valid concerns Hussein.
>> >>
>> >> And I am happy to share some more information on that. I did not want
>> to
>> >> put all of that in the original email, but I see that might be
>> interesting
>> >> for you and possibly others.
>> >>
>> >> I am closely following the numbers now. One of the reasons I am doing /
>> >> proposing it now is that finally (after almost 3 years of waiting) we
>> >> finally have access to some metrics that we can check. As of last week
>> I
>> >> got access to the ASF metrics (
>> >> https://issues.apache.org/jira/browse/INFRA-25662).
>> >>
>> >> I have access to "organisation" level information. Infra does not want
>> to
>> >> open it to everyone - even to every member -  but since I got very
>> active
>> >> and been helping with a number I got the access granted as an
>> exception.
>> >> Also I saw a small dashboard the INFRA prepares to open to everyone
>> once
>> >> they sort the access where we will be able to see the "per-project"
>> usage.
>> >>
>> >> Some stats that I can share (they asked not to share too much).
>> >>
>> >> From what I looked at I can tell that we are right now (the whole ASF
>> >> organisation) safely below the total capacity. With a large margin -
>> enough
>> >> to handle spikes, but of course the growth of usage is there and if
>> >> uncontrolled - we can again reach the same situation that triggered
>> getting
>> >> self-hosted runners a few years ago.
>> >>
>> >> Luckily - INRA gets it under control this time |(and metrics will
>> help).
>> >> In the last INFRA newsletter, they announced some limitations that will
>> >> apply to the projects (effective as of end of April) - so once those
>> will
>> >> be followed, we should be "safe" from being impacted by others (i.e.
>> >> noisy-neighbour effect). Some of the projects (not Airflow (!) ) were
>> >> exceeding those so far and they will be capped - they will need to
>> optimize
>> >> their builds eventually.
>> >>
>> >> Those are the rules:
>> >>
>> >> * All workflows MUST have a job concurrency level less than or equal to
>> >> 20. This means a workflow cannot have more than 20 jobs running at the
>> same
>> >> time across all matrices.
>> >> * All workflows SHOULD have a job concurrency level less than or equal
>> to
>> >> 15. Just because 20 is the max, doesn't mean you should strive for 20.
>> >> * The average number of minutes a project uses per calendar week MUST
>> NOT
>> >> exceed the equivalent of 25 full-time runners (250,000 minutes, or
>> 4,200
>> >> hours).
>> >> * The average number of minutes a project uses in any consecutive
>> five-day
>> >> period MUST NOT exceed the equivalent of 30 full-time runners (216,000
>> >> minutes, or 3,600 hours).
>> >> * Projects whose builds consistently cross the maximum use limits will
>> >> lose their access to GitHub Actions 

[VOTE] AIP-67 Multi-team deployment of Airflow components

[Jarek Potiuk]

Hello here.

I have not not heard a lot of feedback after my last update, so let me
start a vote, hoping that the last changes proposed addressed most of the
concerns.

Just to recap. the proposal is here:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components

Summarizing the most recent changes - responding to comments and doubts
raised during the discussion:

* renamed it to be multi-team to clarify that this is the only case it
addresses
* splitting it into two phases: without and with internal API AIP-44 (also
named as GRPC server)
* implifying the approach for Variables and Connections, where no changes
in Airflow will be needed to handle the DB updates.

This makes phase 1 simpler and not depending on AIP-44.

The vote will last till next Friday 26th of April 2024 Noon CEST.

J.

Re: [VOTE] Airflow Providers prepared on April 16, 2024

[Jarek Potiuk]

+1 (binding): checked reproducibility, licences, signatures, checksums -
all look good.

On Thu, Apr 18, 2024 at 12:27 PM Pankaj Singh 
wrote:

> +1 (non-binding) tested my changes.
>
> Thanks
> Pankaj
>
>
>
> On Wed, Apr 17, 2024 at 12:24 PM Amogh Desai 
> wrote:
>
> > +1 non binding
> >
> > Tested few example DAGs with cncf. Works fine.
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Wed, 17 Apr 2024 at 12:21 PM, Pankaj Koti
> >  wrote:
> >
> > > +1 (non-binding) Concurring with Wei.
> > >
> > >
> > > Best regards,
> > >
> > > *Pankaj Koti*
> > > Senior Software Engineer (Airflow OSS Engineering team)
> > > Location: Pune, Maharashtra, India
> > > Timezone: Indian Standard Time (IST)
> > >
> > >
> > > On Wed, Apr 17, 2024 at 9:01 AM Wei Lee  wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > I tested my change and ran example DAGs with cncf, databircks
> providers
> > > > RCs, and it worked fine.
> > > >
> > > > Best,
> > > > Wei
> > > >
> > > > > On Apr 16, 2024, at 8:40 PM, Elad Kalif 
> wrote:
> > > > >
> > > > > Hey all,
> > > > >
> > > > > I have just cut the new wave Airflow Providers packages. This email
> > is
> > > > > calling a vote on the release,
> > > > > which will last for 72 hours - which means that it will end on
> April
> > > 19,
> > > > > 2024 12:40 PM UTC and until 3 binding +1 votes have been received.
> > > > >
> > > > >
> > > > > Consider this my (binding) +1.
> > > > >
> > > > > Airflow Providers are available at:
> > > > > https://dist.apache.org/repos/dist/dev/airflow/providers/
> > > > >
> > > > > *apache-airflow-providers--*.tar.gz* are the binary
> > > > > Python "sdist" release - they are also official "sources" for the
> > > > provider
> > > > > packages.
> > > > >
> > > > > *apache_airflow_providers_-*.whl are the binary
> > > > > Python "wheel" release.
> > > > >
> > > > > The test procedure for PMC members is described in
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> > > > >
> > > > > The test procedure for and Contributors who would like to test this
> > RC
> > > is
> > > > > described in:
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
> > > > >
> > > > >
> > > > > Public keys are available at:
> > > > > https://dist.apache.org/repos/dist/release/airflow/KEYS
> > > > >
> > > > > Please vote accordingly:
> > > > >
> > > > > [ ] +1 approve
> > > > > [ ] +0 no opinion
> > > > > [ ] -1 disapprove with the reason
> > > > >
> > > > > Only votes from PMC members are binding, but members of the
> community
> > > are
> > > > > encouraged to test the release and vote with "(non-binding)".
> > > > >
> > > > > Please note that the version number excludes the 'rcX' string.
> > > > > This will allow us to rename the artifact without modifying
> > > > > the artifact checksums when we actually release.
> > > > >
> > > > > The status of testing the providers by the community is kept here:
> > > > > https://github.com/apache/airflow/issues/39063
> > > > >
> > > > > The issue is also the easiest way to see important PRs included in
> > the
> > > RC
> > > > > candidates.
> > > > > Detailed changelog for the providers will be published in the
> > > > documentation
> > > > > after the
> > > > > RC candidates are released.
> > > > >
> > > > > You can find the RC packages in PyPI following these links:
> > > > >
> > > > >
> > > >
> > >
> >
> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.1.1rc1/
> > > > >
> > https://pypi.org/project/apache-airflow-providers-databricks/6.3.0rc3/
> > > > > https://pypi.org/project/apache-airflow-providers-fab/1.0.4rc1/
> > > > >
> > > > > Cheers,
> > > > > Elad Kalif
> > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > > > For additional commands, e-mail: dev-h...@airflow.apache.org
> > > >
> > > >
> > >
> >
>

Re: [VOTE] Airflow Providers prepared on April 16, 2024

[Pankaj Singh]

+1 (non-binding) tested my changes.

Thanks
Pankaj



On Wed, Apr 17, 2024 at 12:24 PM Amogh Desai 
wrote:

> +1 non binding
>
> Tested few example DAGs with cncf. Works fine.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Wed, 17 Apr 2024 at 12:21 PM, Pankaj Koti
>  wrote:
>
> > +1 (non-binding) Concurring with Wei.
> >
> >
> > Best regards,
> >
> > *Pankaj Koti*
> > Senior Software Engineer (Airflow OSS Engineering team)
> > Location: Pune, Maharashtra, India
> > Timezone: Indian Standard Time (IST)
> >
> >
> > On Wed, Apr 17, 2024 at 9:01 AM Wei Lee  wrote:
> >
> > > +1 (non-binding)
> > >
> > > I tested my change and ran example DAGs with cncf, databircks providers
> > > RCs, and it worked fine.
> > >
> > > Best,
> > > Wei
> > >
> > > > On Apr 16, 2024, at 8:40 PM, Elad Kalif  wrote:
> > > >
> > > > Hey all,
> > > >
> > > > I have just cut the new wave Airflow Providers packages. This email
> is
> > > > calling a vote on the release,
> > > > which will last for 72 hours - which means that it will end on April
> > 19,
> > > > 2024 12:40 PM UTC and until 3 binding +1 votes have been received.
> > > >
> > > >
> > > > Consider this my (binding) +1.
> > > >
> > > > Airflow Providers are available at:
> > > > https://dist.apache.org/repos/dist/dev/airflow/providers/
> > > >
> > > > *apache-airflow-providers--*.tar.gz* are the binary
> > > > Python "sdist" release - they are also official "sources" for the
> > > provider
> > > > packages.
> > > >
> > > > *apache_airflow_providers_-*.whl are the binary
> > > > Python "wheel" release.
> > > >
> > > > The test procedure for PMC members is described in
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> > > >
> > > > The test procedure for and Contributors who would like to test this
> RC
> > is
> > > > described in:
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
> > > >
> > > >
> > > > Public keys are available at:
> > > > https://dist.apache.org/repos/dist/release/airflow/KEYS
> > > >
> > > > Please vote accordingly:
> > > >
> > > > [ ] +1 approve
> > > > [ ] +0 no opinion
> > > > [ ] -1 disapprove with the reason
> > > >
> > > > Only votes from PMC members are binding, but members of the community
> > are
> > > > encouraged to test the release and vote with "(non-binding)".
> > > >
> > > > Please note that the version number excludes the 'rcX' string.
> > > > This will allow us to rename the artifact without modifying
> > > > the artifact checksums when we actually release.
> > > >
> > > > The status of testing the providers by the community is kept here:
> > > > https://github.com/apache/airflow/issues/39063
> > > >
> > > > The issue is also the easiest way to see important PRs included in
> the
> > RC
> > > > candidates.
> > > > Detailed changelog for the providers will be published in the
> > > documentation
> > > > after the
> > > > RC candidates are released.
> > > >
> > > > You can find the RC packages in PyPI following these links:
> > > >
> > > >
> > >
> >
> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.1.1rc1/
> > > >
> https://pypi.org/project/apache-airflow-providers-databricks/6.3.0rc3/
> > > > https://pypi.org/project/apache-airflow-providers-fab/1.0.4rc1/
> > > >
> > > > Cheers,
> > > > Elad Kalif
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > > For additional commands, e-mail: dev-h...@airflow.apache.org
> > >
> > >
> >
>

Re: [CALL FOR HELP] Help on Connexion 3 migration needed

[Jarek Potiuk]

Just to make it clear I marked this PR as Draft - and I would really
appreciate some comments about the approach and direction (in the light of
Connexion 2 being dead-end holding us from security-related upgrades and
generally holding us back).

Surely - there are things to be improved in this PR, and we already got
quite a deal of comments there (and yes, it's quite obvious it's not
mergeable in the current state - I made a number of comments and TODOs
there, but likely there will be more - there are already some).

And if we feel, we need a separate AIP for that one - we can definitely
think about writing and approving one - describing in more detail the
architectural changes introduced (which as I understand it, is mostly
following best directions of where Python apps of the kind are heading
towards with WSGI => ASGI move for quite a long time already). But mostly
it's a call for help from those who **could** help and move it into a "PROD
Ready" solution, if we think it's a good direction (and possibly help to
write the AIP).

Forming a small task force of people who would like to make it "good to go"
- people who know what they are doing about such change, and can bring some
more "testing" capacity - especially from those who host Airflow as managed
services.

Or maybe that could be one of the "internal" changes for the elusive
"Airflow 3" we've been talking about for such a long time but we were never
able to define what it is going to be? Dunno what others think :).
But it would be good to hear what others think, as this one is one of the
things that definitely holds us back.

J.



On Tue, Apr 16, 2024 at 10:36 PM Jarek Potiuk  wrote:

> I don't think AIP is needed (because it's not really a user facing change
> and generally the change is backwards compatible.
>
> But yeah - the call for help is mostly to see / review / discuss it when
> we really know the scope of the change after we have a PR that proves that
> yes - it can be done.
>
> It might indeed need voting when we get to the act of considering merging
> the change, but I am not sure if we need AIP describing it. We probably
> would not be able to write the AIP upfront - before attempting to migrate
> it to be honest.
>
> I think what I am really looking for is to achieve the level of confidence
> that might let us decide "yep it's good to go" (hopefully). The important
> thing here is while the stack under-the-hood is changing a lot, the actual
> number of changes in airflow code (except the tests) is rather small -
> mostly initialization and wiring the stack together.
>
> We do not have to merge it now or even soon or maybe even never (though
> Connexion 2 is a dead-end and sooner or later we will have to replace it
> with **something** when it comes to serving our API).
> Connexion 3 seems to be a good - and natural - candidate .
>
> I think with this change in this stage -  we more or less know what
> it really means to migrate to Connexion 3 (additionally with having some
> comments from the Connexion maintainer who already looks in more details at
> the comments and questions in the PR) I think it's where we can see if that
> is something we want to move forward with. And especially if we will
> have feedback from those who know more about the involved technology stack.
>
> That's why I mostly opened this thread :)
>
> J.
>
>
> On Tue, Apr 16, 2024 at 5:15 PM Ryan Hatter
>  wrote:
>
>> Does the scope of this PR warrant an AIP?
>>
>> On Tue, Apr 16, 2024 at 6:40 AM Jarek Potiuk  wrote:
>>
>> > Hello here,
>> >
>> > I have a kind request for help from maintainers (and other contributors
>> who
>> > are not maintainers) - on the Connexion 3 migration for Airflow. PR here
>> > (unfortunately - it's one big PR and cannot be split):
>> > https://github.com/apache/airflow/pull/39055.
>> >
>> > I would love some general comments on this - especially from those who
>> are
>> > more experts than me on those web frameworks - is it safe and ok to
>> > migrate, do we need to do some more testing on that? What do other
>> > maintainers think?
>> >
>> > This is not a "simple" change - it introduces a pretty fundamental
>> change
>> > in how our web app is handled - It changes from WSGI to ASGI interface
>> > (though we use gunicorn as WSGI). But it's also absolutely needed -
>> > because we already had some security issues connected with old
>> > dependencies (Werkzeug) - raised - and Connexion 3 migration seems to be
>> > the easiest way to get to the latest, maintained versions of the
>> > dependencies.
>> >
>> > That's why I'd really like a few more maintainers - and people from the
>> > Astronomer, Google and AWS to take it for a spin and help to test that
>> > change and say "yep. It looks good, we can merge it".  I would
>> especially
>> > appreciate some more "scale" testing on it. It seems that performance
>> and
>> > resource usage is not affected and ASGI interface and uvicorn should
>> nicely
>> > replace all the different worker types 

Re: [DISCUSS] Consider disabling self-hosted runners for commiter PRs

[Jarek Potiuk]

PR switching it here: https://github.com/apache/airflow/pull/39106 - sorry
for the delay in following up on that one.

J.

On Fri, Apr 5, 2024 at 6:08 PM Wei Lee  wrote:

> +1 for this. I do not yet have enough chance to experience many job
> failures, but it won’t harm us to test them out. Plus, it saves some of the
> cost.
>
> Best,
> Wei
>
> > On Apr 5, 2024, at 11:36 PM, Jarek Potiuk  wrote:
> >
> > Seeing no big "no's" - I will prepare and run the experiment - starting
> > some time next week, after we get 2.9.0 out - I do not want to break
> > anything there. In the meantime, preparatory PR to add "use self-hosted
> > runners" label is out https://github.com/apache/airflow/pull/38779
> >
> > On Fri, Apr 5, 2024 at 4:21 PM Bishundeo, Rajeshwar
> >  wrote:
> >
> >> +1 with trying this out. I agree with keeping the canary builds
> >> self-hosted in order to validate the usage for the PRs.
> >>
> >> -- Rajesh
> >>
> >>
> >> From: Jarek Potiuk 
> >> Reply-To: "dev@airflow.apache.org" 
> >> Date: Friday, April 5, 2024 at 8:36 AM
> >> To: "dev@airflow.apache.org" 
> >> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Consider disabling
> >> self-hosted runners for commiter PRs
> >>
> >>
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you can confirm the sender and
> know
> >> the content is safe.
> >>
> >>
> >>
> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe.
> >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> >> le contenu ne présente aucun risque.
> >>
> >>
> >> Yeah. Valid concerns Hussein.
> >>
> >> And I am happy to share some more information on that. I did not want to
> >> put all of that in the original email, but I see that might be
> interesting
> >> for you and possibly others.
> >>
> >> I am closely following the numbers now. One of the reasons I am doing /
> >> proposing it now is that finally (after almost 3 years of waiting) we
> >> finally have access to some metrics that we can check. As of last week I
> >> got access to the ASF metrics (
> >> https://issues.apache.org/jira/browse/INFRA-25662).
> >>
> >> I have access to "organisation" level information. Infra does not want
> to
> >> open it to everyone - even to every member -  but since I got very
> active
> >> and been helping with a number I got the access granted as an exception.
> >> Also I saw a small dashboard the INFRA prepares to open to everyone once
> >> they sort the access where we will be able to see the "per-project"
> usage.
> >>
> >> Some stats that I can share (they asked not to share too much).
> >>
> >> From what I looked at I can tell that we are right now (the whole ASF
> >> organisation) safely below the total capacity. With a large margin -
> enough
> >> to handle spikes, but of course the growth of usage is there and if
> >> uncontrolled - we can again reach the same situation that triggered
> getting
> >> self-hosted runners a few years ago.
> >>
> >> Luckily - INRA gets it under control this time |(and metrics will help).
> >> In the last INFRA newsletter, they announced some limitations that will
> >> apply to the projects (effective as of end of April) - so once those
> will
> >> be followed, we should be "safe" from being impacted by others (i.e.
> >> noisy-neighbour effect). Some of the projects (not Airflow (!) ) were
> >> exceeding those so far and they will be capped - they will need to
> optimize
> >> their builds eventually.
> >>
> >> Those are the rules:
> >>
> >> * All workflows MUST have a job concurrency level less than or equal to
> >> 20. This means a workflow cannot have more than 20 jobs running at the
> same
> >> time across all matrices.
> >> * All workflows SHOULD have a job concurrency level less than or equal
> to
> >> 15. Just because 20 is the max, doesn't mean you should strive for 20.
> >> * The average number of minutes a project uses per calendar week MUST
> NOT
> >> exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200
> >> hours).
> >> * The average number of minutes a project uses in any consecutive
> five-day
> >> period MUST NOT exceed the equivalent of 30 full-time runners (216,000
> >> minutes, or 3,600 hours).
> >> * Projects whose builds consistently cross the maximum use limits will
> >> lose their access to GitHub Actions until they fix their build
> >> configurations.
> >>
> >> Those numbers on their own do not tell much, but we can easily see what
> >> they mean when we put them side-by-side t with "our" current numbers.
> >>
> >> * Currently - with all the "public" usage we are at 8 full-time runners.
> >> This is after some of the changes I've done, With the recent changes I
> >> already moved a lot of the non-essential build components that do not
> >> require a lot of parallelism to public runners.
> >> * The 20/15 jobs limit is