Re: [LAZY CONSENSUS] "Edge" as the name for AIP-69

[Wei Lee]

I like what Daniel suggested. But I guess we’ll probably need to split a few 
executors from existing providers for this? If that’s the case, are we going to 
have those executors exist in both side for some time? or is there a better for 
it?

Best,
Wei

> On Sep 17, 2024, at 1:16 AM, Jens Scheffler  
> wrote:
> 
> Hi all,
> 
> yes, diverged a bit. I'll add an agenda item for the next Airflow 3 dev
> call.
> 
> Main question as outcome would be: Shall we split the Providers if we
> are (ongoing) also planning to split the deployment dependencies of
> Scheduler, Worker, Webserver etc?
> 
> -->
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-19September2024
> 
> Jens
> 
> On 16.09.24 16:54, Vincent Beck wrote:
>> On one side it makes sense to me, and I actually like the thinking 
>> "providers should only be for DAG authors". That makes it simple to figure 
>> whether something should belong to providers. If we go that way then FAB 
>> would no longer be a provider but a plugin which would be one step closer to 
>> not install providers on the webserver.
>> 
>> On the other side, I can see some concerns/questions:
>> 
>> - Where would these plugins be in the codebase? And more importantly, how 
>> would they be released? As part of Airflow? As a separate release management 
>> like apache-airflow-plugins?
>> 
>> - Some executors (e.g. AwsEcsExecutor), depend on hooks (AWS hook). If we 
>> move executors outside of providers, then we need the plugin to depend on 
>> some providers?
>> 
>> PS: we diverged from the original topic, we might want to create a separate 
>> thread for that.
>> 
>> On 2024/09/14 12:52:25 Ash Berlin-Taylor wrote:
 Provider package name: "edge", so gets to package "airflow.providers.edge"
>>> I'm not so sure about this name. I have no problems with the"edge" part, I 
>>> am mostly questioning if this should be a provider, or should it be another 
>>> kind of module. For instance it won't have an Operator, Sensor or Hook, nor 
>>> need anything else provided by the Provider Manager.
>>> 
>>> Similarly now I think about it for Celery executor, it's not got anything 
>>> for use in dags does it?
>>> 
>>> My thinking here is: if a dag author uses it directly -> "provider"
>>> If it's only used by the deployment manager -> something else, possibly 
>>> "plugin"?
>>> 
>>> Things that I think should fall in this second category would be the FAB, 
>>> Celery and Edge executors. cncf.kubernetes is mixed as it has the executor 
>>> and an operator.
>>> 
>>> What do others think?
>>> 
>>> On 8 September 2024 14:05:51 BST, Jens Scheffler 
>>>  wrote:
 Hi Devs,
 
 as we had a naming discussion for AIP-69 in
 https://lists.apache.org/thread/br1jfoc8p1wjzk74c09srjgr29spytfy and PRs
 are ready, Elad proposed to have at least a lazy consensus to close the
 naming before merge.
 
 Not having real counts leave "Distributed" and "Edge" close-by with most
 positive feedback. As nobody objected and the term "Edge" is shorter...
 That means:
 
 - Provider package name: "edge", so gets to package 
 "airflow.providers.edge"
 
 - Executor class: "EdgeExecutor"
 
 - Remotely running worker name: "Edge Worker" - called via `airflow edge
 worker [options]` (similar like celery)
 
 
 If somebody wants to have a look to the implementation, several PRs are
 split for better review:
 
 - "Mothership" PR with the complete implementation if you want to have a
 test drive: https://github.com/apache/airflow/pull/40900 (=3800 LoC)
 
   - Split 1: (Full) Edge provider package:
 https://github.com/apache/airflow/pull/41729 (3700 LoC)
 
 - Split 1.1: (Empty) Provider package (Empty boilerplate)
 https://github.com/apache/airflow/pull/42046 (450 LoC)
 
 - Split 1.2: DB Models for Edge Provider
 https://github.com/apache/airflow/pull/42047 (1.1 + 950 LoC)
 
 - Split 1.3: EdgeExecutor
 https://github.com/apache/airflow/pull/42048 (1.1+1.2 + 650 LoC)
 
 - Split 1.4: Rest API Endpoint
 https://github.com/apache/airflow/pull/42049 (1.1+1.2 + 1050 LoC)
 
 - Split 1.5: Worker CLI
 https://github.com/apache/airflow/pull/42050 (1.1+1.2 + 650 LoC)
 
 - Split 1.6: Leftover glue to bring all together
 https://github.com/apache/airflow/pull/42051 (1.1-1.5 + 25 LoC)
 
   - Split 2: (Small) Adjustments in Core:
 https://github.com/apache/airflow/pull/41730 (<10 LoC)
 
   - Split 3: Breeze adjustments to develop/start via CLI
 https://github.com/apache/airflow/pull/41731 (<200 LoC)
 
 
 If anyone objects to the consensus here, let me/devlist know - otherwise 
 this will be effective by Thursday 12th of September 2024 18.00 am PST 
 (which coincidentally is when
 Airflow Summit is completed in San Francisco) - Also lo

Re: [VOTE] Release Airflow 2.10.2 from 2.10.2rc1

[Wei Lee]

Tested my changes and a few example dags. Work fine.

+1 (non-binding)

Best,
Wei

> On Sep 18, 2024, at 4:17 AM, Jens Scheffler  
> wrote:
> 
> Hi,
> 
> I tested reporoducibility, SVN, licenses and signatures. All OK.
> 
> Also I tested the fixed I applied/contributes/merged in the patch
> release as commented in #42279. Some simple example DAGs executed. All
> seem to be fine
> 
> +1 (binding) from my side.
> 
> Jens
> 
> On 17.09.24 21:04, Ephraim Anierobi wrote:
>> Hey fellow Airflowers,
>> 
>> I have cut Airflow 2.10.2rc1. This email is calling a vote on the release,
>> which will last at least 72 hours, from Tuesday, September 17, 2024 at 6:40
>> pm UTC
>> until Friday, September 20, 2024, at 6:40 pm UTC
>> ,
>> and until 3 binding +1 votes have been received.
>> 
>> The status of testing of the release is kept at
>> https://github.com/apache/airflow/issues/42279
>> 
>> Consider this my (binding) +1.
>> 
>> Airflow 2.10.2rc1 is available at:
>> https://dist.apache.org/repos/dist/dev/airflow/2.10.2rc1/
>> 
>> *apache-airflow-2.10.2-source.tar.gz* is a source release that comes with
>> INSTALL instructions.
>> *apache-airflow-2.10.2.tar.gz* is the binary Python "sdist" release.
>> *apache_airflow-2.10.2-py3-none-any.whl* is the binary Python wheel
>> "binary" release.
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> Only votes from PMC members are binding, but all members of the community
>> are encouraged to test the release and vote with "(non-binding)".
>> 
>> The test procedure for PMC members is described in:
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for contributors and members of the community who would
>> like to test this RC is described in:
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
>> 
>> 
>> Please note that the version number excludes the `rcX` string, so it's now
>> simply 2.10.2. This will allow us to rename the artifact without modifying
>> the artifact checksums when we actually release.
>> 
>> Release Notes:
>> https://github.com/apache/airflow/blob/2.10.2rc1/RELEASE_NOTES.rst
>> 
>> For information on what goes into a release please see:
>> https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md
>> 
>> Changes since 2.10.1:
>> 
>> *Bug Fixes*
>> - Revert "Fix: DAGs are not marked as stale if the dags folder change"
>> (#42220, #42217)
>> - Add missing open telemetry span and correct scheduled slots documentation
>> (#41985)
>> - Fix require_confirmation_dag_change (#42063) (#42211)
>> - Only treat null/undefined as falsy when rendering XComEntry (#42199)
>> (#42213)
>> - Add extra and ``renderedTemplates`` as keys to skip ``camelCasing``
>> (#42206) (#42208)
>> - Do not ``camelcase`` xcom entries (#42182) (#42187)
>> - Fix task_instance and dag_run links from list views (#42138) (#42143)
>> - Support multi-line input for Params of type string in trigger UI form
>> (#40414) (#42139)
>> - Fix details tab log url detection (#42104) (#42114)
>> - Add new type of exception to catch timeout (#42064) (#42078)
>> - Rewrite how DAG to dataset / dataset alias are stored (#41987) (#42055)
>> - Allow dataset alias to add more than one dataset events (#42189) (#42247)
>> 
>> *Miscellaneous*
>> - Limit universal-pathlib below ``0.2.4`` as it breaks our integration
>> (#42101)
>> - Auto-fix default deferrable with ``LibCST`` (#42089)
>> - Deprecate ``--tree`` flag for ``tasks list`` cli command (#41965)
>> 
>> *Doc Only Changes*
>> - Update ``security_model.rst`` to clear unauthenticated endpoints
>> exceptions (#42085)
>> - Add note about dataclasses and attrs to XComs page (#42056)
>> - Improve docs on markdown docs in DAGs (#42013)
>> - Add warning that listeners can be dangerous (#41968)
>> 
>> Cheers,
>> Ephraim
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] August 2024 PR of the Month

[Wei Lee]

+1 for #41390

Best,
Wei

> On Aug 29, 2024, at 2:32 AM, Vishnu Chilukoori  
> wrote:
> 
> +1 for #41390 (Non-binding)
> 
> 
> --
> Regards,
> Vishnu C.
> 
> On Wed, Aug 28, 2024 at 8:51 AM Buğra Öztürk 
> wrote:
> 
>> +1 for #41390
>> 
>> On Wed, 28 Aug 2024, 15:10 Brent Bovenzi, 
>> wrote:
>> 
>>> +1 for #41390
>>> 
>>> On Wed, Aug 28, 2024 at 5:03 AM Pierre Jeambrun 
>>> wrote:
>>> 
 +1 for PR #41390
 
 On Tue, Aug 27, 2024 at 10:59 PM Scheffler Jens (XC-AS/EAE-ADA-T)
  wrote:
 
> +1 binding for 41390 - subdags gone is a great step forward!
> 
> Sent from Outlook for iOS
> 
> From: Briana Okyere 
> Sent: Tuesday, August 27, 2024 10:40:07 PM
> To: dev@airflow.apache.org 
> Subject: [VOTE] August 2024 PR of the Month
> 
> Hey All,
> 
> It’s once again time to vote for the PR of the Month!
> 
> With the help of the `get_important_pr_candidates` script in
>> dev/stats,
> we've identified the following candidates:
> 
> PR #41039: Send context using in venv operator <
> 
> 
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F41039&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cdeed61adb6a74d7625d708dcc6d88d80%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638603880598025351%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=CBk7sNhDIp4RjFrHMuC6QMp8MsQnVNO0zsXPc%2BCxa0g%3D&reserved=0
>> 
> 
> PR #41554: feat(providers/openai): support batch api in
> hook/operator/trigger <
> 
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F41554&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cdeed61adb6a74d7625d708dcc6d88d80%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638603880598036094%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=e0fR7Y8DBdCwIbQ7ywp%2F0LhDVE5fxg%2BPG1rsQELb04M%3D&reserved=0
> >
> 
> PR #40008: Add `CloudRunServiceHook` and
>>> `CloudRunCreateServiceOperator`
 <
> 
> 
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F40008&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cdeed61adb6a74d7625d708dcc6d88d80%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638603880598043192%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=w1MeMWLjCxzqzqqpHTJ3ZmeafBV71t2l%2FuAg4xucH9w%3D&reserved=0
>> 
> 
> PR #41304: Add incremental export and cross account export
>>> functionality
 in
> `DynamoDBToS3Operator`
> <
> 
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F41304&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cdeed61adb6a74d7625d708dcc6d88d80%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638603880598047636%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=8yC4rYLoVKAhA9Mdp91kBLxrSr1q%2BaG%2FvYZ34vFg3xQ%3D&reserved=0
> >
> 
> PR #41390: Remove deprecated SubDags <
> 
> 
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F41390&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cdeed61adb6a74d7625d708dcc6d88d80%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638603880598052011%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=R8gsHfvgwuqJulFGc0nSh0JB%2BDVnrw2m321HdLIxX6I%3D&reserved=0
>> 
> 
> Please reply to this thread with your selection or offer your own
> nominee(s).
> 
> Voting will close on Friday, August 30th at 10 AM PST. The winner(s)
>>> will
> be featured in the next issue of the Airflow newsletter.
> 
> Also, if there’s an article or event that you think should be
>> included
>>> in
> this or a future issue of the newsletter, please drop me a line at <
> briana.oky...@astronomer.io>
> 
> --
> Briana Okyere
> Community Manager
> Astronomer
> 
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [DISCUSS] allow_trigger_in_future setting: keep or chop?

[Wei Lee]

I also lean toward allowing it, and removing the configuration sounds good.

Best,
Wei

> On Aug 22, 2024, at 4:00 AM, Jarek Potiuk  wrote:
> 
> weak allow as well.
> 
> On Wed, Aug 21, 2024 at 8:31 PM Oliveira, Niko 
> wrote:
> 
>> I don't feel too strongly about this one (I suppose I also lean allow) but
>> I agree with removing as many of these configs as possible! So I'm all for
>> this one, either way.
>> 
>> 
>> From: Daniel Standish 
>> Sent: Thursday, August 15, 2024 6:13:31 PM
>> To: dev@airflow.apache.org
>> Subject: [EXT] [DISCUSS] allow_trigger_in_future setting: keep or chop?
>> 
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>> 
>> 
>> 
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>> 
>> 
>> 
>> This setting, which affects scheduler behavior in a few places, seems like
>> the type of thing that should not be configurable.
>> 
>> I would propose we either always allow or always don't allow.  Does anyone
>> remember why we made it configurable?
>> 
>> I don't have a strong feeling re allow or don't, but I guess I lean allow.
>> 
>> Opinions?
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

[Discuss] Airflow 2 to 3 migration rules

[Wei Lee]

Hey everyone,

We are currently implementing some breaking changes to the main branch.
I've created this issue to keep track of the items that need to be included
in our migration tool from version 2 to 3. I've provided two examples and
will add more in the upcoming days. It would be great if we could consider
migration rules and add them to this list whenever we make breaking
changes. Thanks Jarek for pinning it in the GitHub issue. Currently, this
process is manual. We'll explore easier ways to track it as we move
forward. Thanks!

https://github.com/apache/airflow/issues/41641

Best regards,
Wei

Re: [DISCUSS] Airflow 2.11 as bridge release

[Wei Lee]

+1

Best,
Wei

> On Aug 19, 2024, at 9:59 PM, Pierre Jeambrun  wrote:
> 
> +1
> 
> On Mon, Aug 19, 2024 at 3:29 PM Jed Cunningham 
> wrote:
> 
>> +1
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] New Provider for Core operators/sensor

[Wei Lee]

Same here.

1. +1 essential/essentials
2. -1 under common

Best,
Wei

> On Aug 19, 2024, at 9:54 PM, Bas Harenslak  wrote:
> 
> +1 for essential (saves one letter)
> -1 under common (feels like that would lead to path convention confusion)
> 
> Bas
> 
>> On 19 Aug 2024, at 15:40, Vincent Beck  wrote:
>> 
>> Same here,
>> 
>> +1 essential/essentials
>> -1 under common
>> 
>> Binding
>> 
>> On 2024/08/19 13:38:38 Pavankumar Gopidesu wrote:
>>> Yes, I agree with Jarek :)
>>> 
>>> +1 essential or essentials
>>> -1 under common
>>> (non-binding)
>>> 
>>> Regards,
>>> Pavan
>>> 
>>> On Mon, Aug 19, 2024 at 2:32 PM Jarek Potiuk  wrote:
>>> 
 +1 essential (or essentials)
 -1 under common
 
 (binding)
 
 Sorry for a bit of modification here, but I think
 `apache-airflow-providers-essentials` (with `s` at the end) would be more
 appropriate - showing also that it's about various "essentials". But I am
 good with either. This is a nuance.
 
 J
 
 
 On Mon, Aug 19, 2024 at 3:25 PM rom sharon  wrote:
 
> Migrate all operators/sensors from core to dedicated provider.
> 
> *Discussion thread*
> https://lists.apache.org/thread/2dmlqkcmyomm4q7rrovygs6bw655zx07
> 
> This vote concerns two key decisions.
> 
> 1. Provider name selection, options are:
> - essential
> - standard
> - builtin
> - primary
> - core
> - base
> - shared
> 
> 2. Placement under common. should the provider be categorized under
 common
> .
> 
> For the provider name, please cast your vote using the following format
> 
> [ ] +1 
> [ ] +0 no opinion
> [ ] -1 
> 
> For the second decision regarding placement under common, please vote
 using
> this format
> 
> [ ] +1 under common
> [ ] +0 no opinion under common
> [ ] -1 under common
> 
> Everyone is encouraged to vote, although only votes from committers and
 PMC
> members are considered binding.
> 
> The vote will run for 3 days and last until 2024-08-23 at 12 AM UTC.
> 
> Please consider this as my own voting:
> 
> *+1 essential*
> *-1 under common*
> *(binding)*
> 
 
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> For additional commands, e-mail: dev-h...@airflow.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
> 

Re: [DISCUSS] New provider Common.time

[Wei Lee]

I like "core” to include these commonly used operators. I also love "shared", 
but it might mean we need to change "common" to "shared". I'm not sure if it's 
worth the effort, though.

Best,
Wei 

> On Aug 16, 2024, at 4:50 AM, Kaxil Naik  wrote:
> 
> Yeah I would favour a single "core provider":
> `apache-airflow-providers-core-modules` or just
> `apache-airflow-providers-core` sounds more apt.
> 
> Example:
> 
> from airflow.providers.core.sensors.datetime import DateTimeSensor
> from airflow.providers.core.operators.python import PythonOperator
> from airflow.providers.core.triggers.temporal import TimeDeltaTrigger
> 
> On Thu, 15 Aug 2024 at 18:43, Ferruzzi, Dennis 
> wrote:
> 
>> Personally, I like "common" but if we decide to look for alternative
>> names, how about calling them "core providers"?  Or does that feel like an
>> oxymoron since we always make a distinction between 'core" and "provider"?
>> I also like Amogh's suggestion of "foundation".
>> 
>>  - ferruzzi
>> 
>> 
>> 
>> From: Amogh Desai 
>> Sent: Thursday, August 15, 2024 4:17 AM
>> To: dev@airflow.apache.org
>> Subject: RE: [EXT] [DISCUSS] New provider Common.time
>> 
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>> 
>> 
>> 
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>> 
>> 
>> 
>> +1 for this idea.
>> 
>> I like the idea of moving the core operators entirely out to the providers.
>> When it comes to naming, I do not have a strong preference but if we are
>> looking at some
>> alternatives to "common", my suggestions would be to try something like:
>> - "shared": this will make some sense as we plan to share these across more
>> than one provider(s)
>> - "foundation": so that it can serve as a "base" or foundation for the
>> other provider(s). My issue with "base"
>> is that we do use "base" in some other places, for example, BaseOperator,
>> could be confusing.
>> 
>> Thanks & Regards,
>> Amogh Desai
>> 
>> 
>> On Thu, Aug 15, 2024 at 12:37 PM Wei Lee  wrote:
>> 
>>> +1 for moving core operators to providers.
>>> 
>>> I also agree with Hussein's statement that common should only include
>>> things that are used at least twice in other providers. For this specific
>>> case, would `airflow.providers.time` probably work?
>>> 
>>> Best,
>>> Wei
>>> 
>>>> On Aug 15, 2024, at 3:46 AM, Hussein Awala  wrote:
>>>> 
>>>> +1 for moving all the core operators, sensors, and triggers to
>>> new/existing
>>>> providers.
>>>> 
>>>>> New provider Common.time
>>>> 
>>>> I agree with others who comment on the name. IMHO common providers
>> should
>>>> have abstract (or generic) providers/sensors/triggers used by at least
>>> two
>>>> other providers, which is not the case here, but it's a good
>> opportunity
>>> to
>>>> discuss the policy to create a new common provider.
>>>> 
>>>> 
>>>> On Wed, Aug 14, 2024 at 8:53 PM Jarek Potiuk  wrote:
>>>> 
>>>>>> It can mean multiple things, but I’d like it if we used it to mean
>> one
>>>>> thing in Airflow :)
>>>>> 
>>>>> It does not have to be common. But Ash, if you have a proposal there
>>> that
>>>>> does not conflict with any other meaning used in Airflow already -
>>> don't be
>>>>> shy and constructively propose it :)
>>>>> 
>>>>> On Wed, Aug 14, 2024 at 7:42 PM Ferruzzi, Dennis
>>>>>  wrote:
>>>>> 
>>>>>> I like it.
>>>>>> 
>>>>>> - ferruzzi
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> From: Ash Berlin-Taylor 
>>>>>> Sent: Wednesday, August 14, 2024 7:50 AM
>>>>>&

Re: [DISCUSS] New provider Common.time

[Wei Lee]

+1 for moving core operators to providers.

I also agree with Hussein's statement that common should only include things 
that are used at least twice in other providers. For this specific case, would 
`airflow.providers.time` probably work?

Best,
Wei

> On Aug 15, 2024, at 3:46 AM, Hussein Awala  wrote:
> 
> +1 for moving all the core operators, sensors, and triggers to new/existing
> providers.
> 
>> New provider Common.time
> 
> I agree with others who comment on the name. IMHO common providers should
> have abstract (or generic) providers/sensors/triggers used by at least two
> other providers, which is not the case here, but it's a good opportunity to
> discuss the policy to create a new common provider.
> 
> 
> On Wed, Aug 14, 2024 at 8:53 PM Jarek Potiuk  wrote:
> 
>>> It can mean multiple things, but I’d like it if we used it to mean one
>> thing in Airflow :)
>> 
>> It does not have to be common. But Ash, if you have a proposal there that
>> does not conflict with any other meaning used in Airflow already - don't be
>> shy and constructively propose it :)
>> 
>> On Wed, Aug 14, 2024 at 7:42 PM Ferruzzi, Dennis
>>  wrote:
>> 
>>> I like it.
>>> 
>>> - ferruzzi
>>> 
>>> 
>>> 
>>> From: Ash Berlin-Taylor 
>>> Sent: Wednesday, August 14, 2024 7:50 AM
>>> To: dev@airflow.apache.org
>>> Subject: RE: [EXT] [DISCUSS] New provider Common.time
>>> 
>>> CAUTION: This email originated from outside of the organization. Do not
>>> click links or open attachments unless you can confirm the sender and
>> know
>>> the content is safe.
>>> 
>>> 
>>> 
>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
>> pouvez
>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
>> que
>>> le contenu ne présente aucun risque.
>>> 
>>> 
>>> 
 On 14 Aug 2024, at 15:11, Vincent Beck  wrote:
 
 "common" can refer to "common use cases" or "common usage" which makes
>>> sense (at least to me).
>>> 
>>> It can mean multiple things, but I’d like it if we used it to mean one
>>> thing in Airflow :)
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [DISCUSS] Sensor Improvements With Tirggers

[Wei Lee]

Thank you for bringing this up! I have added some comments to the document. I'm 
unsure if we really want or need to implement more complex logic for this. What 
I have in mind is simply adding helper functions to InternalSensorTrigger and 
continuing to use the run method in BaseTrigger. The main purpose of those 
methods is to allow operator authors to change the sensors to leverage this 
start/end from trigger more easily. 

Best,
Wei

> On Aug 14, 2024, at 5:19 PM, Pavankumar Gopidesu  
> wrote:
> 
> Hi Jarek,
> 
> Thanks for the questions :) ,
> 
> I completely agree with you , from 2.10 we have the start_from_trigger
> parameter, which , when set ,
> allows a task to be executed directly from the triggerrer without worker
> involvement. I believe that for
> any sensor to be executed in the triggerer, a corresponding trigger
> implementation must be in place.
> Good thing is , most of the required trigger implementations are already
> available in current airflow providers.
> 
> Additionally, i agree that there is no real difference between a deferrable
> sensor and a deferrable operator,
> generally , users utilize the BaseSensorOperator to create custom sensors,
> However if they want to run
> these sensors in triggerer, they must implement the triggerer's run method
> by extending the base trigger class.
> 
> The key difference with this proposal is the introduction of a common
> trigger implementation (referred to in this
> POC as InternalSensorTrigger )[1]. This would allow users to use the
> new feature start_from_trigger param
> with their custom sensor and eliminates the need for individual trigger
> implementations for each custom sensor
> when they create.
> 
> Alternatively, we could leave the start_from_trigger parameter in
> BaseSensorOperator with a default
> value as false and let users decide whether they want to run the sensor in
> triggerrer. If they choose to do so, they
> can simply set the parameter to true and triggerrer uses the
> InternalSensorTrigger class.
> 
> Apologies if my answer is too confusing :)
> 
> [1]:
> https://github.com/apache/airflow/pull/41355/files#diff-7486f32e385d7ad0376cccda08d80e54939aa901a24616d11fb1f5cba6af7f83R144
> 
> Regards,
> Pavan Kumar
> 
> On Wed, Aug 14, 2024 at 12:29 AM Jarek Potiuk  wrote:
> 
>> How does it differ from the upcoming 2.10 feature?
>> 
>> * Deferrable operators can now execute directly from the triggerer without
>> needing to go through the worker. This is especially efficient for certain
>> operators, like sensors, and can help teams save both time and money.
>> 
>> As of 2.10 - Sensors already can run mostly in Triggerrer and basically
>> there is no big difference any more between deferrable sensor and
>> deferrable operator.
>> 
>> 
>> On Mon, Aug 12, 2024 at 3:59 AM Kaxil Naik  wrote:
>> 
>>> Thanks for putting this together, I will take a look this week.
>>> 
>>> On Fri, 9 Aug 2024 at 13:12, Pavankumar Gopidesu <
>> gopidesupa...@gmail.com>
>>> wrote:
>>> 
 Hi All,
 
 I have created a draft document for Sensor Improvements using triggers.
 
 Details:
 
 
>>> 
>> https://docs.google.com/document/d/1Kb_wL-T1DHkOpmR_QNa3O5p_2hMTLzM-sb_HzQ0PYCo/edit?usp=sharing
 
 POC:
 https://github.com/apache/airflow/pull/41355
 
 This is my first draft post, apologies if any mistakes in the
>> document. I
 would greatly appreciate your insights and suggestions on draft.
 
 Thank you very much for your time.
 
 Regards,
 Pavan
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] AIP-81 Enhanced Security in CLI via Integration of API

[Wei Lee]

+1 binding.

Best,
Wei

> On Aug 12, 2024, at 9:58 AM, Kaxil Naik  wrote:
> 
> +1 binding
> 
> On Sun, 11 Aug 2024 at 19:30, Jarek Potiuk  wrote:
> 
>> +1 (binding) - there are likely some small details to work out during the
>> implementation, but I think they don't impact the overall AIP.
>> 
>> On Fri, Aug 9, 2024 at 11:42 PM Scheffler Jens (XC-AS/EAE-ADA-T)
>>  wrote:
>> 
>>> Added a few comments which shall not block vote - just some technical
>>> details.
>>> 
>>> Otherwise +1 for the AIP from my side. (Binding)
>>> 
>>> Sent from Outlook for iOS
>>> 
>>> From: Buğra Öztürk 
>>> Sent: Thursday, August 8, 2024 10:28:35 PM
>>> To: dev@airflow.apache.org 
>>> Subject: [VOTE] AIP-81 Enhanced Security in CLI via Integration of API
>>> 
>>> Hey all,
>>> 
>>> I would like to call for a vote on AIP-81 Enhanced Security in CLI via
>>> Integration of API.
>>> 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fx%2F4wvOEg&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C7e706c2873174a50cb9908dcb7e8cdea%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638587457711451273%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=a7WRntGoGt3hwP20hrCDuDybpOFI5bXRsvDni5%2FjK20%3D&reserved=0
>>> 
>>> 
>>> I would like to thank everyone for providing valuable feedback to shape
>>> this AIP to its current state.
>>> This proposal aims to integrate the Apache Airflow API into the Apache
>>> Airflow CLI commands to enhance security by utilizing security policies
>> for
>>> API endpoints.
>>> 
>>> Discussion Thread:
>>> 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2F57zpjqvxstm45ro1zdj7c2gltjlttfrp&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C7e706c2873174a50cb9908dcb7e8cdea%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638587457711461907%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=rE1PckdoiqR4OQVMouCbnR%2Boxxl4nt%2B%2FUOQqZyjqOZA%3D&reserved=0
>>> 
>>> 
>>> Please vote accordingly:
>>> 
>>> [ ] +1 approve
>>> [ ] +0 no opinion
>>> [ ] -1 disapprove with the reason
>>> 
>>> Votes from PMC members and committers are binding, but everyone in the
>>> community is also encouraged to vote.
>>> 
>>> The vote will run for 5 days and last until 2024-08-14 at 12 AM UTC.
>>> 
>>> Consider this as my vote of +1 (non-binding).
>>> 
>>> Best Regards,
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [ANNOUNCE] New PMC member: Jens Schaffler

[Wei Lee]

Congrats Jens! 🎉

Best,
Wei

> On Aug 6, 2024, at 10:20 PM, Ash Berlin-Taylor  wrote:
> 
> Well deserved!
> 
> On 6 August 2024 13:12:28 BST, Pankaj Singh  wrote:
>> Congrats Jens 🎉
>> 
>> On Tue, Aug 6, 2024 at 4:44 PM Bowrna Prabhakaran 
>> wrote:
>> 
>>> Congrats Jens!
>>> 
>>> On Tue, Aug 6, 2024 at 3:51 PM Aritra Basu 
>>> wrote:
>>> 
 Congrats Jens!
 --
 Regards,
 Aritra Basu
 
 On Tue, 6 Aug 2024, 3:47 pm rom sharon,  wrote:
 
> !!Congratulations Jens
> 
> ‫בתאריך יום ג׳, 6 באוג׳ 2024 ב-13:00 מאת ‪Utkarsh Sharma‬‏
> <‪utkarsh.sha...@astronomer.io.invalid‬‏>:‬
> 
>> Congratulations Jens! :)
>> 
>> On Tue, Aug 6, 2024 at 2:42 PM Kaxil Naik 
>>> wrote:
>> 
>>> Congratulations Jens, very well deserved!
>>> 
>>> On Tue, 6 Aug 2024 at 09:24, Pierre Jeambrun <
>>> pierrejb...@gmail.com>
>>> wrote:
>>> 
 Congratulations Jens!
 
 On Tue 6 Aug 2024 at 10:08, Buğra Öztürk <
>>> ozturkbugr...@gmail.com>
>>> wrote:
 
> Congratulations Jens 🎊🎉
> 
> On Tue, 6 Aug 2024, 10:04 Pankaj Koti, <
 pankaj.k...@astronomer.io
 .invalid>
> wrote:
> 
>> Many congratulations, Jens!! 🎉🎉🎉
>> 
>> Best regards,
>> 
>> *Pankaj Koti*
>> Senior Software Engineer (Airflow OSS Engineering team)
>> Location: Pune, Maharashtra, India
>> Timezone: Indian Standard Time (IST)
>> 
>> Register for Airflow Summit!
>> > 
>> 
>> 
>> On Tue, Aug 6, 2024 at 1:30 PM Christian Schilling
>>  wrote:
>> 
>>> Congrats Jens 🎉
>>> 
>>> Jarek Potiuk  schrieb am Di., 6. Aug.
 2024,
>>> 09:57:
>>> 
 And [PROJECT] => Airflow of course :)
 
 On Tue, Aug 6, 2024 at 9:54 AM Abhishek Bhakat
  wrote:
 
> Congratulations Jens 🎉
> 
> On Tue, Aug 6, 2024 at 7:52 AM Michał Modras
>  wrote:
> 
>> Congratulations Jens, well deserved!
>> 
>> On Tue, Aug 6, 2024 at 9:51 AM Jarek Potiuk <
>>> ja...@potiuk.com>
>>> wrote:
>> 
>>> The Project Management Committee (PMC) for Apache
>> [PROJECT]
>>> has invited Jens to become a PMC member and we are
>> pleased
>>> to announce that they have accepted.
>>> 
>>> Jens has contributed for a long time already in
 quite a
>> few
> areas
>>> 
>>> 
>> 
> 
 
>>> 
>> 
> 
 
>>> 
>> 
> 
 
>>> https://github.com/apache/airflow/pulls?q=is%3Apr+author%3Ajscheffl+is%3Aclosed
>>> - UI, core, xcoms, important providers, but also
>>> dev-
> env
>>> and
> CI,
>> logging.
>>> 
>>> He has been very involved in a lot of discussions
> around
> Airflow
>>> 3 (and many more before) and led a number of things
>> there.
>>> 
>>> Jens is spear-heading AIP-68 (Extending Plugins)
 AIP-69
 (Remote
> executor)
>>> and
>>> he also helps to complete AIP-44.
>>> 
>>> He also was involved in solving and diagnosing
>> (reporting)
>>> a
>> number
 of
>>> security issues.
>>> 
>>> A PMC member helps manage and guide the direction
>>> of
> the
> project.
>>> 
>>> Congrats Jens!
>>> 
>>> J.
>>> 
>> 
> 
 
>>> 
>> 
> 
 
>>> 
>> 
> 
 
>>> 
>>> 
>>> --
>>> Regards
>>> 
>>> Bowrna Prabhakaran
>>> 

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

[Wei Lee]

+1 (binding) for both APIs

Best,
Wei

> On Aug 5, 2024, at 4:00 AM, Shahar Epstein  wrote:
> 
> +1 (binding) for both
> 
> On Thu, Aug 1, 2024 at 9:20 PM Brent Bovenzi 
> wrote:
> 
>> Hi,
>> 
>> Jed and I prepared two AIPs to flesh out how to finish AIP-38 Modern Web
>> Application for Airflow 3.0. We would like to open it up for a vote.
>> 
>> AIP-79
>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-79%3A+Remove+Flask+AppBuilder+as+Core+dependency
>> 
>> AIP-84 UI REST API
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-84+UI+REST+API
>> 
>> The vote will run for 5 days until next Monday August 5th 19:00 UTC.
>> 
>> Consider this my +1 for both.
>> 
>> - Brent
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] AIP-83 Remove Execution Date Unique Constraint from DAG Run

[Wei Lee]

+1 binding.

Best,
Wei

> On Aug 2, 2024, at 5:06 AM, Vishnu Chilukoori  
> wrote:
> 
> +1 non-binding
> 
> On Thu, Aug 1, 2024 at 2:22 PM Ankit Chaurasia  wrote:
> 
>> +1 non-binding
>> 
>> *Ankit Chaurasia*
>> 
>> 
>> 
>> On Fri, 2 Aug 2024 at 01:45, Vishnu Chilukoori 
>> wrote:
>> 
>>> +1
>>> 
>>> On Thu, Aug 1, 2024 at 12:09 AM Tzu-ping Chung >> 
>>> wrote:
>>> 
 Hi,
 
 This is the actual vote thread for AIP-83
 
 AIP-83 Remove Execution Date Unique Constraint from DAG Run
 https://cwiki.apache.org/confluence/x/SA3OEg
 
 Discussion thread:
 https://lists.apache.org/thread/8rokf8ogp7g5pf0koqksxhcwqlhr9217
 
 As mentioned before, I’m adding the following +1 votes from the
>> previous
 thread:
 Vikram Koka
 Jarek Potiuk
 Phani Kumar
 Jens Scheffler
 
 Also cc-ing Hussein for a review!
 
 TP
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: AWS Open Source Credits Program

[Wei Lee]

Awesome!

> On Jul 29, 2024, at 3:08 AM, Utkarsh Sharma 
>  wrote:
> 
> Sounds great! :)
> 
> On Sun, Jul 28, 2024 at 4:17 PM Pierre Jeambrun 
> wrote:
> 
>> Great news!
>> 
>> On Sat 27 Jul 2024 at 01:15, Sadha Chilukoori 
>> wrote:
>> 
>>> That's wonderful news.
>>> 
>>> On Fri, Jul 26, 2024 at 12:34 PM Oliveira, Niko
>>> 
>>> wrote:
>>> 
 Hello folks,
 
 This is a short note to announce that the AWS Open Source Credits
>> Program
 has approved us for another round of credits this year. $31,000 of
>>> credits
 have been deposited in our Apache Airflow AWS account as of yesterday!
 
 These credits help to run portions of our hosted testing
>> infrastructure.
 
 Cheers,
 Niko
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] AIP-76: Asset Partitions

[Wei Lee]

+1 (binding)

Best,
Wei

> On Aug 2, 2024, at 9:09 AM, Kaxil Naik  wrote:
> 
> We should resolve the open comments on the AIP before we conclude the vote
> though
> 
> On Fri, 2 Aug 2024 at 02:09, Kaxil Naik  wrote:
> 
>> +1 binding
>> 
>> On Thu, 1 Aug 2024 at 18:07, Buğra Öztürk  wrote:
>> 
>>> +1 non-binding
>>> 
>>> On Thu, 1 Aug 2024, 16:07 Brent Bovenzi, 
>>> wrote:
>>> 
 +1 (binding)
 
 On Thu, Aug 1, 2024 at 3:10 AM Tzu-ping Chung >>> 
 wrote:
 
> Hi all,
> 
> Kicking start the vote on the final data awareness AIP targeting 3.0.
> 
> AIP-76 Asset Partitions
> https://cwiki.apache.org/confluence/x/2QyTEg
> 
> Discussion thread
> https://lists.apache.org/thread/3d80921801j6p23x3xgq6yy09gy8yvm4
> 
> The vote will run until Thursday, 2024-08-08, 08:00 UTC. This makes
>>> the
> duration roughly a week.
> 
> Everyone is encouraged to vote, although only votes from committers
>>> and
> PMC members are considered binding.
> 
> Please consider this as my own +1
> 
> TP
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] AIP-68 Extended Plugin Interface for React Views

[Wei Lee]

+1 (binding)

Best,
Wei

> On Jul 27, 2024, at 7:23 AM, Pavankumar Gopidesu  
> wrote:
> 
> +1 (non-binding).
> 
> 
> Regards,
> Pavan Kumar
> 
> On Fri, Jul 26, 2024 at 7:05 PM Aritra Basu 
> wrote:
> 
>> +1 (non-binding)
>> --
>> Regards,
>> Aritra Basu
>> 
>> On Fri, 26 Jul 2024, 10:32 pm Sadha Chilukoori, 
>> wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> On Fri, Jul 26, 2024, 9:04 AM Scheffler Jens (XC-AS/EAE-ADA-T)
>>>  wrote:
>>> 
 Hi,
 
 Brent and me have revised the AIP-68 based on the other Airflow 3.x
 discussions we had and as no further discussions are open we assume it
>> is
 ready to vote:
 
 
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+React+Views
 
 Quote:
 “After the completion of AIP-79 Remove Flask App Builder<
 
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-79+Replace+Flask+App+Builder
 
 we need to explore new ways to build UI plugins. Out-of-the-box, custom
 plugin views and menu items will no longer be supported.
 
 Additionally, after the completion of AIP-38 Modern Web Application<
 
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application
 
 we are looking forward to close gaps in UI for a better business
 integration. From a perspective of business users the UI might be very
 technical. You can see task details, logs and run-times. This is very
>>> good
 for a pipeline developer and operations teams, business users would
>> also
 like to see custom details of the workflow that is represented behind
>> the
 scene.”
 
 We target the implementation of this AIP to be in Airflow 3 (not as
 initially scoped to be 2.10). The removal of old plugins is within the
 scope of AIP-79
 
 
 The vote will run for 5 days and last till next Wednesday 31st of July
 2024 18:00 UTC.
 
 
 
 Everyone is encouraged to vote, although only PMC members and
>> Committer's
 votes are considered binding.
 
 
 
 This is my +1.
 
 Mit freundlichen Grüßen / Best regards
 
 Jens Scheffler
 
 Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
 Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
 GERMANY | www.bosch.com
 Tel. +49 711 811-91508 | Mobil +49 160 90417410 |
 jens.scheff...@de.bosch.com
 
 Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
 Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
 Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
 Forschner,
 Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
 ​
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: More ruff style rules

[Wei Lee]

+1 to not include D102, D103.
+1 to enforcing E731. (I just introduced this in one of my latest PRs 🤦‍♂️ I 
will fix it)
+1 to not-enforcing D107.

I actually think we could enforce TCH003. This kind of pattern seems to be 
already applied to the 3rd party library. It could apply to the standard 
library unless we have a good reason not to.

For PT004 and PT005, I think we probably should either enforce both or exclude 
both for less confusiton.

For PT019, I think it’s something we should enforce. (I also love PT006, PT007, 
and PT011, but yep, they’re already in TODO.)

Best,
Wei

> On Jul 17, 2024, at 3:31 PM, Amogh Desai  wrote:
> 
> +1 to what Ash has to say.
> 
> +1 to enforcing E731 too.
> 
> Just wondering what others think of D107 here?
> 
> Thanks & Regards,
> Amogh Desai
> 
> 
> On Tue, Jul 16, 2024 at 6:40 AM Tzu-ping Chung 
> wrote:
> 
>> +1 to enforcing E731, using lambdas is objectively worse in every way
>> except you save two lines and a few characters at most. The lambda is only
>> “needed” in the sense that you can’t do (say) functool.partial due to
>> variable scoping. A nested function does exactly the same thing.
>> 
>> --
>> Sent from my iPhone
>> 
>>> On 16 Jul 2024, at 00:14, Ferruzzi, Dennis 
>> wrote:
>>> 
>>> I forgot the formatting was going to get stripped on the email, sorry
>> about that.  How do we feel about these two?
>>> 
>>> 
>>> "E731", Use a method instead of a lambda when declaring a variable,
>> currently 3 violations
>>> 
>>> Seems easy enough, just replace a lambda with a getter function but
>> someone would need to look more into one spot [1] where there is a comment
>> which seems to say it needs a lambda, but I have not looked into it yet.
>>> 
>>> "PT019", Use @pytest.mark.usefixtures instead of passing a fixture as a
>> parameter, currently 268 violations
>>> 
>>> This one I really just don't know about.  Seems like we gain some
>> clarification vs passing fixtures in as test parameters, which I kind of
>> like, but the more explicit declaration is pretty clunky.  I'm inclined to
>> say it's more trouble than it is worth?
>>> 
>>> 
>>> 
>>> To clean up the formatting mess from the original message and update it
>> with the discussion so far:
>>>   To Discuss:
>>> 
>>>   "E731", Use a method instead of a lambda when declaring a
>> variable, currently 3 violations
>>> 
>>>   "PT019", Use @pytest.mark.usefixtures instead of passing a
>> fixture as a parameter, currently 268 violations
>>> 
>>>   To Do:
>>> 
>>>   "D214", Consistent multi-line docstring indentation, currently 0
>> violations
>>> 
>>>   "D215", Enforce number of underscores if you use underlines in a
>> multi-line docstring, currently 0 violations
>>>   "PT004", Fixture does not return anything, add leading
>> underscore, currently 311 violations
>>> 
>>>   "PT006", @pytest.mark.parametrize names should be a tuple,
>> currently 1215 violations
>>> 
>>>   "PT007", @pytest.mark.parametrize values should be a tuple,
>> currently 391 violations
>>>   "PT011", pytest.raises() is too broad, set the match parameter,
>> currently 153 violations
>>>   "TCH003", stdlib imports which are only used for typechecking go
>> in to TYPE_CHECKING block, currently 38 violations
>>> 
>>>   To Don't:
>>> 
>>>   "D100", # Docstring at the top of every file.
>>> 
>>>   "D102", # Missing docstring in public method, currently 2473
>> violations
>>>   "D103", # Missing docstring in public function, currently 318
>> violations
>>>   "D104", # Docstring at the top of every `__init__.py` file.
>>>   "D105", # See
>> https://lists.apache.org/thread/8jbg1dd2lr2cfydtqbjxsd6pb6q2wkc3
>>>   "D107", # Docstring in every constructor is unnecessary if the
>> class has a docstring.
>>>   "D203", # Conflicts with D211.  Both can not be enabled.
>>>   "D212", # Conflicts with D213.  Both can not be enabled.
>>>   "PT005", # Fixture returns a value, remove leading underscore,
>> currently 1 violation
>>> 
>>> 
>>> 
>>> [1]
>> https://github.com/apache/airflow/blob/d029e77f2fd704bec4f4797b09d54c5c824a8536/dev/perf/scheduler_dag_execution_timing.py#L293
>>> 
>>> 
>>> 
>>> - ferruzzi
>>> 
>>> 
>>> 
>>> From: Kaxil Naik 
>>> Sent: Sunday, July 14, 2024 11:30 AM
>>> To: dev@airflow.apache.org
>>> Cc: Ferruzzi, Dennis
>>> Subject: RE: [EXT] More ruff style rules
>>> 
>>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>> 
>>> 
>>> 
>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>> certain que le contenu ne présente aucun risque.
>>> 
>>> 
>>> 
>>> Yup agreed with both: going ahead with new rules and excluding D102 and
>> D103
>>> 
 On Sun,

Re: [VOTE] AIPs 73 74 75: Data Awareness with Assets

[Wei Lee]

+1 binding for all the 3 AIPs

Best,
Wei

> On Jul 22, 2024, at 10:49 PM, Aritra Basu  wrote:
> 
> +1 non-binding
> --
> Regards,
> Aritra Basu
> 
> On Mon, Jul 22, 2024, 7:40 PM Jarek Potiuk  wrote:
> 
>> +1 binding
>> 
>> On Mon, Jul 22, 2024 at 4:05 PM Phani Kumar
>>  wrote:
>> 
>>> +1 binding
>>> 
>>> On Mon, Jul 22, 2024 at 7:22 PM Maciej Obuchowski <
>> mobuchow...@apache.org>
>>> wrote:
>>> 
 Committers vote for AIPs is also binding according to
 
 
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
 
 So, this is my +1 (binding) vote for all three AIPs.
 
 pon., 22 lip 2024 o 15:24 Tzu-ping Chung 
 napisał(a):
 
> Hi all,
> 
> Since feedback has died down on these AIPs, I’m calling for a vote on
>>> the
> first three Data Awareness AIPs listed above.
> 
> AIP-73 Expanded Data Awareness
> https://cwiki.apache.org/confluence/x/TAyTEg
> 
> AIP-74 Introducing Data Assets
> https://cwiki.apache.org/confluence/x/QQ2TEg
> 
> AIP-75 New Asset-Centric Syntax
> https://cwiki.apache.org/confluence/x/RA2TEg
> 
> Discussion thread
> https://lists.apache.org/thread/6rp4jhflwg3czhtvjszoctdry85vfv8r
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> Only votes from PMC members are binding, but everyone in the
>> community
>>> is
> also encouraged to vote.
> 
> The vote will run for 5 days and last till next Saturday,  2024-07-27
> 14:00 UTC.
> 
> Consider this as my vote as +1.
> 
> TP
 
>>> 
>> 

Re: [VOTE] AIP-65: Improve DAG history in UI

[Wei Lee]

+1 (binding)

Best,
Wei

> On Jul 20, 2024, at 5:18 AM, Vikram Koka  wrote:
> 
> Sorry, should have said +1 (binding)
> 
> On Fri, Jul 19, 2024 at 2:18 PM Vikram Koka  wrote:
> 
>> +1
>> 
>> This is possibly the highest voted feature from the Airflow survey.
>> 
>> 
>> On Fri, Jul 19, 2024 at 1:50 PM Jed Cunningham 
>> wrote:
>> 
>>> I’m calling for a vote on AIP 65:
>>> https://cwiki.apache.org/confluence/x/T4qSEQ
>>> 
>>> Discussion thread:
>>> https://lists.apache.org/thread/vvm43tfchyo92hmf40fqvmq0f5845bjr
>>> 
>>> The vote will run for 5 days and last till next Wednesday, 2024-07-24
>>> 21:00
>>> UTC.
>>> 
>>> Everyone is encouraged to vote, although only PMC members and Committer's
>>> votes are considered binding.
>>> 
>>> This is my +1.
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] AIP-66: DAG Bundles and Parsing

[Wei Lee]

+1 binding.

Best,
Wei

> On Jul 20, 2024, at 10:27 AM, Phani Kumar  
> wrote:
> 
> +1 binding
> 
> On Sat, 20 Jul 2024, 02:50 Vikram Koka, 
> wrote:
> 
>> +1 binding
>> 
>> 
>> On Fri, Jul 19, 2024 at 1:43 PM Jed Cunningham 
>> wrote:
>> 
>>> I’m calling for a vote on this AIP 66:
>>> https://cwiki.apache.org/confluence/x/ZIqSEQ
>>> 
>>> Discussion thread:
>>> https://lists.apache.org/thread/l8ksl144xd43jfk1wk3kz77t1xgbbq7z
>>> 
>>> The vote will run for 5 days and last till next Wednesday,  2024-07-24
>>> 21:00 UTC.
>>> 
>>> Everyone is encouraged to vote, although only PMC members and Committer's
>>> votes are considered binding.
>>> 
>>> This is my +1.
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [ANNOUNCE] New committers: Rom Sharon & Shahar Epstein

[Wei Lee]

Congratulations! 🎉 🙌

Best regards,
Wei

> On Jul 15, 2024, at 9:31 PM, Abhishek Bhakat 
>  wrote:
> 
> Congrats Rom & Sharan 🥳
> 
> On Mon, Jul 15, 2024 at 1:28 PM utkarsh sharma 
> wrote:
> 
>> Congratulations! 🎉🎉.
>> 
>> Thanks,
>> Utkarsh Sharma
>> 
>> On Mon, Jul 15, 2024 at 6:38 PM Jarek Potiuk  wrote:
>> 
>>> Wooohooo! Congrats!
>>> 
>>> On Mon, Jul 15, 2024 at 2:29 PM Pavankumar Gopidesu <
>>> gopidesupa...@gmail.com>
>>> wrote:
>>> 
 Congratulations Rom and Shahar 🎉🎉.
 
 Regards,
 Pavan
 
 On Mon, Jul 15, 2024 at 1:24 PM Pankaj Koti
  wrote:
 
> Congratulations both 🎉🎉🎉
> 
> 
> Best regards,
> 
> *Pankaj Koti*
> Senior Software Engineer (Airflow OSS Engineering team)
> Location: Pune, Maharashtra, India
> Timezone: Indian Standard Time (IST)
> 
> 
> On Mon, Jul 15, 2024 at 5:52 PM Pankaj Singh <
>> ags.pankaj1...@gmail.com
 
> wrote:
> 
>> Congratulations 🎉🎉
>> 
>> On Mon, Jul 15, 2024 at 5:50 PM Kaxil Naik 
 wrote:
>> 
>>> Congrats Rom & Sharan
>>> 
>>> On Mon, 15 Jul 2024 at 17:46, Amogh Desai <
>>> amoghdesai@gmail.com>
>>> wrote:
>>> 
 This is awesome news!
 
 Congratulations Rom and Shahar 🎊! Well deserved!
 
 Thanks & Regards,
 Amogh Desai
 
 
 On Mon, Jul 15, 2024 at 5:33 PM Elad Kalif >> 
> wrote:
 
> The Project Management Committee (PMC) for Apache Airflow
> has invited Rom Sharon & Shahar Epstein to become a
>> committers
 and
> we
>>> are
> pleased
> to announce that they have accepted.
> 
> Congratulations Rom & Shahar and welcome!
> 
 
>>> 
>> 
> 
 
>>> 
>> 

Re: [VOTE] Airflow Providers prepared on July 12, 2024

[Wei Lee]

+1, non-binding

Best,
Wei

> On Jul 14, 2024, at 6:58 PM, Rahul Vats  wrote:
> 
> +1 for Weaviate provider, verified running our example DAGS.
> 
> Regards,
> Rahul Vats
> 9953794332
> 
> 
> On Sun, 14 Jul 2024 at 01:44, Hussein Awala  wrote:
> 
>> +1 (binding) checked licences, checksums, signatures, and sources and
>> tested my changes. All looks good.
>> 
>> On Sat, Jul 13, 2024 at 6:12 PM Jarek Potiuk  wrote:
>> 
>>> +1 (binding): checked signatures, checksums, reproducibility, licences.
>> All
>>> looks good.
>>> 
>>> On Fri, Jul 12, 2024 at 7:26 PM Vincent Beck 
>> wrote:
>>> 
 +1 non binding. All AWS system tests are running successfully against
 apache-airflow-providers-amazon==8.26.0rc2. You can see the results
>> here:
 
>>> 
>> https://aws-mwaa.github.io/#/open-source/system-tests/version/a89514ec38d368efa9733c8376953024c8da9f1a_8.26.0rc2.html
 
 On 2024/07/12 14:21:54 Elad Kalif wrote:
> Valid links for testing:
> 
> The test procedure for PMC members is described in
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> <
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md%5C#verify-the-release-candidate-by-pmc-members
> 
> 
> The test procedure for and Contributors who would like to test this
>> RC
>>> is
> described in:
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
> <
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md%5C#verify-the-release-candidate-by-contributors
> 
> 
> On Fri, Jul 12, 2024 at 5:20 PM Elad Kalif 
>> wrote:
> 
>> Hey all,
>> 
>> I have just cut rc2 for Amazon and Weaviate Providers packages.
>> This
 email
>> is calling a vote on the release, which will last for 24 hours -
>>> which
>> means that it will end on July 13, 2024 14:20 PM UTC and until 3
 binding +1
>> votes have been received.
>> 
>> 
>> Consider this my (binding) +1.
>> 
>> Airflow Providers are available at:
>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>> 
>> *apache-airflow-providers--*.tar.gz* are the binary
>> Python "sdist" release - they are also official "sources" for the
>> provider packages.
>> 
>> *apache_airflow_providers_-*.whl are the binary
>> Python "wheel" release.
>> 
>> The test procedure for PMC members is described in
>> 
>> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md\#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for and Contributors who would like to test this
>>> RC
 is
>> described in:
>> 
>> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md\#verify-the-release-candidate-by-contributors
>> 
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> Only votes from PMC members are binding, but members of the
>> community
 are
>> encouraged to test the release and vote with "(non-binding)".
>> 
>> Please note that the version number excludes the 'rcX' string.
>> This will allow us to rename the artifact without modifying
>> the artifact checksums when we actually release.
>> 
>> The status of testing the providers by the community is kept here:
>> https://github.com/apache/airflow/issues/40752
>> 
>> The issue is also the easiest way to see important PRs included in
>>> the
 RC
>> candidates.
>> Detailed changelog for the providers will be published in the
>> documentation after the
>> RC candidates are released.
>> 
>> You can find the RC packages in PyPI following these links:
>> 
>> 
>> https://pypi.org/project/apache-airflow-providers-amazon/8.26.0rc2/
>> 
>> https://pypi.org/project/apache-airflow-providers-weaviate/2.0.0rc2/
>> 
>> Cheers,
>> Elad Kalif
>> 
> 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
 For additional commands, e-mail: dev-h...@airflow.apache.org
 
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on July 09, 2024

[Wei Lee]

+1 for providers other than weaviate. Ran our example DAGs and worked fine.

Best,
Wei


> On Jul 10, 2024, at 6:42 AM, Vincent Beck  wrote:
> 
> +1 non binding. All AWS system tests are running successfully against 
> apache-airflow-providers-amazon==8.26.0rc1
> 
> On 2024/07/09 15:01:36 Elad Kalif wrote:
>> weaviate provider is excluded from rc1
>> please continue testing and voting on rest of the providers
>> 
>> On Tue, Jul 9, 2024 at 4:57 PM Wei Lee  wrote:
>> 
>>> -1 for weaviate due to https://github.com/apache/airflow/pull/40670
>>> (Thanks, Tamara, for fixing the issue!)
>>> 
>>> Best,
>>> Wei
>>> 
>>> 
>>>> On Jul 9, 2024, at 9:43 PM, Jarek Potiuk  wrote:
>>>> 
>>>> Yep: -1 on weaviate following 40670
>>>> 
>>>> On Tue, Jul 9, 2024 at 3:25 PM Hussein Awala  wrote:
>>>>> 
>>>>> -1 (binding) for Weaviate provider -
>>>>> https://github.com/apache/airflow/pull/40670
>>>>> 
>>>>> +1 (binding) for the rest - checked licences, checksums, signatures, and
>>>>> sources. All looks good.
>>>>> 
>>>>> On Tue, Jul 9, 2024 at 10:10 AM Jarek Potiuk  wrote:
>>>>> 
>>>>>> +1 (binding) - checked all my changes, checked reproducibility,
>>>>>> signatures, licences, checksums, all looks good.
>>>>>> 
>>>>>> One small thing I found was that changelog for weaviate was slightly
>>>>>> broken (2.0.0 changelog swallowed already released 1.4.2) - I created
>>>>>> a PR fixing it https://github.com/apache/airflow/pull/40663 but it
>>>>>> does not affect the packagesand can be fixed directly by regenerating
>>>>>> the docs without regenerating the package, so it does not affect
>>>>>> voting status.
>>>>>> 
>>>>>> On Tue, Jul 9, 2024 at 9:16 AM Elad Kalif  wrote:
>>>>>>> 
>>>>>>> Hey all,
>>>>>>> 
>>>>>>> I have just cut the new wave Airflow Providers packages. This email is
>>>>>>> calling a vote on the release,
>>>>>>> which will last for 72 hours - which means that it will end on July
>>> 12,
>>>>>>> 2024 07:15 AM UTC and until 3 binding +1 votes have been received.
>>>>>>> 
>>>>>>> Consider this my (binding) +1.
>>>>>>> 
>>>>>>> Airflow Providers are available at:
>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>>>>>> 
>>>>>>> *apache-airflow-providers--*.tar.gz* are the binary
>>>>>>> Python "sdist" release - they are also official "sources" for the
>>>>>> provider
>>>>>>> packages.
>>>>>>> 
>>>>>>> *apache_airflow_providers_-*.whl are the binary
>>>>>>> Python "wheel" release.
>>>>>>> 
>>>>>>> The test procedure for PMC members is described in
>>>>>>> 
>>>>>> 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>>>>>>> 
>>>>>>> The test procedure for and Contributors who would like to test this
>>> RC is
>>>>>>> described in:
>>>>>>> 
>>>>>> 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>>>>>>> 
>>>>>>> 
>>>>>>> Public keys are available at:
>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>>>>> 
>>>>>>> Please vote accordingly:
>>>>>>> 
>>>>>>> [ ] +1 approve
>>>>>>> [ ] +0 no opinion
>>>>>>> [ ] -1 disapprove with the reason
>>>>>>> 
>>>>>>> Only votes from PMC members are binding, but members of the community
>>> are
>>>>>>> encouraged to test the release and vote with "(non-binding)".
>>>>>>> 
>>>>>>> Please note that the version number excludes the 'rcX' string.
>>>>>>> This will allow us to rename the arti

Re: [VOTE] Airflow Providers prepared on July 09, 2024

[Wei Lee]

-1 for weaviate due to https://github.com/apache/airflow/pull/40670
(Thanks, Tamara, for fixing the issue!)

Best,
Wei


> On Jul 9, 2024, at 9:43 PM, Jarek Potiuk  wrote:
> 
> Yep: -1 on weaviate following 40670
> 
> On Tue, Jul 9, 2024 at 3:25 PM Hussein Awala  wrote:
>> 
>> -1 (binding) for Weaviate provider -
>> https://github.com/apache/airflow/pull/40670
>> 
>> +1 (binding) for the rest - checked licences, checksums, signatures, and
>> sources. All looks good.
>> 
>> On Tue, Jul 9, 2024 at 10:10 AM Jarek Potiuk  wrote:
>> 
>>> +1 (binding) - checked all my changes, checked reproducibility,
>>> signatures, licences, checksums, all looks good.
>>> 
>>> One small thing I found was that changelog for weaviate was slightly
>>> broken (2.0.0 changelog swallowed already released 1.4.2) - I created
>>> a PR fixing it https://github.com/apache/airflow/pull/40663 but it
>>> does not affect the packagesand can be fixed directly by regenerating
>>> the docs without regenerating the package, so it does not affect
>>> voting status.
>>> 
>>> On Tue, Jul 9, 2024 at 9:16 AM Elad Kalif  wrote:
 
 Hey all,
 
 I have just cut the new wave Airflow Providers packages. This email is
 calling a vote on the release,
 which will last for 72 hours - which means that it will end on July 12,
 2024 07:15 AM UTC and until 3 binding +1 votes have been received.
 
 Consider this my (binding) +1.
 
 Airflow Providers are available at:
 https://dist.apache.org/repos/dist/dev/airflow/providers/
 
 *apache-airflow-providers--*.tar.gz* are the binary
 Python "sdist" release - they are also official "sources" for the
>>> provider
 packages.
 
 *apache_airflow_providers_-*.whl are the binary
 Python "wheel" release.
 
 The test procedure for PMC members is described in
 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
 
 The test procedure for and Contributors who would like to test this RC is
 described in:
 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
 
 
 Public keys are available at:
 https://dist.apache.org/repos/dist/release/airflow/KEYS
 
 Please vote accordingly:
 
 [ ] +1 approve
 [ ] +0 no opinion
 [ ] -1 disapprove with the reason
 
 Only votes from PMC members are binding, but members of the community are
 encouraged to test the release and vote with "(non-binding)".
 
 Please note that the version number excludes the 'rcX' string.
 This will allow us to rename the artifact without modifying
 the artifact checksums when we actually release.
 
 The status of testing the providers by the community is kept here:
 https://github.com/apache/airflow/issues/40661
 The issue is also the easiest way to see important PRs included in the RC
 candidates.
 Detailed changelog for the providers will be published in the
>>> documentation
 after the
 RC candidates are released.
 
 You can find the RC packages in PyPI following these links:
 
 https://pypi.org/project/apache-airflow-providers-amazon/8.26.0rc1/
 
>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.3.3rc1/
 https://pypi.org/project/apache-airflow-providers-common-sql/1.14.2rc1/
 https://pypi.org/project/apache-airflow-providers-databricks/6.7.0rc1/
 https://pypi.org/project/apache-airflow-providers-docker/3.12.2rc1/
 https://pypi.org/project/apache-airflow-providers-fab/1.2.1rc1/
 https://pypi.org/project/apache-airflow-providers-google/10.21.0rc1/
 https://pypi.org/project/apache-airflow-providers-influxdb/2.6.0rc1/
 
>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/10.2.0rc1/
 https://pypi.org/project/apache-airflow-providers-openlineage/1.9.1rc1/
 https://pypi.org/project/apache-airflow-providers-oracle/3.10.3rc1/
 https://pypi.org/project/apache-airflow-providers-snowflake/5.6.0rc1/
 https://pypi.org/project/apache-airflow-providers-teradata/2.4.0rc1/
 https://pypi.org/project/apache-airflow-providers-weaviate/2.0.0rc1/
 https://pypi.org/project/apache-airflow-providers-ydb/1.1.0rc1/
 
 Cheers,
 Elad Kalif
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>>> For additional commands, e-mail: dev-h...@airflow.apache.org
>>> 
>>> 
> 

Re: [VOTE] Airflow Providers prepared on July 02, 2024 (openlineage rc2)

[Wei Lee]

+1 (non-binding) 

Best,
Wei

> On Jul 3, 2024, at 8:14 PM, Rahul Vats  wrote:
> 
> +1 (non-binding)
> 
> Verified with our example DAGs.
> 
> Regards,
> Rahul Vats
> 9953794332
> 
> 
> On Wed, 3 Jul 2024 at 14:30, Maciej Obuchowski 
> wrote:
> 
>> +1 (non binding)
>> 
>> Verified by running some example DAGs.
>> 
>> wt., 2 lip 2024 o 18:16 Jarek Potiuk  napisał(a):
>> 
>>> Hey all,
>>> 
>>> I have just cut the new wave Airflow Providers packages.
>>> 
>>> This email is calling a vote on the release,
>>> which will last for 24 hours - which means that it will end on Wed, 3rd
>> of
>>> July at 18:30 CEST.
>>> 
>>> Consider this my (binding) +1.
>>> 
>>> This is an accelerated vote as this is just openlineage rc2 after rc1 was
>>> withdrawn from previous vote after a bug was found.
>>> 
>>> Airflow Providers are available at:
>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>> 
>>> *apache-airflow-providers--*.tar.gz* are the binary
>>> Python "sdist" release - they are also official "sources" for the
>> provider
>>> packages.
>>> 
>>> *apache_airflow_providers_-*.whl are the binary
>>> Python "wheel" release.
>>> 
>>> The test procedure for PMC members is described in
>>> 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>>> 
>>> The test procedure for and Contributors who would like to test this RC is
>>> described in:
>>> 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>>> 
>>> Public keys are available at:
>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>> 
>>> Please vote accordingly:
>>> 
>>> [ ] +1 approve
>>> [ ] +0 no opinion
>>> [ ] -1 disapprove with the reason
>>> 
>>> Only votes from PMC members are binding, but members of the community are
>>> encouraged to test the release and vote with "(non-binding)".
>>> 
>>> Please note that the version number excludes the 'rcX' string.
>>> This will allow us to rename the artifact without modifying
>>> the artifact checksums when we actually release.
>>> 
>>> The status of testing the providers by the community is kept here:
>>> https://github.com/apache/airflow/issues/40555
>>> 
>>> The issue is also the easiest way to see important PRs included in the RC
>>> candidates.
>>> Detailed changelog for the providers will be published in the
>> documentation
>>> after the
>>> RC candidates are released.
>>> 
>>> You can find the openlineage RC packages in PyPI following this link:
>>> 
>>> https://pypi.org/project/apache-airflow-providers-openlineage/1.9.0rc2/
>>> 
>>> Cheers,
>>> 
>>> J.
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [DISCUSS] @remove provide_bucket_name and @unify_bucket_name_and_key?

[Wei Lee]

It looks like only Alibaba and Amazon providers use these decorators. Removing 
them seems like simplifying things, but I’m not the actual user of these 
features. I would vote a +0 for removing them. We probably only need to bump 
the major version of these providers.

Best,
Wei

> On Jul 2, 2024, at 12:23 PM, Amogh Desai  wrote:
> 
> I understand that having multiple ways of doing the same thing is not
> needed and we need
> to slowly and eventually clean up these things to simplify the codebase and
> also make it
> maintainable.
> 
> Aligning with one of the principles of Airflow3, we need to simplify
> things, nuke unnecessary
> stuff, remove deprecated code etc. This will have multiple benefits for the
> ecosystem:
> 1. Maintainability is not a headache. We will know for sure that users are
> using certain
> things in a certain way.
> 2. It will make it easier for us to make some decisions like this one
> without major discussions
> and edge case considerations because we know how users are using certain
> patterns (aligns with
> point 1)
> 3. Easier for new contributors to contribute to :)
> 
> I am all up for cleaning up the codebase if we have something of
> this nature. My 2c.
> 
> Thanks & Regards,
> Amogh Desai
> 
> 
> On Tue, Jul 2, 2024 at 4:29 AM Daniel Standish
>  wrote:
> 
>> Sometimes in software, we try to be helpful by doing a bunch of work for
>> the user, but it ends up causing a bunch of confusion and maintenance
>> headaches.  I think these decorators are an example of that.
>> 
>> What `unify_bucket_name_and_key` does is, it tries to make it so you could
>> pass the full S3 URI to `key`, or you can provide them separately as bucket
>> and key, and the decorator will "figure it out".
>> 
>> Then this decorator can also be combined with another one,
>> @provide_bucket_name, which, if you don't provide the bucket name, it will
>> grab it from an airflow connection.
>> 
>> Then, sometimes the params are again passed through another function
>> `get_s3_bucket_key`.
>> 
>> This requires us to add tests to ensure that the decorators are applied in
>> the correct order, and now, to handle the various async cases (the
>> decorated fn may be either coroutine or async iterator).
>> 
>> It's a bunch of code and complexity that, really does not need to exist.
>> I.e. what if bucket were simply bucket, and key were simply key, always?
>> 
>> If it were up to me, I'd just remove all of it.  Just force the user to
>> supply a bucket and key separately.
>> 
>> We can of course do this since semver allows it.  But some portion of users
>> will be depending on this change and will have to change their code.
>> 
>> Curious what others think.
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [ANNOUNCE] New committer: Ryan Hatter

[Wei Lee]

Congratulations, Ryan! Welcome! 

Best,
Wei

> On Jun 29, 2024, at 3:20 AM, Scheffler Jens (XC-AS/EAE-ADA-T) 
>  wrote:
> 
> Welcome to the club!
> 
> Mit freundlichen Grüßen / Best regards
> 
> Jens Scheffler
> 
> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY 
> | www.bosch.com
> Tel. +49 711 811-91508 | Mobil +49 160 90417410 | jens.scheff...@de.bosch.com
> 
> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; 
> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus 
> Forschner, 
> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
> 
> -Original Message-
> From: Aritra Basu  
> Sent: Friday, June 28, 2024 8:52 PM
> To: dev@airflow.apache.org
> Subject: Re: [ANNOUNCE] New committer: Ryan Hatter
> 
> Congrats Ryan! Awesome to see it!
> --
> Regards,
> Aritra Basu
> 
> On Fri, Jun 28, 2024, 11:58 PM Elad Kalif  wrote:
> 
>> Congrats!
>> 
>> On Fri, Jun 28, 2024 at 9:17 PM Vikram Koka 
>> 
>> wrote:
>> 
>>> Awesome!
>>> Congratulations Ryan!
>>> 
>>> On Fri, Jun 28, 2024 at 11:07 AM Pierre Jeambrun 
>>> 
>>> wrote:
>>> 
 Well done Ryan :)
 
 Le ven. 28 juin 2024 à 19:42, Ferruzzi, Dennis
>>>  
 a écrit :
 
> Hey, congrats!
> 
> 
> - ferruzzi
> 
> 
> 
> From: Pankaj Koti 
> Sent: Friday, June 28, 2024 10:21 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [ANNOUNCE] New committer: Ryan Hatter
> 
> CAUTION: This email originated from outside of the organization. 
> Do
>> not
> click links or open attachments unless you can confirm the 
> sender and
 know
> the content is safe.
> 
> 
> 
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>>> externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si 
> vous ne
 pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>> certain
 que
> le contenu ne présente aucun risque.
> 
> 
> 
> Congratulations Ryan, very happy for you and well deserved !! 
> 🥳🥳🥳
> 
> On Fri, 28 Jun 2024, 22:45 Vincent Beck, 
>> wrote:
> 
>> Congrats!!
>> 
>> On 2024/06/28 17:04:36 Amogh Desai wrote:
>>> Congratulations Ryan!
>>> 
>>> Welcome onboard :)
>>> 
>>> Thanks & Regards,
>>> Amogh Desai
>>> 
>>> 
>>> On Fri, Jun 28, 2024 at 10:32 PM Jarek Potiuk 
>>> 
> wrote:
>>> 
 Congrats!
 
 On Fri, Jun 28, 2024 at 6:59 PM Jed Cunningham <
>> jedcunning...@apache.org>
 wrote:
 
> The Project Management Committee (PMC) for Apache 
> Airflow has invited Ryan Hatter to become a committer 
> and we are
>>> pleased
> to announce that they have accepted.
> 
> Ryan has been involved in the Airflow community for a 
> few
>> years
> now -
> contributing code, triaging and creating issues, 
> answering
> questions
>> on
> Slack, etc.
> He also contributed the custom names for mapped tasks 
> feature
>>> in
>> Airflow
> 2.9,
> which was a long time ask from the community.
> 
> Congratulations Ryan, and welcome!
> 
 
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> For additional commands, e-mail: dev-h...@airflow.apache.org
>> 
>> 
> 
 
>>> 
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] June 2024 PR of the Month

[Wei Lee]

+1 for #39355. Love dark mode!

Best,
Wei

> On Jun 28, 2024, at 11:00 AM, Mehta, Shubham  
> wrote:
> 
> +1 for #39355. 
> 놀라운 작업, 윤현수 님. 계속 힘내세요!
> 
> Shubham
> 
> On 2024-06-27, 7:49 PM, "Vikram Koka"  LID> 
> wrote:
> 
> 
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you can confirm the sender and know the 
> content is safe.
> 
> 
> 
> 
> 
> 
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
> cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
> confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
> contenu ne présente aucun risque.
> 
> 
> 
> 
> 
> 
> Voting for #39355. Love the dark mode feature!
> 
> 
> On Wed, Jun 26, 2024 at 9:45 PM Amogh Desai  >
> wrote:
> 
> 
>> My vote would definitely go to
>> https://github.com/apache/airflow/pull/39355!
>> 
>> Developers love dark mode, and I mean it is done fantastically!
>> 
>> Thanks & Regards,
>> Amogh Desai
>> 
>> 
>> On Thu, Jun 27, 2024 at 3:57 AM Kaxil Naik > > wrote:
>> 
>>> My vote would be biased to #39355 because it is DARK MODE :
>>> https://github.com/apache/airflow/pull/39355
>>>  :)
>>> 
>>> On Wed, 26 Jun 2024 at 21:28, Jarek Potiuk >> > wrote:
>>> 
 IF (and only IF) the votes from Jens and Vincent go to
 https://github.com/apache/airflow/pull/39355
  (#39*3*55) then it's my
>> +1
>>> as
 well. Otherwise I am NOT voting for
 https://github.com/apache/airflow/pull/39555
  but
 https://github.com/apache/airflow/pull/39355
  has still my vote
 
 :P
 
 On Wed, Jun 26, 2024 at 10:22 PM Vincent Beck >>> >
>>> wrote:
 
> Same! +1 for #39555. Quite an impactful contribution for a first time
> contributor!!
> 
> On 2024/06/26 20:17:47 "Scheffler Jens (XC-AS/EAE-ADA-T)" wrote:
>> Thanks for the proposals...
>> 
>> There was a "late arrival" and month is not over... as it was a
>> long
> runner and we had a first-time contributor with a long breath making
>> it
 to
> the finish line "never gonna give you up" (-->
> https://www.youtube.com/watch?v=dQw4w9WgXcQ)...
>> 
>> I'd like to add and +1 vote for #39555
>> 
>> Mit freundlichen Grüßen / Best regards
>> 
>> Jens Scheffler
>> 
>> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
>> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565
>> Stuttgart-Vaihingen |
> GERMANY | http://www.bosch.com/
>> Tel. +49 711 811-91508 | Mobil +49 160 90417410 |
> jens.scheff...@de.bosch.com 
> 
>> 
>> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
>> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
>> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr.
>>> Markus
> Forschner,
>> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
>> 
>> -Original Message-
>> From: Briana Okyere > LID>
>> Sent: Tuesday, June 25, 2024 9:25 PM
>> To: dev@airflow.apache.org 
>> 
>> Subject: [VOTE] June 2024 PR of the Month
>> 
>> Hey All,
>> 
>> It's once again time to vote for the PR of the Month!
>> 
>> With the help of the `get_important_pr_candidates` script in
>>> dev/stats,
> we've identified the following candidates:
>> 
>> PR #37948: [AIP-49] OpenTelemetry Traces for Apache Airflow <
> https://github.com/apache/airflow/pull/37948> 
> ;>
>> 
>> PR #39936: add git-sync-ssh secret template <
> https://github.com/apache/airflow/pull/39936> 
> ;>
>> 
>> PR #39585: Prevent start trigger initialization in scheduler <
> https://github.com/apache/airflow/pull/39585> 
> ;>
>> 
>> PR #38524: Chart: Allow AWS ECS Executor <
> https://github.com/apache/airflow/pull/38524> 
> ;>
>> 
>> PR #40034: AIP-61 - Add executor field to the task instance API <
> https

Re: Using AI / Dosu to help us with triaging issues

[Wei Lee]

+1 for this idea. I remember we had this discussion during PyCon US, haha.

So, in the first phase, we'll need a maintainer's approval before the AI 
response is sent. I'm not sure whether this will be a burden to maintainers. 
I'm thinking of adding a label to that AI response answer; the user can decide 
whether to accept that even before the maintainers' approval. If maintainers 
check the response and it's ok, they can remove that label.

Best,
Wei

> On Jun 27, 2024, at 3:07 PM, Jarek Potiuk  wrote:
> 
>> 
>> 
>> I guess it's too early to discuss what the workflow we're picturing for it
>> is. But I would like it if a reporter is unable to see the comment until
>> it's approved (which I guess is exactly what you're also suggesting Jarek).
>> 
> 
> Absolutely. Actually - we (few of maintainers) can already see the answers
> generated in a DoSU interface as "drafts" - and once we enable labelling we
> can on-board more maintainers and triage team and work with DoSu team on
> implementing a workflow that might be good to start with.


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [PROPOSAL] Use Trusted Publishing workflow for Airflow releases to PyPI

[Wei Lee]

This proposal is great! PyPI security has been valued a lot these days. Glad 
we're also joining.

Best,
Wei

> On Jun 25, 2024, at 8:01 PM, Jarek Potiuk  wrote:
> 
> Yes and no :)
> 
> We publish alpha/betas - yes. No change there. But for RCs what we publish
> in SVN currently are the packages that are built fro RC tag but without rc
> suffix - so that when they pass the voting we upload them to PyPI without
> regenerating them (RC becomes final).
> 
> But we do not publish the PYPI RCs - since PYPI uploads are immutable, we
> need to publish PYPI RCs with the rc suffixes. So far we just generated
> them and published to PyPI for testing but we did not upload them to SVN.
> 
> 
> So if we want to pull RCs from SVN - we need to upload there both: the RC
> version for PyPI (with RC suffix) and the no-suffix candidate that might
> become the final version once voted.
> 
> J


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on June 22, 2024

[Wei Lee]

+ 1 non-binding. Verified my change and tested the RCs with some example DAGs.

Best,
Wei

> On Jun 24, 2024, at 12:38 PM, Rahul Vats  wrote:
> 
> +1 non-binding. Verified running some example dags. LGTM!
> 
> Regards,
> Rahul Vats
> 9953794332
> 
> 
> On Mon, 24 Jun 2024 at 09:41, Amogh Desai  wrote:
> 
>> +1 non binding
>> 
>> Tested some example dags in cncf and hive providers. My changes work well
>> too!
>> 
>> Thanks & Regards,
>> Amogh Desai
>> 
>> 
>> On Mon, Jun 24, 2024 at 1:02 AM Marcus Eagan  wrote:
>> 
>>> Amazing work!
>>> 
>>> On Sat, Jun 22, 2024 at 07:47 Jarek Potiuk  wrote:
>>> 
 +1 (binding): tested signatures, checksums, licences, reproducibility,
 checked my changes. All looks good.
 
 On Sat, Jun 22, 2024 at 3:26 PM Elad Kalif  wrote:
 
> Hey all,
> 
> I have just cut the new wave Airflow Providers packages. This email
>> is
> calling a vote on the release,
> which will last for 72 hours - which means that it will end on June
>> 25,
> 2024 13:25 PM UTC and until 3 binding +1 votes have been received.
> 
> Consider this my (binding) +1.
> 
> Airflow Providers are available at:
> https://dist.apache.org/repos/dist/dev/airflow/providers/
> 
> *apache-airflow-providers--*.tar.gz* are the binary
> Python "sdist" release - they are also official "sources" for the
 provider
> packages.
> 
> *apache_airflow_providers_-*.whl are the binary
> Python "wheel" release.
> 
> The test procedure for PMC members is described in
> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> 
> The test procedure for and Contributors who would like to test this
>> RC
>>> is
> described in:
> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
> 
> 
> Public keys are available at:
> https://dist.apache.org/repos/dist/release/airflow/KEYS
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> Only votes from PMC members are binding, but members of the community
>>> are
> encouraged to test the release and vote with "(non-binding)".
> 
> Please note that the version number excludes the 'rcX' string.
> This will allow us to rename the artifact without modifying
> the artifact checksums when we actually release.
> 
> The status of testing the providers by the community is kept here:
> https://github.com/apache/airflow/issues/40382
> 
> The issue is also the easiest way to see important PRs included in
>> the
>>> RC
> candidates.
> Detailed changelog for the providers will be published in the
 documentation
> after the
> RC candidates are released.
> 
> You can find the RC packages in PyPI following these links:
> https://pypi.org/project/apache-airflow-providers-amazon/8.25.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-drill/2.7.2rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-flink/1.4.2rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.4.2rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-hive/8.1.2rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-kafka/1.5.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-kylin/3.6.2rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-spark/4.8.2rc1/
> https://pypi.org/project/apache-airflow-providers-cloudant/3.5.2rc1/
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.3.2rc1/
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-common-compat/1.0.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-common-sql/1.14.1rc1/
> 
>> https://pypi.org/project/apache-airflow-providers-databricks/6.6.0rc1/
> 
>> https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.9.0rc1/
> https://pypi.org/project/apache-airflow-providers-docker/3.12.1rc1/
> https://pypi.org/project/apache-airflow-providers-exasol/4.5.2rc1/
> https://pypi.org/project/apache-airflow-providers-fab/1.2.0rc1/
> https://pypi.org/project/apache-airflow-providers-facebook/3.5.2rc1/
> https://pypi.org/project/apache-airflow-providers-ftp/3.10.0rc1/
> https://pypi.org/project/apache-airflow-providers-github/2.6.2rc1/
> https://pypi.org/project/apache-airflow-providers-google/10.20.0rc1/
> https://pypi.org/project/apache-airflow-providers-grpc/3.5.2rc1/
> https://pypi.org/project/apache-airflow-providers-http/4.12.0rc1/
> 
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/10.1.2rc1/
> 
 
>>> 
>> https:/

Re: [Meeting Notes] Airflow 3.0 Dev call - 13 June 2024

[Wei Lee]

Thanks Kaxil for summarising! Also tried to update the invitations through the 
link. It works perfectly.

Best,
Wei

> On Jun 17, 2024, at 3:18 AM, Kaxil Naik  wrote:
> 
> Cool, thanks
> 
> On Sun, 16 Jun 2024 at 20:15, Jarek Potiuk  wrote:
> 
>> The summary looks good !
>> 
>> BTW. Maybe attaching an .ics from the Zoom meeting to the meeting summary
>> page might help with re-adding calendar meetings without re-registering. I
>> added it as attachment and updated the doc with link to it.
>> 
>> On Sun, Jun 16, 2024 at 9:07 PM Kaxil Naik  wrote:
>> 
>>> I have also added a meeting for this week, so if you don't have it on
>> your
>>> calendar, just open the registration link again. Adding it again should
>>> update your calendar. In case you have any issues, drop me a note here or
>>> on Slack.
>>> 
>>> The Airflow 3 Home page on cWiki
>>>  has
>> also
>>> been updated to simplify and reflect discussions from the last 2 calls.
>>> 
>>> Regards,
>>> Kaxil
>>> 
>>> 
>>> On Sun, 16 Jun 2024 at 19:56, Kaxil Naik  wrote:
>>> 
 Hey all,
 
 I have updated our meeting notes document to summarize the discussion
 from our 13th June dev call for Airflow 3.0.
 
 Link:
 
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-Summary.1
 
 To all those who attended, can you please double-check and add if I
>> have
>>> missed
 anything?
 
 To all those who didn't join, if you disagree with anything in the
 Summary, please voice your opinion.
 
 I will send a separate email for the agenda for the next meeting on
>> 20th
 June.
 
 Regards,
 Kaxil
 
 --
 
 Including the Summary here too (might break formatting):
 
 *Drafting proposals & getting feedback*
 
   - The team discussed the challenges of Apache Confluence access and
   considered using Google Docs for initial feedback before moving to
   Confluence.
   - The general consensus was that developers could utilize the tool
>> of
   their choice to get initial feedback (on their high-level draft) as
>>> long as
   the final AIP is published on Airflow's Confluence space for AIPs
   <
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
 
   .
   - If someone does not have an account on Apache Confluence to
>> comment
   on existing proposals or create a new proposal, reach out on
   #airflow-3-dev Slack channel
   <
>>> https://apache-airflow.slack.com/archives/C07813CNKA8/p1718292608810799>
   .
 
 
 *Personas for the Airflow User*
 
   - Elad prepared a document on Airflow Actors
   >> 
>>> that
   the group discussed to get initial feedback. Several good
>> suggestions
>>> were
   made, so it was decided to take the discussion offline/async and
>>> continue
   the discussion on that page.
 
 
 *Task Context draft AIP*
 
   - Ash presented the initial draft of the Task Execution
   InterfaceContext AIP (formerly known as Task Context SDK).
   - The draft AIP is now published at AIP-72 Task Execution Interface
   <
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK
>>> 
>>> and
   ready for review, although some technical details still need to be
>>> worked
   out.
 
 
 *Frequency of the Next Dev calls*
 
   - The general consensus was to make the dev calls weekly for the
>> next
   2-3 weeks
 
 
 *Timeline to release Airflow 3*
 
   - Based on the discussions on the first dev call, we also agreed
>> that
   we should target *March-April 2025* to release Airflow 3.
   - We will announce the dates and discuss some of the feature sets
>> with
   the community at the Airflow Summit in September this year.
 
 
 *Action items*
 
   - Docs needing reviews:
  - AIP-72 Task Execution Interface
  <
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK
 
  - Enhanced Data Awareness proposal
  <
>>> 
>> https://docs.google.com/document/d/1Sra65yjbAIZ2mZIbSUL9YMPrW73ltDEPWTCD4J3j2hQ/edit?usp=sharing
 
  - Airflow Actors
  <
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Actors>
   - Updating AIP template & Housekeeping of AIPs & label additions
>>> (Kaxil
   Naik  )
 
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: Google Provider Package System Tests Dashboard

[Wei Lee]

This is great!

Best,
Wei

> On Jun 12, 2024, at 9:47 PM, Bishundeo, Rajeshwar 
>  wrote:
> 
> Fantastic job from the Google team. Love it!!
> 
> -- Rajesh 
> 
> 
> 
> 
> 
> 
> On 2024-06-12, 9:20 AM, "Pankaj Koti"   
> LID> wrote:
> 
> 
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you can confirm the sender and know the 
> content is safe.
> 
> 
> 
> 
> 
> 
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
> cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
> confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
> contenu ne présente aucun risque.
> 
> 
> 
> 
> 
> 
> yes, indeed a great one!
> 
> 
> 
> 
> Best regards,
> 
> 
> *Pankaj Koti*
> Senior Software Engineer (Airflow OSS Engineering team)
> Location: Pune, Maharashtra, India
> Timezone: Indian Standard Time (IST)
> 
> 
> 
> 
> On Wed, Jun 12, 2024 at 6:42 PM Vincent Beck   > wrote:
> 
> 
>> Really nice!!
>> 
>> On 2024/06/12 11:38:13 Jarek Potiuk wrote:
 I believe we should include a reference to it in the Google provider
>>> documentation.
>>> 
>>> Already there: https://github.com/apache/airflow/pull/40102 
>>>  in the
>> README
>>> documentation (which is where it should be as it is mostly
>>> maintainer/contributor docs rather than user's docs.
>>> 
>>> J.
>>> 
>>> 
>>> On Wed, Jun 12, 2024 at 1:34 PM Pankaj Singh >>  >
>>> wrote:
>>> 
 Nice! Thanks for sharing. I believe we should include a reference to
>> it in
 the Google provider documentation.
 
 On Wed, Jun 12, 2024 at 3:24 PM Freddy Demiane
>> mailto:fdemi...@google.com.inva> 
>> lid
> 
 wrote:
 
> Hi Team,
> 
> At Google Cloud Composer, we developed a public dashboard that shows
>> the
> results of the System Tests of the head revision of the Google
>> Provider
> Package against the head revision of Apache Airflow. This will help
 detect
> regressions caused by modifying an operator or a system test. At the
> moment, the system tests are executed every 6 hours. Here is the
>> link for
> the dashboard:
> 
>> https://storage.googleapis.com/providers-dashboard-html/dashboard.html 
>>  .
> We hope this eases the development process!
> 
> Best,
> Freddy
> 
 
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org 
>>  
>> 
>> For additional commands, e-mail: dev-h...@airflow.apache.org 
>>  
>> 
>> 
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org 
> 
> For additional commands, e-mail: dev-h...@airflow.apache.org 
> 

Re: Call with Nielsen team demoing their DAG debugging feature

[Wei Lee]

Hi Jarek,

Love to hear about it! But it’s way too late for my time. 😞 Would it be 
possible for us to record it?

Best,
Wei

> On Jun 11, 2024, at 7:12 AM, Julian LaNeve  
> wrote:
> 
> +1 I'd love to hear about it but unfortunately can't make the meeting!
> 
> --
> 
> *Julian LaNeve*
> CTO
> 
> Email: jul...@astronomer.io 
> ( jul...@astronomer.io  ) Mobile: 330 509 5792
> 
> On Mon, Jun 10, 2024 at 3:05 PM, Constance Martineau < 
> consta...@astronomer.io.invalid  > 
> wrote:
> 
>> 
>> 
>> 
>> Hello again,
>> 
>> 
>> 
>> Given all the enthusiasm - assuming Nielsen is ok with this - what if
>> someone recorded the meeting so that it could be shared with those that
>> are interested?
>> 
>> 
>> 
>> Constance
>> 
>> 
>> 
>> On Mon, Jun 10 , 2024 at 1:48 PM Constance Martineau < constance@ astronomer.
>> io ( consta...@astronomer.io  ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Hi Jarek,
>>> 
>>> 
>>> 
>>> Same :)
>>> 
>>> 
>>> 
>>> Thanks,
>>> Constance
>>> 
>>> 
>>> 
>>> On Mon, Jun 10 , 2024 at 9:57 AM Amogh Desai < amoghdesai. oss@ gmail. com
>>> ( amoghdesai@gmail.com  ) > wrote:
>>> 
>>> 
 
 
 Hello Jarek,
 
 
 
 Please add me to the invite as well.
 
 
 
 Thanks & Regards,
 Amogh Desai
 
 
 
 On Mon, Jun 10 , 2024 at 11:22 AM Abhishek Bhakat
 < abhishek. bhakat@ astronomer. io. invalid (
 abhishek.bha...@astronomer.io.invalid 
  ) > wrote:
 
 
> 
> 
> Hi Jarek,
> 
> 
> 
> I would also like to join as well, please.
> 
> 
> 
> Thanks,
> Avi
> 
> 
> 
> On Sat, Jun 8 , 2024 at 3:32 PM Buğra Öztürk < ozturkbugra93@ gmail. com (
> ozturkbugr...@gmail.com  ) > wrote:
> 
> 
>> 
>> 
>> Hello Jarek,
>> 
>> 
>> 
>> Thanks for sharing! It sounds very interesting. I would like to join.
>> 
>> 
> 
> 
> 
> Could
> 
> 
>> 
>> 
>> you please forward to me as well?
>> 
>> 
>> 
>> Thanks!
>> 
>> 
>> 
>> On Sat , 8 Jun 2024 , 17:28 Jed Cunningham, < jedcunningham@ apache. org 
>> (
>> jedcunning...@apache.org  ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Interesting. Can you forward to me as well Jarek? Thanks!

Re: [DISCUSS] Restore the SQL server backend

[Wei Lee]

I agree with Jed and the following comments. If my memory serves me right, this 
topic has been discussed a few times in the past. 5% doesn't seem very 
convincing. Even if it's biased, I'm still not persuaded that there are a large 
number of users that are worth the community's effort. And Jarek pointed out a 
great solution for forking Airflow and adding MSSQL support to it.

Best,
Wei

> On May 31, 2024, at 7:50 PM, Elad Kalif  wrote:
> 
> I agree with Jarek
> 
> I am a bit worried about the mental model of this proposal as you are
> offering to deliver a feature but you are not offering being a community
> member.
> I had a lot of frustration with the MsSQL backend tests, it really caused
> me pain as a contributor. According to your mental model - will you
> actively review community PRs, triage Airflow issues and offer guidance and
> help when needed about MsSQL or will the maintainers have to track these
> problems and actively tag you/your team for assistance?
> 
> Let me give an example: User opens a Github issue about HA scheduler. Will
> your team participate in the issue triage? Or do you expect the community
> to triage the issue and only after some discussion when it turns out that
> it's MsSQL specific issue then we need to notify you?
> 
> On Fri, May 31, 2024 at 10:05 AM Jarek Potiuk  wrote:
> 
>>> We also understand and are ready to address the concerns stated in the
>> vote about support and resolving CI issues
>> 
>> Hello James,
>> 
>> Could you please explain how exactly are you planning to help a number of
>> maintainers who are working on developing new feature to make sure
>> they know and realise unobvious consequences of some of the DB changes they
>> might have when some of the features of MYSQL are causing - for example
>> heavy slowdown of  inserts because of rebalancing B-TREES on UUID index for
>> databases (that unlike Postgres and MariaDB) lack native UUID support (see
>> . How would you help with discovering similar type of issues see here
>> https://lists.apache.org/thread/7235o1bc3w4694sw8q9m4p58g3tdcjj7
>> 
>> Could you please explain how many people, effort and dedicated resources
>> (i.e. continuous testing of stability and performance you are going to
>> spend on fixing those)?
>> 
>> IMHO. If you see a LOT of users that want MsSQL support - you are
>> absolutely free to spend those money, effort and resources on making a fork
>> of Airflow with MsSQL support and charge a premium for that (and a large
>> one). That seems like a very good business model to make if you see a lot
>> of interest there.
>> 
>> This is all perfectly fine according to our licence and community would be
>> really thankful for someone who would take the burden of maintaining MSSQL
>> while also making it possible for MSSQL users. Maybe that's the way to go
>> for you?
>> 
>> J,
>> 
>> 
>> 
>> On Fri, May 31, 2024 at 8:32 AM James Duong
>>  wrote:
>> 
>>> Many of the MSSQL customers using Airflow with MSSQL as the backend are
>>> unlikely to participate in those types of surveys, unfortunately, so I
>> fear
>>> the numbers are biased.  We have had direct feedback from multiple very
>>> large MSSQL customers who see the removal of this support as a large
>>> blocker to using Airflow.
>>> 
>>> Although yes, Microsoft does support PostgreSQL (and MySQL), MSSQL is an
>>> extremely widely used and popular Database platform across different
>>> segments whether Enterprise, Government, Major or SMC. Various Oracle,
>> IBM
>>> and OSS customers are diversifying their Database platform with SQL and
>> it
>>> is important for Airflow-type products to support SQL.
>>> 
>>> We also understand and are ready to address the concerns stated in the
>>> vote about support and resolving CI issues.
>>> 
>>> From: Jarek Potiuk 
>>> Date: Thursday, May 30, 2024 at 3:47 PM
>>> To: dev@airflow.apache.org 
>>> Subject: Re: [DISCUSS] Restore the SQL server backend
>>> Agree with all comments above. Also I think bringing MySQL back is going
>> to
>>> make it way more complex to implement some of the improvements we thought
>>> about - mostly async DB operations (only recently - November 2023 async
>>> support has been added to MSSQL and we know from the history that MSSQL
>>> gave us a lot of headache while developing it and there is no reason to
>>> believe it will be different. And "helping in CI" is not going to cut it
>> -
>>> we need every maintainer who wants to implement a new DB change to become
>>> expert on what is different in MSSQL.
>>> 
>>> Honestly - if I'd lose 5% of users because their internal rules say
>>> MSSQL-only (and no Postgres, which as mentioned above is widely supported
>>> and popular including Azure) at the expense of better performance, less
>>> resource usage (as we expect with asyncio) delivered faster to remaining
>>> 95% users, then I know what my decision is.
>>> 
>>> BTW. That's not really a criteria we use for such decisions about
>>> technology, but unlike Amazon and Google, Micro

Re: [VOTE] Airflow Providers prepared on May 30, 2024

[Wei Lee]

+1 (non-binding)

Tested my changes and our example DAGs without encountering issues.

Best,
Wei

> On May 30, 2024, at 9:28 PM, Elad Kalif  wrote:
> 
> Hey all,
> 
> I have just cut the new wave Airflow Providers packages. This email is
> calling a vote on the release,
> which will last for 72 hours - which means that it will end on June 02,
> 2024 13:25 PM UTC and until 3 binding +1 votes have been received.
> 
> Consider this my (binding) +1.
> 
> Airflow Providers are available at:
> https://dist.apache.org/repos/dist/dev/airflow/providers/
> 
> *apache-airflow-providers--*.tar.gz* are the binary
> Python "sdist" release - they are also official "sources" for the provider
> packages.
> 
> *apache_airflow_providers_-*.whl are the binary
> Python "wheel" release.
> 
> The test procedure for PMC members is described in
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> 
> The test procedure for and Contributors who would like to test this RC is
> described in:
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
> 
> 
> Public keys are available at:
> https://dist.apache.org/repos/dist/release/airflow/KEYS
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> Only votes from PMC members are binding, but members of the community are
> encouraged to test the release and vote with "(non-binding)".
> 
> Please note that the version number excludes the 'rcX' string.
> This will allow us to rename the artifact without modifying
> the artifact checksums when we actually release.
> 
> The status of testing the providers by the community is kept here:
> https://github.com/apache/airflow/issues/39947
> 
> The issue is also the easiest way to see important PRs included in the RC
> candidates.
> Detailed changelog for the providers will be published in the documentation
> after the
> RC candidates are released.
> 
> You can find the RC packages in PyPI following these links:
> 
> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.3.0rc2/
> https://pypi.org/project/apache-airflow-providers-amazon/8.24.0rc1/
> https://pypi.org/project/apache-airflow-providers-teradata/2.2.0rc1/
> 
> Cheers,
> Elad Kalif


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] May 2024 PR of the Month

[Wei Lee]

Hard to choose only one, but my vote goes to #39336.

Best,
Wei

> On May 29, 2024, at 4:45 AM, Scheffler Jens (XC-AS/EAE-ADA-T) 
>  wrote:
> 
> I can only agree on all the excellent options but I am ver much like 
> Constance and thank @dstandish for the cleanup in #39336… but close by my +1 
> (binding) this time goes to #39650!
> 
> Sent from Outlook for iOS
> 
> From: Pankaj Koti  >
> Sent: Tuesday, May 28, 2024 8:58:20 PM
> To: dev@airflow.apache.org  
> mailto:dev@airflow.apache.org>>
> Subject: Re: [VOTE] May 2024 PR of the Month
> 
> Yes, indeed, all the nominations are excellent.
> 
> Since I have only one vote, I give a +1 to #39336.
> 
> Best regards,
> 
> *Pankaj Koti*
> Senior Software Engineer (Airflow OSS Engineering team)
> Location: Pune, Maharashtra, India
> Timezone: Indian Standard Time (IST)
> 
> 
> On Wed, May 29, 2024 at 12:18 AM Brent Bovenzi 
> wrote:
> 
>> +1 on #39336
>> 
>> On Tue, May 28, 2024 at 2:14 PM Daniel Standish
>>  wrote:
>> 
>>> As Jed said, many good contenders.  Here's one vote for Kaxil's scarf PR
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F39510&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C529c38715b954506dab608dc7f483320%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638525195265199063%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=UJV8VROGf%2FG8tpxfgp0ZRpOniSnvUveH%2FYpbMjq1XHQ%3D&reserved=0.
>>>It will be really cool
>> to
>>> have some more data on what users are doing.
>>> 
>>> On Tue, May 28, 2024 at 11:10 AM Kaxil Naik >> > wrote:
>>> 
 #39336 -- pain since a long long long time!
 
 On Tue, 28 May 2024 at 19:05, Jed Cunningham >>> 
>>> 
 wrote:
 
> Some good contenders this month, but my vote also goes to #39336.

Re: [VOTE] Airflow Providers prepared on May 26, 2024

[Wei Lee]

-1 (non-binding) for apache-airflow-providers-cncf-kubernetes, as Rahul found 
an issue.
The issue is fixed in , as 
mentioned.

Best,
Wei


> On May 28, 2024, at 12:17 AM, Vincent Beck  wrote:
> 
> +1 non binding. All AWS system tests are running successfully against 
> apache-airflow-providers-amazon==8.23.0rc1. You can see the results here: 
> https://aws-mwaa.github.io/#/open-source/system-tests/version/98c5a3a2c6d1df722d56bb3748dfbc810d5952aa_8.23.0rc1.html
> 
> On 2024/05/27 14:01:48 Rahul Vats wrote:
>> -1 (non-binding) for cncf-kubernetes-provider. Our example DAGs are failing
>> due to an unexpected argument 'pod' in the read_namespaced_pod_log call.
>> Thanks to Wei for raising the fix PR
>> .
>> 
>> Regards,
>> Rahul Vats
>> 9953794332
>> 
>> 
>> On Mon, 27 May 2024 at 01:31, Jarek Potiuk  wrote:
>> 
>>> +1 (binding) - checked reproducibility, checksums, licences. signatures.
>>> Checked my changes.
>>> 
>>> On Sun, May 26, 2024 at 11:03 AM Elad Kalif  wrote:
>>> 
 Hey all,
 
 I have just cut the new wave Airflow Providers packages. This email is
 calling a vote on the release, which will last for 72 hours - which means
 that it will end on May 29, 2024 09:00 AM UTC and until 3 binding +1
>>> votes
 have been received.
 
 Consider this my (binding) +1.
 
 Airflow Providers are available at:
 https://dist.apache.org/repos/dist/dev/airflow/providers/
 
 *apache-airflow-providers--*.tar.gz* are the binary
 Python "sdist" release - they are also official "sources" for the
>>> provider
 packages.
 
 *apache_airflow_providers_-*.whl are the binary
 Python "wheel" release.
 
 The test procedure for PMC members is described in
 
 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
 
 The test procedure for and Contributors who would like to test this RC is
 described in:
 
 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
 
 
 Public keys are available at:
 https://dist.apache.org/repos/dist/release/airflow/KEYS
 
 Please vote accordingly:
 
 [ ] +1 approve
 [ ] +0 no opinion
 [ ] -1 disapprove with the reason
 
 Only votes from PMC members are binding, but members of the community are
 encouraged to test the release and vote with "(non-binding)".
 
 Please note that the version number excludes the 'rcX' string.
 This will allow us to rename the artifact without modifying
 the artifact checksums when we actually release.
 
 The status of testing the providers by the community is kept here:
 https://github.com/apache/airflow/issues/39842
 
 The issue is also the easiest way to see important PRs included in the RC
 candidates.
 Detailed changelog for the providers will be published in the
>>> documentation
 after the
 RC candidates are released.
 
 You can find the RC packages in PyPI following these links:
 
 https://pypi.org/project/apache-airflow-providers-airbyte/3.8.1rc1/
 https://pypi.org/project/apache-airflow-providers-alibaba/2.8.1rc1/
 https://pypi.org/project/apache-airflow-providers-amazon/8.23.0rc1/
 https://pypi.org/project/apache-airflow-providers-apache-beam/5.7.1rc1/
 
 
>>> https://pypi.org/project/apache-airflow-providers-apache-cassandra/3.5.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-drill/2.7.1rc1/
 
>>> https://pypi.org/project/apache-airflow-providers-apache-druid/3.10.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-flink/1.4.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.4.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-hive/8.1.1rc1/
 
>>> https://pypi.org/project/apache-airflow-providers-apache-impala/1.4.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-kafka/1.4.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-kylin/3.6.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-livy/3.8.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-pig/4.4.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-pinot/4.4.1rc1/
 https://pypi.org/project/apache-airflow-providers-apache-spark/4.8.1rc1/
 https://pypi.org/project/apache-airflow-providers-apprise/1.3.1rc1/
 https://pypi.org/project/apache-airflow-providers-arangodb/2.5.1rc1/
 https://pypi.org/project/apache-airflow-providers-asana/2.5.1rc1/
 
>>> https://pypi.org/project/apache-airflow-providers-atlassian-jira/2.6.1rc1/
 https://pypi.org/project/apache-airflow-providers-celery/3.7.1rc1/
 https://pypi.org/project/apache-airflow-provi

Re: [VOTE] Airflow Providers prepared on May 12, 2024

[Wei Lee]

+1 (non-binding)

Test a few examples DAGs with Amazon, google, and azure providers without 
encountering issues.

Best,
Wei

> On May 13, 2024, at 9:44 PM, Hussein Awala  wrote:
> 
> +1 (binding): checked the licences, the signatures, the checksums, and ran
> some testing dags for Amazon provider.
> 
> On Monday, May 13, 2024, Jarek Potiuk  wrote:
> 
>> +1 (binding): verified reproducibility, signatures, checksums, licences.
>> 
>> On Sun, May 12, 2024 at 9:34 PM Elad Kalif  wrote:
>> 
>>> Hey all,
>>> 
>>> I have just cut the new wave Airflow Providers packages. This email is
>>> calling a vote on the release,
>>> which will last for 72 hours - which means that it will end on May 15,
>> 2024
>>> 19:30 PM UTC and until 3 binding +1 votes have been received.
>>> 
>>> Consider this my (binding) +1.
>>> 
>>> Airflow Providers are available at:
>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>> 
>>> *apache-airflow-providers--*.tar.gz* are the binary
>>> Python "sdist" release - they are also official "sources" for the
>> provider
>>> packages.
>>> 
>>> *apache_airflow_providers_-*.whl are the binary
>>> Python "wheel" release.
>>> 
>>> The test procedure for PMC members is described in
>>> 
>>> https://github.com/apache/airflow/blob/main/dev/README_
>> RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>>> 
>>> The test procedure for and Contributors who would like to test this RC is
>>> described in:
>>> 
>>> https://github.com/apache/airflow/blob/main/dev/README_
>> RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>>> 
>>> 
>>> Public keys are available at:
>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>> 
>>> Please vote accordingly:
>>> 
>>> [ ] +1 approve
>>> [ ] +0 no opinion
>>> [ ] -1 disapprove with the reason
>>> 
>>> Only votes from PMC members are binding, but members of the community are
>>> encouraged to test the release and vote with "(non-binding)".
>>> 
>>> Please note that the version number excludes the 'rcX' string.
>>> This will allow us to rename the artifact without modifying
>>> the artifact checksums when we actually release.
>>> 
>>> The status of testing the providers by the community is kept here:
>>> https://github.com/apache/airflow/issues/39578
>>> 
>>> The issue is also the easiest way to see important PRs included in the RC
>>> candidates.
>>> Detailed changelog for the providers will be published in the
>> documentation
>>> after the
>>> RC candidates are released.
>>> 
>>> You can find the RC packages in PyPI following these links:
>>> 
>>> https://pypi.org/project/apache-airflow-providers-amazon/8.22.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-
>> apache-iceberg/1.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-google/10.18.0rc2/
>>> 
>>> https://pypi.org/project/apache-airflow-providers-
>> microsoft-azure/10.1.0rc2/
>>> https://pypi.org/project/apache-airflow-providers-pinecone/2.0.0rc2/
>>> https://pypi.org/project/apache-airflow-providers-tabular/1.5.1rc1/
>>> 
>>> Cheers,
>>> Elad Kalif
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Proposal for adding Telemetry via Scarf

[Wei Lee]

Got it. Thanks Jarek for pointing out!

Best,
Wei

> On May 9, 2024, at 3:59 PM, Ankit Chaurasia  wrote:
> 
> +1 non-binding
> 
> *Ankit Chaurasia*
> 
> 
> 
> 
> 
> 
> On Thu, May 9, 2024 at 11:16 AM Aritra Basu 
> wrote:
> 
>> +1 non-binding
>> 
>> --
>> Regards,
>> Aritra Basu
>> 
>> On Thu, May 9, 2024, 10:36 AM Amogh Desai 
>> wrote:
>> 
>>> +1 binding
>>> 
>>> Thanks & Regards,
>>> Amogh Desai
>>> 
>>> 
>>> On Thu, May 9, 2024 at 10:29 AM Jarek Potiuk  wrote:
>>> 
>>>> Short reminder and correction :).
>>>> 
>>>> Wei Lee - as a committer, your vote is binding for any votes except
>>>> releases. Releases are special - they are a legal act of the
>>>> Apache Software Foundation, so when you vote on releases, only PMC
>> votes
>>>> are binding.
>>>> 
>>>> Generally "releasing software" is what the ASF Foundation does. The
>>>> foundation does not create software, only releases it ("for the public
>>>> good").
>>>> The PMC member is an official role in the ASF bylaws
>>>> https://www.apache.org/foundation/bylaws.html  (Apache Airflow is a
>>> PMC).
>>>> This is according to Delaware laws - that's where Foundation is
>>> registered
>>>> as https://en.wikipedia.org/wiki/501(c)(3)_organization non-profit
>>>> organisation,
>>>> 
>>>> This allows us to release software without the fear that someone will
>> sue
>>>> us personally if they are harmed by it, because if - as PMC members -
>> we
>>>> follow ASF rules (minimum 3 PMC members, reproducibility check,
>>> signatures,
>>>> etc.) ASF indemnifies us personally from any harm done to anyone using
>>> that
>>>> released software.
>>>> 
>>>> But all the other decisions in Airflow are voted by committers:
>>>> https://github.com/apache/airflow?tab=readme-ov-file#voting-policy -
>> so
>>>> committer votes are binding (except releases).
>>>> 
>>>> J.
>>>> 
>>>> On Thu, May 9, 2024 at 6:46 AM Jarek Potiuk  wrote:
>>>> 
>>>>> +1 (binding)
>>>>> 
>>>>> On Thu, May 9, 2024 at 4:47 AM Wei Lee  wrote:
>>>>> 
>>>>>> +1 non-binding
>>>>>> 
>>>>>> Best,
>>>>>> Wei
>>>>>> 
>>>>>>> On May 9, 2024, at 10:39 AM, Phani Kumar <
>> phani.ku...@astronomer.io
>>>> .INVALID>
>>>>>> wrote:
>>>>>>> 
>>>>>>> +1 binding, looking forward to add Scarf
>>>>>>> 
>>>>>>> On Thu, May 9, 2024 at 7:42 AM Kaxil Naik 
>>>> wrote:
>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Discussion thread:
>>>>>>>> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m
>>>>>>>> 
>>>>>>>> I would like to officially call for a vote to add Scarf as a
>>>> Telemetry
>>>>>>>> tool. Some other things:
>>>>>>>> 
>>>>>>>>  - Opt-in by default
>>>>>>>>  - Explicit documentation that we collect the telemetry data
>>>>>>>>  - Opt-out via airflow.cfg and env var
>>>>>>>>  - Works for air-gapped environments
>>>>>>>>  - Initial access to only PMC members
>>>>>>>> 
>>>>>>>> I have created a free account on Scarf and added it to the shared
>>>>>> 1password
>>>>>>>> (only PMC members have access to it). For now, I am just playing
>>>> around
>>>>>>>> with how the information can be shown.
>>>>>>>> 
>>>>>>>> I have a draft PR: https://github.com/apache/airflow/pull/39510
>>> that
>>>>>>>> collects some basic info, adds docs & tests.
>>>>>>>> 
>>>>>>>> Looking forward to releasing this for Airflow 2.10.
>>>>>>>> 
>>>>>>>> Consider this my +1 binding vote.
>>>>>>>> 
>>>>>>>> The vote will last until 04:20 GMT/UTC on May 16, 2024, and until
>>> at
>>>>>>>> least 3 binding votes have been cast.
>>>>>>>> 
>>>>>>>> Please vote accordingly:
>>>>>>>> 
>>>>>>>> [ ] + 1 approve
>>>>>>>> [ ] + 0 no opinion
>>>>>>>> [ ] - 1 disapprove with the reason
>>>>>>>> 
>>>>>>>> Only votes from PMC members and committers are binding, but other
>>>>>> members
>>>>>>>> of the community are encouraged to check the AIP and vote with
>>>>>>>> "(non-binding)".
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Kaxil
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>> -
>>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org
>>>>>> 
>>>>>> 
>>>> 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Proposal for adding Telemetry via Scarf

[Wei Lee]

+1 non-binding

Best,
Wei

> On May 9, 2024, at 10:39 AM, Phani Kumar  
> wrote:
> 
> +1 binding, looking forward to add Scarf
> 
> On Thu, May 9, 2024 at 7:42 AM Kaxil Naik  wrote:
> 
>> Hi all,
>> 
>> Discussion thread:
>> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m
>> 
>> I would like to officially call for a vote to add Scarf as a Telemetry
>> tool. Some other things:
>> 
>>   - Opt-in by default
>>   - Explicit documentation that we collect the telemetry data
>>   - Opt-out via airflow.cfg and env var
>>   - Works for air-gapped environments
>>   - Initial access to only PMC members
>> 
>> I have created a free account on Scarf and added it to the shared 1password
>> (only PMC members have access to it). For now, I am just playing around
>> with how the information can be shown.
>> 
>> I have a draft PR: https://github.com/apache/airflow/pull/39510 that
>> collects some basic info, adds docs & tests.
>> 
>> Looking forward to releasing this for Airflow 2.10.
>> 
>> Consider this my +1 binding vote.
>> 
>> The vote will last until 04:20 GMT/UTC on May 16, 2024, and until at
>> least 3 binding votes have been cast.
>> 
>> Please vote accordingly:
>> 
>> [ ] + 1 approve
>> [ ] + 0 no opinion
>> [ ] - 1 disapprove with the reason
>> 
>> Only votes from PMC members and committers are binding, but other members
>> of the community are encouraged to check the AIP and vote with
>> "(non-binding)".
>> 
>> Regards,
>> Kaxil
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on May 01, 2024

[Wei Lee]

Concur with Pankaj Koti, -1 (non-binding) for google provider

Best,
Wei

> On May 6, 2024, at 2:24 PM, Pankaj Koti  
> wrote:
> 
> -1 (non-binding) for the Google RC based on comment
> https://github.com/apache/airflow/issues/36090#issuecomment-2094972855
> risking some PRs included in the Google RC.
> 
> 
> Best regards,
> 
> *Pankaj Koti*
> Senior Software Engineer (Airflow OSS Engineering team)
> Location: Pune, Maharashtra, India
> Timezone: Indian Standard Time (IST)
> 
> 
> On Sun, May 5, 2024 at 12:56 PM rom sharon  wrote:
> 
>> +1 (non-binding)
>> 
>> ‫בתאריך שבת, 4 במאי 2024 ב-22:43 מאת ‪Kaxil Naik‬‏ <‪kaxiln...@gmail.com
>> ‬‏>:‬
>> 
>>> +1 binding for all except Pinecone one that Ankit is solving.
>>> 
>>> 
>>> 
>>> On Sat, 4 May 2024 at 18:02, Andrey Anshin 
>>> wrote:
>>> 
>>>> +1 (binding)
>>>> 
>>>> Check licences, signatures, checksums, and changes
>>>> 
>>>> 
>>>> On Fri, 3 May 2024 at 15:58, Pankaj Koti >>> .invalid>
>>>> wrote:
>>>> 
>>>>> +1 (non-binding)
>>>>> 
>>>>> Tested my set of changes.
>>>>> 
>>>>> Best regards,
>>>>> 
>>>>> *Pankaj Koti*
>>>>> Senior Software Engineer (Airflow OSS Engineering team)
>>>>> Location: Pune, Maharashtra, India
>>>>> Timezone: Indian Standard Time (IST)
>>>>> 
>>>>> 
>>>>> On Fri, May 3, 2024 at 4:25 PM Hussein Awala 
>> wrote:
>>>>> 
>>>>>> +1 (binding) checked licences, checksums, signatures and sources,
>> and
>>>> ran
>>>>>> some testing dags for cncf.kuberenetes and amazon providers.
>>>>>> 
>>>>>> On Fri, May 3, 2024 at 9:29 AM Wei Lee 
>> wrote:
>>>>>> 
>>>>>>> +1 (non-binding)
>>>>>>> 
>>>>>>> Best,
>>>>>>> Wei
>>>>>>> 
>>>>>>>> On May 3, 2024, at 2:53 PM, Pankaj Singh <
>>> ags.pankaj1...@gmail.com
>>>>> 
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> +1 (non-binding)
>>>>>>>> 
>>>>>>>> On Thu, May 2, 2024 at 6:03 PM Elad Kalif 
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I will exclude pinecone from this release.
>>>>>>>>> Please continue voting excluding pinecone provider
>>>>>>>>> 
>>>>>>>>> On Thu, May 2, 2024 at 3:19 PM Ankit Chaurasia <
>>>> sunank...@gmail.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> -1 non-binding for Pinecone: 2.0.0rc1
>>>>>>>>>> 
>>>>>>>>>> There is an issue with Pinecone: 2.0.0rc1 with the system
>> test.
>>>>> This
>>>>>>> bug
>>>>>>>>> is
>>>>>>>>>> introduced as part of the following PR Pinecone provider
>>> support
>>>>> for
>>>>>>>>>> pinecone-client>=3 (#37307): @rawwar
>>>>>>>>>> <https://github.com/apache/airflow/pull/37307>
>>>>>>>>>> 
>>>>>>>>>> PR #39365  <https://github.com/apache/airflow/pull/39365
>>>> should
>>>>> fix
>>>>>>> it.
>>>>>>>>>> 
>>>>>>>>>> *Ankit Chaurasia*
>>>>>>>>>> HomePage <https://ankitchaurasia.info/> |  LinkedIn
>>>>>>>>>> <https://www.linkedin.com/in/sunank200/>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Thu, May 2, 2024 at 10:47 AM Amogh Desai <
>>>>>> amoghdesai@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> +1 non binding
>>>>>>>>>>> 
>>>>>>>>&

Re: [DISCUSS] simplifying try_number handling

[Wei Lee]

Thanks, Daniel! +1 for this one. This was confusing when I worked on the 
starting from triggerer stuff. 

Best,
Wei


> On May 3, 2024, at 11:59 AM, Amogh Desai  wrote:
> 
> Looks good to me.
> 
> Personally I never ran into any issues with this so far but I agree with
> the issues it solves.
> Thanks & Regards,
> Amogh Desai
> 
> 
> On Fri, May 3, 2024 at 2:50 AM Vincent Beck  wrote:
> 
>> I am all +1 on this one. This thing gave me headaches when working on
>> AIP-44 and I could not understand the difference between the private
>> "_try_number" and the public "try_number". Thanks for simplifying it!
>> 
>> This is obviously assuming it does not break anything I am not aware of :)
>> 
>> On 2024/05/02 19:37:32 Daniel Standish wrote:
>>> TLDR
>>> * changing handling of try_number in
>>> https://github.com/apache/airflow/pull/39336
>>> * no more private attr
>>> * no more getter that changes value based on state of task
>>> * no more decrementing
>>> * try number now only handled by scheduler
>>> * hope that sounds good to all of you
>>> 
>>> For more detail read on...
>>> 
>>> In https://github.com/apache/airflow/pull/39336 I am doing some work to
>>> resolve some longstanding pain and frustration caused by try_number.
>>> 
>>> The way we handle try_number has for quite some time been messy and
>>> problematic.
>>> 
>>> For example, if you access `ti.try_number` and then change the state to
>> or
>>> from RUNNING, you will get a different value if you access it again!
>>> 
>>> And the responsibility for managing this number has been distributed
>>> throughout the codebase.  For example the task itself always increments
>>> when it starts running.  But then if it defers or reschedules itself, it
>>> decrements it back down so that when it runs again and naively
>> increments,
>>> then it will be right again.
>>> 
>>> Recently more issues have become visible as I have worked with AIP-44
>>> because for example pydantic does not like private attrs and it's just
>>> awkward to know *what value to use* when serializing it when the TI will
>>> give you a different answer depending on the state of the task!
>>> 
>>> And there's yet another edge case being solved in this community PR
>>> .
>>> And then when we start looking at try history and AIP-64, it also
>> forces a
>>> look at this.
>>> 
>>> So it all sounds bad and indeed it is bad but I think I have a solution.
>>> 
>>> What I do is, just have the scheduler increment try_number at the moment
>>> when it schedules the task.  It alone will have the responsibility for
>>> incrementing try_number.  And no where will it ever be decremented.  It
>>> will not be incremented when resuming after deferral or reschedule.  And
>>> that's about all there is to it.
>>> 
>>> I've tested it out and it works.  But I'm working through many test
>>> failures that need to be resolved (there's lots of asserts re
>> try_number).
>>> 
>>> One small thing I just want to point out is that if a user were
>> previously
>>> to be doing `task.run()` sort of manually without the task having been
>>> scheduled by the scheduler, well now their try_number won't be
>>> automatically incremented.  Same if they just do `airflow tasks run` --
>>> because now the responsibility is going to be solely with the scheduler.
>>> But airflow was never designed to assume that tasks will be run without
>>> having been scheduled, so I do not think that counts as a breaking
>> change.
>>> So I don't think that's a blocker for this.
>>> 
>>> Thanks for the consideration.  Let me know if you have any concerns.
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> For additional commands, e-mail: dev-h...@airflow.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on May 01, 2024

[Wei Lee]

+1 (non-binding)

Best,
Wei

> On May 3, 2024, at 2:53 PM, Pankaj Singh  wrote:
> 
> +1 (non-binding)
> 
> On Thu, May 2, 2024 at 6:03 PM Elad Kalif  wrote:
> 
>> I will exclude pinecone from this release.
>> Please continue voting excluding pinecone provider
>> 
>> On Thu, May 2, 2024 at 3:19 PM Ankit Chaurasia 
>> wrote:
>> 
>>> -1 non-binding for Pinecone: 2.0.0rc1
>>> 
>>> There is an issue with Pinecone: 2.0.0rc1 with the system test. This bug
>> is
>>> introduced as part of the following PR Pinecone provider support for
>>> pinecone-client>=3 (#37307): @rawwar
>>> 
>>> 
>>> PR #39365  should fix it.
>>> 
>>> *Ankit Chaurasia*
>>> HomePage  |  LinkedIn
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Thu, May 2, 2024 at 10:47 AM Amogh Desai 
>>> wrote:
>>> 
 +1 non binding
 
 Installed the tarball and ran some example DAGs for hive and cncf
>>> provider,
 works as expected.
 
 Thanks & Regards,
 Amogh Desai
 
 
 On Wed, May 1, 2024 at 10:49 PM Vincent Beck 
>>> wrote:
 
> +1 non binding. All AWS system tests are working successfully against
> apache-airflow-providers-amazon==8.21.0rc1. You can see the results
>>> here:
> 
 
>>> 
>> https://aws-mwaa.github.io/#/open-source/system-tests/version/fe4605a10e26f1b8a180979ba5765d1cb7fb0111_8.21.0rc1.html
 .
> The only failure (example_bedrock) is a known issue in the test
>> itself
 and
> is currently being worked on.
> 
> On 2024/05/01 13:03:07 Elad Kalif wrote:
>> Hey all,
>> 
>> I have just cut the new wave Airflow Providers packages. This email
>>> is
>> calling a vote on the release,
>> which will last for 72 hours - which means that it will end on May
>>> 04,
> 2024
>> 13:00 PM UTC and until 3 binding +1 votes have been received.
>> 
>> 
>> Consider this my (binding) +1.
>> 
>> *Note some of the providers are rc2 and some rc1.*
>> 
>> Airflow Providers are available at:
>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>> 
>> *apache-airflow-providers--*.tar.gz* are the binary
>> Python "sdist" release - they are also official "sources" for the
> provider
>> packages.
>> 
>> *apache_airflow_providers_-*.whl are the binary
>> Python "wheel" release.
>> 
>> The test procedure for PMC members is described in
>> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for and Contributors who would like to test this
>>> RC
 is
>> described in:
>> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>> 
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> Only votes from PMC members are binding, but members of the
>> community
 are
>> encouraged to test the release and vote with "(non-binding)".
>> 
>> Please note that the version number excludes the 'rcX' string.
>> This will allow us to rename the artifact without modifying
>> the artifact checksums when we actually release.
>> 
>> The status of testing the providers by the community is kept here:
>> https://github.com/apache/airflow/issues/39346
>> 
>> The issue is also the easiest way to see important PRs included in
>>> the
 RC
>> candidates.
>> Detailed changelog for the providers will be published in the
> documentation
>> after the
>> RC candidates are released.
>> 
>> You can find the RC packages in PyPI following these links:
>> 
>> 
>> https://pypi.org/project/apache-airflow-providers-airbyte/3.8.0rc1/
>> 
>> https://pypi.org/project/apache-airflow-providers-alibaba/2.8.0rc1/
>> 
>> https://pypi.org/project/apache-airflow-providers-amazon/8.21.0rc1/
>> 
 
>> https://pypi.org/project/apache-airflow-providers-apache-beam/5.7.0rc1/
>> 
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-apache-cassandra/3.5.0rc1/
>> 
 
>> https://pypi.org/project/apache-airflow-providers-apache-drill/2.7.0rc1/
>> 
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-apache-druid/3.10.0rc1/
>> 
 
>> https://pypi.org/project/apache-airflow-providers-apache-flink/1.4.0rc1/
>> 
 
>> https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.4.0rc2/
>> 
 
>> https://pypi.org/project/apache-airflow-providers-apache-hive/8.1.0rc1/
>> 
> 
 
>>> 
>>

Re: [VOTE] April 2024 PR of the Month

[Wei Lee]

Thanks Kaxil for nominating me and thank you to everyone for your support 🙏

Best,
Wei

> On May 2, 2024, at 12:06 AM, Ankit Chaurasia  wrote:
> 
> My vote goes to #38674 .
> Awesome work Wei!
> 
> Ankit Chaurasia
> 
> 
> 
> 
> 
> 
> 
> *Ankit Chaurasia*
> HomePage  |  LinkedIn
>  |  +91-9987351649
> 
> 
> 
> 
> 
> 
> On Wed, May 1, 2024 at 9:37 PM Amogh Desai  wrote:
> 
>> My vote goes to #38674. Awesome work
>> 
>> Thanks & Regards,
>> Amogh Desai
>> 
>> 
>> On Wed, 1 May 2024 at 9:21 PM, Jed Cunningham 
>> wrote:
>> 
>>> I'll also vote for 38674 - cool stuff!
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on April 16, 2024

[Wei Lee]

+1 (non-binding)

I tested my change and ran example DAGs with cncf, databircks providers RCs, 
and it worked fine.

Best,
Wei

> On Apr 16, 2024, at 8:40 PM, Elad Kalif  wrote:
> 
> Hey all,
> 
> I have just cut the new wave Airflow Providers packages. This email is
> calling a vote on the release,
> which will last for 72 hours - which means that it will end on April 19,
> 2024 12:40 PM UTC and until 3 binding +1 votes have been received.
> 
> 
> Consider this my (binding) +1.
> 
> Airflow Providers are available at:
> https://dist.apache.org/repos/dist/dev/airflow/providers/
> 
> *apache-airflow-providers--*.tar.gz* are the binary
> Python "sdist" release - they are also official "sources" for the provider
> packages.
> 
> *apache_airflow_providers_-*.whl are the binary
> Python "wheel" release.
> 
> The test procedure for PMC members is described in
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> 
> The test procedure for and Contributors who would like to test this RC is
> described in:
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
> 
> 
> Public keys are available at:
> https://dist.apache.org/repos/dist/release/airflow/KEYS
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> Only votes from PMC members are binding, but members of the community are
> encouraged to test the release and vote with "(non-binding)".
> 
> Please note that the version number excludes the 'rcX' string.
> This will allow us to rename the artifact without modifying
> the artifact checksums when we actually release.
> 
> The status of testing the providers by the community is kept here:
> https://github.com/apache/airflow/issues/39063
> 
> The issue is also the easiest way to see important PRs included in the RC
> candidates.
> Detailed changelog for the providers will be published in the documentation
> after the
> RC candidates are released.
> 
> You can find the RC packages in PyPI following these links:
> 
> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.1.1rc1/
> https://pypi.org/project/apache-airflow-providers-databricks/6.3.0rc3/
> https://pypi.org/project/apache-airflow-providers-fab/1.0.4rc1/
> 
> Cheers,
> Elad Kalif


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [PROPOSAL] Keep >= LATEST MINOR for all providers using common.sql provider

[Wei Lee]

I feel like this is the responsibility of the provider contributor 🤔 Maybe we 
could check whether the PR contains common.sql usage and raise a warning?

Best,
Wei

> On Apr 12, 2024, at 12:31 AM, Andrey Anshin  wrote:
> 
> There are some drawbacks I saw here, it would force to upgrade other
> providers to the latest version.
> Some scenarios from the my head:
> 
> End users use amazon provider and google provider, and do not use
> common.sql and both of them have mandatory common.sql as dependency.
> if everything works fine there is no problem, but it could be a situation
> that a new release of one of the providers is major or contains bugs and
> there is no possible way to downgrade one of them and keep a new version of
> another without breaking dependencies.
> 
> Other case if we need to exclude common.sql provider from provider release
> wave than we also need to exclude all providers which depends on common.sql
> even if it problem not affected directly, e.g. it is not a sql only
> provider, e.g. amazon, google, microsoft.azure
> 
> If it is not a big deal I have no objections to adding this because I do
> not have a better solution which could be implemented "Here and Now".
> 
> It could be resolved if we might run tests against released versions of
> common.sql, but after some brief investigation to run tests
> against installed versions and not a main version this might require
> changing quite a few things, which might take a time.
> 
> 
> 
> 
> On Thu, 11 Apr 2024 at 12:41, Jarek Potiuk  > wrote:
> 
>> Hello here,
>> 
>> I have a proposal to add a general policy that all our providers depending
>> on common.sql provider always use >= LATEST_MNOR version of the provider.
>> 
>> For example, following the change here
>> https://github.com/apache/airflow/pull/38707  by David we will update all
>> sql providers to have: airflow-providers-common-sql >= 1.12. We could of
>> course automate that with pre-commit so that we do not have to remember
>> about it.
>> 
>> Why ?
>> 
>> Because it's very easy by a provider to accidentally use a new feature in
>> common.sql without bumping the version. Current situation is like in the
>> image attached (thanks David), but we cannot be certain that the
>> "min-versions" there are "good"..
>> 
>> A bit more context:
>> 
>> Generally speaking - common.sql **should** be backwards compatible - like
>> - always. We should not make any changes in it's API (which is for-now
>> captured here:
>> https://github.com/apache/airflow/blob/main/airflow/providers/common/sql/hooks/sql.pyi
>> ). And from time to time we add new things to common.sql that providers
>> might start using.
>> 
>> Example:
>> 
>> From the https://lists.apache.org/thread/lzcpllwcgo7pc8g18l3b905kn8f9k4w8
>> thread the new "suppress_and_warn"  method is going to be added.
>> 
>> Currently we have no good mechanism to verify if the min version in
>> provider dependencies is really "good". For example when we add
>> `suppress_and_warn` today and someone starts using  `suppress_and_warn` in
>> the "presto" (for example) provider tomorrow, without bumping common-sql to
>>> = 1.12 - it will fail with 1.11 or earlier installed.
>> 
>> On the other hand - all the tests we run in `main` use the "latest"
>> version of the `common.sql` and all the constraints we produce also use the
>> latest version released, so we can be rather sure that the new providers
>> actually work well with the latest `common.sql` version. If there will be a
>> bug about compatibility (happened in the past), then it should be fixed by
>> fixing it in a new patchlevel of the `common.sql`.
>> 
>> So in-general it should be "safe" to add >= LATEST MINOR of common.sql in
>> all providers.
>> 
>> Should we do (and automate) it ? Any other comments / proposals maybe ?
>> 
>> Here is the current state of dependencies:
>> 
>> 
>> [image: image.png]

Re: [VOTE] Airflow Providers prepared on April 10, 2024

[Wei Lee]

+1 (non-binding)

Verified my changes and tested with a few DAGs.

Best,
Wei

> On Apr 13, 2024, at 3:51 PM, Utkarsh Sharma 
>  wrote:
> 
> +1 (non-binding)
> test a few example dags.
> 
> Thanks,
> Utkarsh Sharma
> 
> On Sat, Apr 13, 2024 at 11:46 AM Aritra Basu 
> wrote:
> 
>> +1 (non-binding)
>> tested against some sample dags
>> 
>> --
>> Regards,
>> Aritra Basu
>> 
>> On Sat, Apr 13, 2024, 4:21 AM Hussein Awala  wrote:
>> 
>>> +1 (binding) checked licences, checksums, signatures and sources, tested
>> my
>>> changes and run some testing dags for cncf.kuberenetes and amazon
>>> providers.
>>> 
>>> On Friday, April 12, 2024, Rahul Vats  wrote:
>>> 
 +1 (non-binding)
 Verified running our example DAGS
 
 Regards,
 Rahul Vats
 9953794332
 
 
 On Fri, 12 Apr 2024 at 12:35, Amogh Desai 
 wrote:
 
> +1 non binding
> 
> Installed and RC with breeze and tested my changes:
> - Setting use_beeline by default for hive cli connection (#38763)
> - Removing deprecated code in hive provider (#38859)
> - Adding support to hive hook for high availability Hive
>> installations
> (#38651)
> - Fix TRY002 for apache hive provider (#38781)
> - Add ssl context for verification of certs in FTPS hook (#38266)
> 
> All these changes are working as expected.
> 
> Thanks & Regards,
> Amogh Desai
> 
> 
> On Thu, Apr 11, 2024 at 10:53 PM Elad Kalif 
>>> wrote:
> 
>> databricks and yandex providers are excluded from rc1.
>> Please continue testing/voting without considering these two
>>> providers.
>> 
>> On Thu, Apr 11, 2024 at 3:25 PM Pankaj Koti
>>  wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> Verified my changes and our provider's example DAGs are running
>>> fine
>>> on the RCs.
>>> 
>>> Best regards,
>>> 
>>> *Pankaj Koti*
>>> Senior Software Engineer (Airflow OSS Engineering team)
>>> Location: Pune, Maharashtra, India
>>> Timezone: Indian Standard Time (IST)
>>> 
>>> 
>>> On Thu, Apr 11, 2024 at 3:01 PM Jarek Potiuk 
 wrote:
>>> 
 +1 (binding). Checked all my changes, including the celery
>>> provider
> bug
 that prevented 3.6.1 from running on Airflow 2.7.* (celery
>>> 3.6.2rc1
> now
 works just fine for Airflow 2.7). Checked reproducibility,
 licences,
 signatures, checksums. All good.
 
 On Wed, Apr 10, 2024 at 8:11 PM Vincent Beck <
>>> vincb...@apache.org>
>>> wrote:
 
> +1 non binding. All AWS system tests are running successfully
> against
> apache-airflow-providers-amazon==8.20.0rc1 with the exception
>>> of
> example_bedrock that is failing due to a bug in the test
>> itself
> (fix
 here:
> https://github.com/apache/airflow/pull/38887). You can see
>> the
>> results
> here:
> 
 
>>> 
>> 
> https://aws-mwaa.github.io/#/open-source/system-tests/version/
 3d804351aa7a875dfdba824c2b27300cc5ce9e92_8.20.0rc1.html
> 
> On 2024/04/10 16:37:12 Elad Kalif wrote:
>> Hey all,
>> 
>> I have just cut the new wave Airflow Providers packages.
>> This
> email
>>> is
>> calling a vote on the release,
>> which will last for 72 hours - which means that it will end
>>> on
>> April
 13,
>> 2024 16:35 PM UTC and until 3 binding +1 votes have been
> received.
>> 
>> 
>> Consider this my (binding) +1.
>> 
>> Airflow Providers are available at:
>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>> 
>> *apache-airflow-providers--*.tar.gz* are the
>> binary
>> Python "sdist" release - they are also official "sources"
>>> for
> the
> provider
>> packages.
>> 
>> *apache_airflow_providers_-*.whl are the binary
>> Python "wheel" release.
>> 
>> The test procedure for PMC members is described in
>> 
> 
 
>>> 
>> 
> https://github.com/apache/airflow/blob/main/dev/README_
 
>> RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for and Contributors who would like to
>>> test
> this
>>> RC
 is
>> described in:
>> 
> 
 
>>> 
>> 
> https://github.com/apache/airflow/blob/main/dev/README_
 
>> RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>> 
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> Only votes from PMC members are b

Re: [ANNOUNCE] New committer: Wei Lee

[Wei Lee]

Thanks for all your support 🙏 Can’t be more excited 🤩

Best,
Wei

> On Apr 8, 2024, at 9:15 PM, Vincent Beck  wrote:
> 
> Congrats Wei! Well deserved!
> 
> On 2024/04/08 13:03:50 Hemkumar Chheda wrote:
>> Congratulations Wei! Best news ever 🤩🥳
>> 
>>> On 8 Apr 2024, at 6:10 PM, Bishundeo, Rajeshwar 
>>>  wrote:
>>> 
>>> Congratulations Wei!! Good job and well deserved!!
>>> 
>>> -- Rajesh 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 2024-04-08, 8:36 AM, "Abhishek Bhakat" 
>>> >> <mailto:abhishek.bha...@astronomer.io.inva>LID> wrote:
>>> 
>>> 
>>> CAUTION: This email originated from outside of the organization. Do not 
>>> click links or open attachments unless you can confirm the sender and know 
>>> the content is safe.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. 
>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez 
>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que 
>>> le contenu ne présente aucun risque.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Yay! Congrats Wei!
>>> 
>>> 
>>> On Mon, Apr 8, 2024 at 11:36 AM Aritra Basu >> <mailto:aritrabasu1...@gmail.com>>
>>> wrote:
>>> 
>>> 
>>>> Congrats wei! Great job!
>>>> 
>>>> --
>>>> Regards,
>>>> Aritra Basu
>>>> 
>>>> On Mon, Apr 8, 2024, 4:17 PM >>> <mailto:consta...@astronomer.io.inva>lid> wrote:
>>>> 
>>>>> Congrats Wei!
>>>>> 
>>>>>> On Apr 8, 2024, at 5:31 AM, Pankaj Singh >>>>> <mailto:ags.pankaj1...@gmail.com>>
>>>>> wrote:
>>>>>> 
>>>>>> Congrats Wei, very well deserved!
>>>>>> 
>>>>>>> On Mon, Apr 8, 2024 at 2:33 PM Rahul Vats >>>>>> <mailto:rah.sharm...@gmail.com>>
>>>>> wrote:
>>>>>>> 
>>>>>>> Congrats Wei, very well deserved!
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Rahul Vats
>>>>>>> 
>>>>>>> 
>>>>>>>> On Mon, 8 Apr 2024 at 14:27, Amogh Desai >>>>>>> <mailto:amoghdesai@gmail.com>>
>>>>> wrote:
>>>>>>>> 
>>>>>>>> Many congratulations, Wei!
>>>>>>>> 
>>>>>>>> Welcome onboard!
>>>>>>>> 
>>>>>>>> Thanks & Regards,
>>>>>>>> Amogh Desai
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Mon, 8 Apr 2024 at 2:26 PM, Jarek Potiuk >>>>>>>> <mailto:ja...@potiuk.com>>
>>>> wrote:
>>>>>>>> 
>>>>>>>>> Congrats Wei! Indeed, well deserved!
>>>>>>>>> 
>>>>>>>>> On Mon, Apr 8, 2024 at 10:54 AM Hussein Awala >>>>>>>> <mailto:huss...@awala.fr>>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Congrats Wei, well deserved!
>>>>>>>>>> 
>>>>>>>>>> On Monday, April 8, 2024, Ash Berlin-Taylor >>>>>>>>> <mailto:a...@apache.org>>
>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> The Project Management Committee (PMC) for Apache Airflow has
>>>>>>> invited
>>>>>>>>> Wei
>>>>>>>>>>> Lee to become a committer and
>>>>>>>>>>> we are pleased to announce that they have accepted.
>>>>>>>>>>> 
>>>>>>>>>>> Wei has been contributing for a number of months he also
>>>>>>>> participated a
>>>>>>>>>>> lot in discussions and decisions
>>>>>>>>>>> on many aspects of Airflow but also helps a lot our users and
>>>>>>>>>> contributors
>>>>>>>>>>> on Slack, Github, Discuss

Re: [DISCUSS] Consider disabling self-hosted runners for commiter PRs

[Wei Lee]

+1 for this. I do not yet have enough chance to experience many job failures, 
but it won’t harm us to test them out. Plus, it saves some of the cost.

Best,
Wei

> On Apr 5, 2024, at 11:36 PM, Jarek Potiuk  wrote:
> 
> Seeing no big "no's" - I will prepare and run the experiment - starting
> some time next week, after we get 2.9.0 out - I do not want to break
> anything there. In the meantime, preparatory PR to add "use self-hosted
> runners" label is out https://github.com/apache/airflow/pull/38779
> 
> On Fri, Apr 5, 2024 at 4:21 PM Bishundeo, Rajeshwar
>  wrote:
> 
>> +1 with trying this out. I agree with keeping the canary builds
>> self-hosted in order to validate the usage for the PRs.
>> 
>> -- Rajesh
>> 
>> 
>> From: Jarek Potiuk 
>> Reply-To: "dev@airflow.apache.org" 
>> Date: Friday, April 5, 2024 at 8:36 AM
>> To: "dev@airflow.apache.org" 
>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Consider disabling
>> self-hosted runners for commiter PRs
>> 
>> 
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>> 
>> 
>> 
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>> 
>> 
>> Yeah. Valid concerns Hussein.
>> 
>> And I am happy to share some more information on that. I did not want to
>> put all of that in the original email, but I see that might be interesting
>> for you and possibly others.
>> 
>> I am closely following the numbers now. One of the reasons I am doing /
>> proposing it now is that finally (after almost 3 years of waiting) we
>> finally have access to some metrics that we can check. As of last week I
>> got access to the ASF metrics (
>> https://issues.apache.org/jira/browse/INFRA-25662).
>> 
>> I have access to "organisation" level information. Infra does not want to
>> open it to everyone - even to every member -  but since I got very active
>> and been helping with a number I got the access granted as an exception.
>> Also I saw a small dashboard the INFRA prepares to open to everyone once
>> they sort the access where we will be able to see the "per-project" usage.
>> 
>> Some stats that I can share (they asked not to share too much).
>> 
>> From what I looked at I can tell that we are right now (the whole ASF
>> organisation) safely below the total capacity. With a large margin - enough
>> to handle spikes, but of course the growth of usage is there and if
>> uncontrolled - we can again reach the same situation that triggered getting
>> self-hosted runners a few years ago.
>> 
>> Luckily - INRA gets it under control this time |(and metrics will help).
>> In the last INFRA newsletter, they announced some limitations that will
>> apply to the projects (effective as of end of April) - so once those will
>> be followed, we should be "safe" from being impacted by others (i.e.
>> noisy-neighbour effect). Some of the projects (not Airflow (!) ) were
>> exceeding those so far and they will be capped - they will need to optimize
>> their builds eventually.
>> 
>> Those are the rules:
>> 
>> * All workflows MUST have a job concurrency level less than or equal to
>> 20. This means a workflow cannot have more than 20 jobs running at the same
>> time across all matrices.
>> * All workflows SHOULD have a job concurrency level less than or equal to
>> 15. Just because 20 is the max, doesn't mean you should strive for 20.
>> * The average number of minutes a project uses per calendar week MUST NOT
>> exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200
>> hours).
>> * The average number of minutes a project uses in any consecutive five-day
>> period MUST NOT exceed the equivalent of 30 full-time runners (216,000
>> minutes, or 3,600 hours).
>> * Projects whose builds consistently cross the maximum use limits will
>> lose their access to GitHub Actions until they fix their build
>> configurations.
>> 
>> Those numbers on their own do not tell much, but we can easily see what
>> they mean when we put them side-by-side t with "our" current numbers.
>> 
>> * Currently - with all the "public" usage we are at 8 full-time runners.
>> This is after some of the changes I've done, With the recent changes I
>> already moved a lot of the non-essential build components that do not
>> require a lot of parallelism to public runners.
>> * The 20/15 jobs limit is a bit artificial (not really enforceable on
>> workflow level) - but in our case as I optimized most PR to run just a
>> subset of the tests, The average will be way below that - no matter if you
>> are committer or not, regular PRs are far smaller subset of the jobs than
>> full "canary" build. And for canary builds we should stay - at least for
>> now - with self-hosted runners

Re: [DISCUSS] Proposal for adding Telemetry via Scarf

[Wei Lee]

+1 to this. It would be really useful. As long as we can opt out, I think we’re 
good.

Best,
Wei

> On Mar 31, 2024, at 12:47 AM, Kaxil Naik  wrote:
> 
> Grammar Correction:
> 
> We should assume that those who deploy and upgrade Airflow - actually read
>> and take into account what is written in the release notes - especially if
>> they have security guys breathing their necks, similarly as we have to
>> assume they follow CVE announcements about security issues fixed. If we
>> are very straightforward and out-going about the change, inform very
>> clearly how to opt-out, I don't see a big problem with opt-out.
> 
> 
> I couldn't agree more; even though we shouldn't collect any data that
> hamper security (and we should aim to do the same), most security concerned
> folks don't just upgrade, and we can rely on them regarding release notes
> or announcements and we can make it very clear in our announcements too;
> and in our installation guides.
> 
> On Sat, 30 Mar 2024 at 16:47, Kaxil Naik  wrote:
> 
>> Grammar crrection:
>> 
>> 
>> On Sat, 30 Mar 2024 at 16:43, Kaxil Naik  wrote:
>> 
>>> Have this at the end of the email too: but if folks don't read until the
>>> end and quoting Maxime from the use-case blog[1]:
>>> 
>>> "I think people often ask ‘how do I contribute to open source?’, ‘I've
>>> got to get into the code’, or ‘ I’ve got to be an engineer.’ Actually, the
>>> very simplest thing that you can do is just say, ‘my organization gets real
>>> value from this piece of software.’ There are a bunch of ways to let the
>>> people know about it – and now Scarf is there. If your organization is
>>> getting a lot of value from a piece of open source software, make sure the
>>> devs know about it."
>>> 
>>> What kind of edge cases are you thinking about? I don't think it makes
>>> sense to have "opt-in" at all. As the goal is to collect data for most
>>> Airflow installations except for those that don't want to give data, then
>>> "opt-out" is the only way to maximize it. As long as we don't collect any
>>> PII data, this is in-compliance as well.
>>> 
>>> Imagine someone learning Airflow, if they have to opt-in via a config,
>>> they wouldn't even know or care about it, hence us losing most of the data.
>>> I understand why some orgs & individuals may want to opt-out.
>>> 
>>> Scarf Provides tracking pixels (essentially an HTML image tag) that you
>>> can place in your website or product to track visitors to that URL. If
>>> there were any concerns about Privacy, ASF wouldn't have approved it at all.
>>> 
>>> A few key details to note about the pixel:
>>> 
>>> 
>>>   - No PII is tracked… Scarf does not capture/retain IP information…
>>>   this information is discarded by the platform upon processing/aggregating
>>>   - Scarf pixels respect the Do Not Track (DNT) settings of browsers -
>>>   these users will not be tracked whatsoever.
>>> 
>>> 
>>> All the ASF projects I had listed (whether they use Scarf gateway or
>>> Scarf pixel in product) are using opt-out.
>>> 
>>> 1. Short opt-in period before opt-out. Test this feature with users who
 trust and if it works great - make it public. I think it's wise to handle
 edge cases and configure collected data more accurately.
>>> 
>>> 
>>> 
>>> It would be a pixel in the webserver, should affect nothing at all even
>>> in an air-gapped environment.
>>> 
 2. It should not affect anything if access to the internet is restricted
 which is default for many companies.
>>> 
>>> 
>>> 
>>> 100% agreed on the below:
>>> 
 I think we have a very good blueprint to follow including at least 5
 other
 ASF projects that also passed the review of the privacy@asf. And while I
 understand (and concur) the urge for opt-in by default coming from
 consumer
 market (where it makes perfect sense) Airflow is not a consumer
 software and is used in "corporate environment" which has a little
 different expectations and broad assumption that the company can make
 decisions on such telemetry on behalf of the employees using it.
>>> 
>>> 
>>> Couldn't agree more; even though there shouldn't we collect hamper
>>> security (and we should aim to do the same), most security concerned folks
>>> don't just
>>> upgrade, and we can rely on them regarding release notes or announcements
>>> and we can make it very clear in our announcements too; and in our
>>> installation guides.
>>> 
>>> We should assume that those who deploy and upgrade Airflow - actually read
 and take into account what is written in the release notes - especially
 if
 they have security guys breathing their necks, similarly as we have to
 assume they follow CVE announcements about security issues fixed. If we
 are very straightforward and out-going about the change, inform very
 clearly how to opt-out, I don't see a big problem with opt-out.
>>> 
>>> 
>>> 
>>> To be clear, the collection of data, or at least the data we should
>>> gather here shoul

Re: [VOTE] AIP-64: Keep TaskInstance try history

[Wei Lee]

+1 (non-binding), looking forward to it!

Best,
Wei

> On Mar 26, 2024, at 6:17 AM, Jarek Potiuk  wrote:
> 
> +1 (binding)
> 
> On Mon, Mar 25, 2024 at 10:51 PM Scheffler Jens (XC-AS/EAE-ADA-T)
>  wrote:
> 
>> +1 binding
>> 
>> Sent from Outlook for iOS
>> 
>> From: Pierre Jeambrun 
>> Sent: Monday, March 25, 2024 9:19:36 PM
>> To: dev@airflow.apache.org 
>> Subject: Re: [VOTE] AIP-64: Keep TaskInstance try history
>> 
>> +1 (binding)
>> 
>> On Mon 25 Mar 2024 at 19:32, Igor Kholopov 
>> wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> On Mon, Mar 25, 2024 at 7:28 PM Tzu-ping Chung >> 
>>> wrote:
>>> 
 +1 binding.
 
 This is something we should just do even outside of the DAG versioning
 context (which we also should do, to be clear).
 
 TP
 
 
> On Mar 26, 2024, at 02:12, Aritra Basu 
>>> wrote:
> 
> +1 (non-binding)
> This is quite good, I'd thought in passing that it'd be useful to
>> have.
> 
> --
> Regards,
> Aritra Basu
> 
> On Mon, Mar 25, 2024, 10:46 PM Jed Cunningham <
>>> jedcunning...@apache.org>
> wrote:
> 
>> Hello Airflow Community,
>> 
>> I would like to start a vote on AIP-64: Keep TaskInstance try
>> history.
>> 
>> You can find the AIP here:
>> 
>> 
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-64%253A%2BKeep%2BTaskInstance%2Btry%2Bhistory&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cb9bc3ca48fe64e90cfa508dc4d08eed3%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469947955266031%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=kFyVySPHT9GnX0nPEWT4S9l3nBzulv30%2FL1Q56Uuisc%3D&reserved=0
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-64%3A+Keep+TaskInstance+try+history
>>> 
>> 
>> Discussion Thread:
>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fvvm43tfchyo92hmf40fqvmq0f5845bjr&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cb9bc3ca48fe64e90cfa508dc4d08eed3%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469947955274281%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=UeOEK7OfNbM4ka7GXziO4n6D9xdL7e7Zs%2Fd55MH5QZM%3D&reserved=0
>> 
>> 
>> This is the first step in the AIP-63 DAG Versioning journey, though
>>> this
>> provides value in isolation as well:
>> 
>> 
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-63%253A%2BDAG%2BVersioning&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cb9bc3ca48fe64e90cfa508dc4d08eed3%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469947955279994%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=bubkyBIPWxDYqTRyCQbozLtVuw1bmK4KXkxnBM2bccA%3D&reserved=0
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63%3A+DAG+Versioning
>>> 
>> 
>> The vote will last until 2024-03-28 17:30 UTC and until at least 3
 binding
>> votes have been cast.
>> 
>> Consider this my binding +1.
>> 
>> Please vote accordingly:
>> 
>> [ ] + 1 approve
>> [ ] + 0 no opinion
>> [ ] - 1 disapprove with the reason
>> 
>> Only votes from PMC members and committers are binding, but other
 members
>> of the community are encouraged to check the AIP and vote with
>> "(non-binding)".
>> 
>> Thanks,
>> Jed
>> 
 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
 For additional commands, e-mail: dev-h...@airflow.apache.org
 
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] March 2024 PR of the Month

[Wei Lee]

+1 for #36755

Best,
Wei

> On Mar 26, 2024, at 5:50 AM, Scheffler Jens (XC-AS/EAE-ADA-T) 
>  wrote:
> 
> My vote is +1 for #36755 - mainly because it was really a long - runner and 
> important we made it!
> 
> Sent from Outlook for iOS
> 
> From: Briana Okyere  >
> Sent: Monday, March 25, 2024 8:53:32 PM
> To: dev@airflow.apache.org  
> mailto:dev@airflow.apache.org>>
> Subject: [VOTE] March 2024 PR of the Month
> 
> Hey All,
> 
> It’s once again time to vote for the PR of the Month!
> 
> With the help of the `get_important_pr_candidates` script in dev/stats,
> we've identified the following candidates:
> 
> PR #37937: refactor: Refactored __new__ magic method of BaseOperatorMeta
> to avoid bad mixing classic and decorated operators <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F37937&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0a80d271194c4cb01fc508dc4d055a9e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469932608262236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=PgitHoOv1s7k1lqG9oaqog4Ajo2gm7K9bdPlkITx5yE%3D&reserved=0>
> 
> PR #37821: Add `ADLSCreateObjectOperator` <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F37821&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0a80d271194c4cb01fc508dc4d055a9e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469932608272501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=qV6tNfyonXsa0R7j%2FNJExEWIfQWdc2NWmpzokCY27jA%3D&reserved=0>
> 
> PR #37458: Add Yandex Query support from Yandex.Cloud <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F37458&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0a80d271194c4cb01fc508dc4d055a9e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469932608279527%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=vodH9CqvR%2FQeDvkUt0YgkuS9WRJKD7BIDojjcrZETTU%3D&reserved=0>
> 
> PR #36935: Adding ability to automatically set DAG to off after X times it
> failed sequentially
> 
>  
> >
> 
> PR #36755: Python3.12 
> 
>  
> >
> 
> Please reply to this thread with your selection or offer your own
> nominee(s).
> 
> Voting will close on Friday, March 29th at 10 AM PST. The winner(s) will be
> featured in the next issue of the Airflow newsletter.
> 
> Also, if there’s an article or event that you think should be included in
> this or a future issue of the newsletter, please drop me a line at <
> briana.oky...@astronomer.io>
> 
> --
> Briana Okyere
> Community Manager
> Astronomer

Re: [VOTE] Release Airflow 2.8.4 from 2.8.4rc1

[Wei Lee]

+1 (non-binding)
Ran a few example DAGs and worked fine.

Best,
Wei


> On Mar 25, 2024, at 3:30 PM, Amogh Desai  wrote:
> 
> +1 non binding
> 
> Verified my changes, and ran a few example DAGs. I see no regressions.
> 
> Thanks & Regards,
> Amogh Desai
> 
> 
> On Sat, Mar 23, 2024 at 8:11 PM Andrey Anshin 
> wrote:
> 
>> +1 binding
>> 
>> Checked files, licences, signatures and also my changes.
>> 
>> One small nit that I've found is that the key which is used for sign
>> releases is expired, I'm not sure if it should be considered as a
>> showstopper.
>> 
>> ❯ gpg --verify apache_airflow-2.8.4-py3-none-any.whl.asc
>> apache_airflow-2.8.4-py3-none-any.whl
>> gpg: Signature made Wed Mar 20 19:17:26 2024 +04
>> gpg:using RSA key E1A1E984F55B8F280BD9CBA20BB7163892A2E48E
>> gpg:issuer "jedcunning...@apache.org"
>> gpg: Good signature from "Jed Cunningham "
>> [ultimate]
>> gpg: Note: This key has expired!
>> Primary key fingerprint: A020 DD36 34F1 A4D1 26A1  C554 7774 A4E5 90CB 0351
>> Subkey fingerprint: E1A1 E984 F55B 8F28 0BD9  CBA2 0BB7 1638 92A2 E48E
>> 
>> ❯ gpg --list-options show-unusable-subkeys --list-keys
>> A020DD3634F1A4D126A1C5547774A4E590CB0351
>> pub   rsa4096 2022-01-05 [C]
>>  A020DD3634F1A4D126A1C5547774A4E590CB0351
>> uid   [ultimate] Jed Cunningham 
>> sub   rsa4096 2022-01-05 [A] [expired: 2024-01-05]
>> sub   rsa4096 2022-01-05 [E] [expired: 2024-01-05]
>> sub   rsa4096 2022-01-05 [S] [expired: 2024-01-05]
>> 
>> 
>> 
>> 
>> On Fri, 22 Mar 2024 at 15:24, Rahul Vats  wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> Verified running our regression DAGS and API tests. LGTM
>>> 
>>> Regards,
>>> Rahul Vats
>>> 9953794332
>>> 
>>> 
>>> On Fri, 22 Mar 2024 at 15:24, Hussein Awala  wrote:
>>> 
 +1 (binding) checked licences, checksums, signatures, and source codes,
>>> and
 ran some testing dags. All looks good.
 
 On Wed, Mar 20, 2024 at 7:54 PM Jarek Potiuk  wrote:
 
> +1 (binding) - tested / verified all changes I was involved (either
>> as
> fixer, bug introducer or both, particularly when both), verified
> reproducibility, licences, checksums, signatures, run a few DAGs -
>> all
> looks good.
> 
> On Wed, Mar 20, 2024 at 4:56 PM Jed Cunningham <
>>> jedcunning...@apache.org
> 
> wrote:
> 
>> Hey fellow Airflowers,
>> 
>> I have cut Airflow 2.8.4rc1. This email is calling a vote on the
 release,
>> which will last at least 72 hours, from Wednesday, March 20, 2024
>> at
 4:00
>> pm UTC
>> until Saturday, March 23, 2024 at 4:00 pm UTC, and until 3 binding
>> +1
> votes
>> have been received.
>> 
>> 
>> 
> 
 
>>> 
>> https://www.timeanddate.com/worldclock/fixedtime.html/?msg=8&iso=20240323T1600&p1=1440
>> 
>> Status of testing of the release is kept in
>> https://github.com/apache/airflow/issues/38334
>> 
>> Consider this my (binding) +1.
>> 
>> Airflow 2.8.4rc1 is available at:
>> https://dist.apache.org/repos/dist/dev/airflow/2.8.4rc1/
>> 
>> *apache-airflow-2.8.4-source.tar.gz* is a source release that comes
 with
>> INSTALL instructions.
>> *apache-airflow-2.8.4.tar.gz* is the binary Python "sdist" release.
>> *apache_airflow-2.8.4-py3-none-any.whl* is the binary Python wheel
> "binary"
>> release.
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> Only votes from PMC members are binding, but all members of the
 community
>> are encouraged to test the release and vote with "(non-binding)".
>> 
>> The test procedure for PMC members is described in:
>> 
>> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for and Contributors who would like to test this
>>> RC
 is
>> described in:
>> 
>> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#verify-the-release-candidate-by-contributors
>> 
>> 
>> Please note that the version number excludes the `rcX` string, so
>>> it's
> now
>> simply 2.8.4. This will allow us to rename the artifact without
 modifying
>> the artifact checksums when we actually release.
>> 
>> Release Notes:
>> https://github.com/apache/airflow/blob/2.8.4rc1/RELEASE_NOTES.rst
>> 
>> For information on what goes into a release please see:
>> 
>> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md
>> 
>> Changes since 2.8.3:
>> *Bugs*:
>> - Fix incorrect serialization of ``FixedTimezone`` (#38139)
>> - Fix excessive permission c

Re: [DISCUSS] Applying D105 rule for our codebase ("undocumented magic methods") ?

[Wei Lee]

+1 for removing this rule. If docstring is needed for some cases, we can still 
do that or add comments to PRs.

Best,
Wei

> On Mar 25, 2024, at 1:31 PM, Amogh Desai  wrote:
> 
> After reading the emails in this thread, I too think that this rule is not
> generally very useful, but we should have
> it for cases which are special (where we are overriding some magic method
> to do something different or more
> than its reserved meaning)
> 
> +1 for removal of the rule, with special cases being handled separately.
> 
> Thanks & Best Regards,
> Amogh Desai
> 
> On Wed, Mar 20, 2024 at 11:46 PM Oliveira, Niko 
> wrote:
> 
>> I'm -1 to enabling D105
>> 
>> 
>> I don't think it will lead to helpful documentation. I think for the rare
>> cases it is required it can left up to the developer or caught in PR review.
>> 
>> Cheers,
>> Niko
>> 
>> 
>> From: Vincent Beck 
>> Sent: Wednesday, March 20, 2024 5:51:43 AM
>> To: dev@airflow.apache.org
>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Applying D105 rule
>> for our codebase ("undocumented magic methods") ?
>> 
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>> 
>> 
>> 
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>> 
>> 
>> 
>> +1 for not enforcing as well. Let's leave to maintainers the flexibility
>> to chose whether a given method should be documented.
>> 
>> On 2024/03/20 08:28:51 Ash Berlin-Taylor wrote:
>>> I'm for not enforcing this rule - as others have said its very unlikely
>> to result in more useful docs for developers or end users.
>>> 
>>> -asg
>>> 
>>> On 20 March 2024 08:12:40 GMT, Andrey Anshin 
>> wrote:
 ±0 from my side
 
 Maybe we have to review all current methods which do not follow this
>> rule
 to find a really useful meaning, and do not enforce (disable it).
 So for avoid unnecessary changes we might close
 https://github.com/apache/airflow/issues/37523 and remove/mark
>> completed
 into the https://github.com/apache/airflow/issues/10742
 
 
 
 
 
 
 On Wed, 20 Mar 2024 at 10:41, Pankaj Koti > .invalid>
 wrote:
 
> +1 to what Aritra is saying.
> 
> 
> Best regards,
> 
> *Pankaj Koti*
> Senior Software Engineer (Airflow OSS Engineering team)
> Location: Pune, Maharashtra, India
> Timezone: Indian Standard Time (IST)
> Phone: +91 9730079985
> 
> 
> On Wed, Mar 20, 2024 at 12:05 PM Aritra Basu <
>> aritrabasu1...@gmail.com>
> wrote:
> 
>> I'm in general not a huge fan of documenting for the sake of
>> documenting,
>> so I'd be in agreement of not enforcing it via code but rather be
> enforced
>> by the reviewers in cases they believe certain methods need
>> documenting.
>> 
>> --
>> Regards,
>> Aritra Basu
>> 
>> On Wed, Mar 20, 2024, 9:39 AM Jarek Potiuk 
>> wrote:
>> 
>>> Hey here,
>>> 
>>> I wanted to quickly poll what people think about applying the
>>> https://docs.astral.sh/ruff/rules/undocumented-magic-method/
>> rule in
> our
>>> codebase. There are many uncontroversial rules - but that one is
> somewhat
>>> more controversial than others.
>>> 
>>> See
> https://github.com/apache/airflow/pull/37602#issuecomment-2001951402
>>> and
>>> 
>> 
> 
>> https://github.com/apache/airflow/pull/38277#pullrequestreview-1945745542
>>> for example
>>> 
>>> I think that even in the ruff example, but also in many cases
>> requiring
>> to
>>> document the methods will lead to rather useless documentation:
>>> 
>>> class Cat(Animal):
>>>def __str__(self) -> str:
>>>"""Return a string representation of the cat."""
>>>return f"Cat: {self.name}"
>>> 
>>> There is IMHO very little value in having such documentation. It
>> might
> be
>>> useful in some cases where we have a really good reason to add
>> such a
>> magic
>>> method and it is important to document it, but in many cases - the
>>> documentation will be just documenting what the magic method
>> already
>>> explains well (like the case above).
>>> 
>>> This actually reminds me the early days of java documentation
>> where
>> javadoc
>>> looks more or less like this:
>>> 
>>> "Paints the object"
>>> func paint()
>>> 
>>> "Repaints the object"
>>> fund repaint()
>>> 
>>> However - maybe I am wrong :). Maybe it's worth documenting those
> methods
>>> in bulk, even if in many cases it will not bring much value?
>>> 
>>> WDYT ? Should

Re: [VOTE] AIP-59 Performance testing framework

[Wei Lee]

+1 (non-binding)

btw I notice the content of "Date Created" is some kind of script instead
of a real date. not sure whether it's expected.

Best,
Wei

Aritra Basu  於 2024年3月13日 週三 上午10:32寫道：

> +1 (non-binding)
>
> --
> Regards,
> Aritra Basu
>
> On Wed, Mar 13, 2024, 7:27 PM Bishundeo, Rajeshwar
>  wrote:
>
> > +1 (non-binding)
> >
> > -- Rajesh
> >
> >
> >
> >
> >
> >
> > On 2024-03-13, 9:40 AM, "Vincent Beck"  > vincb...@apache.org>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > +1 (binding)
> >
> >
> > On 2024/03/13 10:26:31 "Scheffler Jens (XC-AS/EAE-ADA-T)" wrote:
> > > +1 (binding)
> > >
> > > Mit freundlichen Grüßen / Best regards
> > >
> > > Jens Scheffler
> > >
> > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> > > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
> > GERMANY | www.bosch.com
> > > Tel. +49 711 811-91508 | Mobil +49 160 90417410 |
> > jens.scheff...@de.bosch.com 
> > >
> > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> > > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> > > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> > Forschner,
> > > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
> > >
> > > -Original Message-
> > > From: Jarek Potiuk mailto:ja...@potiuk.com>>
> > > Sent: Mittwoch, 13. März 2024 10:05
> > > To: dev@airflow.apache.org 
> > > Subject: Re: [VOTE] AIP-59 Performance testing framework
> > >
> > > +1 (binding)
> > >
> > > On Wed, Mar 13, 2024 at 7:40 AM Bartosz Jankiewicz
> > mailto:bjankiew...@google.com.inva>lid>
> > wrote:
> > > >
> > > > Hi folks,
> > > >
> > > > The AIP for performance testing has been in review for quite some
> time
> > > > and I've included your feedback in the document.
> > > >
> > > > I'd like to call a vote, and if you agree I'd start a development of
> > > > the framework.
> > > >
> > > > The AIP can be found below:
> > > >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> > 
> > > > i.apache.org
> %2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-59%2BPerformance%2
> > > > Btests%2Bframework&data=05%7C02%7CJens.Scheffler%40de.bosch.com
> %7Cc981
> > > >
> 7bb7ec0f4a3e31f108dc433cbebc%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C
> > > >
> 0%7C638459175397057760%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
> > > >
> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=GNR0pQz
> > > > mSsFEI%2FHDZ7MbI%2FuViioLIV%2BTWp6LcBqWCY0%3D&reserved=0
> > > >
> > > > Please vote accordingly:
> > > >
> > > > [ ] + 1 approve
> > > > [ ] + 0 no opinion
> > > > [ ] - 1 disapprove with the reason
> > > >
> > > > Thank you!
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org  > dev-unsubscr...@airflow.apache.org>
> > > For additional commands, e-mail: dev-h...@airflow.apache.org  > dev-h...@airflow.apache.org>
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org  > dev-unsubscr...@airflow.apache.org>
> > > For additional commands, e-mail: dev-h...@airflow.apache.org  > dev-h...@airflow.apache.org>
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org  > dev-unsubscr...@airflow.apache.org>
> > For additional commands, e-mail: dev-h...@airflow.apache.org  > dev-h...@airflow.apache.org>
> >
> >
> >
> >
> >
> >
>

Re: [VOTE] January 2024 PR of the Month

[Wei Lee]

 +1 for #37101

Best,
Wei

Kenten Danas  於 2024年2月27日 週二 上午5:28寫道：

> +1 for #37101 - sooo many requests for this, the community will be very
> happy to have this implemented!
>
> On Mon, Feb 26, 2024 at 11:52 AM Jarek Potiuk  wrote:
>
> > +1 on #37101 -> I think all the changes that make the dataset feature
> more
> > powerful (and especially one that our users definitely asked for) is a
> > really cool way to make our users look at Airflow through the lens of
> > "modern and slick". This one is definitely taking us closer to this goal.
> >
> > On Mon, Feb 26, 2024 at 8:13 PM Briana Okyere
> >  wrote:
> >
> > > Hey All,
> > >
> > > It’s once again time to vote for the PR of the Month.
> > >
> > > With the help of the `get_important_pr_candidates` script in dev/stats,
> > > we've identified the following candidates:
> > >
> > >  PR #37058: AIP-58: Add object storage backend for xcom <
> > > https://github.com/apache/airflow/pull/37058>
> > >
> > >  PR #37101: Introducing Logical Operators for dataset conditional
> logic <
> > > https://github.com/apache/airflow/pull/37101>
> > >
> > >  PR #37545: Refrain from passing `encoding` to the SQL engine in
> > SQLAlchemy
> > > v2 
> > >
> > >  PR #34225: Add extra operator links for EMR Serverless
> > > 
> > >
> > >  PR #37176: add "queuedEvent" endpoint to get/delete
> DatasetDagRunQueue <
> > > https://github.com/apache/airflow/pull/37176>
> > >
> > > Please reply to this thread with your selection or offer your own
> > > nominee(s).
> > >
> > > Voting will close on Thursday, February 29th at 12 PM PST. The
> winner(s)
> > > will be featured in the next issue of the Airflow newsletter.
> > >
> > > Also, if there’s an article or event that you think should be included
> in
> > > this or a future issue of the newsletter, please drop me a line at <
> > > briana.oky...@astronomer.io>
> > >
> > > --
> > > Briana Okyere
> > > Community Manager
> > > Astronomer
> > >
> >
>

Re: [VOTE] Airflow Providers prepared on February 19, 2024

[Wei Lee]

+1 (non-binding)

Best,
Wei

> On Feb 19, 2024, at 6:59 PM, Phani Kumar  
> wrote:
> 
> +1 non-binding.
> 
> On Mon, Feb 19, 2024 at 4:15 PM Pankaj Singh 
> wrote:
> 
>> +1 (non-binding)
>> 
>> Tested my changes. looking good. Thanks for the RC's and testing it.
>> 
>> On Mon, Feb 19, 2024 at 4:01 PM Pankaj Koti
>>  wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> Tested my latest PR in RC3 fixing task to be failed when pod times out
>>> to start https://github.com/apache/airflow/pull/37514 and works as
>>> expected.
>>> Tested other scenarios like successful run of KPO and works as expected.
>>> Also have tested the dependent GKEStartPodOperator successful and failure
>>> scenario where tasks should be marked as success/failed correspondingly
>>> and should not be stuck, works as expected.
>>> 
>>> 
>>> Best regards,
>>> 
>>> *Pankaj Koti*
>>> Senior Software Engineer (Airflow OSS Engineering team)
>>> Location: Pune, Maharashtra, India
>>> Timezone: Indian Standard Time (IST)
>>> Phone: +91 9730079985
>>> 
>>> 
>>> On Mon, Feb 19, 2024 at 3:05 PM Elad Kalif  wrote:
>>> 
 Hey all,
 
 I have just cut the new wave Airflow Providers packages. This email is
 calling a vote on the release,
 which will last for *24 hours* - which means that it will end on
>> February
 20, 2024 09:35 AM UTC and until 3 binding +1 votes have been received.
>>> Vote
 duration is according to a shortened voting period policy as accepted
>> in
 this thread
 .
 
 Consider this my (binding) +1.
 
 Airflow Providers are available at:
 https://dist.apache.org/repos/dist/dev/airflow/providers/
 
 *apache-airflow-providers--*.tar.gz* are the binary
 Python "sdist" release - they are also official "sources" for the
>>> provider
 packages.
 
 *apache_airflow_providers_-*.whl are the binary
 Python "wheel" release.
 
 The test procedure for PMC members is described in
 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
 
 The test procedure for and Contributors who would like to test this RC
>> is
 described in:
 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
 
 
 Public keys are available at:
 https://dist.apache.org/repos/dist/release/airflow/KEYS
 
 Please vote accordingly:
 
 [ ] +1 approve
 [ ] +0 no opinion
 [ ] -1 disapprove with the reason
 
 Only votes from PMC members are binding, but members of the community
>> are
 encouraged to test the release and vote with "(non-binding)".
 
 Please note that the version number excludes the 'rcX' string.
 This will allow us to rename the artifact without modifying
 the artifact checksums when we actually release.
 
 The status of testing the providers by the community is kept here:
 https://github.com/apache/airflow/issues/37534
 
 The issue is also the easiest way to see important PRs included in the
>> RC
 candidates.
 Detailed changelog for the providers will be published in the
>>> documentation
 after the
 RC candidates are released.
 
 You can find the RC packages in PyPI following these links:
 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.0.0rc3/
 
 Cheers,
 Elad Kalif
 
>>> 
>> 

Re: [VOTE] Airflow Providers prepared on February 17, 2024

[Wei Lee]

+1 (non-binding)

Tested both providers with our example DAGs without encountering an error. 
Verified #36330 
Best,

Wei


> On Feb 17, 2024, at 5:51 PM, ayaan rayan  wrote:
> 
> ‫بتاریخ ہفتہ، 17 فروری، 2024 بوقت 1:22 PM‏ ‪Elad Kalif‬‏
> <‪elad...@apache.org‬‏> نے لکھا:‬
>> 
>> Hey all,
>> 
>> I have just cut the RC2/RC3 wave Airflow Providers packages. This email is
>> calling a vote on the release,
>> which will last for *24 hours* - which means that it will end on February
>> 18, 2024 08:20 AM UTC and until 3 binding +1 votes have been received. Vote
>> duration is according to a shortened voting period policy as accepted in
>> this thread
>> .
>> 
>> 
>> Consider this my (binding) +1.
>> 
>> Airflow Providers are available at:
>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>> 
>> *apache-airflow-providers--*.tar.gz* are the binary
>> Python "sdist" release - they are also official "sources" for the provider
>> packages.
>> 
>> *apache_airflow_providers_-*.whl are the binary
>> Python "wheel" release.
>> 
>> The test procedure for PMC members is described in
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for and Contributors who would like to test this RC is
>> described in:
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>> 
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> Only votes from PMC members are binding, but members of the community are
>> encouraged to test the release and vote with "(non-binding)".
>> 
>> Please note that the version number excludes the 'rcX' string.
>> This will allow us to rename the artifact without modifying
>> the artifact checksums when we actually release.
>> 
>> The status of testing the providers by the community is kept here:
>> https://github.com/apache/airflow/issues/37504
>> 
>> The issue is also the easiest way to see important PRs included in the RC
>> candidates.
>> Detailed changelog for the providers will be published in the documentation
>> after the
>> RC candidates are released.
>> 
>> You can find the RC packages in PyPI following these links:
>> 
>> https://pypi.org/project/apache-airflow-providers-amazon/8.18.0rc2/
>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.0.0rc2/
>> https://pypi.org/project/apache-airflow-providers-common-sql/1.11.0rc3/
>> 
>> Cheers,
>> Elad Kalif
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
> 

Re: [VOTE] Airflow Providers prepared on February 12, 2024

[Wei Lee]

Hi,

I want to change my vote on Amazon to +1. The issue I mentioned is no longer 
reproducible.
Thanks!

Best,
Wei

> On Feb 15, 2024, at 1:19 AM, Pankaj Singh  wrote:
> 
> - 1 (non-binding) for cncf.kubernetes and common.sql since bug has been
> reported.
> 
> I have raised a PR for the CNCF provider at
> https://github.com/apache/airflow/pull/37363 to address the issues raised
> by @vchiapaikeo. The PR has been tested by @vchiapaikeo for his use case.
> 
> +1 (non-binding) for other providers
> 
> Thanks you
> Pankaj
> 
> On Wed, Feb 14, 2024 at 2:38 PM Pankaj Koti
>  wrote:
> 
>> -1 (non-binding) only for cncf.kubernetes: 8.0.0rc1 and common.sql:
>> 1.11.0rc2.
>> 
>> 1. cncf.kubernetes: Pankaj Singh (@pankajastro) is actively working on
>> resolving bugs reported in the resolution PR
>> https://github.com/apache/airflow/pull/37363.
>> He will update here and in the status of RC testing issue once the PR is
>> ready
>> for review. Hope the community can test then whether it fixes the issue.
>> 
>> 2. common.sql: Jarek has already fixed the issue real quick in PR
>> https://github.com/apache/airflow/pull/37359
>> 
>> 
>> +1 (non-binding) for all other providers including amazon: 8.18.0rc1.
>> I was unable to reproduce the issue that Wei Lee (@lee-w)  has mentioned
>> about https://github.com/apache/airflow/pull/36685. The DAG works as
>> expected.
>> Wei is on vacation, but if available sometime could you please retest and
>> change
>> your vote if you do not encounter an issue.
>> Also Vincent has given a +1 for the Amazon RC based on the Amazon system
>> tests.
>> 
>> 
>> Best regards,
>> 
>> *Pankaj Koti*
>> Senior Software Engineer (Airflow OSS Engineering team)
>> Location: Pune, Maharashtra, India
>> Timezone: Indian Standard Time (IST)
>> Phone: +91 9730079985
>> 
>> 
>> On Wed, Feb 14, 2024 at 1:03 PM Elad Kalif  wrote:
>> 
>>> *Update:* Providers amazon, cncf.kubernetes and common.sql will be
>> excluded
>>> from RC1.
>>> 
>>> 
>>> 
>>> On Tue, Feb 13, 2024 at 8:33 PM Vincent Beck 
>> wrote:
>>> 
>>>> +1 non binding. All AWS system tests are working fine against
>> 8.18.0rc1.
>>>> You can the results here:
>>>> 
>>> 
>> https://aws-mwaa.github.io/#/open-source/system-tests/version/2e1561015c44a0acfd86b63360fc17ad477a8d3b_8.18.0rc1.html
>>>> 
>>>> On 2024/02/13 05:08:17 Wei Lee wrote:
>>>>> -1 (non-binding) for amazon: 8.18.0rc1, cncf.kubernetes: 8.0.0rc1,
>>>> common.sql: 1.11.0rc2.
>>>>> 
>>>>> * Amazon: https://github.com/apache/airflow/pull/36685 doesn’t work
>> as
>>>> expected in the RC. It holds the DAG in the deferred state forever.
>>>>> * Kubernetes: issue mentioned in
>>>> https://github.com/apache/airflow/pull/37279 breaks
>>>> https://github.com/apache/airflow/pull/37306
>>>>> * common.sql: Encountered "ModuleNotFoundError: No module named
>>>> ‘more_itertools’”
>>>>> 
>>>>> +1 (non-binding) for the rest
>>>>> 
>>>>> Best,
>>>>> Wei
>>>>> 
>>>>>> On Feb 13, 2024, at 1:02 PM, Amogh Desai >> 
>>>> wrote:
>>>>>> 
>>>>>> - 0 (non binding) due to
>>> https://github.com/apache/airflow/pull/37279
>>>> for
>>>>>> the CNCF provider
>>>>>> 
>>>>>> +1 (non binding) for the rest
>>>>>> 
>>>>>> On Tue, Feb 13, 2024 at 5:34 AM Jarek Potiuk 
>>> wrote:
>>>>>> 
>>>>>>> +1 (binding), checked signatures, checksums, licences,
>>>> reproducibility.
>>>>>>> Checked all my changes are present in the packages.
>>>>>>> 
>>>>>>> -1 (common.sql missing more-itertools dependency - already merged
>> in
>>>> main).
>>>>>>> 
>>>>>>> On Mon, Feb 12, 2024 at 9:13 PM Hussein Awala 
>>>> wrote:
>>>>>>> 
>>>>>>>> I checked the source code, the licences, the signatures, and the
>>>>>>> checksums
>>>>>>>> and tested my changes; all look good.
>>>>>>>> 
>>>>>>>> -0 (binding) for cncf.kubernetes provider; #37279
>>>>>>>> <https://github.com/a

Re: [VOTE] Airflow Providers prepared on February 12, 2024

[Wei Lee]

-1 (non-binding) for amazon: 8.18.0rc1, cncf.kubernetes: 8.0.0rc1, common.sql: 
1.11.0rc2.

* Amazon: https://github.com/apache/airflow/pull/36685 doesn’t work as expected 
in the RC. It holds the DAG in the deferred state forever.
* Kubernetes: issue mentioned in https://github.com/apache/airflow/pull/37279 
breaks https://github.com/apache/airflow/pull/37306
* common.sql: Encountered "ModuleNotFoundError: No module named 
‘more_itertools’”

+1 (non-binding) for the rest

Best,
Wei

> On Feb 13, 2024, at 1:02 PM, Amogh Desai  wrote:
> 
> - 0 (non binding) due to https://github.com/apache/airflow/pull/37279 for
> the CNCF provider
> 
> +1 (non binding) for the rest
> 
> On Tue, Feb 13, 2024 at 5:34 AM Jarek Potiuk  wrote:
> 
>> +1 (binding), checked signatures, checksums, licences, reproducibility.
>> Checked all my changes are present in the packages.
>> 
>> -1 (common.sql missing more-itertools dependency - already merged in main).
>> 
>> On Mon, Feb 12, 2024 at 9:13 PM Hussein Awala  wrote:
>> 
>>> I checked the source code, the licences, the signatures, and the
>> checksums
>>> and tested my changes; all look good.
>>> 
>>> -0 (binding) for cncf.kubernetes provider; #37279
>>>  is too risky and needs
>> some
>>> testing, I will try to find some time to test all the cases (especially
>> the
>>> compatibility with GKE and EKS operators) and then change my vote to +1
>> or
>>> -1
>>> +1 (binding) for the rest
>>> 
>>> Hussein
>>> 
>>> On Mon, Feb 12, 2024 at 12:29 PM Elad Kalif  wrote:
>>> 
 Hey all,
 
 I have just cut the new wave Airflow Providers packages. This email is
 calling a vote on the release,
 which will last for 72 hours - which means that it will end on February
>>> 15,
 2024 11:30 AM UTC and until 3 binding +1 votes have been received.
 
 Consider this my (binding) +1.
 
 
 Airflow Providers are available at:
 https://dist.apache.org/repos/dist/dev/airflow/providers/
 
 *apache-airflow-providers--*.tar.gz* are the binary
 Python "sdist" release - they are also official "sources" for the
>>> provider
 packages.
 
 *apache_airflow_providers_-*.whl are the binary
 Python "wheel" release.
 
 The test procedure for PMC members is described in
 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
 
 The test procedure for and Contributors who would like to test this RC
>> is
 described in:
 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
 
 
 Public keys are available at:
 https://dist.apache.org/repos/dist/release/airflow/KEYS
 
 Please vote accordingly:
 
 [ ] +1 approve
 [ ] +0 no opinion
 [ ] -1 disapprove with the reason
 
 Only votes from PMC members are binding, but members of the community
>> are
 encouraged to test the release and vote with "(non-binding)".
 
 Please note that the version number excludes the 'rcX' string.
 This will allow us to rename the artifact without modifying
 the artifact checksums when we actually release.
 
 The status of testing the providers by the community is kept here:
 https://github.com/apache/airflow/issues/37358
 
 The issue is also the easiest way to see important PRs included in the
>> RC
 candidates.
 Detailed changelog for the providers will be published in the
>>> documentation
 after the
 RC candidates are released.
 
 You can find the RC packages in PyPI following these links:
 
 https://pypi.org/project/apache-airflow-providers-amazon/8.18.0rc1/
 
>> https://pypi.org/project/apache-airflow-providers-apache-beam/5.6.1rc1/
 
>> https://pypi.org/project/apache-airflow-providers-apache-drill/2.6.1rc1/
 
>> https://pypi.org/project/apache-airflow-providers-apache-druid/3.8.1rc1/
 
>> https://pypi.org/project/apache-airflow-providers-apache-hive/7.0.0rc1/
 
>> https://pypi.org/project/apache-airflow-providers-apache-livy/3.7.2rc1/
 https://pypi.org/project/apache-airflow-providers-apprise/1.2.2rc1/
 https://pypi.org/project/apache-airflow-providers-celery/3.6.0rc1/
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.0.0rc1/
 https://pypi.org/project/apache-airflow-providers-common-io/1.3.0rc1/
 
>> https://pypi.org/project/apache-airflow-providers-common-sql/1.11.0rc2/
 https://pypi.org/project/apache-airflow-providers-databricks/6.2.0rc1/
 https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.6.1rc1/
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-elasticsearch/5.3.3rc1/
 https://pypi.org/project/apache-airflow-providers-google/10.15.0rc1/
 https://pypi.org/project/apache-airflow-providers-hashicorp

Re: [VOTE] "Require conversation resolution" setting in PRs as permanent solution

[Wei Lee]

+0 (non-binding)

Best,
Wei

> On Feb 8, 2024, at 11:15 PM, Elad Kalif  wrote:
> 
> I'm -0 (binding)
> 
> 
> On Thu, Feb 8, 2024 at 10:51 AM Akash Sharma <2akash111...@gmail.com> wrote:
> 
>> +1 (non binding)
>> 
>> On Thu, 8 Feb 2024, 13:30 Jarek Potiuk,  wrote:
>> 
>>> Since this is a controversial topic, just a kind reminder - the voting
>> ends
>>> today in ~ 8 hrs. If you feel (even somewhat) strongly about either
>>> direction - that's the last time to cast the vote.
>>> 
>>> And yes - fractional votes are possible :) - see
>>> 
>>> 
>> https://www.apache.org/foundation/voting.html#expressing-votes-1-0-1-and-fractions
>>> For Jed's case it **might** mean:  "-0.5: 'I don't like this idea, but I
>>> can't find any rational justification for my feelings.'"
>>> 
>>> J.
>>> 
>>> On Wed, Feb 7, 2024 at 5:12 AM Amogh Desai 
>>> wrote:
>>> 
 Love the vote from @Jed Cunningham  here!
 
 On Wed, Feb 7, 2024 at 3:05 AM Jed Cunningham <
>> jedcunning...@apache.org>
 wrote:
 
> -0.5
> 
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Add the ability to report slack messages that don't meet code of conduct

[Wei Lee]

I believe that anonymity is important in this case. It would be great if we 
could have the bot suggested by Aritra, and option #2 is favorable. 
Alternatively, I would prefer option #1. Thanks!

Best regards,
Wei Lee

> On Feb 4, 2024, at 12:37 AM, Aritra Basu  wrote:
> 
> I'm very much for anonymity here, so if we could have some kind of bot
> that'd be great. Though even without that I think the combination of
> channel and form also works out fine for talking to a person vs wanting
> anonymity.
> 
> --
> Regards,
> Aritra Basu
> 
> On Sat, Feb 3, 2024, 9:12 PM Andrey Anshin  wrote:
> 
>> I’m also for some bot/application, after quick search I’ve found this one
>> https://github.com/slackapi/slack-reporting-tool
>> 
>> 
>> I’m not sure is it suits our requirements and it required instance where
>> this code would be deployed but it worthwhile to check it, especially if
>> someone familiar with Typescript
>> 
>>> On 3 Feb 2024, at 19:19, Amogh Desai  wrote:
>>> 
>>> I would also like to go with #2.
>>> 
>>> But just to bring up, how easy / hard is it to have a slack bot that
>> users
>>> can use to report
>>> bad users or spam users. Never done it before myself, but it would be a
>>> great interface to report issues.
>>> 
>>> The issues would then come to few admins who can take regulatory action.
>>> (Might be an overkill, but thinking out loud here)
>>> 
>>> Thanks & Regards,
>>> Amogh Desai
>>> 
>>>> On Sat, Feb 3, 2024 at 2:55 AM Briana Okyere
>>>>  wrote:
>>>> 
>>>> Hey All,
>>>> 
>>>> I'm breaking this out into another conversation because I think it
>> warrants
>>>> its own decision before we move forward.
>>>> 
>>>> Last week, I proposed we add a code of conduct for Airflow Slack and
>>>> in-person meetups.
>>>> Thread: <
>> https://lists.apache.org/thread/jy16tb09qoc7lxb03pdj3dfnmrsmjq87>
>>>> 
>>>> In the Code of Conduct I shared, there is a conversation taking place
>> about
>>>> HOW to report someone in Slack if their message does not abide by the
>> code
>>>> of conduct.
>>>> 
>>>> There is a general consensus that a committee should be formed that
>> people
>>>> can report code of conduct violations to.
>>>> 
>>>> Two options were proposed about the reporting process:
>>>> 1. Create a google form that people can fill out anonymously to report
>> code
>>>> of conduct violations
>>>> 2. Create a channel in slack called #slack-admins where people can post
>> and
>>>> report a violation.
>>>> 
>>>> I would like to know which option you all vote for. I vote for option
>> #2.
>>>> With the addition that we should have a #slack-admins channel that
>> links to
>>>> the google form for anonymous reporting, but also serves as a space
>> where
>>>> people can post questions about the code of conduct and receive clarity
>> and
>>>> answers. But I strongly feel there should be an option to anonymously
>>>> report, which I think a form would solve.
>>>> 
>>>> Curious what others think, or if you have additional ideas!
>>>> 
>>>> ---
>>>> Briana Okyere
>>>> Community Manager
>>>> Astronomer
>>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] AIP 61 - Hybrid Executors

[Wei Lee]

+1 (non-binding)

Looking forward to it!

Best,
Wei

> On Feb 1, 2024, at 7:57 PM, Aritra Basu  wrote:
> 
> +1 (non-binding)
> 
> This would definitely be a good value add. Looked through the proposal and
> it looks solid! Great job!
> 
> --
> Regards,
> Aritra Basu
> 
> On Thu, Feb 1, 2024, 12:58 PM Amogh Desai  wrote:
> 
>> +1 binding
>> 
>> Good work on the proposal, Niko.
>> 
>> The most important part of this vote for me is the scoping you have done.
>> Great work on that!
>> 
>> Thanks & Regards,
>> Amogh Desai
>> 
>> On Thu, Feb 1, 2024 at 12:12 PM Jarek Potiuk  wrote:
>> 
>>> +1 (binding) . I think we know the scope (more importantly we also know
>>> what's out of the scope)
>>> 
>>> On Thu, Feb 1, 2024 at 6:45 AM Scheffler Jens (XC-AS/EAE-ADA-T)
>>>  wrote:
>>> 
 +1 binding - looking forward to implementation!
 
 Sent from Outlook for iOS
 
 From: Oliveira, Niko 
 Sent: Thursday, February 1, 2024 6:12:02 AM
 To: dev@airflow.apache.org 
 Subject: [VOTE] AIP 61 - Hybrid Executors
 
 Hey folks,
 
 
 The AIP for Hybrid Executors has been out for a few weeks now. Some
>> great
 feedback came in and some challenges to scope which I think have all
>> been
 addressed, and the AIP document has been updated where applicable.
 
 
 At this point I'd like to call a vote, and if all goes well, begin
 development soon!
 
 
 You can find the AIP here:
 
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-61%2BHybrid%2BExecution&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0ee4ec946cc24c1a378408dc22e45aa4%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638423611370691730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=FAu5DPEq1%2FFmqdEKl06ZB%2F%2FfJ7ymgHgovKMvgzE8U2U%3D&reserved=0
 <
 
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-61+Hybrid+Execution
> 
 
 
 Discussion threads:
 
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2F94sg7l4m3qjk4b3vfq3lr94oc5fs9q4j&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0ee4ec946cc24c1a378408dc22e45aa4%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638423611370702702%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=xJYMlsb8u9pvDWuMwJP6If2ZIeva6XYmYa9%2Fac8007Q%3D&reserved=0
 
 
 The voting will last for 6 days (until 6th of February 2024, 22:00
>> PST),
 and until at least 3 binding votes have been cast.
 
 Please vote accordingly:
 
 [ ] + 1 approve
 [ ] + 0 no opinion
 [ ] - 1 disapprove with the reason
 
 Only votes from PMC members and committers are binding, but other
>> members
 of the community are encouraged to check the AIP and vote with
 "(non-binding)".
 
 Thanks!
 
 
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on January 30, 2024

[Wei Lee]

+1 (non-binding)

Tested my changes with our example DAG without encountering error

Best,
Wei

> On Jan 31, 2024, at 6:34 AM, Jarek Potiuk  wrote:
> 
> +1 (binding) - checked reproducibility, licences, signatures, checksums -
> all good.
> 
> On Tue, Jan 30, 2024 at 5:42 PM Elad Kalif  wrote:
> 
>> Hey all,
>> 
>> I have just cut an ad-hoc release for the microsoft.azure provider package.
>> This email is calling a vote on the release,
>> which will last for 72 hours - which means that it will end on February 02,
>> 2024 16:40 PM UTC and until 3 binding +1 votes have been received.
>> 
>> 
>> Consider this my (binding) +1.
>> 
>> Airflow Providers are available at:
>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>> 
>> *apache-airflow-providers--*.tar.gz* are the binary
>> Python "sdist" release - they are also official "sources" for the provider
>> packages.
>> 
>> *apache_airflow_providers_-*.whl are the binary
>> Python "wheel" release.
>> 
>> The test procedure for PMC members is described in
>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for and Contributors who would like to test this RC is
>> described in:
>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>> 
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> Only votes from PMC members are binding, but members of the community are
>> encouraged to test the release and vote with "(non-binding)".
>> 
>> Please note that the version number excludes the 'rcX' string.
>> This will allow us to rename the artifact without modifying
>> the artifact checksums when we actually release.
>> 
>> The status of testing the providers by the community is kept here:
>> https://github.com/apache/airflow/issues/37102
>> 
>> The issue is also the easiest way to see important PRs included in the RC
>> candidates.
>> Detailed changelog for the providers will be published in the documentation
>> after the
>> RC candidates are released.
>> 
>> You can find the RC packages in PyPI following these links:
>> 
>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/9.0.0rc1/
>> 
>> Cheers,
>> Elad Kalif
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [ANNOUNCE] Starting experimenting with "Require conversation resolution" setting

[Wei Lee]

I didn't notice much of a difference as a contributor. +1 vote

Best,
Wei

> On Jan 30, 2024, at 11:41 AM, Amogh Desai  wrote:
> 
> Contrary to my initial expectation of the trouble this would bring in for
> reviewers, it has been
> pretty nice. I have not faced any issues in marking the conversations as
> resolved for the pull
> requests I have reviewed and it has even given me a chance to re review
> prior to approval.
> 
> I am happy with this overall and my vote will be a +1
> 
> Thanks & Regards,
> Amogh Desai
> 
> On Mon, Jan 29, 2024 at 7:56 PM Aritra Basu 
> wrote:
> 
>> I personally haven't had too much friction due to the change and it has
>> helped me keep track of any comments people have made. I remain +1 to the
>> change so far.
>> 
>> --
>> Regards,
>> Aritra Basu
>> 
>> On Mon, Jan 29, 2024, 6:11 PM Jarek Potiuk  wrote:
>> 
>>> Just wanted to remind everyone, we are nearing the end of the trial
>> period
>>> for "require conversation" feature to be enabled. I have my own
>>> observations and examples, but since I was the one to propose it, I am
>>> likely biased, so I'd love to hear from others what their feedback and
>>> assessment is. Or maybe we need more time to assess it ?
>>> 
>>> I would love to hear your thoughts.
>>> 
>>> J,
>>> 
>>> 
>>> On Sat, Dec 30, 2023 at 2:20 PM Jarek Potiuk  wrote:
>>> 
 After an initial indentation problem in .asf.yaml it's not working as
 expected. So  let's see how resolving conversations will work for
>> us.
 
 On Sat, Dec 30, 2023 at 12:17 PM Amogh Desai >> 
 wrote:
 
> Wooho! Looking to see how this turns out for airflow 😃
> 
> On Sat, 30 Dec 2023 at 1:35 PM, Jarek Potiuk 
>> wrote:
> 
>> Hello everyone,
>> 
>> As discussed in
>> https://lists.apache.org/thread/cs6mcvpn2lk9w2p4oz43t20z3fg5nl7l I
>>> just
>> enabled "require conversation resolution" for our main/stable
>>> branches.
> We
>> have not used it in the past so it might not work as we think or we
> might
>> need to tweak something.
>> 
>> Generally speaking (if all works) all conversations on PRs should be
>> resolved before we can merge the PR. This "resolving" is encouraged
>> to
> be
>> done by the author when they think the conversation is resolved, but
>>> it
> can
>> also be done by reviewers or the maintainer who wants to merge the
>> PR.
>> 
>> We attempted to describe some basic rules and expectations here:
>> 
>> 
> 
>>> 
>> https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#step-5-pass-pr-review
>> but undoubtedly there will be questions and issues that we might
>> want
>>> to
>> solve - so feel free to discuss it here or raise question/issues in
>> #development channel in slack (I am also happy to be pinged directly
> about
>> it and help to resolve any issues/gather feedback).
>> 
>> J.
>> 
> 
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on January 22, 2024

[Wei Lee]

+1 non-binding

Tested my changes and example DAGs without encountering errors.

Best,
Wei

> On Jan 24, 2024, at 4:18 PM, rom sharon  wrote:
> 
> +1 non-binding
> 
> ‫בתאריך יום ד׳, 24 בינו׳ 2024 ב-10:17 מאת ‪Rahul Vats‬‏ <‪
> rah.sharm...@gmail.com‬‏>:‬
> 
>> +1 non-binding
>> 
>> Verified below providers with our example DAGS.
>> 
>>   -  apache-airflow-providers-amazon==8.17.0rc1
>>   -  apache-airflow-providers-apache-hive==6.5.0rc1
>>   -  apache-airflow-providers-common-sql==1.11.0rc1
>>   -  apache-airflow-providers-databricks==6.1.0rc1
>>   -  apache-airflow-providers-dbt-cloud==3.6.0rc1
>>   -  apache-airflow-providers-elasticsearch==5.3.1
>>   -  apache-airflow-providers-google==10.14.0rc1
>>   -  apache-airflow-providers-http==4.9.0rc1
>>   -  apache-airflow-providers-snowflake==5.3.0rc1
>>   -  apache-airflow-providers-ftp==3.8.0rc1
>>   -  apache-airflow-providers-mysql==5.5.2rc1
>>   -  apache-airflow-providers-cohere==1.2.0rc1
>>   -  apache-airflow-providers-pinecone==1.2.0rc1
>>   -  apache-airflow-providers-weaviate==1.3.1rc1
>> 
>> 
>> Regards,
>> Rahul Vats
>> 9953794332
>> 
>> 
>> On Wed, 24 Jan 2024 at 09:56, Amogh Desai 
>> wrote:
>> 
>>> +1 non binding
>>> 
>>> Tested some example dags in CNCF provider and Hive Provider. Looks good!
>>> 
>>> Thanks & Regards,
>>> Amogh Desai
>>> 
>>> On Tue, Jan 23, 2024 at 8:44 PM Jarek Potiuk  wrote:
>>> 
 +1 (binding) - tested my changes, checked binary reproducibility,
>>> licences,
 signatures, checksums - all looks good.
 
 On Mon, Jan 22, 2024 at 1:11 PM Elad Kalif  wrote:
 
> Hey all,
> 
> I have just cut the new wave Airflow Providers packages. This email
>> is
> calling a vote on the release,
> which will last for 72 hours - which means that it will end on
>> January
 25,
> 2024 12:10 PM UTC and until 3 binding +1 votes have been received.
> 
> Consider this my (binding) +1.
> 
> Airflow Providers are available at:
> https://dist.apache.org/repos/dist/dev/airflow/providers/
> 
> *apache-airflow-providers--*.tar.gz* are the binary
> Python "sdist" release - they are also official "sources" for the
 provider
> packages.
> 
> *apache_airflow_providers_-*.whl are the binary
> Python "wheel" release.
> 
> The test procedure for PMC members is described in
> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> 
> The test procedure for and Contributors who would like to test this
>> RC
>>> is
> described in:
> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
> 
> 
> Public keys are available at:
> https://dist.apache.org/repos/dist/release/airflow/KEYS
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> Only votes from PMC members are binding, but members of the community
>>> are
> encouraged to test the release and vote with "(non-binding)".
> 
> Please note that the version number excludes the 'rcX' string.
> This will allow us to rename the artifact without modifying
> the artifact checksums when we actually release.
> 
> The status of testing the providers by the community is kept here:
> https://github.com/apache/airflow/issues/36948
> 
> The issue is also the easiest way to see important PRs included in
>> the
>>> RC
> candidates.
> Detailed changelog for the providers will be published in the
 documentation
> after the
> RC candidates are released.
> 
> You can find the RC packages in PyPI following these links:
> 
> https://pypi.org/project/apache-airflow-providers-airbyte/3.6.0rc1/
> https://pypi.org/project/apache-airflow-providers-alibaba/2.7.2rc1/
> https://pypi.org/project/apache-airflow-providers-amazon/8.17.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-beam/5.6.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-druid/3.8.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.4.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-hive/6.5.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-kafka/1.4.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-kylin/3.6.0rc1/
> 
>> https://pypi.org/project/apache-airflow-providers-apache-pig/4.4.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-pinot/4.4.0rc1/
> 
>>> https://pypi.org/project/apache-airflow-providers-apache-spark/4.8.0rc1/
> https://pypi.org/project/apache-airflow-providers-apprise/1.3.0rc1/
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-atlassian-jira/2.6.0rc1/

Re: [VOTE] January 2024 PR of the Month

[Wei Lee]

My vote is for #36537. Love this packaging improvement!

Best,
Wei

> On Jan 23, 2024, at 10:30 PM, Ryan Hatter  
> wrote:
> 
> Gotta agree with Constance and go with 22253 -- how cool that the author
> stuck with it all this time!
> 
> On Tue, Jan 23, 2024 at 12:25 AM Aritra Basu 
> wrote:
> 
>> My vote is for #36537 it's been a huge effort and it makes huge
>> improvements in our packaging. Great to see it make it into airflow.
>> 
>> --
>> Regards,
>> Aritra Basu
>> 
>> On Tue, Jan 23, 2024, 10:13 AM Amogh Desai 
>> wrote:
>> 
>>> Is there a possibility to vote for more than one? I guess not :/
>>> 
>>> My vote goes to #36537 for the enhancements that have come in with it. I
>>> have followed the discussions
>>> at a higher level and it surely wasn't easy :)
>>> (If I could vote again, it would surely be #36537 for the endless
>>> perseverance and dedication of the author)
>>> 
>>> Thanks & Regards,
>>> Amogh Desai
>>> 
>>> On Tue, Jan 23, 2024 at 3:21 AM Jarek Potiuk  wrote:
>>> 
 Heck, why not. I will shamelessly vote on my #36537. While it took
>> just a
 few weeks to merge, It leapfrogged our legacy packaging setup to
 more-or-less bleeding edge from what was there since the beginning of
 Airflow (almost 10 years) and was already "old-ish" when I joined the
 project more than 4 years ago. And with hatch and cleanups in extras,
>> it
 has a positive impact on both - contributors and users (or so I hope).
 
 On Mon, Jan 22, 2024 at 7:48 PM Constance Martineau
  wrote:
 
> +1 #22253
> 
> The PR was opened in March 2022, and was finally merged last week! I
 admire
> the author's persistence in getting this merged in, and think the
> simplifications to the interface make the Operator more user-friendly
>>> for
> our Data Science users.
> 
> On Mon, Jan 22, 2024 at 1:29 PM Briana Okyere
>  wrote:
> 
>> Hey All,
>> 
>> It’s once again time to vote for the PR of the Month.
>> 
>> With the help of the `get_important_pr_candidates` script in
>>> dev/stats,
>> we've identified the following candidates:
>> 
>> PR #36513: Include plugins in the architecture diagrams.
>> 
>> 
>> PR #32867: Sanitize the conn_id to disallow potential script
 execution. <
>> https://github.com/apache/airflow/pull/32867>
>> 
>> PR #22253: Add SparkKubernetesOperator crd implementation.
>> 
>> 
>> PR #36171: Implement AthenaSQLHook.
>> 
>> 
>> PR #36537: Standardize airflow build process and switch to
>> Hatchling
> build
>> backend. 
>> 
>> Please reply to this thread with your selection or offer your own
>> nominee(s).
>> 
>> Voting will close on Jan. 26th at 1 PM PST. The winner(s) will be
> featured
>> in the next issue of the Airflow newsletter.
>> 
>> Also, if there’s an article or event that you think should be
>>> included
 in
>> this or a future issue of the newsletter, please drop me a line at
>> <
>> briana.oky...@astronomer.io>.
>> 
>> --
>> Briana Okyere
>> Community Manager
>> *Astronomer*
>> 
> 
> 
> --
> 
> Constance Martineau
> Senior Product Manager
> 
> Email: consta...@astronomer.io
> Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
> 
> 
> 
> 
 
>>> 
>> 

Re: [VOTE] Accept AIP-60 (Standard URI representation for Airflow Datasets)

[Wei Lee]

+1 (non-binding)

Best,
Wei

> On Jan 22, 2024, at 8:16 AM, Jarek Potiuk  wrote:
> 
> +1 (binding).
> 
> On Fri, Jan 19, 2024 at 12:38 AM Igor Kholopov 
> wrote:
> 
>> +1 (non-binding)
>> 
>> On Fri, Jan 19, 2024 at 12:03 AM Maciej Obuchowski >> 
>> wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> On Thu, Jan 18, 2024, 22:51 Scheffler Jens (XC-AS/EAE-ADA-T)
>>>  wrote:
>>> 
 +1 binding as discussed - looking forward for this and THANKS!
 
 Sent from Outlook for iOS
 
 From: Tzu-ping Chung 
 Sent: Thursday, January 18, 2024 9:07:42 AM
 To: dev@airflow.apache.org 
 Subject: [VOTE] Accept AIP-60 (Standard URI representation for Airflow
 Datasets)
 
 AIP page:
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-60%2BStandard%2BURI%2Brepresentation%2Bfor%2BAirflow%2BDatasets&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cfeb80ec5aa4b4fa99c7a08dc17fc9ae4%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638411620901637967%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZJ3XtYmB5k5NsO%2Ft%2F05QSxH9CIrYEiP4td09LZZ8rcI%3D&reserved=0
 <
 
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-60+Standard+URI+representation+for+Airflow+Datasets
> 
 Discussion thread:
 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Frf6c80ljjkml0l15h2jys7k713q3os1d&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cfeb80ec5aa4b4fa99c7a08dc17fc9ae4%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638411620901637967%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QQqR1gaRDfC9udbGbMubPfkyt73jSUmB7uPU%2BukCE2s%3D&reserved=0
 
 
 Reaction on the proposal seems to mostly positive, with most comments
 around what documentation should be added, and the exact criteria the
>> AIP
 should be considered “done”. I believe I have addressed most of them;
>>> most
 notably, additional sentences have been added to the What defines this
>>> AIP
 as "done"? section to require the best practice to be demonstrated by
 example DAGs and tutorials in the documentation.
 
 One comment I left unaddressed is about auto-generating documentation
>>> from
 providers. This is mostly because I’m not quite sure how it can be
 practical. We can generate a list of supported protocols (s3, gcs,
>> file,
 etc.), but that is not particularly useful to users without the actual
 format the URI would use. In the current implementation, each URI
>> handler
 is a simple Python function, and it is not viable to extract logic from
>>> it
 unless we adopt some kind of rule-based parser (like regex, and even
>> that
 is too complex to automatically generate documentation from). I am open
>>> to
 suggestions on this, so feel free to give a -1 with an idea on this.
 Otherwise I would move the proposal forward without auto documentation
 generation.
 
 This vote will be kept open for more than 72 hours even if three +1s
>> are
 reached, to gather potential ideas on the documentation thing mentioned
 above. I intend to start implementation (including the example DAGs) in
>>> the
 mean time.
 
 TP
 
>>> 
>> 

Re: [VOTE] New Airflow Community Provider: Teradata

[Wei Lee]

+1 non binding

Best,
Wei

> On Jan 17, 2024, at 2:54 AM, Josh Fell  
> wrote:
> 
> +1 (binding)
> 
> On Tue, Jan 16, 2024 at 1:53 PM Vincent Beck  wrote:
> 
>> +1 binding. Makes sense to me.
>> 
>> On 2024/01/16 18:21:39 Jarek Potiuk wrote:
>>> +1 binding
>>> 
>>> On Tue, Jan 16, 2024 at 6:20 PM Phani Kumar
>>>  wrote:
>>> 
 +1 non binding
 
 On Tue, Jan 16, 2024 at 10:43 PM K Mallam, Sunil
  wrote:
 
> Hello Airflow Community,
> 
> Thank you very much for your comments/feedback on the Discussion
>> Thread.
> 
> I’m creating this voting thread for Teradata to be Airflow’s new
 community
> provider.
> 
> We have one 1 binding vote from Kaxil and I request the other
>> community
> members to share their votes.
> 
> Below are initial implementation links -
> 
> Implementation:
> 
 
>> https://github.com/Teradata/airflow/tree/td_develop/airflow/providers/teradata
> 
> Documentation:
> 
 
>> https://github.com/Teradata/airflow/tree/td_develop/docs/apache-airflow-providers-teradata
> 
> Unit Tests:
> 
 
>> https://github.com/Teradata/airflow/tree/td_develop/tests/providers/teradata
> 
> System Tests:
> 
 
>> https://github.com/Teradata/airflow/tree/td_develop/tests/system/providers/teradata
> System Tests Dashboard: https://teradata.github.io/airflow/
> 
 
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> For additional commands, e-mail: dev-h...@airflow.apache.org
>> 
>> 

Re: [ANNOUNCE] New PMC member: Andrey Anshin (taragolis)

[Wei Lee]

Congrats Andrey! 🎉

Best,
Wei

> On Jan 16, 2024, at 4:06 AM, Bishundeo, Rajeshwar 
>  wrote:
> 
> Congrats Andrey!! Well deserved!!
> 
> -- Rajesh 
> 
> 
> 
> 
> 
> 
> On 2024-01-15, 3:01 PM, "Oliveira, Niko"  LID> wrote:
> 
> 
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you can confirm the sender and know the 
> content is safe.
> 
> 
> 
> 
> 
> 
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
> cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
> confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
> contenu ne présente aucun risque.
> 
> 
> 
> 
> 
> 
> Congrats Andrey!
> 
> 
> 
> From: Pankaj Singh  >
> Sent: Monday, January 15, 2024 11:11:01 AM
> To: dev@airflow.apache.org 
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [ANNOUNCE] New PMC member: Andrey 
> Anshin (taragolis)
> 
> 
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you can confirm the sender and know the 
> content is safe.
> 
> 
> 
> 
> 
> 
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
> cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
> confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
> contenu ne présente aucun risque.
> 
> 
> 
> 
> 
> 
> Congrats Andrey 🎉
> 
> 
> On Tue, Jan 16, 2024 at 12:30 AM Aritra Basu  >
> wrote:
> 
> 
>> Congrats Andrey!!
>> 
>> --
>> Regards,
>> Aritra Basu
>> 
>> On Tue, Jan 16, 2024, 12:24 AM Vincent Beck > > wrote:
>> 
>>> Congrats Andrey!
>>> 
>>> On 2024/01/15 18:46:32 ambika garg wrote:
 Congrats Andrey!!
 
 On Mon, Jan 15, 2024 at 1:44 PM Amogh Desai >>> >
 wrote:
 
> Congrats Andrey!!
> 
> On Tue, 16 Jan 2024 at 12:08 AM, Hussein Awala  >
>>> wrote:
> 
>> Congratulations Andrey, very well deserved!
>> 
>> On Mon 15 Jan 2024 at 19:35, Jarek Potiuk > >
>> wrote:
>> 
>>> I have the pleasure to announce that The Project Management
>>> Committee
>>> (PMC) for Apache Airflow has invited Andrey Anshin (taragolis) to
> become
>>> Apache
>>> Airflow PMC Member and we are pleased to announce that he has
>>> kindly
>>> accepted it.
>>> 
>>> Being a PMC member enables assistance with the management and to
>>> guide
>>> the direction of the project.
>>> 
>>> Congratulations Andrey, I'd say it's been very well deserved !.
>>> 
>>> Regards,
>>> Jarek on behalf of the Apache Airflow PMC
>>> 
>> 
> 
 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org 
>>> 
>>> For additional commands, e-mail: dev-h...@airflow.apache.org 
>>> 
>>> 
>>> 
>> 
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on January 07, 2024

[Wei Lee]

+1 (non-binding)

We've tested with the following providers with our example DAGs without 
encountering issues.

* apache-airflow-providers-amazon==8.16.0rc1
* apache-airflow-providers-apache-hive==6.4.1rc1
* apache-airflow-providers-apache-livy==3.7.1rc1

Best,
Wei


> On Jan 8, 2024, at 2:22 AM, Jarek Potiuk  wrote:
> 
> +1 (binding) - > checked reproducible builds, signatures, licences,
> checksums, all look good.
> 
> On Sun, Jan 7, 2024 at 1:11 PM Elad Kalif  wrote:
>> 
>> Hey all,
>> 
>> I have just cut the new wave Airflow Providers packages. This email is
>> calling a vote on the release,
>> which will last for 72 hours - which means that it will end on January 10,
>> 2024 12:10 PM UTC and until 3 binding +1 votes have been received.
>> 
>> Consider this my (binding) +1.
>> 
>> 
>> Airflow Providers are available at:
>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>> 
>> *apache-airflow-providers--*.tar.gz* are the binary
>> Python "sdist" release - they are also official "sources" for the provider
>> packages.
>> 
>> *apache_airflow_providers_-*.whl are the binary
>> Python "wheel" release.
>> 
>> The test procedure for PMC members is described in
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for and Contributors who would like to test this RC is
>> described in:
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>> 
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> Only votes from PMC members are binding, but members of the community are
>> encouraged to test the release and vote with "(non-binding)".
>> 
>> Please note that the version number excludes the 'rcX' string.
>> This will allow us to rename the artifact without modifying
>> the artifact checksums when we actually release.
>> 
>> The status of testing the providers by the community is kept here:
>> https://github.com/apache/airflow/issues/36644
>> 
>> The issue is also the easiest way to see important PRs included in the RC
>> candidates.
>> Detailed changelog for the providers will be published in the documentation
>> after the
>> RC candidates are released.
>> 
>> You can find the RC packages in PyPI following these links:
>> 
>> https://pypi.org/project/apache-airflow-providers-amazon/8.16.0rc1/
>> https://pypi.org/project/apache-airflow-providers-apache-hive/6.4.1rc1/
>> https://pypi.org/project/apache-airflow-providers-apache-livy/3.7.1rc1/
>> https://pypi.org/project/apache-airflow-providers-apache-spark/4.7.0rc1/
>> https://pypi.org/project/apache-airflow-providers-atlassian-jira/2.5.0rc1/
>> https://pypi.org/project/apache-airflow-providers-common-io/1.2.0rc1/
>> https://pypi.org/project/apache-airflow-providers-mysql/5.5.1rc1/
>> https://pypi.org/project/apache-airflow-providers-openlineage/1.4.0rc1/
>> https://pypi.org/project/apache-airflow-providers-opsgenie/5.5.0rc1/
>> https://pypi.org/project/apache-airflow-providers-pagerduty/3.6.0rc1/
>> https://pypi.org/project/apache-airflow-providers-redis/3.6.0rc1/
>> https://pypi.org/project/apache-airflow-providers-samba/4.5.0rc1/
>> 
>> Cheers,
>> Elad Kalif
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [ANNOUNCEMENT] Looking for potential new security team members

[Wei Lee]

Hi Jarek,

I’m also interested in learning security stuff, but I have no related 
experience. I'm not sure whether I fit the criteria.

Best,
Wei

> On Jan 6, 2024, at 12:56 PM, Amogh Desai  wrote:
> 
> Hi Jarek,
> 
> I have personally never done much on security, neither followed a lot of
> blogs/learning materials regarding
> it so far. I want to explore this untouched territory by myself, but I am
> not so sure if this will be the right forum for "experimentation".
> 
> So, not so sure if I cut the criteria here.
> 
> Thanks & Regards,
> Amogh Desai
> 
> On Sat, Jan 6, 2024 at 2:14 AM Jarek Potiuk  wrote:
> 
>> * we have to know the candidate  - they have to be either a committer
>> or someone who has contributed a lot and we know who the person is.
>> Stakeholders and community members that we know and can trust might
>> also nominate some people who have security experience and already
>> work on security (especially if they work on Airflow) outside of the
>> community - we know our stakeholders have dedicated security people
>> who have a good experience and they are not known to us simply due to
>> "secrecy" around security.
>> 
>> 
>> I'd say you both fulfill the criteria :)
>> 
>> On Fri, Jan 5, 2024 at 9:22 PM utkarsh sharma 
>> wrote:
>>> 
>>> Hey Jarek,
>>> 
>>> I'm very interested in security-related stuff, but not sure if I fit the
>>> bill. :)
>>> 
>>> Thanks,
>>> Utkarsh
>>> 
>>> On Sat, Jan 6, 2024 at 1:43 AM Ryan Hatter
>>>  wrote:
>>> 
 What are the criteria? Just curious, as I'm quite confident I do not
>> fit
 the criteria 😀
 
 On Fri, Jan 5, 2024 at 9:21 AM Jarek Potiuk  wrote:
 
> Hello everyone,
> 
> TL;DR; In short - we are looking for candidates to join our security
> team. Please send a message to priv...@airflow.apache.org if you
>> would
> like to be added to the team.
> 
> Following this:
> 
> 
 
>> https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#periodic-security-team-rotation
> I wanted to make a call for new security team members. Some of the
> people will rotate out the team as well (we want to keep the team
> small and lean and focused).
> 
> First of all I have a great pleasure - in the name of the community -
> to thank for all the work the current security team has accomplished.
> 
> When we started discussions at the beginning of last year we had ~ 20
> outstanding issues, some of them older than 6 months and the process
> of fixing them was not really cool. Today we have 0 (yes - 0)
> unhandled issues. And we had >50 issues raised since so we not only
> managed to fix the backlog but also we handled incoming issues. We
> have much better understanding on how to handle them, we've improved
> and clarified our security model, and we even have some standard ways
> on handling and responding to similar issues when they come. And we
> have learning material for new team members to take a look at.
> 
> What's going to happen now?
> 
> We want to partially rotate the team - first of all to give the
> experienced and recognized community members an opportunity to learn
> and participate in our security process, but also to distribute a bit
> more knowledge on handling security issues in the community.
> 
> I personally believe that security will become increasingly more
> important in the years to come - things like Cyber Resilience Act
> 
>> https://digital-strategy.ec.europa.eu/en/library/cyber-resilience-act
> will create a lot of opportunities to make use of the knowledge you
> can gain by becoming part of security team so I think it's also good
> to have an experience in it in a professional portflolio.
> 
> What does it mean to be in a security team?
> 
> * You will be subscribed to receive reports from security researchers
> 
> * You will take part in the discussions when we assess the issues -
> whether they are real issues, what severity they have, how we can
> address them
> 
> * You will take part in discussing on how we can improve current
> processes and even how to improve our security model  and whether we
> need to apply some systematic fixes
> 
> * You will possibly volunteer to fix or review, or talk to other
> community members to fix it  help with handling some of the security
> issues
> 
> * The traffic on our security list (after we got through the backlog)
> is moderate to small - there are maybe 1 new issue a week (usually
> less than one) and we have occasional discussions that might be more
> frequent
> 
> * For the new team members - we have learning materials to get to
> understand how things work - I will prepare some "on-boarding"
> packages.
> 
> * This is not a permanent "assignment" - as you see now we are doing
>> a
> partial rotation to get 

Re: [DISCUSSION] Enabling `pre-commit.ci` application for Airflow

[Wei Lee]

Same as Amogh. Even though I would like to fix that myself, it would make it 
much easier for those who aren’t familiar with these tools and still be able to 
contribute. But we might need to doc this behavior somewhere (GitHub PR issue 
might make more sense 🤔). Otherwise, the contributor might be surprised by the 
new commit.

Best,
Wei

> On Jan 3, 2024, at 12:21 AM, Vincent Beck  wrote:
> 
> I like the concept! +1
> 
> On 2023/12/30 11:16:35 Amogh Desai wrote:
>> I am aligning here with Pierre, but I am not against the idea of enabling
>> the pre commit ci application.
>> 
>> I’d rather have myself fix the issue as it sometimes also lets me have
>> second,third or multiple passes at my code which is up for review. This is
>> a personal choice where I feel that we are trying to fix a problem that is
>> not too problematic.
>> 
>> Again, only a personal choice but not against it. If it makes lives of the
>> stakeholders involved easier, I am all up for it!
>> 
>> Thanks & Regards,
>> Amogh Desai
>> 
>> On Sat, 30 Dec 2023 at 2:35 PM, Pierre Jeambrun 
>> wrote:
>> 
>>> I like the idea, but in practice auto fixable static checks are very
>>> obvious to fix and doesn’t require much work.
>>> 
>>> On the other hand most of static failure are ‘real issues’ and not auto
>>> fixable, for instance mypy, spelling, sphinx, db session usage etc…. (And
>>> ruff fix is a little aggressive IMO regarding linting).
>>> 
>>> So I would say that in practice it solves a painless problem (formatting,
>>> import sorting and other obvious things) and can’t do much about other
>>> issues.
>>> 
>>> This is why I am not sure it is worth the confusion for users. (But I am
>>> not against it)
>>> 
>>> On Sat 30 Dec 2023 at 09:19, Scheffler Jens (XC-DX/PJ-PACE-E03)
>>>  wrote:
>>> 
 I‘d also like to have auto-fixing included in CI. It is a classic pitfall
 and all that can be automated does not need to be a manual burden.
 Though I am not sure whether the plugin is able to use all the custom
 stuff as we also depend during execution on the CI image and docker.
 Besides security things this would be something that needs testing if it
 works.
 
 TLDR: +1 opinion
 
 Sent from Outlook for iOS
 
 From: Pankaj Koti 
 Sent: Saturday, December 30, 2023 7:50:10 AM
 To: dev@airflow.apache.org 
 Subject: Re: [DISCUSSION] Enabling `pre-commit.ci` application for
>>> Airflow
 
 I very much like the concept. We have been using it actively for
>>> Astronomer
 code repositories for 1+ year already and it has helped us greatly
>>> (Thanks
 to Felix Uellendall for introducing this back then 🙂)
 
 On Sat, 30 Dec 2023, 12:10 Jarek Potiuk,  wrote:
 
> FYI - Just now INFRA rejected the request on the basis of "code write"
> access permissions the app needs.
> 
> I'd still love to get feedback though on the concept -  I am not giving
 up
> that easily. We might still get it approved easily. We likely have some
> ways we can get "auto-fixing" working for us.
> 
> 1) I believe Github Applications now can use a bit different mechanism
>>> to
> ask for permissions and it can have "required" and "optional"
>>> permissions
> and I believe "Pull request write" should be enough (and I might
>>> attempt
 to
> convince the maintainers of it to adapt it to our needs).
> 2) Also, there is a "Pre-commit Lite Github Action" that we **might**
>>> be
> able to use to achieve a similar effect (with some added complexity to
 our
> `Pull Request Target` workflow.
> 
> So I would still love to hear from others :)
> 
> J.
> 
> On Fri, Dec 29, 2023 at 11:52 PM Jarek Potiuk 
>>> wrote:
> 
>> Hello everyone,
>> 
>> TL;DRl; I'd like to propose that we enable the pre-commit-ci GitHub
>> application for Airflow repo. According to how I understand it works,
 it
>> should greatly reduce friction (especially for new contributors) for
>> passing the quality gates for our pre-commits. That is - if we get
>>> the
>> approval for pre-commit-ci application from the ASF infra team.
>> 
>> Some more context:
>> 
>> We use and love (well some of us do, some of us likely hate) the
>> pre-commit as a quality gate for our checks. We have been using it
>>> for
>> years for local checks and CI integration and we have ~60 custom
> precommits
>> and in total we use about 100 pre-commit checks as our "quality"
>>> gates
>> 
>> However, using `standard` pre-commit (that is a de-facto standard in
>> Python world) has a nice property of 'standing on the shoulders of
> giants'.
>> There is one thing that few of us are aware of, that there is a way
>>> to
>> reduce friction for pre-commits that are not only flagging errors but
 can
>> also fix them. If we get the `pre-commit-

Re: [VOTE] December PR of the Month

[Wei Lee]

I want to vote for https://github.com/apache/airflow/pull/35926, and thank 
Jarek for bringing it up. https://github.com/apache/airflow/pull/35719 is also 
great, so it's a difficult choice.

Best,
Wei

> On Jan 3, 2024, at 6:04 AM, Jarek Potiuk  wrote:
> 
> I'd like to propose another one:
> https://github.com/apache/airflow/pull/35926 - Create FAB provider and move
> FAB auth manager in it
> 
> I think that one is something some of us have been talking about getting
> rid of FAB for a long time. While FAB was really cool when Airflow started,
> it has not aged well and showed some properties that made our life much
> harder and we shot ourselves in the foot by our decision to partially
> vendor it in. It resulted in pinning Airflow to a specific FAB version
> which causes a complex process of updating to newer versions and makes it
> generally difficult for security and dependency management. While moving
> FAB to the provider does not solve all of that, it's a step that makes it
> far easier to manage and release security fixes (they will be able to be
> fixed without bumping Airflow). And also opens up a way to future FAB-less
> Airflow installation.
> 
> It gets +1 from my side.
> 
> 
> J.
> 
> 
> On Tue, Jan 2, 2024 at 6:23 PM Kenten Danas 
> wrote:
> 
>> +1 for #35719
>> 
>> On Tue, Jan 2, 2024 at 8:59 AM Scheffler Jens (XC-AS/EAE-ADA-T)
>>  wrote:
>> 
>>> I‘d vote for #35719 - because I like external contributions especially to
>>> move ahead in UI!
>>> 
>>> Sent from Outlook for iOS
>>> 
>>> From: Briana Okyere 
>>> Sent: Tuesday, January 2, 2024 5:51:12 PM
>>> To: dev@airflow.apache.org 
>>> Subject: [VOTE] December PR of the Month
>>> 
>>> Happy New Year Everyone :).
>>> 
>>> It’s once again time to vote for the PR of the Month.
>>> 
>>> I decided to wait to release the vote until folks were back from their
>>> holidays so that it was not missed.
>>> 
>>> With the help of the `get_important_pr_candidates` script in dev/stats,
>>> we've identified the following candidates:
>>> 
>>> PR #36472: Review and update our Pull Request review process and policy.
>> <
>>> 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F36472&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Ca7d1a1618a1b49b697a708dc0bb3248c%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638398111234079685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=acRTMNpT%2BJ4wOZkVFDSy8IUtW8fW%2FJjjpIR6kwKaGaM%3D&reserved=0
 
>>> 
>>> PR #35970: New breeze command to clean up previous provider artifacts. <
>>> 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F35970&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Ca7d1a1618a1b49b697a708dc0bb3248c%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638398111234079685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=96bsdH9QO%2BxRbSuxxvcorrSCOp6LP32JSE0ey122GQ4%3D&reserved=0
 
>>> 
>>> PR #36205: Make databricks sql hook return a namedtuple. <
>>> 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F36205&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Ca7d1a1618a1b49b697a708dc0bb3248c%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638398111234079685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=s3R2p6ETr7FwLA9R164Twwo4NDKfnRusaz1TiKolUzY%3D&reserved=0
 
>>> 
>>> PR #35919: Add helper function for CRUD operations on weaviate’s schema
>> and
>>> class objects. <
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F35919&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Ca7d1a1618a1b49b697a708dc0bb3248c%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638398111234079685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=10GCt4Ob0NJf5pxnFkvflH56qS20%2Fxk%2BiUg9GgGN1Bs%3D&reserved=0
>>> >
>>> 
>>> PR #35719: Add XCom tab to Grid. <
>>> 
>>> 
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F35719&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Ca7d1a1618a1b49b697a708dc0bb3248c%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638398111234079685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MbCgA72xc52Hh6uttguryFxesNnErDh5sGZxP0dRkXI%3D&reserved=0
 
>>> 
>>> Please reply to this thread with your selection or offer your own
>

Re: [DISCUSSION] Enabling `pre-commit.ci` application for Airflow

[Wei Lee]

Same as Amogh. Even though I would like to fix that myself, it would make it 
much easier for those who aren’t familiar with these tools and still be able to 
contribute. But we might need to doc this behavior somewhere (GitHub PR issue 
might make more sense 🤔). Otherwise, the contributor might be surprised by the 
new commit.

Best,
Wei

> On Jan 3, 2024, at 12:21 AM, Vincent Beck  wrote:
> 
> I like the concept! +1
> 
> On 2023/12/30 11:16:35 Amogh Desai wrote:
>> I am aligning here with Pierre, but I am not against the idea of enabling
>> the pre commit ci application.
>> 
>> I’d rather have myself fix the issue as it sometimes also lets me have
>> second,third or multiple passes at my code which is up for review. This is
>> a personal choice where I feel that we are trying to fix a problem that is
>> not too problematic.
>> 
>> Again, only a personal choice but not against it. If it makes lives of the
>> stakeholders involved easier, I am all up for it!
>> 
>> Thanks & Regards,
>> Amogh Desai
>> 
>> On Sat, 30 Dec 2023 at 2:35 PM, Pierre Jeambrun 
>> wrote:
>> 
>>> I like the idea, but in practice auto fixable static checks are very
>>> obvious to fix and doesn’t require much work.
>>> 
>>> On the other hand most of static failure are ‘real issues’ and not auto
>>> fixable, for instance mypy, spelling, sphinx, db session usage etc…. (And
>>> ruff fix is a little aggressive IMO regarding linting).
>>> 
>>> So I would say that in practice it solves a painless problem (formatting,
>>> import sorting and other obvious things) and can’t do much about other
>>> issues.
>>> 
>>> This is why I am not sure it is worth the confusion for users. (But I am
>>> not against it)
>>> 
>>> On Sat 30 Dec 2023 at 09:19, Scheffler Jens (XC-DX/PJ-PACE-E03)
>>>  wrote:
>>> 
 I‘d also like to have auto-fixing included in CI. It is a classic pitfall
 and all that can be automated does not need to be a manual burden.
 Though I am not sure whether the plugin is able to use all the custom
 stuff as we also depend during execution on the CI image and docker.
 Besides security things this would be something that needs testing if it
 works.
 
 TLDR: +1 opinion
 
 Sent from Outlook for iOS
 
 From: Pankaj Koti 
 Sent: Saturday, December 30, 2023 7:50:10 AM
 To: dev@airflow.apache.org 
 Subject: Re: [DISCUSSION] Enabling `pre-commit.ci` application for
>>> Airflow
 
 I very much like the concept. We have been using it actively for
>>> Astronomer
 code repositories for 1+ year already and it has helped us greatly
>>> (Thanks
 to Felix Uellendall for introducing this back then 🙂)
 
 On Sat, 30 Dec 2023, 12:10 Jarek Potiuk,  wrote:
 
> FYI - Just now INFRA rejected the request on the basis of "code write"
> access permissions the app needs.
> 
> I'd still love to get feedback though on the concept -  I am not giving
 up
> that easily. We might still get it approved easily. We likely have some
> ways we can get "auto-fixing" working for us.
> 
> 1) I believe Github Applications now can use a bit different mechanism
>>> to
> ask for permissions and it can have "required" and "optional"
>>> permissions
> and I believe "Pull request write" should be enough (and I might
>>> attempt
 to
> convince the maintainers of it to adapt it to our needs).
> 2) Also, there is a "Pre-commit Lite Github Action" that we **might**
>>> be
> able to use to achieve a similar effect (with some added complexity to
 our
> `Pull Request Target` workflow.
> 
> So I would still love to hear from others :)
> 
> J.
> 
> On Fri, Dec 29, 2023 at 11:52 PM Jarek Potiuk 
>>> wrote:
> 
>> Hello everyone,
>> 
>> TL;DRl; I'd like to propose that we enable the pre-commit-ci GitHub
>> application for Airflow repo. According to how I understand it works,
 it
>> should greatly reduce friction (especially for new contributors) for
>> passing the quality gates for our pre-commits. That is - if we get
>>> the
>> approval for pre-commit-ci application from the ASF infra team.
>> 
>> Some more context:
>> 
>> We use and love (well some of us do, some of us likely hate) the
>> pre-commit as a quality gate for our checks. We have been using it
>>> for
>> years for local checks and CI integration and we have ~60 custom
> precommits
>> and in total we use about 100 pre-commit checks as our "quality"
>>> gates
>> 
>> However, using `standard` pre-commit (that is a de-facto standard in
>> Python world) has a nice property of 'standing on the shoulders of
> giants'.
>> There is one thing that few of us are aware of, that there is a way
>>> to
>> reduce friction for pre-commits that are not only flagging errors but
 can
>> also fix them. If we get the `pre-commit-

Re: [DISCUSS] Interested in joining/leaving the triage team ?

[Wei Lee]

Hi Jarek,

I would also want to continue being part of the team as well.

Best,
Wei

> On Jan 2, 2024, at 10:47 PM, utkarsh sharma  wrote:
> 
> I can move out of triage team now :)
> 
> On Tue, 2 Jan 2024 at 8:15 PM, Rahul Vats  wrote:
> 
>> Hi Jarek,
>> 
>> As discussed in #issue-triage, I would also like to nominate myself for the
>> triage team.
>> 
>> 
>> Regards,
>> 
>> Rahul Vats
>> 
>> On Tue, 2 Jan, 2024, 18:29 Ankit Chaurasia,  wrote:
>> 
>>> Hi Jarek,
>>> 
>>> I would want to continue being a "collaborators" team for the triage.
>>> 
>>> *Ankit Chaurasia*
>>> HomePage  |  LinkedIn
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Sat, Dec 30, 2023 at 1:36 PM Jarek Potiuk  wrote:
>>> 
 A regular question if there are people who are interested in joining
>> the
 "collaboators" team for our triage team (or would like to be removed) ?
 
 We regularly review/cleanup the "collaborators" team of ours
 https://github.com/apache/airflow/blob/main/.asf.yaml#L78 .
 
 Being collaborator gives a bit more capabilities for our issues
>>> (labelling,
 closing, converting to discussion etc.) - described in
 
 
>>> 
>> https://github.com/apache/airflow/blob/main/ISSUE_TRIAGE_PROCESS.rst#actions-that-can-be-taken-by-the-issue-triager
 
 The team has to be small (<10 people) and If you are interested to join
>>> the
 team - let us know - privately to me or in #issue-triage channel on
>>> Slack.
 
 I already have a few people interested but asking here in case others
>>> also
 are interested, we have Utkarsh to free it after becoming committer and
 maybe someone would like to be removed from the team to free the place
>>> for
 others - this is not a "lifetime" job :D
 
 We usually add people there who are already actively reviewing and
 commenting other's code, so if you have not done it before, starting
>> with
 doing it (and then even actively asking to join the tem in the slack
 channel) is a better idea than joining straight away.
 
 J.
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on December 31, 2023

[Wei Lee]

+1 (non binding)

Best,
Wei

> On Jan 1, 2024, at 1:09 AM, Amogh Desai  wrote:
> 
> +1 non binding
> 
> Tested if the RC is fine with other providers. Things look good 👍
> 
> On Sun, 31 Dec 2023 at 10:01 PM, Phani Kumar
>  wrote:
> 
>> +1 non binding.
>> 
>> On Sun, Dec 31, 2023 at 9:48 PM Hussein Awala  wrote:
>> 
>>> +1 (binding)
>>> 
>>> On Sun, Dec 31, 2023 at 5:14 PM Jarek Potiuk  wrote:
>>> 
 Cool. Now I just need 2 binding +1s :)
 
 On Sun, Dec 31, 2023 at 5:04 PM utkarsh sharma >> 
 wrote:
 
> +1(non-binding) tested my changes.
> 
> On Sun, Dec 31, 2023 at 6:16 PM Jarek Potiuk 
>> wrote:
> 
>> Hey all,
>> 
>> I have just cut the new wave Airflow Providers packages. This email
>>> is
>> calling a vote on the release,
>> which will last for 24 hours - accelerated vote which means that it
 will
>> end on January 01, 2024 12:42 PM UTC and until 3 binding +1 votes
>>> have
> been
>> received.
>> 
>> Consider this my (binding) +1.
>> 
>> This is an ad-hoc release of rc2 weaviate provider that has been
 removed
>> from previous wave due to a bug found (and fixed since).
>> 
>> Airflow Providers are available at:
>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>> 
>> *apache-airflow-providers--*.tar.gz* are the binary
>> Python "sdist" release - they are also official "sources" for the
> provider
>> packages.
>> 
>> *apache_airflow_providers_-*.whl are the binary
>> Python "wheel" release.
>> 
>> The test procedure for PMC members is described in
>> 
>> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for and Contributors who would like to test this
>>> RC
 is
>> described in:
>> 
>> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>> 
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> 
>> Only votes from PMC members are binding, but members of the
>> community
 are
>> encouraged to test the release and vote with "(non-binding)".
>> 
>> Please note that the version number excludes the 'rcX' string.
>> This will allow us to rename the artifact without modifying
>> the artifact checksums when we actually release.
>> 
>> The status of testing the providers by the community is kept here:
>> 
>> https://github.com/apache/airflow/issues/36511
>> 
>> You can the RC packages in PyPI following these links:
>> 
>> 
>> https://pypi.org/project/apache-airflow-providers-weaviate/1.3.0rc2/
>> 
>> Cheers,
>> J.
>> 
> 
 
>>> 
>> 

Re: [VOTE] Airflow Providers prepared on December 28, 2023

[Wei Lee]

+1 (non-binding) for providers other than Weavaite

Tested our example DAGs with the following providers without encountering issues

apache-airflow-providers-amazon==8.15.0rc1
apache-airflow-providers-cncf-kubernetes==7.13.0rc1
apache-airflow-providers-google==10.13.1rc1
apache-airflow-providers-microsoft-azure==8.5.1rc1

Best,
Wei

> On Dec 30, 2023, at 4:32 PM, Jarek Potiuk  wrote:
> 
> Need 2 binding (+1) to proceed on that one (and get RC2 for Waeviate)
> 
> On Thu, Dec 28, 2023 at 6:48 PM Jarek Potiuk  wrote:
> 
>> We will get  an rc2 for weaviate. Depending on how many other changes will
>> be merged in the meantime I will either release ad-hoc for just weaviate
>> (and potentially other problematic providers) or prepare a new wave.
>> 
>> On Thu, Dec 28, 2023 at 4:01 PM utkarsh sharma 
>> wrote:
>> 
>>> -1 (non-binding) for Weavaite provider. I found an issue with the
>>> provider,
>>> which I unfortunately missed in my initial testing. Raised a PR
>>>  for the same.
>>> 
>>> Thanks,
>>> Utkarsh Sharma
>>> 
>>> On Thu, Dec 28, 2023 at 2:40 PM Jarek Potiuk  wrote:
>>> 
 Hey all,
 
 I have just cut the new wave Airflow Providers packages. This email is
 calling a vote on the release,
 which will last for 72 hours - which means that it will end on
 December 31, 2023 09:08 AM UTC and until 3 binding +1 votes have been
 received.
 
 Consider this my (binding) +1.
 
 
 
 Airflow Providers are available at:
 https://dist.apache.org/repos/dist/dev/airflow/providers/
 
 *apache-airflow-providers--*.tar.gz* are the binary
 Python "sdist" release - they are also official "sources" for the
 provider packages.
 
 *apache_airflow_providers_-*.whl are the binary
 Python "wheel" release.
 
 The test procedure for PMC members is described in
 
 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
 
 The test procedure for and Contributors who would like to test this RC
 is described in:
 
 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
 
 
 Public keys are available at:
 https://dist.apache.org/repos/dist/release/airflow/KEYS
 
 Please vote accordingly:
 
 [ ] +1 approve
 [ ] +0 no opinion
 [ ] -1 disapprove with the reason
 
 
 Only votes from PMC members are binding, but members of the community
>>> are
 encouraged to test the release and vote with "(non-binding)".
 
 Please note that the version number excludes the 'rcX' string.
 This will allow us to rename the artifact without modifying
 the artifact checksums when we actually release.
 
 The status of testing the providers by the community is kept here:
 https://github.com/apache/airflow/issues/36467
 
 You can find packages as well as detailed changelog following the below
 links:
 
 https://pypi.org/project/apache-airflow-providers-alibaba/2.7.1rc1/
 https://pypi.org/project/apache-airflow-providers-amazon/8.15.0rc1/
 https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.3.2rc1/
 
>>> https://pypi.org/project/apache-airflow-providers-apache-kylin/3.5.0rc1/
 
 
>>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/7.13.0rc1/
 
>>> https://pypi.org/project/apache-airflow-providers-elasticsearch/5.3.1rc1/
 https://pypi.org/project/apache-airflow-providers-google/10.13.1rc1/
 
>>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/8.5.1rc1/
 https://pypi.org/project/apache-airflow-providers-weaviate/1.3.0rc1/
 https://pypi.org/project/apache-airflow-providers-zendesk/4.6.0rc1/
 
 
 Cheers,
 J.
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
 For additional commands, e-mail: dev-h...@airflow.apache.org
 
 
>>> 
>> 

Re: [VOTE] Airflow Providers prepared on December 23, 2023

[Wei Lee]

+1 (non-binding)

We've tested the following provider RCs with our example DAGs without 
encountering issues.

apache-airflow-providers-microsoft-azure==8.5.0rc1
apache-airflow-providers-sftp==4.8.1rc1
apache-airflow-providers-amazon==8.14.0rc1
apache-airflow-providers-apache-hive==6.4.0rc1
apache-airflow-providers-cncf-kubernetes==7.12.0rc1
apache-airflow-providers-databricks==6.0.0rc1
apache-airflow-providers-google==10.13.0rc3
apache-airflow-providers-sftp==4.8.1rc1
apache-airflow-providers-microsoft-azure==8.5.0rc1

Best,
Wei

> On Dec 23, 2023, at 10:30 AM, Jarek Potiuk  wrote:
> 
> Hey all,
> 
> I have just cut the new wave Airflow Providers packages. This email is
> calling a vote on the release,
> which will last for 72 hours - which means that it will end on December
> 27th, 2023 02:24 AM UTC and until 3 binding +1 votes have been received.
> This accounts for upcoming Holidays.
> 
> Consider this my (binding) +1.
> 
> Airflow Providers are available at:
> https://dist.apache.org/repos/dist/dev/airflow/providers/
> 
> *apache-airflow-providers--*.tar.gz* are the binary
> Python "sdist" release - they are also official "sources" for the provider
> packages.
> 
> *apache_airflow_providers_-*.whl are the binary
> Python "wheel" release.
> 
> The test procedure for PMC members is described in
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> 
> The test procedure for and Contributors who would like to test this RC is
> described in:
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
> 
> Public keys are available at:
> https://dist.apache.org/repos/dist/release/airflow/KEYS
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> 
> Only votes from PMC members are binding, but members of the community are
> encouraged to test the release and vote with "(non-binding)".
> 
> Please note that the version number excludes the 'rcX' string.
> This will allow us to rename the artifact without modifying
> the artifact checksums when we actually release.
> 
> The status of testing the providers by the community is kept here:
> 
> https://github.com/apache/airflow/issues/36384
> 
> (I removed all changes for that issue
> https://github.com/apache/airflow/pull/36086 )
> 
> You can find packages as well as detailed changelog following the below
> links:
> 
> https://pypi.org/project/apache-airflow-providers-airbyte/3.5.1rc1/
> https://pypi.org/project/apache-airflow-providers-amazon/8.14.0rc1/
> https://pypi.org/project/apache-airflow-providers-apache-beam/5.5.0rc1/
> https://pypi.org/project/apache-airflow-providers-apache-cassandra/3.4.1rc1/
> https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.3.1rc1/
> https://pypi.org/project/apache-airflow-providers-apache-hive/6.4.0rc1/
> https://pypi.org/project/apache-airflow-providers-apache-kafka/1.3.1rc1/
> https://pypi.org/project/apache-airflow-providers-apache-spark/4.6.0rc1/
> https://pypi.org/project/apache-airflow-providers-apache-sqoop/4.2.1rc1/
> https://pypi.org/project/apache-airflow-providers-apprise/1.2.1rc1/
> https://pypi.org/project/apache-airflow-providers-arangodb/2.4.1rc1/
> https://pypi.org/project/apache-airflow-providers-asana/2.4.1rc1/
> https://pypi.org/project/apache-airflow-providers-celery/3.5.1rc1/
> https://pypi.org/project/apache-airflow-providers-cloudant/3.4.1rc1/
> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/7.12.0rc1/
> https://pypi.org/project/apache-airflow-providers-cohere/1.1.1rc1/
> https://pypi.org/project/apache-airflow-providers-common-sql/1.10.0rc1/
> https://pypi.org/project/apache-airflow-providers-databricks/6.0.0rc1/
> https://pypi.org/project/apache-airflow-providers-datadog/3.5.1rc1/
> https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.5.1rc1/
> https://pypi.org/project/apache-airflow-providers-docker/3.9.1rc1/
> https://pypi.org/project/apache-airflow-providers-exasol/4.4.1rc1/
> https://pypi.org/project/apache-airflow-providers-fab/1.0.0rc1/
> https://pypi.org/project/apache-airflow-providers-github/2.5.1rc1/
> https://pypi.org/project/apache-airflow-providers-google/10.13.0rc3/
> https://pypi.org/project/apache-airflow-providers-grpc/3.4.1rc1/
> https://pypi.org/project/apache-airflow-providers-hashicorp/3.6.1rc1/
> https://pypi.org/project/apache-airflow-providers-jdbc/4.2.1rc1/
> https://pypi.org/project/apache-airflow-providers-jenkins/3.5.1rc1/
> https://pypi.org/project/apache-airflow-providers-microsoft-azure/8.5.0rc1/
> https://pypi.org/project/apache-airflow-providers-odbc/4.4.0rc1/
> https://pypi.org/project/apache-airflow-providers-openlineage/1.3.1rc1/
> https://pypi.org/project/apache-airflow-providers-opensearch/1.1.1rc1/
> https://pypi.org/project/apache-airflow-providers-oracle/3.9.1rc1/
> https://pypi.org/project/apache-airflow-providers-pagerdut

Re: [DISCUSS] Future of Pendulum in Airflow

[Wei Lee]

I’d vote for 2-a. Removing it in the upcoming minor release may not give enough 
time for airflow users to adjust. On the other hand, we would also want to 
decrease the amount of time we spend supporting various versions of Pendulum. 
2.9 might be a reasonable compromise.

Best,
Wei

> On Dec 19, 2023, at 9:52 PM, Jarek Potiuk  wrote:
> 
> I am for a) with deprecation warning when Pendulum 2 is installed in
> Airflow 2.8.1+.
> 
> I understand we have a long way before we can get rid of Pendulum
> dependency in general as it will be a major pain for our users for a
> number of reasons, so I think getting rid of Pendulum is likely not
> easy / feasible in the short/medium term (though it would likely be
> best).
> Pendulum 3 solves the biggest pain (one of the things that blocks us
> from officially having Python 3.12 support) and when we consider the
> scope of the features we use from Pendulum, it's not very likely it
> will cause us similar pains even if we keep it for quite some time. So
> supporting Pendulum 3 while giving our users (and 3rd-party libraries)
> a chance to upgrade to Pendulum 3 seems a good balance between cost
> and risk - at least for a year or so until Python 3.13 will be out
> (with possibly some more breaking changes) or until some security
> issues will be discovered (and not fixed for some time) in Pendulum.
> 
> J,
> 
> On Tue, Dec 19, 2023 at 2:43 PM Andrey Anshin  
> wrote:
>> 
>> Greetings everyone!
>> 
>> Because pendulum 3 was released we probably need to discuss about support
>> for the previous version of pendulum in Airflow core.
>> 
>> There is not much options here
>> 
>> Option 1: Drop support of next patch version of Airflow, e.g. 2.8.1. It
>> might (or might not) break someone's pipelines. In general we do not count
>> change upstream library version as breaking changes, however this could be
>> an exclusion, because `pendulum 2` first time released in May 9, 2018 and
>> there is potentially could be a lot of user code which do not work properly
>> with `pendulum 3`
>> 
>> 
>> Option 2: Keep support of both pendulum 2 and pendulum 3 for a while. There
>> are different suboptions
>> a). Support pendulum 2 until Airflow 2.9
>> b). Support pendulum 2 until the min Airflow version in providers reaches
>> Airflow 2.9, a bit more than 1 year. In Airflow 2.9 add user warning about
>> limited time of support
>> c). Support pendulum 2 until it is not painful
>> d). Support pendulum 2 until Airflow 3 (Forever?)
>> 
>> 
>> On Mon, 18 Dec 2023 at 16:53, Bolke de Bruin  wrote:
>> 
>>> Pendulum 3 was released.
>>> 
>>> https://pypi.org/project/pendulum/
>>> 
>>> Bolke
>>> 
>>> On Fri, 1 Dec 2023 at 18:38, Aritra Basu  wrote:
>>> 
 I agree with wanting to stay away from having to deal with dst but I am
>>> on
 the side of biting the bullet and transitioning to datetime and testing
 thoroughly, especially considering our current predicament with pendulum
 and the potential of us being in the same place again potentially in the
 future.
 
 --
 Regards,
 Aritra Basu
 
 On Fri, Dec 1, 2023, 10:27 PM Jarek Potiuk  wrote:
 
> Yeah. Agree it's a challenge but I think it's also an opportunity to
> fix those - because whether we admit it, or not - we already have those
> problems - regardless of Pendulum's use.
> 
> If you look at the issues (especially ones created around DST time) -
 there
> are always one or two issues (even in latest versions of Airflow) where
 DST
> transition is causing people problems. And they are rarely
> solved/addressed.
> 
> Example from last October:
> 
> https://github.com/apache/airflow/issues/35272
> 
> Example from May 2022:
> 
> https://github.com/apache/airflow/issues/23400
> 
> And Example from March 2020 ( which might or might not be fixed by the
 open
> PR):
> 
> https://github.com/apache/airflow/issues/7999
> 
> J.
> 
> 
> On Fri, Dec 1, 2023 at 5:41 PM Bolke de Bruin 
>>> wrote:
> 
>> The alternative is to vendor in or fork pendulum as part of the
>>> Airflow
>> project. Also a major undertaking but maybe less effort.
>> 
>> We shouldn't just address this as a technical challenge. The question
>> becomes: does that combination fornally do DST transitions correctly,
> does
>> it do leap seconds correctly, does it add dates correctly, does it do
>> timezone traversal correctly and many many other things.
>> 
>> So if we switch to the datetime + zone info + timezone we need to
>>> have
 a
>> test suite that verifies this. Otherwise we potentially expose our
 users
> to
>> different behavior IN their data. Not just a schedule by in their SQL
 or
> in
>> some scripting where logical date / execution date is used or where
 date
>> arithmetic is applied.
>> 
>> So it is not changing one car for another. I

Re: [DISCUSS] "Require conversation resolution" in our PRs before merge?

[Wei Lee]

+1 for trying and observing how it works. My concern is that adding an 
additional obstacle might lead to more unfinished PRs. It might be helpful to 
give the contributor some guidance on when we can resolve the comments.

Best,
Wei

> On Dec 19, 2023, at 9:28 PM, Andrey Anshin  wrote:
> 
> We could try and if found it slows down for some reason we always might
> revert it back.
> 
> Just one suggestion, sometimes discussion contains some useful information,
> e.g. "What the reason of finally decision", "Useful information why it
> should works by suggested way, or should not work", which might be useful
> for someone who investigate why this changes was made, so in this case I
> would suggest to create link in the main thread of PR with useful
> discussions.
> 
> 
> 
> 
> On Tue, 19 Dec 2023 at 17:16, Jarek Potiuk  wrote:
> 
>> Hey everyone,
>> 
>> TL;DR; I have a small proposal/discussion proposal to modify a bit the
>> branch protection rules for Airflow. Why don't we add a protection
>> rule in our PRs that requires all the comments in the PRs to be
>> "marked as resolved" before merging the PR ?
>> 
>> I have been following myself  - for quite some time - an approach that
>> whenever there are comments/suggestions/doubts in my PRs I do not
>> merge the PR until I **think** all of those have been addressed
>> (somehow).
>> 
>> The resolution might not be what the person commenting wants directly,
>> it might be "I hear your comment, and there are good reasons to do
>> otherwise" or simply saying - "I know it could be done this way but I
>> think otherwise" etc. etc. But sometimes I miss that there is a
>> comment that I have not reacted to, I skipped it unconsciously etc.
>> 
>> I think having "some" kind of reaction to comments and deliberate "I
>> believe the conversation is resolved" is a very good thing and having
>> the author making a deliberate effort to "mark the conversation as
>> resolved" is a sign it's been read, though about and consciously
>> reacted to.
>> 
>> I've learned recently that you can add protection rule that will
>> require all conversations on PR to be resolved before merging it, I
>> even went to a great length to create (and get merged) a PR to ASF
>> infra to enable it via .asf.yml feature
>> (https://github.com/apache/infrastructure-p6/pull/1740) - so we can
>> enable it now by a simple PR to our .asf.yaml enabling it.
>> 
>> I'd love to try it  - but of course it will have to change a bit the
>> workflow of everyone, where the author (or reviewer, or maintainer)
>> will have to mark all conversations as resolved deliberately before
>> merging PR.
>> 
>> I'd love to enable it - at least to try and see how it can work - but
>> I understand it might add a bit of burden for everyone, however, I
>> think it might be worth it.
>> 
>> WDYT?
>> 
>> J.
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> For additional commands, e-mail: dev-h...@airflow.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Release Airflow Python Client 2.8.0 from 2.8.0rc1

[Wei Lee]

+1 (non-binding)

Best,
Wei

> On Dec 19, 2023, at 8:20 PM, utkarsh sharma  wrote:
> 
> +1 non binding
> 
> On Tue, 19 Dec 2023 at 5:23 PM, Amogh Desai 
> wrote:
> 
>> +1 non binding
>> 
>> Installed the RC bits and ran example DAGs.
>> 
>> 
>> 
>> On Tue, 19 Dec 2023 at 5:22 PM, Phani Kumar
>>  wrote:
>> 
>>> +1 non binding
>>> 
>>> On Tue, Dec 19, 2023 at 5:10 PM Pankaj Koti
>>>  wrote:
>>> 
 +1 (non-binding)
 
 Installed the RC and was able to trigger a DAG successfully using the
 client.
 
 Best regards,
 
 *Pankaj Koti*
 Senior Software Engineer (Airflow OSS Engineering team)
 Location: Pune, Maharashtra, India
 Timezone: Indian Standard Time (IST)
 Phone: +91 9730079985
 
 
 On Tue, Dec 19, 2023 at 3:49 PM Ephraim Anierobi <
 ephraimanier...@apache.org>
 wrote:
 
> Hey fellow Airflowers,
> 
> I have cut the release candidate for the Airflow Python Client
>>> 2.8.0rc1.
> The client consists of APIs corresponding to REST APIs available in
> *Apache Airflow 2.8.0*. This email is calling for a vote on
> the release, which will last for 72 hours. Consider this my (binding)
>>> +1.
> 
> Airflow Client 2.8.0rc1 is available at:
> 
>>> https://dist.apache.org/repos/dist/dev/airflow/clients/python/2.8.0rc1/
> 
> Or also available at PyPI:
> https://pypi.org/project/apache-airflow-client/2.8.0rc1/
> 
> *apache-airflow-client-2.8.0rc1-source.tar.gz* is a source release
>> that
> comes with
> INSTALL instructions.
> *apache-airflow-client-2.8.0rc1-bin.tar.gz* is the binary Python
>>> "sdist"
> release.
> 
> Public keys are available at:
> https://dist.apache.org/repos/dist/release/airflow/KEYS
> 
> Only votes from PMC members are binding, but the release manager
>> should
> encourage members of the community to test the release and vote with
> "(non-binding)".
> 
> *Changelog:*
> 
> ### Major changes:
>  - Allow filtering event logs by attributes ([#34417](
> https://github.com/apache/airflow/pull/34417))
>  - Add extra fields to plugins endpoint ([#34913](
> https://github.com/apache/airflow/pull/34913))
>  - Let auth managers provide their own API endpoints ([#34349](
> https://github.com/apache/airflow/pull/34349))
>  - Enable pools to consider deferred tasks ([#32709](
> https://github.com/apache/airflow/pull/32709))
>  - Add dag_run_ids and task_ids filter for the batch task instance
>> API
> endpoint ([#32705](https://github.com/apache/airflow/pull/32705))
> 
> ### Major Fixes
>  - Add DagModel attributes before dumping DagDetailSchema for
> get_dag_details API endpoint ([#34947](
> https://github.com/apache/airflow/pull/34947))
>  - Add TriggerRule missing value in rest API ([#35194](
> https://github.com/apache/airflow/pull/35194))
>  - Fix wrong plugin schema ([#34858](
> https://github.com/apache/airflow/pull/34858))
>  - Make dry run optional for patch task instance ([#34568](
> https://github.com/apache/airflow/pull/34568))
>  - OpenAPI Spec fix nullable alongside $ref ([#32887](
> https://github.com/apache/airflow/pull/32887))
>  - Clarify new_state in OpenAPI spec ([#34056](
> https://github.com/apache/airflow/pull/34056))
> 
> ### NEW API supported
>  - NA
> 
> Cheers,
> Ephraim
> 
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Release Airflow 2.8.0 from 2.8.0rc4

[Wei Lee]

+1 (non-binding)

Best,
Wei

> On Dec 18, 2023, at 6:15 PM, Pankaj Koti  wrote:
> 
> +1 (non-binding)
> 
> On 2023/12/18 10:00:50 Rahul Vats wrote:
>> +1 (non-binding). I re-verified tests with rc4.
>> 
>>   - Verified that running our example DAGs did not reveal any regressions.
>>   - Verified the addition of the task context logging feature, enabling
>>   the forwarding of messages to task logs (#32646, #32693, #35857).
>>   - Verified the implementation of AIP-58: Airflow ObjectStore (AFS).
>>   - Verified the addition of listener hooks for Datasets (#34418).
>>   - Verified the addition of the XCom tab to the Grid (#35719).
>>   - Verified the API endpoints and no regressions were found.
>>   - Verified running our example DAG for performance task throughput - I
>>   do not see any degradation from version 2.7.3.
>> 
>> Regards,
>> Rahul Vats
>> 
>> Regards,
>> Rahul Vats
>> 9953794332
>> 
>> 
>> On Sat, 16 Dec 2023 at 16:45, Ephraim Anierobi 
>> wrote:
>> 
>>> Hey fellow Airflowers,
>>> 
>>> I have cut Airflow 2.8.0rc4. This email is calling a vote on the release,
>>> which will last at least 50 hours, from Saturday, December 16, 2023 at
>>> 11:15 am UTC
>>> until Monday, December 18, 2023, at 1:15 pm UTC
>>> <
>>> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20231218T1315&p1=1440
 ,
>>> and until 3 binding +1 votes have been received.
>>> 
>>> Consider this my (binding) +1.
>>> 
>>> Airflow 2.8.0rc4 is available at:
>>> https://dist.apache.org/repos/dist/dev/airflow/2.8.0rc4/
>>> 
>>> *apache-airflow-2.8.0-source.tar.gz* is a source release that comes with
>>> INSTALL instructions.
>>> *apache-airflow-2.8.0.tar.gz* is the binary Python "sdist" release.
>>> *apache_airflow-2.8.0-py3-none-any.whl* is the binary Python wheel "binary"
>>> release.
>>> 
>>> Public keys are available at:
>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>> 
>>> Please vote accordingly:
>>> 
>>> [ ] +1 approve
>>> [ ] +0 no opinion
>>> [ ] -1 disapprove with the reason
>>> 
>>> Only votes from PMC members are binding, but all members of the community
>>> are encouraged to test the release and vote with "(non-binding)".
>>> 
>>> The test procedure for PMC members is described in:
>>> 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
>>> 
>>> The test procedure for and Contributors who would like to test this RC is
>>> described in:
>>> 
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
>>> 
>>> 
>>> Please note that the version number excludes the `rcX` string, so it's now
>>> simply 2.8.0. This will allow us to rename the artifact without modifying
>>> the artifact checksums when we actually release.
>>> 
>>> Release Notes:
>>> https://github.com/apache/airflow/blob/2.8.0rc4/RELEASE_NOTES.rst
>>> 
>>> Changes since 2.8.0rc4:
>>> 
>>> *Bug Fixes*:
>>> - Load consuming_dags attr eagerly before dataset listener (#36247)
>>> 
>>> *Miscellaneous*:
>>> - Change default MySQL client to MariaDB (#36243)
>>> 
>>> Cheers,
>>> Ephraim
>>> 
>> 
> 

Re: [VOTE] Airflow Providers prepared on December 12, 2023

[Wei Lee]

-1 (non-binding) for the google and databricks providers. Sent
https://github.com/apache/airflow/pull/36202 for fixing the google issue

Best,
Wei

Jarek Potiuk  於 2023年12月13日 週三 下午6:34寫道：

> Google and databricks will get RC3 (PRs to fix them are already created).
> I'd still love to have +1 PMC votes on the remaining 3 :)
>
> J
>
> On Wed, Dec 13, 2023 at 1:59 PM Pankaj Singh 
> wrote:
>
> > -1 (non-binding) for google and databricks providers since bug a has been
> > found.
> >
> > On Wed, Dec 13, 2023 at 3:45 PM Utkarsh Sharma
> >  wrote:
> >
> > > -1 (non-binding)
> > >
> > > I looked into Databricks providers and the PR
> > >  introduced breaking
> > changes
> > > I have added my concern in the comment
> > > .
> Also,
> > I
> > > don't think there is an easy way to make row objects serializable.
> > >
> > > On Wed, Dec 13, 2023 at 12:28 PM Rahul Vats 
> > > wrote:
> > >
> > > > -1 (non-binding) our example DAG failing for Google and Databricks
> > > > providers, We are working on fixes.
> > > >
> > > >-
> > > https://pypi.org/project/apache-airflow-providers-google/10.13.0rc2/
> > > >-
> > > >
> https://pypi.org/project/apache-airflow-providers-databricks/5.1.0rc2/
> > > >
> > > > Regards,
> > > > Rahul Vats
> > > > 9953794332
> > > >
> > > >
> > > > On Wed, 13 Dec 2023 at 11:35, Phani Kumar  > > > .invalid>
> > > > wrote:
> > > >
> > > > > -1 non binding. We are finding failures in astro sdk DAGs after
> using
> > > the
> > > > > databricks and google RC.
> > > > > We are working on creating PRs for the fixes.
> > > > >
> > > > > FAILED
> > > > >
> > > > >
> > > >
> > >
> >
> tests_integration/databases/databricks_tests/test_delta.py::test_delta_run_sql
> > > > > - AttributeError: 'list' object has no attribute 'asDict'
> > > > > FAILED
> > > > >
> > > > >
> > > >
> > >
> >
> tests_integration/databases/databricks_tests/test_delta.py::test_delta_run_sql_with_parameters
> > > > > - AttributeError: 'list' object has no attribute 'asDict'
> > > > > FAILED
> > > > >
> > > > >
> > > >
> > >
> >
> tests_integration/databases/databricks_tests/test_delta.py::test_delta_create_table_with_columns[delta]
> > > > > - AssertionError: assert ['id', 'int', None] == Row(col_name='id',
> > > > > data_type='int', comment=None)
> > > > >   Full diff:
> > > > >   - Row(col_name='id', data_type='int', comment=None)
> > > > >   + ['id', 'int', None]
> > > > > ERROR
> > > > >
> > > > >
> > > >
> > >
> >
> tests_integration/databases/databricks_tests/test_delta.py::test_create_table_from_select_statement[delta]
> > > > > - airflow.exceptions.AirflowException: Databricks job failed. Job
> > info
> > > > > ***'job_id': 438976959785934, 'run_id': 268958235083047,
> > > > > 'creator_user_name': 'phani.ku...@astronomer.io', 'number_in_job':
> > > > > 268958235083047, 'state': ***'life_cycle_state': 'TERMINATED',
> > > > > 'result_state': 'FAILED', 'state_message': 'Workload failed, see
> run
> > > > output
> > > > > for details', 'user_cancelled_or_timedout': False***, 'task':
> > > > > ***'spark_python_task': ***'python_file':
> > > > >
> > > > >
> > > >
> > >
> >
> 'dbfs:/mnt/pyscripts/load_file__tmp_en0ra8imcxfee4sp8b9y4hzqj4bjy5y2fvgpo9gv0s3pzvmzqv1tfaza9.py'***,
> > > > > 'cluster_spec': ***'existing_cluster_id': '***'***,
> > 'cluster_instance':
> > > > > ***'cluster_id': '***', 'spark_context_id':
> '4902558347078657686'***,
> > > > > 'start_time': 1702436831591, 'setup_duration': 1000,
> > > > 'execution_duration':
> > > > > 12000, 'cleanup_duration': 0, 'end_time': 1702436844777,
> 'run_name':
> > > > > 'Untitled', 'run_page_url': '
> > > > >
> > > > >
> > > >
> > >
> >
> https://dbc-9c390870-65ef.cloud.databricks.com/?o=4256138892007661#job/438976959785934/run/268958235083047
> > > > > ',
> > > > > 'run_type': 'SUBMIT_RUN', 'attempt_number': 0, 'format':
> > > 'SINGLE_TASK'***
> > > > >
> > > > > On Wed, Dec 13, 2023 at 4:02 AM Jarek Potiuk 
> > wrote:
> > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > [Filling-in for Elad, who had no time to do it this time]
> > > > > >
> > > > > > I have just cut the new wave Airflow Providers packages. This
> email
> > > is
> > > > > > calling a vote on the release,
> > > > > > which will last for 24 hours - which means that it will end on
> > > December
> > > > > 13,
> > > > > > 2023 22:00 PM UTC and until 3 binding +1 votes have been
> received.
> > > > > >
> > > > > > Following our processes, this is an accelerated vote taking into
> > > > account
> > > > > > that the RC1 version has been tested and it only contains
> > incremental
> > > > > > changes.
> > > > > >
> > > > > > Consider this my (binding) +1.
> > > > > >
> > > > > > This release contains fixes to issues found in rc1 in google,
> > docker,
> > > > > odbc,
> > > > > > databricks providers and the last (after scheduling for
> > > > > > removal) daskexecutor provider release.
> > > > > >
> > > > > >

Re: [VOTE] Airflow Providers prepared on December 08, 2023

[Wei Lee]

+1 (none-binding) for providers other than Google and Docker.

Best,
Wei

> On Dec 11, 2023, at 3:07 PM, Rahul Vats  wrote:
> 
> +1 (non-binding) except for Google, Docker and Databricks.
> 
> Regards,
> Rahul Vats
> 9953794332
> 
> 
> On Mon, 11 Dec 2023 at 12:21, Pankaj Koti 
> wrote:
> 
>> +1 (non-binding)
>> 
>> Tested my set of changes.
>> 
>> Best regards,
>> 
>> 
>> On Mon, Dec 11, 2023 at 9:57 AM Amogh Desai 
>> wrote:
>> 
>>> +1 non binding for all the other providers other than Google and Docker.
>>> 
>>> Thanks & Regards,
>>> Amogh Desai
>>> 
>>> On Sun, Dec 10, 2023 at 5:46 PM Jarek Potiuk  wrote:
>>> 
 +1 (binding) for all providers except Google, an Docker where bugs were
 found as noted in the Github Issue.
 
 BTW. I have also found that it was easy to miss that the that removed
 `daskexecutor` provider should also be prepared separately (with
>>> "removal"
 note) - so this one will also need to be added (and I will improve the
 tooling a bit so that removed providers are prepared even if they are
 technically suspended as well). Seems in the next release we will have
>> a
 few more of those :)
 
 J.
 
 
 On Sat, Dec 9, 2023 at 5:33 PM Elad Kalif  wrote:
 
> google and docker providers are excluded from this vote.
> both providers will have rc2
> 
> On Sat, Dec 9, 2023 at 6:14 PM Pankaj Singh <
>> ags.pankaj1...@gmail.com>
> wrote:
> 
>> -1 (non-binding) for Google provider because a bug have been found
>>> that
>> need a fix see
>> 
>> https://github.com/apache/airflow/pull/34919#issuecomment-1846993082
>> 
>> +1 (non-binding) for other providers.
>> 
>> On Sat, Dec 9, 2023 at 9:04 PM Hussein Awala 
>>> wrote:
>> 
>>> Checked signatures, checksums, licences, sources and tested my
 changes;
>> all
>>> look good.
>>> 
>>> -0 (binding) for Google provider, I think
>>> https://github.com/apache/airflow/pull/34919 will impact memory
 usage
> if
>>> there are a lot of files (here you find my assumption
>>> 
>>> https://github.com/apache/airflow/pull/36130#discussion_r1421434993
 ).
>>> Unfortunately, I will not have the time to test it, but anyway,
>>> there
> is
>> a
>>> bug on the mentioned PR, and the provider will probably be
>> excluded
> from
>>> RC1.
>>> 
>>> +1 (binding) for the other providers.
>>> 
>>> On Fri, Dec 8, 2023 at 1:39 AM Elad Kalif 
 wrote:
>>> 
 Hey all,
 
 I have just cut the new wave Airflow Providers packages. This
>>> email
> is
 calling a vote on the release,
 which will last for 72 hours - which means that it will end on
> December
>>> 10,
 2023 23:35 PM UTC and until 3 binding +1 votes have been
>>> received.
 
 Consider this my (binding) +1.
 
 Airflow Providers are available at:
 https://dist.apache.org/repos/dist/dev/airflow/providers/
 
 *apache-airflow-providers--*.tar.gz* are the binary
 Python "sdist" release - they are also official "sources" for
>>> the
>>> provider
 packages.
 
 *apache_airflow_providers_-*.whl are the binary
 Python "wheel" release.
 
 The test procedure for PMC members is described in
 
 
>>> 
>> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
 
 The test procedure for and Contributors who would like to test
>>> this
> RC
>> is
 described in:
 
 
>>> 
>> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
 
 
 Public keys are available at:
 https://dist.apache.org/repos/dist/release/airflow/KEYS
 
 Please vote accordingly:
 
 [ ] +1 approve
 [ ] +0 no opinion
 [ ] -1 disapprove with the reason
 
 
 Only votes from PMC members are binding, but members of the
 community
>> are
 encouraged to test the release and vote with "(non-binding)".
 
 Please note that the version number excludes the 'rcX' string.
 This will allow us to rename the artifact without modifying
 the artifact checksums when we actually release.
 
 The status of testing the providers by the community is kept
>>> here:
 https://github.com/apache/airflow/issues/36117
 
 You can find packages as well as detailed changelog following
>> the
> below
 links:
 
 
 https://pypi.org/project/apache-airflow-providers-airbyte/3.5.0rc1/
 
 https://pypi.org/project/apache-airflow-providers-alibaba/2.7

Re: [DISCUSS] Capturing Architectural decisions (ADRS?)

[Wei Lee]

I like this idea! It would make it easier to follow why these decisions were 
made. But if we are to adopt this idea, we might need to decide which decisions 
we want to include in these ADRs. Or maybe we could include a checkbox on the 
PR template for the committers to decide whether to ask the author to add an 
ADR before merging.

On the other hand, I also like Niko's suggestion a lot. Some basic 
annotation/metadata makes it easier to trace the log, and git blame is really 
handy. It is easier to check compared to the doc. And the commit rule doesn't 
need to be complicated. But yes, this kind of rule would definitely be a 
barrier to new contributors, which will be something we need to consider as 
well.

Best,
Wei

> On Dec 5, 2023, at 6:20 AM, Jarek Potiuk  wrote:
> 
>> 
>> I love this idea!
>> 
>> Another option, that I don't think we as a community are very good at, is
>> putting the context of the change in the git commit message itself. Those
>> messages are already tightly associated into git history and the code
>> itself via blame without needing to introduce an new concept for this
>> purpose. Those commit messages can be viewed with many existing tools that
>> can browse Git blame/log. Some projects like Linux or Git itself have very
>> large commit messages with all the context required, a random example I
>> pulled from the first page of most recent commits:
>> https://github.com/torvalds/linux/commit/d67f39d2b81b6a8259944d2400c1ff4fe283ff72
> 
> 
> Oh absolutely - if the single commit is a "complete" change - I am all for
> long messages.
> Take a look at some of my commit messages :D.
> 
> I'd love some of our commit messages that are detailed and describe the
> context of the PRs. Especially in cases where it affects more than just
> obvious "Adding this or that new operator" for example.
> 
> I think that we do not also need to capture all decisions this way. For me
> this is really a question of having something in between a substantial
> commit (that needs a long message) and AIP. Something that is a "shared"
> and "common" piece of infrastructure that introduces a new "idea" or new
> "common utility" or "concept" to be followed by others in the future.
> 
> The "common.sql" and "serde" are IMHO good examples of such "concepts".
> The consequences of decisions made there will be carried over in multiple
> other commits, and it's great to have it written and recorded in the repo
> as a "markdown" file - so that we do not have to find the original commit
> that introduced it. It is right there, where you need it - in the folder
> where the source code lives that it relates to.
> 
> I am not a big fan of having "one rule" for all commits and changes, I
> really think we should have a gradation:
> 
> 1) simple fix or adding new operator/hook that needs no extra context ->
> fine with single line commit
> 2) bigger fix that needs more explanation why we are doing it/context that
> will not be obvious after some time -> long commit message explaining why
> we are doing it, context, problem it solves, etc.
> 3) a concept that comes and gets refined in probably multiple PRs
> introducing a "shared" way of doing things that we want others to follow ->
> ADR describing the concept, decisions, alternatives, consequences (and can
> evolve over time by adding more records).
> 4) big architectural change which requires a lot of deliberation and voting
> and has wide impact on the architecture of Airflow -> AIP
> 
> I think we do not really want to introduce a lot of bureaucracy, we should
> only add more "documentation" expectations where it really matters to keep
> it for future maintainers (and future selves as well as I often find those
> kind of notes and commit messages very useful months and years from the
> time they were made.
> 
> BTW. Something that made my day recently "If bureaucracy was meant to be
> easy, they'd have made the word easier to spell".
> 
> J.
> 
> 
> 
>> 
>> 
>> You can see the git commit messages is MUCH longer than the code change
>> itself even! So if you're curious why that code is the way it is, you can
>> just git blame it and have all the context there.
>> 
>> The ADR having markdown is nice, it allows you a bit more formatting, but
>> then it also requires a couple more steps to view that formatting.
>> 
>> Cheers,
>> Niko
>> 
>> 
>> From: Vincent Beck 
>> Sent: Monday, December 4, 2023 11:22:54 AM
>> To: dev@airflow.apache.org
>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Capturing
>> Architectural decisions (ADRS?)
>> 
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>> 
>> 
>> 
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’ête

Re: [ANNOUNCE] New committer: Utkarsh Sharma

[Wei Lee]

Congratulations, Utkarsh!

Best,
Wei

> On Dec 5, 2023, at 5:58 AM, Oliveira, Niko  
> wrote:
> 
> Congrats! Very well deserved!
> 
> 
> From: Pankaj Koti 
> Sent: Monday, December 4, 2023 11:28:41 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [ANNOUNCE] New committer: Utkarsh 
> Sharma
> 
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you can confirm the sender and know the 
> content is safe.
> 
> 
> 
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
> cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
> confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
> contenu ne présente aucun risque.
> 
> 
> 
> Great news, many congratulations Utkarsh 🕺🥳✌🏾
> 
> On Tue, 5 Dec 2023, 00:53 Pierre Jeambrun,  wrote:
> 
>> Congratulations!
>> 
>> Le lun. 4 déc. 2023 à 20:18, Andrey Anshin  a
>> écrit :
>> 
>>> Congrats! 🔥💯
>>> 
>>> 
>>> 
>>> 
>>> On Mon, 4 Dec 2023 at 22:45, Amogh Desai 
>> wrote:
>>> 
 This is fantastic! Congratulations Utkarsh.
 
 Well deserved!
 
 Thanks & Best Regards,
 Amogh Desai
 
 
 On Mon, 4 Dec 2023 at 10:56 PM, Pankaj Singh >> 
 wrote:
 
> Congratulations Utkarsh 🎉
> 
> On Mon, Dec 4, 2023 at 10:45 PM Aritra Basu <
>> aritrabasu1...@gmail.com>
> wrote:
> 
>> Amazing Utkarsh, congratulations!
>> 
>> --
>> Regards,
>> Aritra Basu
>> 
>> On Mon, Dec 4, 2023, 10:27 PM Phani Kumar <
>> phani.ku...@astronomer.io
>> .invalid>
>> wrote:
>> 
>>> Congratulations Utkarsh ! Well deserved
>>> 
>>> On Mon, Dec 4, 2023 at 10:13 PM Jarek Potiuk 
 wrote:
>>> 
 Hello everyone,
 
 [filling-in for Kaxil who is less available nowadays and
>>> travelling
>>> without
 much access to computer]
 
 The Project Management Committee (PMC) for Apache Airflow
 has invited Utkarsh Sharma to become a committer and we are
>>> pleased
 to announce that they have accepted.
 
 Utkarsh had been contributing for quite some time and played
 instrumental role in making Airflow ready for the LLM
>> revolution
 and recently embarked on the difficult route of improving our
 documentation contributing experience (which is quite a
 challenge to be honest and a brave commitment :) ) and
 I am looking forward to what the future holds with
 Utkarsh becoming the committer,
 
 Congratulations Utkarsh, and welcome onboard! Certainly well
> deserved.
 
 Being a committer enables easier contribution to the
 project since there is no need to go via the patch
 submission process. This should enable better productivity.
 
 J.
 
>>> 
>> 
> 
 
>>> 
>> 

Re: [VOTE] November PR of the Month

[Wei Lee]

+ 1 for 32646

Best,
Wei

> On Nov 28, 2023, at 5:35 AM, utkarsh sharma  wrote:
> 
> +1 for 32646, great work.
> 
> Thanks,
> Utkarsh Sharma
> 
> On Tue, Nov 28, 2023 at 2:50 AM Collin McNulty 
> wrote:
> 
>> +1 to 32646. Been wanting this for a long time!
>> 
>> On Mon, Nov 27, 2023 at 11:50 AM Briana Okyere
>>  wrote:
>> 
>>> Happy Holidays Everyone :)
>>> 
>>> It’s once again time to vote for the PR of the Month.
>>> 
>>> With the help of the `get_important_pr_candidates` script in dev/stats,
>>> we've identified the following candidates:
>>> 
>>> PR #32646: Add task context logging feature to allow forwarding messages
>> to
>>> task logs. 
>>> 
>>> PR #34964: Add task parameter to set custom logger name. <
>>> https://github.com/apache/airflow/pull/34964>
>>> 
>>> PR #34921: Add Cohere Provider. <
>>> https://github.com/apache/airflow/pull/34921>
>>> 
>>> PR #35488: Implement login and logout in AWS auth manager. <
>>> https://github.com/apache/airflow/pull/35488>
>>> 
>>> PR #35530: feature(providers): added `OpsgenieNotifier`. <
>>> https://github.com/apache/airflow/pull/35530>
>>> 
>>> Please reply to this thread with your selection or offer your own
>>> nominee(s).
>>> 
>>> Voting will close on December 1st at 1 PM PST. The winner(s) will be
>>> featured in the next issue of the Airflow newsletter.
>>> 
>>> Also, if there’s an article or event that you think should be included in
>>> this or a future issue of the newsletter, please drop me a line at <
>>> briana.oky...@astronomer.io>.
>>> 
>>> --
>>> Briana Okyere
>>> Community Manager
>>> Email: briana.oky...@astronomer.io
>>> Mobile: +1 415.713.9943
>>> Time zone: US Pacific UTC
>>> 
>>> 
>>> 
>> 

Re: [VOTE] Airflow Providers prepared on November 24, 2023

[Wei Lee]

+1 (non-binding)

We tested it with our example DAGs with "apache-airflow-providers-amazon", 
"apache-airflow-providers-cncf-kubernetes", 
"apache-airflow-providers-databricks", "apache-airflow-providers-dbt-cloud", 
"apache-airflow-providers-google", "apache-airflow-providers-cncf-kubernetes", 
"apache-airflow-providers-microsoft-azure", 
"apache-airflow-providers-snowflake" and everything works as expected. Thanks!

Best,
Wei

> On Nov 27, 2023, at 7:51 AM, Bolke de Bruin  wrote:
> 
> +1 (binding)
> 
> Had some fun discussion changes. Sorry for the fuss :-).
> 
> B.
> 
> On Sun, 26 Nov 2023 at 21:41, Jarek Potiuk  wrote:
> 
>> +1 (binding)
>> 
>> Tested my template changes, signatures, checksum, licences, sources are all
>> good.
>> 
>> I also tested that reproducible builds work great ! I checked out the tag
>> from sources, ran `breeze release-management prepare-provider-packages` and
>> got the packages built locally which had 100% binary compatibility with the
>> packages Elad prepared - this means that we no longer need to verify
>> sources.
>> 
>> As PMC members we can simply diff the .whl/.tar.gz packages produced by the
>> release manager and those that you can build locally using only sources
>> from our repo and official build tools - and we can call it a day.
>> 
>> One step closer to get SASL compliance and Follow the standards of securing
>> the software build/supply chain process: https://reproducible-builds.org/.
>> 
>> J.
>> 
>> On Sun, Nov 26, 2023 at 7:08 PM Pankaj Koti
>>  wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> Tested my set of changes, all work fine. Thank you for the release
>> efforts!
>>> 
>>> 
>>> Best regards,
>>> 
>>> *Pankaj Koti*
>>> Senior Software Engineer (Airflow OSS Engineering team)
>>> Location: Pune, Maharashtra, India
>>> Timezone: Indian Standard Time (IST)
>>> Phone: +91 9730079985
>>> 
>>> 
>>> On Sun, Nov 26, 2023 at 10:50 PM Hussein Awala  wrote:
>>> 
 +1 (binding): checked signatures, checksums, licences, sources and
>> tested
 my changes; all looks good.
 
 On Fri 24 Nov 2023 at 20:30, Elad Kalif  wrote:
 
> Hey all,
> 
> I have just cut the new wave Airflow Providers packages. This email
>> is
> calling a vote on the release,
> which will last for 72 hours - which means that it will end on
>> November
 27,
> 2023 18:30 PM UTC and until 3 binding +1 votes have been received.
> 
> 
> Consider this my (binding) +1.
> 
> Airflow Providers are available at:
> https://dist.apache.org/repos/dist/dev/airflow/providers/
> 
> *apache-airflow-providers--*.tar.gz* are the binary
> Python "sdist" release - they are also official "sources" for the
 provider
> packages.
> 
> *apache_airflow_providers_-*.whl are the binary
> Python "wheel" release.
> 
> The test procedure for PMC members is described in
> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> 
> The test procedure for and Contributors who would like to test this
>> RC
>>> is
> described in:
> 
> 
 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
> 
> 
> Public keys are available at:
> https://dist.apache.org/repos/dist/release/airflow/KEYS
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> 
> Only votes from PMC members are binding, but members of the community
>>> are
> encouraged to test the release and vote with "(non-binding)".
> 
> Please note that the version number excludes the 'rcX' string.
> This will allow us to rename the artifact without modifying
> the artifact checksums when we actually release.
> 
> The status of testing the providers by the community is kept here:
> https://github.com/apache/airflow/issues/35845
> 
> You can find packages as well as detailed changelog following the
>> below
> links:
> 
> https://pypi.org/project/apache-airflow-providers-amazon/8.12.0rc1/
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-apache-impala/1.2.1rc1/
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-atlassian-jira/2.3.0rc1/
> 
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/7.10.0rc1/
> 
>> https://pypi.org/project/apache-airflow-providers-common-io/1.1.0rc1/
> 
>> https://pypi.org/project/apache-airflow-providers-common-sql/1.8.1rc1/
> 
>> https://pypi.org/project/apache-airflow-providers-databricks/5.0.1rc1/
> 
>> https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.4.1rc1/
> https://pypi.org/project/apache-airflow-providers-docker/3.8.2rc1/
> 
 
>>> 
>> https://pypi.org/project/apache-airflow-pr

Re: [DISCUSS] Suspend/Remove Plexus provider

[Wei Lee]

+1 for removing it.

Best,
Wei

> On Nov 24, 2023, at 7:17 AM, Jarek Potiuk  wrote:
> 
> Let's remove it.
> 
> On Thu, Nov 23, 2023 at 11:52 PM Andrey Anshin 
> wrote:
> 
>> Greetings everyone!
>> 
>> Not much time has passed since the last discussion about suspension/removal
>> providers 🤣
>> 
>> It is time to discuss Plexus Provider.
>> 
>> During check providers links, I've found that link [1] from the Plexus
>> provider description [2] can not be resolved. The same thing happened with
>> link [3] to the API which are hardcoded into the hook [4]
>> 
>> I had no idea what the provider/service was about and also couldn't found
>> any useful information on the Core Scientific site. However after couple of
>> minute in google I've found that Plexus was acquired by AMD [6] in Sep 2022
>> 
>> With combination above and the fact that there are no any valuable changes
>> since migration from 1.10 to Providers - PlexusHook [4] even use Airflow
>> Variables "email" and "password" instead of Connection, I made
>> conclusion that we could suspend/remove this provider, and no one even
>> noticed that
>> 
>> Any chance that someone has more information about this service? Or maybe I
>> miss something and we need to keep this provider?
>> 
>> [1] https://plexus.corescientific.com/
>> [2]
>> 
>> https://github.com/apache/airflow/blob/99534e47f330ce0efb96402629dda5b2a4f16e8f/airflow/providers/plexus/provider.yaml#L22
>> [3] https://apiplexus.corescientific.com/
>> [4]
>> 
>> https://github.com/apache/airflow/blob/e45bee884068399e7265421511e17fed106ce5b4/airflow/providers/plexus/hooks/plexus.py#L30
>> [5] https://corescientific.com/
>> [6]
>> 
>> https://community.amd.com/t5/discussions/amd-acquires-plexus-a-private-cloud-platform-for-hpc-and-ai/td-p/534613
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [DISCUSS] Suspend/Remove Apache Scoop provider

[Wei Lee]

+1

> On Nov 23, 2023, at 11:30 PM, Phani Kumar  
> wrote:
> 
> +1
> 
> On Thu, Nov 23, 2023 at 8:24 PM Vincent Beck  wrote:
> 
>> +1
>> 
>> On 2023/11/23 13:08:09 Jarek Potiuk wrote:
>>> +1
>>> 
>>> czw., 23 lis 2023, 12:39 użytkownik Aritra Basu <
>> aritrabasu1...@gmail.com>
>>> napisał:
>>> 
 +1 sounds like a good reason to suspend it and eventually remove it.
 
 --
 Regards,
 Aritra Basu
 
 On Thu, Nov 23, 2023, 4:45 PM Bolke de Bruin 
>> wrote:
 
> +1
> 
> Go
> 
> Sent from my iPhone
> 
>> On 23 Nov 2023, at 10:52, Andrey Anshin 
> wrote:
>> 
>> Greetings everyone!
>> 
>> Since we began to actively use the mechanism to suspend/remove
 providers
> I
>> want to start the discussion about suspend and potential remove
>> Apache
>> Scoop [1] provider.
>> 
>> Apache Scoop moved into the Attic in July 2021 [2] due to inactive
>> development [3] and Apache Scoop PMS was terminated.
>> According to the Attic page of Scoop there is no forks of this
>> project
>> 
>> Latest releases :
>> Scoop1 (stable): 1.4.7 was released on 2018-01-24
>> Sqoop2: 1.99.7 was released on 2016-08-08
>> 
>> Perhaps the fact that the product is no longer being developed
>> might be
> not
>> a reason to exclude it from the Airflow providers.
>> In that case it would be nice to have a discussion which would use
>> in
>> further cases of suspension/removal providers
>> 
>> [1] https://sqoop.apache.org/
>> [2] https://attic.apache.org/projects/sqoop.html
>> [3] https://whimsy.apache.org/board/minutes/Sqoop.html
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
> 
> 
 
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> For additional commands, e-mail: dev-h...@airflow.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [DISCUSS] Move FAB auth manager to a new provider

[Wei Lee]

+1 for this moving it. It gives us more flexibility on both the core and 
provider sides.

Best,
Wei

> On Nov 17, 2023, at 9:15 AM, Jarek Potiuk  wrote:
> 
> I am all for it. As we saw already and we see it more in the future -
> moving code of out of Airflow core to provider and having separate
> provider's release cycle and lifecycle is generally beneficial:
> 
> * dependencies can be more decoupled - even if we pin FAB with a particular
> version of provider, our users can freely change provider version
> without updating airflow and the other way round - update airflow without
> changing provider version (thus without changing FAB).
> * we will be able to remove FAB in the future from active maintenance -
> similarly to discussed dask executor we can stop maintaining FAB provider
> in the future (no plans now of course - but it might happen - especially if
> we will implement a way to migrate current USER/ROLE of FAB to something
> else
> 
> I think we've learned already a lot on how to manage lifecycle of providers
> and have a number of tools and processes, so I think it will not be too
> problematic.
> 
> J.
> 
> 
> 
> On Tue, Nov 14, 2023 at 9:46 PM Beck, Vincent 
> wrote:
> 
>> Hello,
>> 
>> I am sending this email to gather feedbacks/concerns on what is going to
>> happen regarding AIP-56 (
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management).
>> The majority of the work related to AIP-56 is now completed. As a summary,
>> all the code related to user authentication and user authorization has been
>> packaged in a new component called "FAB auth manager" (
>> https://github.com/apache/airflow/blob/main/airflow/auth/managers/fab/fab_auth_manager.py).
>> Now, all user authentication and user authorization operations are done in
>> this FAB auth manager. The purpose of having this new component is to be
>> able to plug (and/or create) another "auth manager" if desired.
>> 
>> For simplification reasons, this FAB auth manager is still in core Airflow
>> under "airflow.auth.managers.fab" (
>> https://github.com/apache/airflow/tree/main/airflow/auth/managers/fab)
>> but the ultimate plan is to move the module "airflow.auth.managers.fab" to
>> a new provider. Of course, this new provider would still be installed by
>> default in Airflow and will have no impact for the users.
>> 
>> An issue had been created for that work and a discussion has started but I
>> wanted to increase the audience and potentially get more feedbacks/concerns
>> before actually doing so: https://github.com/apache/airflow/issues/32210.
>> In this issue you will also find the motivations behind moving this code to
>> a new provider.
>> 
>> Cheers,
>> Vincent
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Wei Lee]

+1 for removal if there is no active maintainer on this one

Best,
Wei

> On Nov 17, 2023, at 6:00 PM, Andrey Anshin  wrote:
> 
> +1 for suspend and after a while remove
> 
> There is a small chance that things would improve after we suspended this
> provider.
> 
> However we do not have a lot of statistics, AFAIK (correct me if I'm wrong)
> it is only two cases:
> * 1 provider suspended in the past and restored (Yandex), there is no
> process of removal at that moment.
> * 1 provider suspended and removed (Qubole)
> 
> 
> 
> Best Wishes
> *Andrey Anshin*
> 
> 
> 
> On Fri, 17 Nov 2023 at 12:17, Amogh Desai  wrote:
> 
>> Theres very little incentive in maintaining this if theres no one actively
>> maintaining it.
>> 
>> I am totally for the removal +1
>> 
>> 
>> 
>> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty
>> 
>> wrote:
>> 
>>> +1 for removal
>>> 
>>> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala  wrote:
>>> 
> we would do it branching off at the TAG when the last release
>> happened
 and develop/release a fix from there.
 
 Since it would be possible to release security patches, +1 to remove
>> it.
 
 On Thu 16 Nov 2023 at 21:22, Oliveira, Niko
>> >>> 
 wrote:
 
> If no one comes forward willing to support the executor long term I'm
>>> +1
> for removal.
> 
> 
> From: Vincent Beck 
> Sent: Thursday, November 16, 2023 10:59:40 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend
>> (Remove?)
> Daskexecutor provider
> 
> CAUTION: This email originated from outside of the organization. Do
>> not
> click links or open attachments unless you can confirm the sender and
 know
> the content is safe.
> 
> 
> 
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>>> externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
 pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>> certain
 que
> le contenu ne présente aucun risque.
> 
> 
> 
> +1 for removal
> 
> On 2023/11/16 18:54:15 Jarek Potiuk wrote:
>> More detailed comparison:
>> 
>> apache-airflow 2.7.* ~ 255.000 downloads/day
>> apache-aurflow-provider-dask-executor ~ 900/ day
>> 
>> This means that *apache-airflow-providers-daskexecutor * is
>>> downloaded
 in
>> less of *0.3 %* cases, comparing to *apache-airflow*
>> 
>> I'd say it's negligible usage.
>> 
>> My personal vote would go for immediate removal.
>> 
>> WDYT?
>> 
>> J.
>> 
>> 
>> 
>> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk 
 wrote:
>> 
>>> 
>>> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif 
 wrote:
>>> 
 Now that the code is in its own provider we can check the
>> download
> stats
 of
 the library via Pypi stats.
 
 
>>> Good point but this is only an indication.
>>> 
>>> Currently this is only for Airflow 2.7+ and it is a bit difficult
>>> to
>>> compare those. The "fixed" numbers are misleading (we still do
>> not
 know
>>> where the 400.000 downloads/day jump to almost 1Million / day
>> came
 from
>>> mid-October.
>>> The ratio is now 1000/day (daskexecutor): 800.000 day (airflow)
>> ->
> ~0.1
>>> %. But those are only downloads - and not whether it's used -
>>> likely
> there
>>> are a number of cases where CI pulls "all" extras.
>>> 
>>> J.
>>> 
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
> 
> 
 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [PrOPOSAL] Change default docker image to point to "latest supported"

[Wei Lee]

Agreed, as long as users can still use different versions through tags, there 
are no drawbacks or incompatibilities with this great idea!

Best,
Wei

> On Nov 17, 2023, at 1:39 PM, Aritra Basu  wrote:
> 
> Agreed, moving to latest by default sounds like a fine idea. I don't see
> any drawbacks to it and seems like a good enough time as any to make the
> switch with 2.8.0.
> 
> --
> Regards,
> Aritra Basu
> 
> On Fri, Nov 17, 2023, 12:33 AM Vincent Beck  wrote:
> 
>> I agree, by default we should use the latest python version. Like any
>> package manager, if the user does not explicitly specify a version, the
>> latest should be used. If the user wants to use a lower version, he can
>> always pin it.
>> 
>> On 2023/11/16 12:06:17 Jarek Potiuk wrote:
>>> Hello everyone,
>>> 
>>> Since we are close to the Airflow 2.8.0 release, I would like to propose
>> a
>>> change in the approach for our "default" images.
>>> 
>>> Currently there are few images that are considered as "default", for
>>> example:
>>> 
>>> apache/airflow:latest
>>> apache/airflow:2.7.4
>>> 
>>> Currently (according to our process [1] and user documentation [2]) those
>>> point to the "oldest" python version we support (currently they point to
>>> Python 3.8).
>>> 
>>> There is no particular reason why it is like that, and with Airflow 2.8.0
>>> we have an opportunity to change it and point the default images to
>> "latest
>>> supported" (and keep this version as default for the whole MINOR line of
>>> releases.
>>> 
>>> In the case of Airflow 2.8.* - that would be "Python 3.11" being default
>>> for the whole 2.8.* line unless we manage to get Python 3.12 support in
>> our
>>> CI before we release Airflow 2.8.0, then it would be Python 3.12
>>> 
>>> We do not have any SemVer promises about that. Users can still choose to
>>> use the  "2.8.0-python3.8" tag if they want.
>>> 
>>> Generally going to 2.8 should always be a deliberate action, so we have
>>> chance to explain in the release notes that if they want to stick to the
>>> 2.8 release. So they are not "losing" anything, they can have 100%
>>> compatibility by just choosing a different image in their deployment.
>> This
>>> **might** cause a little hassle when they migrate if they find some
>>> incompatibilities, but generally speaking it's a very straightforward and
>>> simple change - just  adding "-python3.8" to your TAG - whatever
>> deployment
>>> option you have. And our users will have to go through it anyway every
>> time
>>> we drop the old Python version (and this change might be even more costly
>>> as they have no choice then) - so it changes very little, just shifts the
>>> time where they will have to do it.
>>> 
>>> There are benefits of doing it - for both our users and well, environment
>>> as well (and I really mean a positive impact on the "world environment"
>> to
>>> be honest. Maybe a little impact - but with Airflow's popularity, it
>> might
>>> make a (small) difference. Python 3.11 is generally 30% faster than
>>> previous versions and using it by default means that 30% less CPU is
>> being
>>> wasted. Also it will mean actual money savings for our users. Also Python
>>> 3.12 comes with even more performance improvements and keeping up with
>>> those being the "default" is a pretty good idea.
>>> 
>>> I cannot think of any other drawbacks of this change.
>>> 
>>> WDYT?
>>> 
>>> [1] Documented versioning approach:
>>> 
>> https://github.com/apache/airflow#base-os-support-for-reference-airflow-images
>>> [2] User documentation
>>> https://airflow.apache.org/docs/docker-stack/index.html
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>> For additional commands, e-mail: dev-h...@airflow.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [VOTE] Airflow Providers prepared on November 08, 2023

[Wei Lee]

+1, non binding

Tested with our example DAGs and azure changes.

Best,
Wei

> On Nov 9, 2023, at 12:58 PM, Amogh Desai  wrote:
> 
> +1 non binding.
> 
> Tested out some test DAGs, mostly around CNCF provider:
> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/7.9.0rc1/,
> no regression found.
> 
> 
> Thanks & Regards,
> Amogh Desai
> 
> On Thu, Nov 9, 2023 at 6:49 AM Jarek Potiuk  wrote:
> 
>> +1 (binding) - checked sources, signatures, checksums, licences. I had just
>> a little change about verifying the structure of provider docs. I also
>> checked that the scheduled-to-remove qubole provider last release contains
>> the "removal warning" :
>> https://pypi.org/project/apache-airflow-providers-qubole/3.4.3rc1/
>> 
>> On Wed, Nov 8, 2023 at 10:10 PM Elad Kalif  wrote:
>> 
>>> Hey all,
>>> 
>>> I have just cut the new wave Airflow Providers packages. This email is
>>> calling a vote on the release,
>>> which will last for 72 hours - which means that it will end on November
>> 11,
>>> 2023 21:10 PM UTC and until 3 binding +1 votes have been received.
>>> 
>>> 
>>> Consider this my (binding) +1.
>>> 
>>> Airflow Providers are available at:
>>> https://dist.apache.org/repos/dist/dev/airflow/providers/
>>> 
>>> *apache-airflow-providers--*.tar.gz* are the binary
>>> Python "sdist" release - they are also official "sources" for the
>> provider
>>> packages.
>>> 
>>> *apache_airflow_providers_-*.whl are the binary
>>> Python "wheel" release.
>>> 
>>> The test procedure for PMC members is described in
>>> 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
>>> 
>>> The test procedure for and Contributors who would like to test this RC is
>>> described in:
>>> 
>>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors
>>> 
>>> 
>>> Public keys are available at:
>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>> 
>>> Please vote accordingly:
>>> 
>>> [ ] +1 approve
>>> [ ] +0 no opinion
>>> [ ] -1 disapprove with the reason
>>> 
>>> 
>>> Only votes from PMC members are binding, but members of the community are
>>> encouraged to test the release and vote with "(non-binding)".
>>> 
>>> Please note that the version number excludes the 'rcX' string.
>>> This will allow us to rename the artifact without modifying
>>> the artifact checksums when we actually release.
>>> 
>>> The status of testing the providers by the community is kept here:
>>> https://github.com/apache/airflow/issues/35540
>>> 
>>> You can find packages as well as detailed changelog following the below
>>> links:
>>> 
>>> https://pypi.org/project/apache-airflow-providers-amazon/8.11.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-apache-spark/4.4.0rc1/
>>> 
>> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/7.9.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-cohere/1.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-common-io/1.0.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-databricks/5.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-discord/3.4.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-docker/3.8.1rc1/
>>> 
>> https://pypi.org/project/apache-airflow-providers-elasticsearch/5.1.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-ftp/3.6.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-google/10.11.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-http/4.7.0rc1/
>>> 
>> https://pypi.org/project/apache-airflow-providers-microsoft-azure/8.2.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-openai/1.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-openlineage/1.2.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-pgvector/1.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-pinecone/1.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-postgres/5.8.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-qubole/3.4.3rc1/
>>> https://pypi.org/project/apache-airflow-providers-salesforce/5.5.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-slack/8.4.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-snowflake/5.1.1rc1/
>>> https://pypi.org/project/apache-airflow-providers-weaviate/1.0.0rc1/
>>> https://pypi.org/project/apache-airflow-providers-yandex/3.6.0rc1/
>>> 
>>> Cheers,
>>> Elad Kalif
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: [ANNOUNCE] New committer: Jens Scheffler

[Wei Lee]

Congratulations Jens!!

Best,
Wei

> On Nov 8, 2023, at 5:20 AM, Oliveira, Niko  
> wrote:
> 
> Congrats Jens!!
> 
> 
> From: Briana Okyere 
> Sent: Tuesday, November 7, 2023 1:00:35 PM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [ANNOUNCE] New committer: Jens 
> Scheffler
> 
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you can confirm the sender and know the 
> content is safe.
> 
> 
> 
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
> cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
> confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
> contenu ne présente aucun risque.
> 
> 
> 
> Congratulations Jens!
> 
> On Tue, Nov 7, 2023 at 12:59 PM Andrey Anshin 
> wrote:
> 
>> Congrats Jens! 🎉 🏆
>> 
>> 
>> Best Wishes
>> *Andrey Anshin*
>> 
>> 
>> 
>> On Tue, 7 Nov 2023 at 23:33, Utkarsh Sharma
>>  wrote:
>> 
>>> Congratulations Jens! :)
>>> 
>>> Thanks,
>>> Utkarsh Sharma
>>> 
>>> On Wed, Nov 8, 2023 at 1:01 AM Vincent Beck  wrote:
 
 Welcome onboard Jens!
 
 On 2023/11/07 19:24:04 Jarek Potiuk wrote:
> The Project Management Committee (PMC) for Apache Airflow
> has invited Jens Scheffler to become a committer and we are pleased
> to announce that they have accepted.
> 
> Jens has been contributing for a number of months
> he also participated a lot in discussions and decisions
> on many aspects of Airflow but also helps a lot
> our users and contributors on Slack, Github, Discussions
> 
> I am looking forward to what the future holds with Jens
> becoming the committer,
> 
> Congratulations Jens, and welcome onboard!
> 
> Being a committer enables easier contribution to the
> project since there is no need to go via the patch
> submission process. This should enable better productivity.
> A PMC member helps manage and guide the direction of the project.
> 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
 For additional commands, e-mail: dev-h...@airflow.apache.org
 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
>>> For additional commands, e-mail: dev-h...@airflow.apache.org
>>> 
>>> 
>> 

Re: [VOTE] Release Airflow 2.7.3 from 2.7.3rc1

[Wei Lee]

+1 (non-binding)

Best,
Wei

> On Nov 6, 2023, at 12:17 PM, Aritra Basu  wrote:
> 
> +1 (non-binding)
> 
> --
> Regards,
> Aritra Basu
> 
> On Mon, Nov 6, 2023, 9:30 AM Rahul Vats  wrote:
> 
>> +1 (non-binding)
>> 
>> Regards,
>> Rahul Vats
>> 9953794332
>> 
>> 
>> On Sun, 5 Nov 2023 at 10:53, Pankaj Singh 
>> wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> my changes look good to me.
>>> 
>>> On Sat, Nov 4, 2023 at 10:11 PM Hussein Awala  wrote:
>>> 
 +1 (binding) I checked the checksums, the signatures, the licences, the
 sources, and I tested all my changes.
 I also ran some testing dags, and all looks good.
 
>>> 
>> 

Re: [VOTE] October 2023 PR of the Month

[Wei Lee]

+1 to 34729 from me as well. 👍

Best,
Wei

> On Nov 1, 2023, at 4:02 AM, Aritra Basu  wrote:
> 
> +1 to 34729 from me as well. Great work on that
> 
> --
> Regards,
> Aritra Basu
> 
> On Tue, Oct 31, 2023, 11:27 PM Amogh Desai  wrote:
> 
>> +1 for 34729 from me.
>> 
>> Good work 👍
>> 
>> On Tue, Oct 31, 2023, 21:40 Jarek Potiuk  wrote:
>> 
>>> +1 to 34729 as well.
>>> 
>>> On Tue, Oct 31, 2023 at 6:38 PM Constance Martineau
>>>  wrote:
>>> 
 Oh, I'm very sorry. I had forgotten about 34729. Apologies, but I'll be
 changing my vote to that.
 
 On Tue, Oct 31, 2023 at 1:08 PM Hussein Awala 
>> wrote:
 
> I vote for 34729 ,
>> which
> implemented Airflow FileSystem.
> 
> On Tue, Oct 31, 2023 at 6:00 PM Jed Cunningham <
>>> jedcunning...@apache.org
> 
> wrote:
> 
>> The new OpenSearch provider gets my vote - 34705.
>> 
> 
 
 
 --
 
 Constance Martineau
 Senior Product Manager
 
 Email: consta...@astronomer.io
 Time zone: US Eastern (EST UTC-5 / EDT UTC-4)
 
 
 
 
>>> 
>>