Re: Call with Nielsen team demoing their DAG debugging feature

[Ephraim Anierobi]

Hi Jarek,

Awesome!! I’m interested to join too

Ephraim

On Mon, 10 Jun 2024 at 23:41, Mehta, Shubham 
wrote:

> This is great, thanks for working with them to share this with the
> community. Interested to join as well.
>
> Shubham
>
> On 2024-06-07, 11:56 PM, "Jarek Potiuk"  ja...@potiuk.com>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
> Hello here,
>
>
> At PyCon US I met a few people from Nielsen who had developed internally
> tooling for IDE/Python debugger integrated debugging of Airflow DAGs.
>
>
> They are thrilled with the opportunity of sharing what they've done and
> possibly maybe even bringing it to Airflow. As one of the Airflow 3
> workstreams I am particularly interested in is to "*Simplify the learning
> curve*" [1] - this sounds pretty interesting in this context.
>
>
> I will have a call with them next week - immediately after the Airflow 3
> dev call (Thu, 13th of June, 6pm CEST), where they will demo what they have
> - so if you would like to join it - let me know, I will invite you to the
> call.
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024
> >
>
>
>
>
>
> J.
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>

Re: Call with Nielsen team demoing their DAG debugging feature

[Mehta, Shubham]

This is great, thanks for working with them to share this with the community. 
Interested to join as well.

Shubham

On 2024-06-07, 11:56 PM, "Jarek Potiuk" mailto:ja...@potiuk.com>> wrote:


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.






AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne 
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas 
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le 
contenu ne présente aucun risque.






Hello here,


At PyCon US I met a few people from Nielsen who had developed internally
tooling for IDE/Python debugger integrated debugging of Airflow DAGs.


They are thrilled with the opportunity of sharing what they've done and
possibly maybe even bringing it to Airflow. As one of the Airflow 3
workstreams I am particularly interested in is to "*Simplify the learning
curve*" [1] - this sounds pretty interesting in this context.


I will have a call with them next week - immediately after the Airflow 3
dev call (Thu, 13th of June, 6pm CEST), where they will demo what they have
- so if you would like to join it - let me know, I will invite you to the
call.


[1]
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024
 






J.




-
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Re: Call with Nielsen team demoing their DAG debugging feature

[Wei Lee]

Hi Jarek,

Love to hear about it! But it’s way too late for my time. 😞 Would it be 
possible for us to record it?

Best,
Wei

> On Jun 11, 2024, at 7:12 AM, Julian LaNeve  
> wrote:
> 
> +1 I'd love to hear about it but unfortunately can't make the meeting!
> 
> --
> 
> *Julian LaNeve*
> CTO
> 
> Email: jul...@astronomer.io 
> ( jul...@astronomer.io  ) Mobile: 330 509 5792
> 
> On Mon, Jun 10, 2024 at 3:05 PM, Constance Martineau < 
> consta...@astronomer.io.invalid  > 
> wrote:
> 
>> 
>> 
>> 
>> Hello again,
>> 
>> 
>> 
>> Given all the enthusiasm - assuming Nielsen is ok with this - what if
>> someone recorded the meeting so that it could be shared with those that
>> are interested?
>> 
>> 
>> 
>> Constance
>> 
>> 
>> 
>> On Mon, Jun 10 , 2024 at 1:48 PM Constance Martineau < constance@ astronomer.
>> io ( consta...@astronomer.io  ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Hi Jarek,
>>> 
>>> 
>>> 
>>> Same :)
>>> 
>>> 
>>> 
>>> Thanks,
>>> Constance
>>> 
>>> 
>>> 
>>> On Mon, Jun 10 , 2024 at 9:57 AM Amogh Desai < amoghdesai. oss@ gmail. com
>>> ( amoghdesai@gmail.com  ) > wrote:
>>> 
>>> 
 
 
 Hello Jarek,
 
 
 
 Please add me to the invite as well.
 
 
 
 Thanks & Regards,
 Amogh Desai
 
 
 
 On Mon, Jun 10 , 2024 at 11:22 AM Abhishek Bhakat
 < abhishek. bhakat@ astronomer. io. invalid (
 abhishek.bha...@astronomer.io.invalid 
  ) > wrote:
 
 
> 
> 
> Hi Jarek,
> 
> 
> 
> I would also like to join as well, please.
> 
> 
> 
> Thanks,
> Avi
> 
> 
> 
> On Sat, Jun 8 , 2024 at 3:32 PM Buğra Öztürk < ozturkbugra93@ gmail. com (
> ozturkbugr...@gmail.com  ) > wrote:
> 
> 
>> 
>> 
>> Hello Jarek,
>> 
>> 
>> 
>> Thanks for sharing! It sounds very interesting. I would like to join.
>> 
>> 
> 
> 
> 
> Could
> 
> 
>> 
>> 
>> you please forward to me as well?
>> 
>> 
>> 
>> Thanks!
>> 
>> 
>> 
>> On Sat , 8 Jun 2024 , 17:28 Jed Cunningham, < jedcunningham@ apache. org 
>> (
>> jedcunning...@apache.org  ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Interesting. Can you forward to me as well Jarek? Thanks!

Re: Call with Nielsen team demoing their DAG debugging feature

[Julian LaNeve]

+1 I'd love to hear about it but unfortunately can't make the meeting!

--

*Julian LaNeve*
CTO

Email: jul...@astronomer.io
( jul...@astronomer.io ) Mobile: 330 509 5792

On Mon, Jun 10, 2024 at 3:05 PM, Constance Martineau < 
consta...@astronomer.io.invalid > wrote:

> 
> 
> 
> Hello again,
> 
> 
> 
> Given all the enthusiasm - assuming Nielsen is ok with this - what if
> someone recorded the meeting so that it could be shared with those that
> are interested?
> 
> 
> 
> Constance
> 
> 
> 
> On Mon, Jun 10 , 2024 at 1:48 PM Constance Martineau < constance@ astronomer.
> io ( consta...@astronomer.io ) > wrote:
> 
> 
>> 
>> 
>> Hi Jarek,
>> 
>> 
>> 
>> Same :)
>> 
>> 
>> 
>> Thanks,
>> Constance
>> 
>> 
>> 
>> On Mon, Jun 10 , 2024 at 9:57 AM Amogh Desai < amoghdesai. oss@ gmail. com
>> ( amoghdesai@gmail.com ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Hello Jarek,
>>> 
>>> 
>>> 
>>> Please add me to the invite as well.
>>> 
>>> 
>>> 
>>> Thanks & Regards,
>>> Amogh Desai
>>> 
>>> 
>>> 
>>> On Mon, Jun 10 , 2024 at 11:22 AM Abhishek Bhakat
>>> < abhishek. bhakat@ astronomer. io. invalid (
>>> abhishek.bha...@astronomer.io.invalid ) > wrote:
>>> 
>>> 
 
 
 Hi Jarek,
 
 
 
 I would also like to join as well, please.
 
 
 
 Thanks,
 Avi
 
 
 
 On Sat, Jun 8 , 2024 at 3:32 PM Buğra Öztürk < ozturkbugra93@ gmail. com (
 ozturkbugr...@gmail.com ) > wrote:
 
 
> 
> 
> Hello Jarek,
> 
> 
> 
> Thanks for sharing! It sounds very interesting. I would like to join.
> 
> 
 
 
 
 Could
 
 
> 
> 
> you please forward to me as well?
> 
> 
> 
> Thanks!
> 
> 
> 
> On Sat , 8 Jun 2024 , 17:28 Jed Cunningham, < jedcunningham@ apache. org (
> jedcunning...@apache.org ) > wrote:
> 
> 
>> 
>> 
>> Interesting. Can you forward to me as well Jarek? Thanks!
>> 
>> 
> 
> 
 
 
>>> 
>>> 
>> 
>> 
> 
> 
>

Re: Call with Nielsen team demoing their DAG debugging feature

[Constance Martineau]

Hello again,

Given all the enthusiasm - assuming Nielsen is ok with this - what if
someone recorded the meeting so that it could be shared with those that are
interested?

Constance

On Mon, Jun 10, 2024 at 1:48 PM Constance Martineau 
wrote:

> Hi Jarek,
>
> Same :)
>
> Thanks,
> Constance
>
> On Mon, Jun 10, 2024 at 9:57 AM Amogh Desai 
> wrote:
>
>> Hello Jarek,
>>
>> Please add me to the invite as well.
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Mon, Jun 10, 2024 at 11:22 AM Abhishek Bhakat
>>  wrote:
>>
>> > Hi Jarek,
>> >
>> > I would also like to join as well, please.
>> >
>> > Thanks,
>> > Avi
>> >
>> > On Sat, Jun 8, 2024 at 3:32 PM Buğra Öztürk 
>> > wrote:
>> >
>> > > Hello Jarek,
>> > >
>> > > Thanks for sharing! It sounds very interesting. I would like to join.
>> > Could
>> > > you please forward to me as well?
>> > >
>> > > Thanks!
>> > >
>> > > On Sat, 8 Jun 2024, 17:28 Jed Cunningham, 
>> > > wrote:
>> > >
>> > > > Interesting. Can you forward to me as well Jarek? Thanks!
>> > > >
>> > >
>> >
>>
>

Re: Call with Nielsen team demoing their DAG debugging feature

[Constance Martineau]

Hi Jarek,

Same :)

Thanks,
Constance

On Mon, Jun 10, 2024 at 9:57 AM Amogh Desai 
wrote:

> Hello Jarek,
>
> Please add me to the invite as well.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Mon, Jun 10, 2024 at 11:22 AM Abhishek Bhakat
>  wrote:
>
> > Hi Jarek,
> >
> > I would also like to join as well, please.
> >
> > Thanks,
> > Avi
> >
> > On Sat, Jun 8, 2024 at 3:32 PM Buğra Öztürk 
> > wrote:
> >
> > > Hello Jarek,
> > >
> > > Thanks for sharing! It sounds very interesting. I would like to join.
> > Could
> > > you please forward to me as well?
> > >
> > > Thanks!
> > >
> > > On Sat, 8 Jun 2024, 17:28 Jed Cunningham, 
> > > wrote:
> > >
> > > > Interesting. Can you forward to me as well Jarek? Thanks!
> > > >
> > >
> >
>

Re: [Meeting Notes] Airflow 3.0 Dev call - 4 June 2024

[Amogh Desai]

Amazing summary @Kaxil Naik !

I reviewed the discussion points and I am happy with them so far.
I only have a few minor comments which I have posted on confluence.

Looking forward to attending these calls in future.

Thanks & Regards,
Amogh Desai


On Sat, Jun 8, 2024 at 12:15 PM Jarek Potiuk  wrote:

> Very good summary Kaxil. Reviewed it and it looks like all things we
> discussed are captured well. Also I am really happy that we are starting to
> formalize and converge on the set of basic principles - they will help us
> to focus on particular "streams" while keeping those in mind when we are
> going to more and more details in each of the "streams" independently.
>
> J.
>
>
> On Sat, Jun 8, 2024 at 4:17 AM Kaxil Naik  wrote:
>
> > Hey all,
> >
> > I have updated our meeting notes document to summarize the discussion
> from
> > our dev call for Airflow 3.0 on 4th June.
> >
> > Link:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024
> >
> > To all those who attended, can you please double-check and add if I have
> > missed anything?
> >
> > To all those who didn't join, if you disagree with anything in the
> Summary,
> > please voice your opinion.
> >
> > I will send a separate email for the agenda for the next meeting on 13
> > June.
> >
> > Regards,
> > Kaxil
> >
> > --
> >
> > Including the Summary here too (might break formatting):
> >
> > The following *principles* were agreed upon to drive Airflow 3
> development:
> > 1) For the features that require breaking changes, ship Airflow 3 with
> the
> > foundational code to allow for iterative development, optimizing for
> speed
> > and a quicker feedback cycle.
> >   - Additional features that do not require breaking changes can be
> > included in minor releases such as 3.1, 3.2, and beyond since we follow
> > SemVer.
> >   - Discussion points:
> > - Examples:
> >   - Add the hook points with a few references (fetcher for GCS, S3,
> > Git) for DAG Versioning’s Execution AIP (AIP-66).
> >   - Removing dependency on FAB in Core: The new plugin framework
> might
> > support only a few functionalities, which are then built upon later.
> >   - For the multi-language AIP, start with only Python + one more
> > language in AF 3.0 and then 3.1 and later minor versions can have more
> > support for languages like Typescript.
> > - Identify users who would be willing to give feedback during dev and
> > beta snapshots. The Astronomer, MWAA, and GCC teams can help identify
> > these.
> >
> > 2) Ensure a smoother migration path between Airflow 2 and 3, particularly
> > for DAG authors using the existing official Airflow providers.
> >   - Directionally, the time required to update DAGs should be measured in
> > hours, not days or months.
> >   - Action Items:
> > - Update the AIP template to ask for the level of effort (manual and
> > automated) needed for the users to adapt to the breaking changes. This
> > should include high-level details on what could go in the upgrade
> > utilities. This will help the AIP authors consciously think through the
> > migration efforts. (Kaxil Naik)
> > - For AIPs, be explicit about what’s for AF 3.0 and what’s for the
> next
> > minor releases (3.1, 3.2, ...).
> >   - Action Items:
> > - Complete the housekeeping of the AIPs (Kaxil Naik):
> >   - Move the AIPs that we don’t plan to work on in Abandoned
> state.
> >   - Add labels if an AIP is for Airflow 2, 3.0 or >= 3.1. These
> > labels will be used via Macros to auto-populate the tables in Airflow
> > Improvement Proposals, making it a good page for Roadmap items.
> >
> > 3) Build features that solidify Airflow as the modern Orchestrator that
> has
> > state-of-the-art support for Data, AI & ML workloads.
> >   - This includes enhancing scalability, performance, and
> enterprise-level
> > security, adhering to the principle of least privilege.
> >   - Making Airflow aware of what’s happening in the task to provide
> better
> > auditability, lineage & observability.
> >
> > 4) Set up the codebase for the next 3-5 years.
> >   - Reducing the matrix of supported combinations for reducing complexity
> > in testing & development. E.g., Remove MySQL support to reduce the test
> > matrix.
> >   - Simplifying codebase & standardize architecture (e.g., consolidating
> > serialization methods).
> >   - Remove deprecations.
> >   - Consider optimizing development workflows (core Airflow vs. provider,
> > chart development).
> >
> > 5) Simplify the Learning Curve for new Airflow users.
> >   - Decrease the time from running the install command to first DAG.
> >   - Decrease the boilerplate code needed to run the first DAG/task.
> >   - Action Items:
> > - Write a first draft of a doc on the different personas of Airflow
> > users & current state. This will help tailor the learning curve via docs
> &
> > tutorials as well as tailor features t

Re: [DISCUSS] common.compat provider (WAS: Common.util provider?)

[Amogh Desai]

I like the idea of having a common.compat provider too. (common.util was
actually kinda confusing)

* when providers get >= airflow 2.10 - we change them to import from
> `airflow.openlineage` rather than from "airflow.providers.common.compat".
>
I am not so sure why this is the case. Can you elaborate?


Thanks & Regards,
Amogh Desai


On Mon, Jun 10, 2024 at 4:15 PM Jakub Dardziński  wrote:

> As the author of https://github.com/apache/airflow/pull/39530 I love the
> idea.
>
> * when providers get >= airflow 2.10 - we change them to import from
> > `airflow.openlineage` rather than from "airflow.providers.common.compat".
> >
> What's the reasoning behind that? How would Airflow core release impact
> providers dependencies?
>
> pon., 10 cze 2024 o 11:08 Maciej Obuchowski 
> napisał(a):
>
> > I think it's a good solution.
> > The only known problem with that idea is that the common code has to live
> > "forever" - as long as someone can use the older providers (or older
> > Airflow version).
> > The solution would be to introduce some explicit deprecation or
> versioning
> > for provider dependencies - but that's not really possible due to lack of
> > constraints
> > for optional dependencies.
> >
> > sob., 8 cze 2024 o 22:00 Jarek Potiuk  napisał(a):
> >
> > > I have an idea about that one, and probably that one will fulfill the
> > > "polyfill" approach discussed earlier.
> > >
> > > I think we should not name the provider "common.util" but
> > "common.compat" -
> > > because all the code that we need to put there is really about keeping
> > > compatibility.
> > >
> > > For example look here https://github.com/apache/airflow/pull/39530
> > >
> > > We have a need to have a "compatibility" code somewhere that a number
> of
> > > providers could use in case we want to keep some backwards
> compatibility.
> > >
> > > So having a "common.compat" provider would likely nicely full-fill the
> > > polyfill approach - It should only contain the code that we aim to keep
> > > backwards compatibility
> > >
> > > Example for https://github.com/apache/airflow/pull/39530
> > >
> > > * we add the complex compatibility code (see
> > > https://github.com/apache/airflow/pull/39530#issuecomment-2145670785)
> in
> > > the "common.compat" provider - and to airflow.openlineage in this case
> > > * we import it from there in all providers that need it (this will
> > > automatically add dependency)
> > > * when providers get >= airflow 2.10 - we change them to import from
> > > `airflow.openlineage` rather than from
> "airflow.providers.common.compat".
> > >
> > > We could apply similar approach for other "compatibility" code
> > >
> > > J.
> > >
> > >
> > >
> > >
> > > On Thu, Apr 11, 2024 at 10:22 AM Jarek Potiuk 
> wrote:
> > >
> > > > Any other  ideas or suggestions here? Can someone explain how the
> > > > "polypill" approach would look like, maybe? How do we imagine this
> > > working?
> > > >
> > > > Just to continue this discussion - another example.
> > > >
> > > > Small thing that David wanted to add for changes in some sql
> providers:
> > > >
> > > > @contextmanager
> > > > def suppress_and_warn(*exceptions: type[BaseException]):
> > > > """Context manager that suppresses the given exceptions and logs
> a
> > > > warning message."""
> > > > try:
> > > > yield
> > > > except exceptions as e:
> > > > warnings.warn(f"Exception suppressed:
> > > > {e}\n{traceback.format_exc()}", category=UserWarning)
> > > >
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/pull/38707/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R115
> > > >
> > > > This is a small thing - but adding it in `airflow` is problematic -
> > > > because it will only be released in 1.10, so we cannot use it in
> > > providers
> > > > if we do.
> > > > Currently - since it is used in sql providers, I suggested using
> > > > `common.sql` for that code (and add >= 1.12 for common-sql-providers
> > for
> > > > those providers that use it). And I will write a separate email
> about a
> > > > proposed versioning approach there.
> > > >
> > > > Do we have a good proposal on how we can solve similar things in the
> > > > future?
> > > > Do we want it at all? It has some challenges - yes it DRY's the code
> > but
> > > > it also introduces coupling.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Sun, Mar 10, 2024 at 6:21 PM Jarek Potiuk 
> wrote:
> > > >
> > > >> Coming back to it - what about the "polypill" :)? What's different
> vs
> > > the
> > > >> "common.sql" way of doing it ? How do we think it can work ?
> > > >>
> > > >> On Thu, Feb 22, 2024 at 1:58 PM Jarek Potiuk 
> > wrote:
> > > >>
> > > >>> > The symbolic link approach seems to disregard all the external
> > > >>> providers, unless I misunderstand it.
> > > >>>
> > > >>> Not really. It just does not make it easy for the external
> providers
> > to
> > > >>> use it "fast".  They can still - if they want to just manually copy
> > >

Re: Call with Nielsen team demoing their DAG debugging feature

[Amogh Desai]

Hello Jarek,

Please add me to the invite as well.

Thanks & Regards,
Amogh Desai


On Mon, Jun 10, 2024 at 11:22 AM Abhishek Bhakat
 wrote:

> Hi Jarek,
>
> I would also like to join as well, please.
>
> Thanks,
> Avi
>
> On Sat, Jun 8, 2024 at 3:32 PM Buğra Öztürk 
> wrote:
>
> > Hello Jarek,
> >
> > Thanks for sharing! It sounds very interesting. I would like to join.
> Could
> > you please forward to me as well?
> >
> > Thanks!
> >
> > On Sat, 8 Jun 2024, 17:28 Jed Cunningham, 
> > wrote:
> >
> > > Interesting. Can you forward to me as well Jarek? Thanks!
> > >
> >
>

Re: [PROPOSAL] Automated managemenet of lower-bounds of airflow dependencies

[Amogh Desai]

Great work!

Thanks & Regards,
Amogh Desai


On Tue, Jun 4, 2024 at 10:42 AM Jarek Potiuk  wrote:

> And ... merged :). We should now have very nice automated upgrading of
> lower-bound constraints for Airflow and Providers working for all PRs. If
> there will be any problems with your PRs related to that (failing
> "LowestDeps" job on CI) - ping me on Slack. I might not be very responsive
> - I am at "Community over Code EU" conference and leading data engineering
> track but I will keep an eye on those in case there are some teething
> problems.
>
> J,
>
> On Mon, Jun 3, 2024 at 3:14 PM Jarek Potiuk  wrote:
>
> > Thanks for that work - I'm sure it will find many bugs before they are
> >> found by users :)
> >>
> >
> > Yep. That's the main reason we want to do it :)
> >
> > BTW. I think I am about to complete iterating on something that Jens
> > noticed - i.e. that we need to expand the concept to test it on all
> Python
> > versions and I hope we will be able to merge it really soon :).
> >
>

Re: [ANNOUNCE] Podcast Launch- The Data Flowcast: Mastering Airflow for Data Engineering & AI

[Amogh Desai]

This is very cool!

Thanks & Regards,
Amogh Desai


On Sat, Jun 1, 2024 at 4:44 PM Jarek Potiuk  wrote:

> Cool
>
> On Thu, May 30, 2024 at 7:02 PM Briana Okyere
>  wrote:
>
> > Hey All,
> >
> > Very excited to announce the relaunch of the Airflow Podcast, now titled
> > "the relaunch of our podcast, now titled "The Data Flowcast: Mastering
> > Airflow for Data Engineering & AI."
> >
> > This podcast is specially designed for the Apache Airflow community and
> > aims to share invaluable insights, useful tips, and engaging discussions
> > about the current and future trends of Airflow.
> >
> > Our first episode features a discussion with Alexander Booth, Director of
> > R&D at The Texas Rangers on how they powered a World Series victory with
> > Airflow. <
> >
> >
> https://open.spotify.com/episode/1wduswN7sZSebUVL6Y5hMH/?utm_source=slack&utm_medium=organicsocial&utm_campaign=podcastlaunch
> > >
> >
> > Listen/Watch on Spotify, Apple Podcasts, and YouTube!
> >
> > Spotify: 
> >
> > Apple: <
> >
> >
> https://podcasts.apple.com/us/podcast/the-data-flowcast-mastering-airflow-for-data/id1337349579
> > >
> >
> > YouTube: <
> > https://www.youtube.com/playlist?list=PLCi-q9vYo4x80aQhkskTqKmLjOPebum_B
> >
> >
> > Webpage: 
> >
> > --
> > Briana Okyere
> > Community Manager
> > Astronomer
> >
>

[LAZY CONSENSUS] Add YDB provider

[horror]

Hello everyone.

As discussed in
https://lists.apache.org/thread/0sdjmnzjghtfwj25mmqcfr1sco5q5hnq
 this
email calls for lazy consensus on adding of the YDB provider.
Initial PR with new code: https://github.com/apache/airflow/pull/39996 .
There is no need to respond to it, if there will be no objections, lazy
consensus will be reached on Thursday, 13th June.

With best regards,
Sergey

Re: [DISCUSS] common.compat provider (WAS: Common.util provider?)

[Jakub Dardziński]

As the author of https://github.com/apache/airflow/pull/39530 I love the
idea.

* when providers get >= airflow 2.10 - we change them to import from
> `airflow.openlineage` rather than from "airflow.providers.common.compat".
>
What's the reasoning behind that? How would Airflow core release impact
providers dependencies?

pon., 10 cze 2024 o 11:08 Maciej Obuchowski 
napisał(a):

> I think it's a good solution.
> The only known problem with that idea is that the common code has to live
> "forever" - as long as someone can use the older providers (or older
> Airflow version).
> The solution would be to introduce some explicit deprecation or versioning
> for provider dependencies - but that's not really possible due to lack of
> constraints
> for optional dependencies.
>
> sob., 8 cze 2024 o 22:00 Jarek Potiuk  napisał(a):
>
> > I have an idea about that one, and probably that one will fulfill the
> > "polyfill" approach discussed earlier.
> >
> > I think we should not name the provider "common.util" but
> "common.compat" -
> > because all the code that we need to put there is really about keeping
> > compatibility.
> >
> > For example look here https://github.com/apache/airflow/pull/39530
> >
> > We have a need to have a "compatibility" code somewhere that a number of
> > providers could use in case we want to keep some backwards compatibility.
> >
> > So having a "common.compat" provider would likely nicely full-fill the
> > polyfill approach - It should only contain the code that we aim to keep
> > backwards compatibility
> >
> > Example for https://github.com/apache/airflow/pull/39530
> >
> > * we add the complex compatibility code (see
> > https://github.com/apache/airflow/pull/39530#issuecomment-2145670785) in
> > the "common.compat" provider - and to airflow.openlineage in this case
> > * we import it from there in all providers that need it (this will
> > automatically add dependency)
> > * when providers get >= airflow 2.10 - we change them to import from
> > `airflow.openlineage` rather than from "airflow.providers.common.compat".
> >
> > We could apply similar approach for other "compatibility" code
> >
> > J.
> >
> >
> >
> >
> > On Thu, Apr 11, 2024 at 10:22 AM Jarek Potiuk  wrote:
> >
> > > Any other  ideas or suggestions here? Can someone explain how the
> > > "polypill" approach would look like, maybe? How do we imagine this
> > working?
> > >
> > > Just to continue this discussion - another example.
> > >
> > > Small thing that David wanted to add for changes in some sql providers:
> > >
> > > @contextmanager
> > > def suppress_and_warn(*exceptions: type[BaseException]):
> > > """Context manager that suppresses the given exceptions and logs a
> > > warning message."""
> > > try:
> > > yield
> > > except exceptions as e:
> > > warnings.warn(f"Exception suppressed:
> > > {e}\n{traceback.format_exc()}", category=UserWarning)
> > >
> > >
> > >
> > >
> >
> https://github.com/apache/airflow/pull/38707/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R115
> > >
> > > This is a small thing - but adding it in `airflow` is problematic -
> > > because it will only be released in 1.10, so we cannot use it in
> > providers
> > > if we do.
> > > Currently - since it is used in sql providers, I suggested using
> > > `common.sql` for that code (and add >= 1.12 for common-sql-providers
> for
> > > those providers that use it). And I will write a separate email about a
> > > proposed versioning approach there.
> > >
> > > Do we have a good proposal on how we can solve similar things in the
> > > future?
> > > Do we want it at all? It has some challenges - yes it DRY's the code
> but
> > > it also introduces coupling.
> > >
> > > J.
> > >
> > >
> > > On Sun, Mar 10, 2024 at 6:21 PM Jarek Potiuk  wrote:
> > >
> > >> Coming back to it - what about the "polypill" :)? What's different vs
> > the
> > >> "common.sql" way of doing it ? How do we think it can work ?
> > >>
> > >> On Thu, Feb 22, 2024 at 1:58 PM Jarek Potiuk 
> wrote:
> > >>
> > >>> > The symbolic link approach seems to disregard all the external
> > >>> providers, unless I misunderstand it.
> > >>>
> > >>> Not really. It just does not make it easy for the external providers
> to
> > >>> use it "fast".  They can still - if they want to just manually copy
> > those
> > >>> utils from the latest version of Airflow if they want to use it.
> > Almost by
> > >>> definition, those will be small, independent modules that can be
> simply
> > >>> copied as needed by whoever releases external providers - and they
> are
> > also
> > >>> free to copy any older version if they want. That is a nice feature
> > that
> > >>> makes them fully decoupled from the version of Airflow they are
> > installed
> > >>> in (same as community providers). Or - if they want they can just
> > import
> > >>> them from "airflow.provider_utils" - but then they have to add >=
> > Airflow
> > >>> 2.9 if that util appeared in Airflow 2.9

[ANNOUNCE] Apache Airflow 2.9.2 Released

[Utkarsh Sharma]

Dear Airflow community,

I'm happy to announce that Airflow 2.9.2 was just released.

The released sources and packages can be downloaded via
https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-sources.html

Other installation methods are described in
https://airflow.apache.org/docs/apache-airflow/stable/installation/

We also made this version available on PyPI for convenience:
`pip install apache-airflow`
https://pypi.org/project/apache-airflow/2.9.2/

The documentation is available at:
https://airflow.apache.org/docs/apache-airflow/2.9.2/

Find the release notes here for more details:
https://airflow.apache.org/docs/apache-airflow/2.9.2/release_notes.html

Container images are published at:
https://hub.docker.com/r/apache/airflow/tags/?page=1&name=2.9.2

Cheers,
Utkarsh

Re: [DISCUSS] common.compat provider (WAS: Common.util provider?)

[Maciej Obuchowski]

I think it's a good solution.
The only known problem with that idea is that the common code has to live
"forever" - as long as someone can use the older providers (or older
Airflow version).
The solution would be to introduce some explicit deprecation or versioning
for provider dependencies - but that's not really possible due to lack of
constraints
for optional dependencies.

sob., 8 cze 2024 o 22:00 Jarek Potiuk  napisał(a):

> I have an idea about that one, and probably that one will fulfill the
> "polyfill" approach discussed earlier.
>
> I think we should not name the provider "common.util" but "common.compat" -
> because all the code that we need to put there is really about keeping
> compatibility.
>
> For example look here https://github.com/apache/airflow/pull/39530
>
> We have a need to have a "compatibility" code somewhere that a number of
> providers could use in case we want to keep some backwards compatibility.
>
> So having a "common.compat" provider would likely nicely full-fill the
> polyfill approach - It should only contain the code that we aim to keep
> backwards compatibility
>
> Example for https://github.com/apache/airflow/pull/39530
>
> * we add the complex compatibility code (see
> https://github.com/apache/airflow/pull/39530#issuecomment-2145670785) in
> the "common.compat" provider - and to airflow.openlineage in this case
> * we import it from there in all providers that need it (this will
> automatically add dependency)
> * when providers get >= airflow 2.10 - we change them to import from
> `airflow.openlineage` rather than from "airflow.providers.common.compat".
>
> We could apply similar approach for other "compatibility" code
>
> J.
>
>
>
>
> On Thu, Apr 11, 2024 at 10:22 AM Jarek Potiuk  wrote:
>
> > Any other  ideas or suggestions here? Can someone explain how the
> > "polypill" approach would look like, maybe? How do we imagine this
> working?
> >
> > Just to continue this discussion - another example.
> >
> > Small thing that David wanted to add for changes in some sql providers:
> >
> > @contextmanager
> > def suppress_and_warn(*exceptions: type[BaseException]):
> > """Context manager that suppresses the given exceptions and logs a
> > warning message."""
> > try:
> > yield
> > except exceptions as e:
> > warnings.warn(f"Exception suppressed:
> > {e}\n{traceback.format_exc()}", category=UserWarning)
> >
> >
> >
> >
> https://github.com/apache/airflow/pull/38707/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R115
> >
> > This is a small thing - but adding it in `airflow` is problematic -
> > because it will only be released in 1.10, so we cannot use it in
> providers
> > if we do.
> > Currently - since it is used in sql providers, I suggested using
> > `common.sql` for that code (and add >= 1.12 for common-sql-providers for
> > those providers that use it). And I will write a separate email about a
> > proposed versioning approach there.
> >
> > Do we have a good proposal on how we can solve similar things in the
> > future?
> > Do we want it at all? It has some challenges - yes it DRY's the code but
> > it also introduces coupling.
> >
> > J.
> >
> >
> > On Sun, Mar 10, 2024 at 6:21 PM Jarek Potiuk  wrote:
> >
> >> Coming back to it - what about the "polypill" :)? What's different vs
> the
> >> "common.sql" way of doing it ? How do we think it can work ?
> >>
> >> On Thu, Feb 22, 2024 at 1:58 PM Jarek Potiuk  wrote:
> >>
> >>> > The symbolic link approach seems to disregard all the external
> >>> providers, unless I misunderstand it.
> >>>
> >>> Not really. It just does not make it easy for the external providers to
> >>> use it "fast".  They can still - if they want to just manually copy
> those
> >>> utils from the latest version of Airflow if they want to use it.
> Almost by
> >>> definition, those will be small, independent modules that can be simply
> >>> copied as needed by whoever releases external providers - and they are
> also
> >>> free to copy any older version if they want. That is a nice feature
> that
> >>> makes them fully decoupled from the version of Airflow they are
> installed
> >>> in (same as community providers). Or - if they want they can just
> import
> >>> them from "airflow.provider_utils" - but then they have to add >=
> Airflow
> >>> 2.9 if that util appeared in Airflow 2.9 (which is the main reason we
> want
> >>> to use symbolic links - because due to our policies and promises, we
> do not
> >>> want community providers to depend on latest version of Airflow in vast
> >>> majority of cases.
> >>>
> >>> So this approach is also fully usable by external providers, but it
> >>> requires some manual effort to copy the modules to their providers.
> >>>
> >>> > I like the polypill idea. A backport provider that brings new
> >>> interfaces to providers without the actual functionalities.
> >>>
> >>> I would love to hear more about this, I think the "common.util" was
> >>> exactly the kind o

[RESULT][VOTE] Release Airflow 2.9.2 from 2.9.2rc1

[Utkarsh Sharma]

Hello,

Apache Airflow 2.9.2 (based on RC1) has been accepted.

3 "+1" binding votes received:
- Hussein Awala
- Jarek Potiuk
- Elad Kalif

4 "+1" non-binding votes received:
- Rom Sharon
- Phani Kumar
- Scheffler Jens
- Rahul Vats

Vote thread:
https://lists.apache.org/thread/zz10dy2kqy0c8c0dq0bz7q11779ywzbl

I'll continue with the release process, and the release announcement will
follow shortly.

Cheers,
Utkarsh