Re: [DISCUSS] Proposal to enhance Backfills

2024-06-04 Thread Vikram Koka
is to become a reality, Backfills need to be handled > by > > > the > > > Airflow Scheduler as a normal DAG execution > > > > > > I think it's a good idea. > > > It should solve natively problems like > > > https://github.com/apache/airflow

[DISCUSS] Proposal around the Injection of Task Execution Secrets

2024-06-04 Thread Vikram Koka
Internal API covers the interaction between the Airflow workers and Airflow metadatabase for heartbeat information, persisting XComs, and so on. -- Best regards, Vikram Koka, Ash Berlin-Taylor, Kaxil Naik, and Constance Martineau

Re: [VOTE] May 2024 PR of the Month

2024-05-28 Thread Vikram Koka
Agree that there were many excellent nominations, but my vote goes to #39336 Vikram On Tue, May 28, 2024 at 6:11 PM Wei Lee wrote: > Hard to choose only one, but my vote goes to #39336. > > Best, > Wei > > > On May 29, 2024, at 4:45 AM, Scheffler Jens (XC-AS/EAE-ADA-T) < >

[DISCUSS] Proposal for Synchronous (aka Interactive) DAG Execution

2024-05-27 Thread Vikram Koka
and - Return the failure status to the invoking API as a key task in the DAG has failed (inc upstream_failed etc.), rather than waiting for DagRun completion (i.e waiting for teardown task completion). --- Best regards, Vikram Koka, Ash Berlin-Taylor, Kaxil Naik, and Constance Martineau

[DISCUSS] Proposal to enhance Backfills

2024-05-24 Thread Vikram Koka
include at least: - Making the Airflow Scheduler responsible for scheduling decisions on all DagRuns (instead of the current where it purposefully ignores backfill runs) - A new API endpoint to submit a "backfill request". -- Best regards, Vikram Koka, Ash Berlin-Taylor, Kaxil Naik, and Constance Martineau

Re: [VOTE] AIP-69 Remote Executor

2024-05-18 Thread Vikram Koka
I agree with Jarek and Ash on this. I believe that the AIP as written documents the “what” and the “why” very well, but is too light on the “how”. I very much would like to see this AIP become reality as well, but I believe that we need some foundational elements such as API-44 and the “task

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

2024-05-13 Thread Vikram Koka
Definitely a fast moving thread on the mailing list. I haven’t been able to respond for a few days and feel very far behind already. A few comments on topics discussed the last few days: - Jarek, in response to your comments around being more aggressive than in Airflow 2 about deprecation and

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

2024-05-04 Thread Vikram Koka
; > the explanation of Vikram I still don't have a clue what we want to > > accomplish there :-P. > > > > I would like to see a mantra or team for Airflow 3. That helps nudging > > people in the same direction. Suggestions in the comments. > > > > Bolke > &

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

2024-05-03 Thread Vikram Koka
Good point Jed. I responded back to your comment in the doc as well and very open to changing the term in the doc. Used the term "interactive DAG run" as the ability to invoke or trigger a DAG run through the API, with the expectation of getting back a result immediately. An alternate term could

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

2024-04-20 Thread Vikram Koka
A wonderful and exciting Saturday morning discussion! Thank you Jarek for bringing the offline conversations into the mailing list. I completely agree on the necessity of Airflow 3. I also agree that Gen AI is the trigger i.e. the answer to "Why now"? Having been thinking about this for a while

Re: [DISCUSS] Redis licencing changes impact

2024-04-11 Thread Vikram Koka
Agree with your assessment Jarek, and that we don't have to do anything here. On Thu, Apr 11, 2024 at 5:29 AM Jarek Potiuk wrote: > Hello here, > > I've raised the discussion on private@ and it seems that there are no > private/controversies there, so I bring the discussion to devlist where it

Re: [ANNOUNCE] New committer: Wei Lee

2024-04-08 Thread Vikram Koka
Awesome! Congratulations Wei! Very well deserved. A tremendous amount of contributions in a relatively short time! On Mon, Apr 8, 2024 at 2:27 PM Oliveira, Niko wrote: > Congrats Wei! Well deserved :) > > > From: Scheffler Jens (XC-AS/EAE-ADA-T) .INVALID> >

Re: [VOTE] AIP-64: Keep TaskInstance try history

2024-03-25 Thread Vikram Koka
+1 binding On Mon, Mar 25, 2024 at 6:19 PM Mehta, Shubham wrote: > +1 (non-binding). > > Shubham > > On 2024-03-25, 5:10 PM, "Wei Lee" weilee...@gmail.com>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can

Re: [VOTE] Remove experimental API

2024-03-16 Thread Vikram Koka
-1 As much as I would like to see this removed, I feel the same way as Jed above. In response to the question raised regarding "Experimental features", the reason why this one seems different is because though this was marked as "experimental", it was the only way to interact with Airflow before

Re: [ANNOUNCE] Apache Airflow 2.8.0 Released

2023-12-18 Thread Vikram Koka
Very cool, Great to see this come together especially after all the challenges last week! Thanks to the team who pulled this together! On Mon, Dec 18, 2023 at 7:27 PM Amogh Desai wrote: > Awesome! 朗 > > On Tue, 19 Dec 2023 at 1:35 AM, Bolke de Bruin wrote: > > > 拾 > > > > B. > > > > Sent from

Re: Initiative on Strengthenig security for Apache Airflow

2023-11-09 Thread Vikram Koka
Thanks for sharing. This is exciting news and I'm happy to see this! On Tue, Nov 7, 2023 at 10:40 AM Jarek Potiuk wrote: > Hello everyone, > > I wanted to share some news (not so much news for us but - it's just now > reached publication stage) that we have nice security / release process >

Re: [VOTE] Add providers for Pinecone, OpenAI & Cohere to enable first-class LLMOps

2023-10-25 Thread Vikram Koka
+1 (Binding) On Wed, Oct 25, 2023 at 7:20 AM Wei Lee wrote: > +1 (non-binding) > > Best, > Wei > > > On Wed, Oct 25, 2023 at 10:44 PM Vincent Beck wrote: > > > +1 binding > > > > On 2023/10/25 13:32:49 Pierre Jeambrun wrote: > > > +1 (binding) > > > > > > Le mer. 25 oct. 2023 à 13:29, Pankaj

Re: [ANNOUNCE] Pankaj Koti and Amogh Desai as committers

2023-09-19 Thread Vikram Koka
Congratulations Pankaj Koti and Amogh Desai! Very well deserved! On Tue, Sep 19, 2023 at 10:32 AM Shubham Raj wrote: > Congratulations Amogh and Pankaj. It’s great. > > On Tue, 19 Sep 2023 at 10:27 AM, utkarsh sharma > wrote: > > > Congratulations @Pankaj and @Amogh :) > > > > On Tue, 19 Sep

Re: [VOTE] Drop MsSQL as supported backend

2023-08-31 Thread Vikram Koka
+1 (binding) On Thu, Aug 31, 2023 at 8:09 AM Josh Fell wrote: > +1 (binding) > > On Thu, Aug 31, 2023 at 8:35 AM Ankit Chaurasia > wrote: > > > +1 (non-binding) > > *Ankit Chaurasia* > > > > > > > > > > > > > > On Thu, Aug 31, 2023 at 5:47 PM Phani Kumar > > wrote: > > > > > +1 (non-binding)

Re: [ANNOUNCE] Apache Airflow 2.7.0 Released

2023-08-18 Thread Vikram Koka
Awesome! Great work team, this is definitely one of the biggest new Airflow releases! On Fri, Aug 18, 2023 at 9:16 AM Jarek Potiuk wrote: > Great! > > On Fri, Aug 18, 2023 at 6:08 PM Ephraim Anierobi < > ephraimanier...@apache.org> > wrote: > > > Dear Airflow community, > > > > I'm happy to

Re: [VOTE] Restore dag_run.conf UI triggering option for 2.7.0

2023-08-13 Thread Vikram Koka
+1 (with emphasis) On Sun, Aug 13, 2023 at 5:23 AM Ephraim Anierobi wrote: > +1 binding > > On Sun, Aug 13, 2023 at 6:57 AM Elad Kalif wrote: > > > +1 binding > > > > On Sat, Aug 12, 2023 at 9:32 PM Jarek Potiuk wrote: > > > > > > The vote doesn't say the release must happen, just that it

Re: [VOTE] Airflow Providers prepared on July 29, 2023

2023-07-31 Thread Vikram Koka
Not trying to hold up the release, but I thought there was a "race condition" bug discovered with the Celery executor as a provider and Airflow main. Is that resolved now? Or, did I mistake the origin of that? On Mon, Jul 31, 2023 at 8:19 AM Beck, Vincent wrote: > +1 (non-binding). I tested

Re: [VOTE] The daskexecutor, cncf.provider, celery providers in 2.7.0. as regular not-preinstalled providers

2023-07-31 Thread Vikram Koka
I am a little torn on this, but with the separation already in place, I would also vote for option (a). On Mon, Jul 31, 2023 at 1:48 PM Ferruzzi, Dennis wrote: > Pretty unanimous so far, but I'll also say [a]. If the intent is to be > executor/service agnostic then this makes the most sense

Re: Support for Datasets with execution-time values for use cases requiring more fine-grained dataset specs

2023-07-14 Thread Vikram Koka
Hi Jeff, Thank you for bringing this up. This is definitely on my radar and part of a larger AIP which I have been in the process of writing up. We have thought of this use case and deliberately deferred it in the earlier AIP. Doing is definitely quite complex and I think it needs a couple of

Re: [VOTE] June 2023 PR of the Month

2023-06-29 Thread Vikram Koka
+1 for #31123. Great work @pierrejeambrun On Thu, Jun 29, 2023 at 8:33 AM Wei Lee wrote: > +1 for #31123  > > > On Jun 29, 2023, at 11:14 PM, Phani Kumar > > > wrote: > > > > +1 for #31123 > > > > On Thu, 29 Jun 2023, 20:28 Kaxil Naik, wrote: > > > >> +1 for #31123  > >> > >> On Thu, 29

Re: [ANNOUNCE] New committers: Vincent Beck, Phani Kumar, Maciej Obuchowski

2023-06-28 Thread Vikram Koka
Congratulations to all of you! Great work and very well deserved! On Wed, Jun 28, 2023 at 1:43 PM Pierre Jeambrun wrote: > Great news!!! Congratulations  > > On Wed 28 Jun 2023 at 18:40, Ankit Chaurasia wrote: > > > Congrats > > > > > > > > > > > > > > > > On Wed, Jun 28, 2023 at 10:09 PM

Re: [ANNOUNCE] New committer: Pankaj Singh

2023-06-12 Thread Vikram Koka
Awesome! Congratulations Pankaj Singh! On Mon, Jun 12, 2023 at 3:37 PM Oliveira, Niko wrote: > Woo! Congrats Prankaj! > > > From: Ankit Chaurasia > Sent: Monday, June 12, 2023 3:12:42 PM > To: dev@airflow.apache.org > Subject: RE: [EXTERNAL][ANNOUNCE] New

Re: [VOTE] Release Airflow 2.6.0 from 2.6.0rc4

2023-04-28 Thread Vikram Koka
+1 (non-binding) On Fri, Apr 28, 2023 at 6:37 PM Kaxil Naik wrote: > +1 binding > > On Fri, 28 Apr 2023 at 14:54, Jed Cunningham > wrote: > > > +1 (binding) > > > > Checked signatures, checksums, licences. Used it with the helm chart > with a > > few different configs > > >

Re: [VOTE] March PR of the Month

2023-03-31 Thread Vikram Koka
What a great set of PRs to choose from. Amazing work everyone! So hard to choose, but my vote goes for 29413 - looks lovely! On Fri, Mar 31, 2023 at 9:52 AM Kaxil Naik wrote: > One more for 29413 > > On Fri, 31 Mar 2023 at 16:15, Jed Cunningham > wrote: > > > Another for 29413. > > >

Re: [ANNOUNCE] New PMC member: Pierre Jeambrun

2023-03-15 Thread Vikram Koka
Excellent! Congratulations Pierre, very well deserved! On Wed, Mar 15, 2023 at 10:55 AM Oliveira, Niko wrote: > Congrats Pierre, well deserved! > > > From: Kaxil Naik > Sent: Wednesday, March 15, 2023 2:47:31 AM > To: dev@airflow.apache.org > Subject: RE:

Re: [ANNOUNCE] New PMC member: Brent Bovenzi

2023-03-15 Thread Vikram Koka
Awesome! Congratulations Brent, well deserved! On Wed, Mar 15, 2023 at 12:10 PM Pierre Jeambrun wrote: > Congratulations Brent, well deserved :) > > Le mer. 15 mars 2023 à 18:55, Oliveira, Niko > a écrit : > > > Congrats Brent!! > > > > > > From: Jorrick

Re: [VOTE] February PR of the Month

2023-02-24 Thread Vikram Koka
+1 to #27758 as well, people have really wanted that! On Fri, Feb 24, 2023 at 3:20 PM Pierre Jeambrun wrote: > +1 to #27758, this is really cool! > > Le ven. 24 févr. 2023 à 23:55, Jed Cunningham > a écrit : > >> +1 to #27758 >> >

Re: [URGENT] Remove old versions of Airflow docs (<1.10.14) as stop-gap measure for doc builds

2023-02-23 Thread Vikram Koka
Thanks Jarek, for resolving this quickly. I agree that it is important to figure out a plan and next steps here before the next release for sure. Will sync and get back. Vikram On Thu, Feb 23, 2023 at 6:28 AM Jarek Potiuk wrote: > The temporary fix worked - for now. The < 1.10.15 docs are

Re: [VOTE] Move K8S / Celery (and related) executors to respective providers

2023-02-21 Thread Vikram Koka
+1 binding On Tue, Feb 21, 2023 at 12:04 PM Mehta, Shubham wrote: > + 1 non-binding > > > > *From: *Elad Kalif > *Reply-To: *"dev@airflow.apache.org" > *Date: *Tuesday, February 21, 2023 at 10:44 AM > *To: *"dev@airflow.apache.org" > *Subject: *RE: [EXTERNAL][VOTE] Move K8S / Celery (and

Re: Seeking Feedback for Airflow Multi-Tenant Model Proposal

2023-02-13 Thread Vikram Koka
Shubham and Vincent, Let me start by saying that I apologize for my delayed response to your original email. I appreciate the detailed write-up and the thought behind it. I completely agree with your use case and understand how this is applicable to enterprises with multiple data teams using

Re: [VOTE] AIP-53 OpenLineage in Airflow

2023-02-13 Thread Vikram Koka
+1 binding. I have been looking at the doc and having lineage integrated with Airflow as a provider makes sense to me. On Mon, Feb 13, 2023 at 2:38 AM Kaxil Naik wrote: > +1 binding , this should make lineage a first-class citizen for Airflow > users. Excited for this one > > On Sun, 12 Feb

Re: [VOTE] AIP-52 Automatic setup and teardown tasks

2023-01-09 Thread Vikram Koka
+1 binding Vikram On Mon, Jan 9, 2023 at 11:23 AM Ping Zhang wrote: > +1 binding > > Thanks, > > Ping > > > On Mon, Jan 9, 2023 at 11:22 AM Ephraim Anierobi < > ephraimanier...@gmail.com> wrote: > >> +1 binding >> >> On Mon, 9 Jan 2023 at 19:55, Frank Cash wrote: >> >>> +1 (non-binding) >>>

Re: [ANNOUNCE] New committer Niko Oliveira (o-nikolas)

2022-12-20 Thread Vikram Koka
Congratulations Niko! On Tue, Dec 20, 2022 at 10:31 AM Andrey Anshin wrote: > Congrats Niko!   > > > Best Wishes > *Andrey Anshin* > > > > On Tue, 20 Dec 2022 at 19:01, Beck, Vincent > wrote: > >> Congrats Niko!!  >> >> >> >> *From: *Pankaj Singh >> *Reply-To:

Re: 【New provider】Inquire

2022-10-06 Thread Vikram Koka
David, Great to hear this interest regarding integration with Airflow and happy to help guide as well. Similar to XD, I would very much like to understand both the user base sizing and areas of interest from an integration standpoint. >From your perspective, at the risk of repeating both XD and

Re: Apache Airflow Newsletter | September 2022

2022-10-05 Thread Vikram Koka
+1 to the comments. Very useful digest. I really like this edition of the newsletter! Good work! On Wed, Oct 5, 2022 at 6:30 AM Michael Robinson wrote: > Thank you so much and glad you are liking the newsletter! FYI: I’ve been > shipping it lately, but producing it is a team effort with Ross

Re: [ANNOUNCE] New Committer: Pierre Jeaambrun

2022-09-22 Thread Vikram Koka
Congratulations Pierre! Very well deserved On Thu, Sep 22, 2022 at 8:35 AM Josh Fell wrote: > Here here! Congrats Pierre! > > On Thu, Sep 22, 2022 at 8:10 AM Brent Bovenzi > wrote: > >> Congratulations Pierre! Very much well deserved! >> >> On Thu, Sep 22, 2022 at 3:51 AM Jarek Potiuk wrote:

Re: [ANNOUNCE] New PMC member: Daniel Standish

2022-09-22 Thread Vikram Koka
Congratulations Daniel! Very well deserved! On Thu, Sep 22, 2022 at 12:57 AM Jarek Potiuk wrote: > Hello Airflow Community, > > I have the pleasure to announce that The Project Management Committee (PMC) > for Apache Airflow has invited Daniel Standish to become Apache Airflow > PMC Member and

Re: [ANNOUNCE] New PMC member: Tzu-ping Chung

2022-09-22 Thread Vikram Koka
Congratulations TP! Very well deserved! Vikram On Thu, Sep 22, 2022 at 12:55 AM Jarek Potiuk wrote: > Hello Airflow Community, > > I have the pleasure to announce that The Project Management Committee (PMC) > for Apache Airflow has invited Tzu-ping Chung to become Apache Airflow PMC > Member

Re: [VOTE] Release Airflow 2.4.0 from 2.4.0rc1

2022-09-16 Thread Vikram Koka
+1 Tested the Data Dependency Management and Data Driven Scheduling (defined in AIP-48), with Datasets created and configured as dependencies between DAGs. Tested the DAG triggering using these. Also tested the new Datasets View and the updated Dag Dependencies View. This looks amazing and so

Re: [VOTE] August 2022 PR of the Month

2022-08-31 Thread Vikram Koka
If I can vote twice it would be for: 25610 and 25888. Hard to choose between them, they are both really good and critical On Wed, Aug 31, 2022 at 10:00 AM Jeambrun Pierre wrote: > Hello, > > I really love the new graph view for the datasets. > > My vote goes to 25707. > > Best, > Pierre > >

Re: Airflow Pluggable Scheduler

2022-08-29 Thread Vikram Koka
Hi Ping, Conceptually, I have a similar reaction to Jarek and Tomek above, but I really want to understand the problem you have described with (2) before I comment further. Can you please elaborate on the problems: Airflow 2.0 treats all DagRuns with the same scheduling priority (see code

Re: [DISCUSS] consolidate dag scheduling params

2022-08-05 Thread Vikram Koka
ert when robots.txt was being scanned, which was >>>> great, because I ignored it for a while, but eventually caved and looked >>>> into it. This I would see as more of the same. I currently get loads of >>>> warnings about things in my config file that have mov

Re: [DISCUSS] Trimming down Airlfow Versions in issues

2022-08-04 Thread Vikram Koka
+1 on this approach with this approach in the PR https://github.com/apache/airflow/pull/25542 On Thu, Aug 4, 2022 at 12:40 PM Jarek Potiuk wrote: > Love it. > > On Thu, Aug 4, 2022 at 9:28 PM Jed Cunningham > wrote: > >> Sorry, coming to this a little late. I tend to agree with Elad that we

Re: Auto-registering of DAGs in DAG file? (no `as dag` needed?)

2022-08-04 Thread Vikram Koka
+1 Similar view to Jed. I view this as adding a feature, rather than breaking compatibility. On Thu, Aug 4, 2022 at 3:20 PM Jed Cunningham wrote: > +1. I view it as adding a feature vs breaking compatibility. > > On Thu, Aug 4, 2022 at 4:15 PM Ferruzzi, Dennis > wrote: > >> I definitely like

Re: [DISCUSS] consolidate dag scheduling params

2022-08-04 Thread Vikram Koka
> Deprecating schedule_interval, it would be more prominent than some >>>>> other deprecations because it's a DAG param. That said, I think it's >>>>> pretty >>>>> straightforward to handle. For one, it's at the cardinality of DAG and not >>>>> *task*, so there

Re: [DISCUSS] consolidate dag scheduling params

2022-07-29 Thread Vikram Koka
-1 from me. Though I agree in principle with the idea of consolidation, I don't think we should be doing this yet until we understand the implications completely. I am really not in favor of deprecation of the existing params, unless there is really no alternative. On Fri, Jul 29, 2022 at 2:37

Re: [DISCUSS] - Grouping of concerns

2022-07-13 Thread Vikram Koka
like the idea of segmenting the release notes for different groups. > > > Thanks, > > Ping > > > On Thu, Jun 30, 2022 at 8:50 PM Vikram Koka > wrote: > >> Hi everyone, >> >> As I have been looking through the recent AIPs, development features, and >> ma

Re: [DISCUSS] Deprecating SLA feature?

2022-07-13 Thread Vikram Koka
t > breaking a lot of compatibility so we can deprecate it even now (and > we can raise warnings and mention that it will be replaced by > something better). I believe - for example - this was a mistake not > doing that with "experimental" API early enough which got more people > u

Re: [DISCUSS] Deprecating SLA feature?

2022-07-13 Thread Vikram Koka
- resending below to keep the same thread as Ping's response. My prior response and Ping's were sent at the same time, but I did not two two email threads -- I understand the frustration with the SLA feature as it stands. I struggled with trying to understand this early on and finally understood

Re: [DISCUSS] Deprecating SLA feature?

2022-07-13 Thread Vikram Koka
I understand the frustration with the SLA feature as it stands. I struggled with trying to understand this early on and finally understood how they were broken. Having said that, I believe that Airflow users strongly care about the timeliness and consequently SLAs of their data. I also believe

Re: Please vote on the PR of the Month for this month's Airflow Newsletter!

2022-07-05 Thread Vikram Koka
And even more importantly, +1 for the voting process and especially for the write-in element! :D On Mon, Jul 4, 2022 at 12:15 AM Jarek Potiuk wrote: > Too late :D. But it seems we could have two winners :) > > On Mon, Jul 4, 2022 at 8:32 AM Rajath Srinivasaiah > wrote: > >> +1 for #24284 >>

[DISCUSS] - Grouping of concerns

2022-06-30 Thread Vikram Koka
Hi everyone, As I have been looking through the recent AIPs, development features, and mailing list discussions, it struck me that we have effectively three different audiences here for Airflow. 1. Individuals and small teams using Airflow for their purpose, 2. Enterprises managing Airflow for

Re: [DISCUSS] Airflow Scheduling Delay Metric Definition

2022-06-30 Thread Vikram Koka
re about "begin executing" and >> how you calculate "its dependencies have been met.". >> >> >> >> If the 'begin executing' means the state of ti becomes running, then the >> 'Scheduling Delay' metric focuses on the overhead introduced

Re: Please vote on the PR of the Month for this month's Airflow Newsletter!

2022-06-29 Thread Vikram Koka
Also voting in favor of #24249 :) On Tue, Jun 28, 2022 at 7:19 AM Abhishek Bhakat wrote: > I also vote in favor of #24249 > > On 28-Jun-2022 at 7:28:55 PM, Phani Kumar > wrote: > >> I also vote for 24249 :) >> >> On Tue, Jun 28, 2022 at 7:23 PM Jarek Potiuk wrote: >> >>> Let the fight begin

Re: [DISCUSS] Airflow Scheduling Delay Metric Definition

2022-06-08 Thread Vikram Koka
metric you are proposing seems to be addressing a somewhat different need. That's what I am trying to understand. On Wed, Jun 8, 2022 at 11:24 AM Vikram Koka wrote: > Ping, > > I am quite interested in this topic and trying to understand the > difference between the "scheduling delay" m

Re: [DISCUSS] Airflow Scheduling Delay Metric Definition

2022-06-08 Thread Vikram Koka
Ping, I am quite interested in this topic and trying to understand the difference between the "scheduling delay" metric articulated as compared to the "task latency" aka "task lag" metric which we have been using before. As you may recall, we have been using two specific metrics to benchmark

Re: [VOTE] AIP-48 Data Driven Scheduling

2022-06-01 Thread Vikram Koka
+1 (binding) On Wed, Jun 1, 2022 at 10:49 AM Jarek Potiuk wrote: > +1 (binding) > > On Wed, Jun 1, 2022 at 7:47 PM Jed Cunningham > wrote: > >> +1 (binding) >> >> I'm looking forward to this, thanks Ash. >> >

Re: [ANNOUNCE] Apache Airflow 2.3.0 Released

2022-04-30 Thread Vikram Koka
> > >>> The documentation is available at: > >>> https://airflow.apache.org/docs/apache-airflow/2.3.0/ > >>> > >>> Find the release notes here for more details: > >>> > https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html >

Re: [VOTE] Release Airflow 2.3.0 from 2.3.0rc2

2022-04-29 Thread Vikram Koka
+1 (non-binding) Dynamic Task Mapping is a huge improvement! On Fri, Apr 29, 2022 at 11:34 AM Josh Fell wrote: > +1 (non-binding) > > Dynamic Task Mapping feels life-changing. > > On Fri, Apr 29, 2022 at 12:41 PM Abhishek Bhakat > wrote: > >> Other than that issue, have tested the version and

Re: K8s version/support policy: as long as K8s project, or as long as cloud providers?

2022-03-07 Thread Vikram Koka
+1 On Mon, Mar 7, 2022 at 3:01 PM Kaxil Naik wrote: > +1 > > On Mon, 7 Mar 2022 at 21:17, Jarek Potiuk wrote: > >> +1 >> >> On Mon, Mar 7, 2022 at 10:12 PM Ash Berlin-Taylor wrote: >> >>> Hey everyone, >>> >>> So Kubernetes 1.20 has now reached end of life in the upstream project, >>> and as

Re: [VOTE] AIP-45 Remove double dag parsing in airflow run

2022-03-07 Thread Vikram Koka
+1 On Mon, Mar 7, 2022 at 12:54 PM Cong Zhu wrote: > +1 (non-binding) > > On Mon, Mar 7, 2022 at 12:47 PM Kaxil Naik wrote: > >> +1 >> >> On Mon, 7 Mar 2022 at 20:37, Yingbo Wang wrote: >> >>> +1(non-binding) >>> >>> On Mon, Mar 7, 2022 at 12:29 PM Jarek Potiuk wrote: >>> +! (binding)

Re: Improving contributor experience for "trusted" users -- faster CI by using self-hosted runners

2022-02-21 Thread Vikram Koka
+1 I really like the road to committership perspective on this as well! I also like the suggestion of a periodic (ideally automated) clean up of this list for inactive users and committers as well. On Fri, Feb 18, 2022 at 7:54 AM Jarek Potiuk wrote: > +1. That will be a huge one and actually

Re: New Commiter: Malthe Borch

2022-02-21 Thread Vikram Koka
Congratulations and welcome Malthe! On Sat, Feb 19, 2022 at 9:22 AM Daniel Standish wrote: > Congrats Malthe! > >>

Re: New committer: Josh Fell

2022-02-21 Thread Vikram Koka
Congratulations and Welcome Josh! On Sat, Feb 19, 2022 at 9:21 AM Daniel Standish wrote: > Congrats! > > On Sat, Feb 19, 2022 at 5:11 AM Kaxil Naik wrote: > >> Congratulations Josh, welcome aboard >> >> On Sat, 19 Feb 2022 at 07:38, Ephraim Anierobi >> wrote: >> >>> Congratulations Josh! >>>

Re: [PROPOSAL] Not merging failed PRs by default

2022-01-06 Thread Vikram Koka
en" PRs. >>>> >>>> I propose that we change our approach and whenever we see a "red" >>>> build every committer's approach should be : >>>> >>>> * investigate the root cause >>>> * if it's main - attempt to fix it

Re: New PMC Member: Jed Cunningham

2022-01-04 Thread Vikram Koka
Congratulations Jed, very well deserved indeed! On Tue, Jan 4, 2022 at 9:39 AM Ryan Hamilton wrote: > Congrats Jed! > > On Tue, Jan 4, 2022 at 12:36 PM Xiaodong Deng wrote: > >> Congrats, Jed! >> >> On Tue, Jan 4, 2022 at 18:35 Elad Kalif wrote: >> >>> Congratulations Jed! >>> >>> On Tue,

Re: [VOTE] AIP-42 Dynamic Task Mapping

2021-11-18 Thread Vikram Koka
+1 (binding) Great work on the AIP, really looking forward to seeing this happen! On Thu, Nov 18, 2021 at 9:41 AM Brent Bovenzi wrote: > +1 (non-binding) > > On Thu, Nov 18, 2021 at 11:21 AM Ry Walker wrote: > >> +1 (non-binding) >> can’t wait to see this elegant AIP materialize! >> >> >> On

Re: [VOTE] Release Airflow 2.2.0 from 2.2.0rc1

2021-10-08 Thread Vikram Koka
Thanks Jed, this is a huge release! On Fri, Oct 8, 2021 at 8:52 AM Kaxil Naik wrote: > Wohoo  > > On Fri, Oct 8, 2021 at 9:20 PM Jed Cunningham > wrote: > >> Hey fellow Airflowers, >> >> I have cut Airflow 2.2.0 RC1. This email is calling a vote on the >> release, which will last for 72

Re: New PMC Members: Ephraim Anierobi & Elad Kalif

2021-08-31 Thread Vikram Koka
Congratulations Elad and Ephraim!! Very well deserved indeed! Vikram On Tue, Aug 31, 2021 at 7:35 AM Jiajie Zhong wrote: > Congratulations! Ephraim and Elad! Well deserved! > > Best Wish > — Jiajie > > >

Re: New Committer: Brent Bovenzi

2021-08-27 Thread Vikram Koka
Congratulations Brent! Very well deserved! On Fri, Aug 27, 2021 at 3:28 PM Jed Cunningham wrote: > Congrats Brent  > > On Fri, Aug 27, 2021 at 2:16 PM Tomasz Urbaszek > wrote: > >> Congrats Brent! > >

Re: Issue form for Bugs/features in GitHub

2021-08-20 Thread Vikram Koka
+1 from me. I would love to get some templates in place for pre-classification of issues. On Fri, Aug 20, 2021 at 2:01 PM Jarek Potiuk wrote: > I believe Beta is just "subject to change" but generally available. I > think the "public preview" is something that needs to be enabled for an >

Re: New Committer: Aneesh Joseph

2021-07-07 Thread Vikram Koka
Congratulations Aneesh! On Wed, Jul 7, 2021 at 2:34 AM Elad Kalif wrote: > Congrats! Welcome aboard :) > > On Tue, Jul 6, 2021 at 5:23 PM Jarek Potiuk wrote: > >> Congratulations! >> >> On Tue, Jul 6, 2021 at 3:29 PM Kaxil Naik wrote: >> >>> Hello Airflow Community, >>> >>> The Project

Re: New Committers: Jed Cunningham & Tzu-ping Chung

2021-06-25 Thread Vikram Koka
Congratulations Jed and TP! On Fri, Jun 25, 2021 at 6:57 PM Xinbin Huang wrote: > Congrats! Well deserved! > > On Fri, Jun 25, 2021 at 6:12 PM Kaxil Naik wrote: > >> Hello Airflow Community, >> >> The Project Management Committee (PMC) for Apache Airflow >> has invited *Jed Cunningham *&

Re: Allyship workshops for open source contributors

2021-06-08 Thread Vikram Koka
+1 On Fri, Jun 4, 2021 at 6:49 AM Jake Ferriero wrote: > +1 > > On Thu, Jun 3, 2021 at 6:09 PM Paola Peraza Calderon > wrote: > >> +1 >> >> On Thu, Jun 3, 2021 at 4:53 AM Jarek Potiuk wrote: >> >>> +1 >>> >>> śr., 2 cze 2021, 23:03 użytkownik Aizhamal Nurmamat kyzy < >>> aizha...@apache.org>

Re: [ANNOUNCE] Apache Airflow 2.1.0 has been released

2021-05-21 Thread Vikram Koka
Awesome! Thank you Ash, James Timmins and everyone else who contributed to this release. On Fri, May 21, 2021 at 8:04 AM Jarek Potiuk wrote: > Cool! > > > pt., 21 maj 2021, 15:29 użytkownik Ash Berlin-Taylor > napisał: > >> Dear Airflow community, >> >> I'm happy to announce that Airflow

Re: [ANNOUNCE] Apache Airflow Helm Chart version 1.0.0 Released

2021-05-19 Thread Vikram Koka
Awesome! Great job Kaxil and everyone who contributed to this! On Wed, May 19, 2021 at 6:47 PM Kaxil Naik wrote: > Dear Airflow community, > > I am pleased to announce that we have released the first version of the > official *Apache Airflow Helm chart 1.0.0* last night   > > The source

Re: More explicit/prominent EOL information for Airflow 1.10?

2021-05-19 Thread Vikram Koka
Jarek, Thanks for bringing this up and vehemently agreed on the need for this. Elad, great start with the PR. I think we can all add comments to the PR and get it published as soon as possible. On Wed, May 19, 2021 at 2:50 AM Kaxil Naik wrote: > Nice, I think the PR is a good start, we should

Re: [VOTE] AIP-40: Deferrable ("Async") Operators

2021-04-30 Thread Vikram Koka
+1 (binding) Great work on this Andrew, looking forward to seeing this happen! On Fri, Apr 30, 2021 at 7:24 AM Tomasz Urbaszek wrote: > +1 binding - love the idea! > > On Fri, 30 Apr 2021 at 10:40, Jarek Potiuk wrote: > > > > +1 (binding) > > > > On Fri, Apr 30, 2021 at 10:29 AM Ash

Re: [DICUSS][AIP-40] Deferrable ("Async") Operators

2021-04-29 Thread Vikram Koka
Thanks Andrew for the detailed responses and Jarek for the great questions. I really like the AIP and love the direction. Vikram On Thu, Apr 29, 2021 at 10:27 AM Andrew Godwin wrote: > Yup, logging and metrics are something I didn't put in the prototype but I > consider an essential part of

REMINDER: Airflow issue triage process call tomorrow

2021-04-13 Thread Vikram Koka
Hey everyone, I would like to remind whoever is interested to help in the Airflow Issue Triage process that we have our call tomorrow (Wednesday). *Date*: April 14th *Time*: 8.30-9.30 AM Pacific / 4.30 PM GMT *Zoom link*:

Re: New Committers: Qian Yu & Xinbin Huang

2021-04-11 Thread Vikram Koka
Awesome! Congratulations to both of you! On Fri, Apr 9, 2021 at 1:27 PM Aizhamal Nurmamat kyzy wrote: > Congratulations to both of you! Looking forward to seeing more > contributions :) > > On Thu, Apr 8, 2021 at 9:26 PM Xinbin Huang wrote: > >> Thank you all! Looking forward to making the

Re: Triage role for Jed Cunnigham

2021-04-10 Thread Vikram Koka
+1 On Sat, Apr 10, 2021 at 4:24 PM Xinbin Huang wrote: > +1 > > On Sat, Apr 10, 2021 at 4:08 PM Tomasz Urbaszek > wrote: > >> +1 >> >> On Sun, 11 Apr 2021 at 00:59, Daniel Imberman >> wrote: >> > >> > +1 >> > >> > On Sat, Apr 10, 2021 at 3:54 PM, Ryan Hamilton < >>

Rescheduling Issue triage process meeting for April 7th

2021-04-04 Thread Vikram Koka
Hello everyone, I apologize for the inconvenience, but I have to reschedule the Airflow Issue Triage process meeting from this Wednesday to next week. I hope that works for you. In the meantime, here is an update on the metrics from last week. - *Issue metrics: * - The results on "bugs"

Re: Releasing Apache Airflow's Python Client

2021-04-04 Thread Vikram Koka
Sumit, Thank you for bringing up this discussion. Reading through the thread, I am definitely far more comfortable with the hybrid approach proposed by Kaxil, than the first approach, primarily because of the same concerns articulated by Jarek regarding end user experience. Prior to that, I was

[Meeting Notes] Airflow issue triage process call - Mar 24, 2021

2021-03-27 Thread Vikram Koka
Hello everyone, Here is the summary of our meeting earlier this week. Thank you all who joined the call, really appreciate your participation and you taking the time! Please correct anything that I missed. To all those who did not join, please voice your opinion if you disagree with anything.

Re: Triage role for active contributors

2021-03-24 Thread Vikram Koka
+1 Would love to get additional help on triaging issues and features too :) On Wed, Mar 24, 2021 at 11:18 AM Jarek Potiuk wrote: > +1 > > On Wed, Mar 24, 2021 at 5:20 PM Kaxil Naik wrote: > >> +1 -- I have no issues with that. I will wait for the end of today to see >> if anyone has any issues

REMINDER: Airflow issue triage process call in an hour

2021-03-24 Thread Vikram Koka
Hey everyone, I would like to remind whoever is interested to help in the Airflow Issue Triage process that we have our call in about an hour. *Date*: March 24th *Time*: 8.30-9.30 AM Pacific / 4.30 PM GMT *Zoom link*: https://astronomer.zoom.us/j/91979682731?pwd=bWNoTFM3ajRidUlzL25RMXZ4WU96Zz09

Re: [VOTE] AIP-39: Richer scheduler_interval

2021-03-21 Thread Vikram Koka
+1 binding Really good work on this James and Ash. Looking forward to seeing you take shape. Best regards, Vikram On Sun, Mar 21, 2021 at 2:43 PM Ryan Hamilton wrote: > +1 binding > > On Sat, Mar 20, 2021 at 8:41 AM Kaxil Naik wrote: > >> +1 (binding). >> >> This looks awesome, good work

CANCELLING: Airflow issue triage process call today

2021-03-10 Thread Vikram Koka
Hello everyone, I have to cancel the Airflow issue triage process call for later today. I cancelled the meeting invite on the dev calendar earlier today, but wanted to send the email out to make sure that everyone was aware. Apologies for the late notice. Best regards, Vikram

Re: [DISCUSS] Guidelines for Releasing Providers packages

2021-03-09 Thread Vikram Koka
I agree with *Batch vs Ad-hoc *and *Frequency.* For doc-only changes, I would prefer NOT to change the version. Primarily because of the end user perspective, as was articulated earlier in the thread. Best regards, Vikram On Tue, Mar 9, 2021 at 6:05 PM Jarek Potiuk wrote: > Fully agree

Re: [DISCUSS] Major Version should contain new features or just removal of deprecation?

2021-03-09 Thread Vikram Koka
I love the thoughtful discussion. I am in favour of (b), because that is the "general understanding" of Semantic Versioning. On Tue, Mar 9, 2021 at 6:14 PM Kaxil Naik wrote: > Interesting points, reminds me of this thread: > https://github.com/semver/semver/issues/411#issuecomment-347050750 

Re: [DISCUSS] TaskGroup in Tree View

2021-03-05 Thread Vikram Koka
Kevin, I am not sure I understand your response to Nathan. I agree that it is also a valid use case, but I don't see how it can be cleanly done while keeping TaskGroup only as a UI concept. Would this require extending the TaskGroup concept to the backend? Best regards, Vikram On Fri, Mar 5,

Re: [DISCUSS] TaskGroup in Tree View

2021-03-03 Thread Vikram Koka
Hey Kevin, One immediate clarifying question: - For your use case, it seems that you want to continue using TaskGroup only as a "pure UI concept". - But, you want it's representation to also be in the Tree View. - You are not proposing any "execution or scheduling" enhancements [again for your

Re: [VOTE] AIP-38: Modern Web Application

2021-03-03 Thread Vikram Koka
Ryan and Brent, Great job on the AIP and looking forward to seeing this happen. +1 (binding) On Wed, Mar 3, 2021 at 8:16 AM Redwan Rahman wrote: > +1 (non-binding) > > On Wed, Mar 3, 2021 at 10:31 AM Ryan Hamilton > wrote: > >> Team, >> >> This email calls for a vote on the project

[Meeting Notes] Airflow issue triage process call - Feb 24, 2021

2021-03-01 Thread Vikram Koka
Hey everyone, Here is the summary of our meeting last week. Thank you all who joined the call, really appreciate your participation and you taking the time! Please correct anything that I missed. To all those who did not join, please voice your opinion if you disagree with anything. Overall

  1   2   >