Re: [DISCUSS] Asynchronous SQLAlchemy

Yeah do we have concrete examples of places where asyncio would be a non-starter? Are there enough of these examples to kill this idea? I really don't like the idea of needing to maintain both sync and async. On Mon, Apr 8, 2024 at 1:39 PM Hussein Awala wrote: > > we definitely need a way to

Re: [DISCUSS] Asynchronous SQLAlchemy

Agreed with Daniel and Ash, I'm a big +1 for "all", I also think there is work we can also do to make some of the executor work async (particularly around the KubernetesExecutor which is slowed down by calls to the k8s API) On Mon, Apr 8, 2024 at 1:08 PM Daniel Standish wrote: > If nothing

Re: [VOTE] Remove experimental API

As everyone above mentioned. I’m all for removing it but we should do so as part of a major breaking release. Perhaps if we haven’t already we should at least add deprecation warnings? -1 but very down to add deprecation warnings On Sat, Mar 16, 2024 at 4:19 PM Bas Harenslak wrote: > -1 for me

Re: [VOTE] Drop MsSQL as supported backend

+1 binding On Wed, Aug 30, 2023 at 10:01 PM Ryan Hatter wrote: > +1 non-binding > > On Mon, Aug 28, 2023 at 3:29 PM Aritra Basu > wrote: > > > +1 (non-binding) > > Based on reading the previous mails, looks like a good idea to drop along > > with the migration support > > > > -- > > Regards, >

Re: [DISCUSS] Maciej Obuchowski for committer?

https://www.youtube.com/watch?v=CQeezCdF4mk On Wed, Jun 14, 2023 at 11:56 PM Ankit Chaurasia wrote: > +1 (non-binding) > > On Thu, 15 Jun 2023 at 07:41, Pankaj Koti .invalid> > wrote: > > > :) > > > > Hahah, yes, +1 (non-binding) > > > > He is super helpful and super responsive. Helped us

Re: 【Airflow】New provider - Huawei Cloud

Hi David, Ultimately what it comes down to is this: when people add new providers to Airflow it becomes the PMCs responsibility to maintain them in perpetuity. If the Huawei cloud provider is broken and we can’t reach you, then it reflects poorly on the project that we have a broken provider. If

Re: [DISCUSS] Exclude some providers that hold us back from releasing

@dennis I think that if there is a bug or feature where we reach out and don’t get a response in a set number of days that could be a good marker. On Wed, Mar 29, 2023 at 16:26 Elad Kalif wrote: > I agree that we should exclude providers that block us. > > > On Tue, Mar 28, 2023 at 9:58 PM

Re: New PMC Member: Jed Cunningham

Congratulations, Jed!On Wed, Jan 05, 2022 at 12:13pm, Jorrick Sleijster wrote:Congratulations Jed! Great news and well deserved On Wed, 5 Jan 2022, 10:32 Tomasz Urbaszek, wrote:Congratulations Jed!On Wed, 5 Jan 2022 at 03:37, Ian Stanton

Re: [VOTE] AIP-42 Dynamic Task Mapping

ronomer.io.invalid; target="_blank" style="">josh.d.f...@astronomer.io.invalid wrote:style="">dir="ltr" style="">+1 (non-binding) Very excitingstyle="">dir="ltr" class="gmail_attr" style="">On Thu, Nov

[VOTE] AIP-42 Dynamic Task Mapping

style="font-size: 15px;">style="">Hi everyone,style="">I would like to call for a vote on AIP-42 Dynamic Task Mapping:style="">href="https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-42%3A+Dynamic+Task+Mapping; style="">AIP-42: Dynamic Task Mapping - Airflow - Apache Software

Re: AIP-42: Dynamic Task Mapping

Hey everyone!style="">Wanted to update a few additions/changes we've made and open up the thread for any remaining comments/questions before we go to a vote:style="">Consideringstyle="">@Jarek Potiuk'scomments, we have changed the mapping_id to an integer to ensure that we are within the range

AIP-42: Dynamic Task Mapping

ly excited about :). Happy Friday! Ash Berlin-Taylor, TP Chung, and Daniel Imberman

Re: New Committer: Brent Bovenzi

Congrats Brent! On Fri, Aug 27, 2021 at 4:23 PM, Luciano Resende wrote: On Fri, Aug 27, 2021 at 12:27 PM Kaxil Naik < kaxiln...@apache.org [kaxiln...@apache.org] > wrote: Hello Airflow Community, The Project Management Committee (PMC) for Apache Airflow had invited Brent Bovenzi to become an

Re: [VOTE] AIP-40: Deferrable ("Async") Operators

+1 (binding) I’m really excited about this. Thanks Andrew! On Thu, Apr 29, 2021 at 5:20 PM, Andrew Godwin wrote: Hi everyone, I would like to call for a vote on AIP-40 Deferrable ("Async") Operators: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177050929

Re: [DICUSS][AIP-40] Deferrable ("Async") Operators

@ash since we’re dropping 3.6 support soon anyways, would we even need this warning? Based on the POC it seems that this involves core changes, so I imagine if this is only 2.2.X compatible then ending py3.6 support for 2.2.X would be sufficient? (unless truly ending 3.6 support would need to

Re: [VOTE] Release Airflow 2.0.2 from RC1

+1 (Binding) On Mon, Apr 19, 2021 at 6:31 AM, Elad Kalif wrote: +1 non binding tested on several of my dags On Mon, Apr 19, 2021 at 3:42 PM Jarek Potiuk < ja...@potiuk.com [ja...@potiuk.com] > wrote: +1 (binding). Great job Ash. Tested validity of the packages/signatures/licences as well as

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

an alternate method as needed. On Wed, Apr 14, 2021 at 6:58 PM Daniel Imberman < daniel.imber...@gmail.com [daniel.imber...@gmail.com] > wrote: Thursday works for me! On Wed, Apr 14, 2021 at 10:05 AM, Ian Buss < ianjb...@gmail.com [ianjb...@gmail.com] > wrote: Hi all, I actually can’t do We

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

ment (2) & (3) would help solve the isolation problem in (1) and can be done with some work now. Happy to talk about it in more detail here or on call, the time Daniel suggested works for me. Regards, Kaxil On Wed, Apr 14, 2021 at 5:35 PM Daniel Imberman < daniel.imber...@gmail.com [daniel.

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

How about Wednesday, April 21 at 8:00AM PST? On Wed, Apr 14, 2021 at 9:33 AM, Xinbin Huang wrote: I am available any days. On Wed, Apr 14, 2021, 9:32 AM Daniel Imberman < daniel.imber...@gmail.com [daniel.imber...@gmail.com] > wrote: Hi everyone! Would people be available around 8AM/9

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

mprove the security of such a system. Food for thought at least. I will start putting some of these thoughts down on paper in a sharable format. Ian On Tue, Apr 13, 2021 at 7:46 PM Daniel Imberman < daniel.imber...@gmail.com [daniel.imber...@gmail.com] > wrote: Hi Ian, Firstly, welcome to the Airf

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Hi Ian, Firstly, welcome to the Airflow community :). I'm glad to hear you've had a positive experience so far. It's great to hear that you want to contribute back, and I think that multi-tenancy/DAG isolation is a pretty fantastic project for the community as a whole (a lot of things are

Re: Triage role for Jed Cunnigham

+1 On Sat, Apr 10, 2021 at 3:54 PM, Ryan Hamilton wrote: +1 On Sat, Apr 10, 2021 at 6:45 PM Kaxil Naik < kaxiln...@gmail.com [kaxiln...@gmail.com] > wrote: Hi folks, I would like to propose that the triage role be given to Jed Cunnigham (Github Handle: jedcunningham): This will allow

Re: [VOTE] AIP-39: Richer scheduler_interval

+1 binding. Really proud of this work, James! On Fri, Mar 19, 2021 at 6:50 PM, Jarek Potiuk wrote: +1 binding. It will make it better for a class of uses. On Sat, Mar 20, 2021 at 2:44 AM Xinbin Huang < bin.huan...@gmail.com [bin.huan...@gmail.com] > wrote: +1 non-binding. Looking forward to

Re: [DISCUSS] TaskGroup in Tree View

I personally think that TaskGroup should go beyond being “just” a UI concept. I think that there are a lot of use-cases where people might want to perform a single operation across an entire group of tasks. I think that Bin points out a few really good examples (default arguments and group

Re: [VOTE] Airflow Providers - release candidates from 2021-02-27

amazon : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica Kolenkaš, JavierLopezT * apache.druid : Xinbin Huang * apache.spark : Igor Khrol * cncf.kubernetes : jpyen, Ash Berlin-Taylor, Daniel Imberman * google : Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan Yuan, morrme, Sam

Re: [VOTE] AIP-38: Modern Web Application

+1 (binding) On Wed, Mar 3, 2021 at 7:46 AM, Constance Martineau wrote: +1 (non-binding) On Wed, Mar 3, 2021 at 10:31 AM Ryan Hamilton wrote: Team, This email calls for a vote on the project proposed in AIP-38:

Re: [VOTE] Airflow Backport Providers 2021.2.5rc1

@jarek does the cncf provider have this recent fix? https://github.com/apache/airflow/pull/14083 Would love to add that if possible On Mon, Feb 8, 2021 at 11:06 AM, Jarek Potiuk wrote: We still need some review/votes on Backport Providers please :). Only two months left to release those :).

Re: Committer PRs and mainline builds now faster: running on our own machines.

Thank you for your work on this Ash! One thing to mention is that while this only directly affects the committer/PMC runs, it should still free up more resources overall. Might also be worth bringing this up to the ASF board as perhaps other projects can consider similar methods. On Tue, Feb

Re: Scoping out a new feature for 2.1: improving schedule_interval

t; Both these values are set by the dag parser process, which has full >> access to run code. What ever interface for defining new schedule >> expression should run in the existing process, much like how James C did in >> a subclass. >> " >> -- >> Dmitri >

Re: Scoping out a new feature for 2.1: improving schedule_interval

21, at 6:37 PM, Kaxil Naik < kaxiln...@gmail.com [kaxiln...@gmail.com] > wrote: "CronBackend" / "ScheduleIntervalBackend" :D similar to Xcom and Secrets Backend Would be definitely good to have Custom Schedule intervals using functions/class that is Serializable too. On Wed, Jan 20, 20

Re: Scoping out a new feature for 2.1: improving schedule_interval

Schedule intervals using functions/class that is Serializable too. On Wed, Jan 20, 2021 at 11:02 PM QP Hou wrote: On Wed, Jan 20, 2021 at 10:22 AM Daniel Imberman < daniel.imber...@gmail.com [daniel.imber...@gmail.com] > wrote: > > I love the idea of allowing users to create their own schedul

Re: Scoping out a new feature for 2.1: improving schedule_interval

@Jarek the problem with just cron is I don’t think cron can handle “every third Thursday” or “the next open market day after the 15th.” I think we need something more flexible than just cron (though agree that cron syntax can get a fair bit of mileage) On Wed, Jan 20, 2021 at 10:24 AM, Jarek

Re: Scoping out a new feature for 2.1: improving schedule_interval

I love the idea of allowing users to create their own scheduling objects/scheduling python functions. They could either live in the scheduler or as a seperate process that trips some value in the DB when it is “true”. Would be great from a “marketplace” standpoint as well as users could post

Re: Measuring community state

@tomek I just added a few questions/fixed grammar. PTAL :). via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.51=10.15.7=email_footer_2] On Thu, Jan 14, 2021 at 4:44 AM, Kaxil Naik wrote: Looks good to me, thanks Tomek. On Thu, Jan 14, 2021 at 12:11 PM Tomasz Urbaszek <

Re: Backfill job is deadlocked

Hi Shiv, Did you mean 1.10.10 or are you running 1.10.0? via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.51=10.15.7=email_footer_2] On Mon, Dec 28, 2020 at 10:45 PM, shivboom q wrote: Hi Team, We have dag composed of subdags, when one or more dags are running in parallel we are

Re: [VOTE] Release Apache Airflow 2.0.0 form 2.0.0rc3

+1 !! (binding) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.51=10.15.7=email_footer_2] On Wed, Dec 16, 2020 at 1:18 AM, Ash Berlin-Taylor wrote: +1 (binding) Third time is the charm, hopefully. -ash On Mon, 14 Dec, 2020 at 23:00, Jarek Potiuk wrote: +1 (binding): RAT,

Re: [VOTE] Release Apache Airflow 1.10.13 based on 1.10.13rc1

+1 binding On Sat, Nov 21, 2020 at 3:08 AM Tomasz Urbaszek wrote: > +1 binding > > Tomek > > On Sat, Nov 21, 2020 at 1:25 AM Kaxil Naik wrote: > >> Hello Apache Airflow Community, >> >> This is a call for the vote to release Apache Airflow version 1.10.13. >> >> The release candidate: >>

Re: Stack Overflow as valuable source of information about Airflow ?

ag. If you want this to change - people must start giving votes up. Take 5 min of your day to read some questions with answers and vote quality questions and quality answers. Sent: Saturday, November 14, 2020 at 7:54 PM From: "Daniel Imberman" To: dev@airflow.apache.org Subject: Re: Stack

Re: Rewriting Breeze in Python ?

I am all for rewriting breeze, but I think waiting until after 2.0 makes the most sense. Python could work, but let’s be intentional about the decision before we choose. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.51=10.15.7=email_footer_2] On Wed, Nov 11, 2020 at 3:12 PM,

Re: New Airflow Committer: Ryan Hamilton

Congratulations Ryan! Well deserved and with high expectations! On Wed, Oct 28, 2020 at 11:01 AM, Deng Xiaodong wrote: Congrats Ryan! The UI improvements you kindly made are definitely a big progress of Airflow  XD On Wed, Oct 28, 2020 at 6:53 PM QP Hou wrote: Congrats Ryan and wellcome!

Re: Add /ready-to-test bot to airflow CI

quot;Approved" label yet. Setting the label and re-running the build will happen at the same time. But I start thinking this label should be named differently - how about " Ready to merge " maybe? Or maybe other ideas? J. On Sat, Oct 24, 2020 at 1:10 AM Daniel Imberman &

Re: More flexible unit tests for the Helm Chart

Honestly having played around with python-based testing in this PR (https://github.com/apache/airflow/pull/11693) I’m in love. It’s so much easier to use than helm-unittest, much more flexible in what we can test for (as we can now use the full python language), and ties in much nicer to our

Re: Add /ready-to-test bot to airflow CI

ps://github.com/apache/airflow/issues/10507 [https://github.com/apache/airflow/issues/10507] "Selective Tests" for which I have an open PR. They will give much bigger improvements - because, in the vast majority of cases, the tests will take very little time - giving feedback about rele

Re: Airflow Feature Design Feedback

This could be pretty valuable for future audits. I’d personally rather avoid adding fields to the DB in general. Could we store it wherever we store normal failure messages? via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.51=10.15.6=email_footer_2] On Fri, Oct 16, 2020 at 9:21

Re: Credits from Google (or other sponsors?) for self-hosted runners

re: security conerns, this is a case where we could require committer approval before running full tests (though leaves the risk that a PR is approved for testing and then the user adds something concerning after). via Newton Mail

Re: Proposal: Change default to LocalExecutor and (eventually) remove SequentialExecutor

+1 to the general notes of the convo not much to add via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.51=10.15.6=email_footer_2] On Wed, Oct 7, 2020 at 6:05 AM, Kaxil Naik wrote: As long as we make sure LocalExecutor works fine with Sqlite, I am fine with that. But we find any

Re: Much more stable CI tests (hopefully!)

Thanks Jarek! This was much needed and should lead to a cleaner dev process via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.51=10.15.6=email_footer_2] On Sun, Oct 11, 2020 at 3:37 PM, Jarek Potiuk wrote: Hello everyone, I have really high hopes for the CI change that we

Re: [VOTE] Airflow Backport Providers 2020.10.5rc1

+1 excited to finally encourage the Kubernetes providers! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.51=10.14.6=email_footer_2] On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk wrote: Hey all, I have cut Airflow Backport Providers 2020.10.5rc1. This email is calling a vote

Re: Add /ready-to-test bot to airflow CI

a review we can start tests on CI. On the other hand, I can see people asking for starting the tests or being even more confused why some PRs have more CI builds than others... Cheers, Tomek On Thu, Oct 1, 2020 at 10:29 PM Daniel Imberman < daniel.imber...@gmail.com [daniel.imber...@gmail.com] >

Re: Consider using stale bot for issues

I tried this out today and I am absolutely in love. It certainly needs some work (i.e. making it work for pull requests as well, faster updates, etc.) but this 100x better than the GitHub system right now. I also think that since it is open source, any airflow user who wants to start helping

Add /ready-to-test bot to airflow CI

Hello all, With the recent uptick in airflow contribution and pull requests, I have a proposal that I hope will ensure that we do not find ourselves in a CI backlog hell. I noticed that on the Kubernetes project, pull requests do not run integration test until a committer submits a "ready to

Re: [VOTE] Enable Github Discussions on Apache Airflow Github Repo

+1 via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.15.6=email_footer_2] On Tue, Sep 22, 2020 at 11:59 AM, Kaxil Naik wrote: Hi all, This email calls for a vote to enable Github Discussion on Apache Airflow repo. Intended Use : A forum for users, you might consider it as

Re: [DISCUSS] Removing Pickling from Airflow 2.0

Are there any use-cases that REQUIRE pickle? Do we have any sense of what % of the Airflow community depends on Pickle? I’m all for killing it if possible but I want to make sure we’re not setting up a major hurdle for migration. via Newton Mail

Re: [DISCUSS] Deprecate SubDags in favor of TaskGroups ?

I agree with Gerard on all fronts. SubDags are difficult, slow, and can cause a lot of strange edge cases. The difficulties in SubDags is a sticking point for Airflow competitors that I want to remove as quickly as possible. I think that 2.0 is a perfect time to introduce TaskGroups as people

Re: Discuss: should we allow HTML emails on dev@ list

+1 (binding) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.15.6=email_footer_2] On Mon, Sep 14, 2020 at 9:03 AM, Jarek Potiuk wrote: Oh noo .. I just started to use Plain Text Mode in my Gmail more often than not. Seriously though: +1 Binding. On Mon,

Re: [VOTE] AIP-8 Split Providers into Separate Packages for Airflow 2.0

+1 (binding). via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.15.6=email_footer_2] On Sun, Sep 13, 2020 at 1:57 PM, Kevin Yang wrote: +1 (binding) On Sun, Sep 13, 2020 at 1:29 PM Jarek Potiuk wrote: > Hello Everyone, > > Last week, at the Airflow 2.0 meeting the people

Re: Import style in Airflow codebase

+1 on my end! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.15.6=email_footer_2] On Wed, Sep 9, 2020 at 7:38 PM, Maxime Beauchemin wrote: +1 On Wed, Sep 9, 2020 at 10:36 AM Vikram Koka wrote: > +1 on absolute import. Honestly, a huge fan of doing it as an absolute

Re: Generic Transfer Operator

at 9:45 PM Daniel Imberman wrote: > > Oof ok yeah. I hadn't realized that beam had a hard JVM requirement. I > think that initially offering a local or block storage based solution with > easy extensions for users is totally in line with airflow philosophy. I > think that offer

Re: Generic Transfer Operator

> >> > > >> >1. Dependency management. If I do `pip install apache-airflow[gcp]` > >> >will it install `apache-beam[gcp]`? What if there's a version clash > >> >between dependencies? > >> > > >> >2. The initial appro

Re: Generic Transfer Operator

ows us to simplify > > > >> > read/write from external sources. Thus, it requires no external > > > >> > dependency other than the library to communicate with the API. In > the > > > >> > case of Beam we need more than that I think. > > > >> &g

Re: Generic Transfer Operator

Interesting! Beam also could potentially allow transfers within Dask/any other system with a java/python SDK? I think @jarek and Polidea do a lot of work with Beam as well so I’d love their thoughts if this a good use-case. via Newton Mail

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 binding via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.15.5=email_footer_2] On Tue, Aug 25, 2020 at 8:45 AM, Gerard Casas Saez wrote: + 1 (non-binding) Gerard Casas Saez Twitter | Cortex | @casassaez On Tue, Aug 25, 2020 at 7:18 AM

Re: Optimize KuberneteExecutor pod labels to task instance key

Hi Ping, I think that’s a great idea! Would be glad to help merge this. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.15.5=email_footer_2] On Sun, Aug 23, 2020 at 11:33 PM, Ping Zhang wrote: Hi everyone, I was evaluating using *KubernetesExcutor* and found the

Re: Simplifying the KubernetesExecutor

> Best wishes > > > > Ping Zhang > > > > > > On Thu, Aug 13, 2020 at 12:58 PM Jarek Potiuk > > wrote: > > > >> +1 for pod mutation in config. It's not a "yamly" thing - it's a python > >> > >> > >> code to run, s

Re: [VOTE] Release Apache Airflow 1.10.12 based on 1.10.12rc1

+1! On Sat, Aug 15, 2020, 1:38 PM Jarek Potiuk wrote: > Cool! Will take a look tomorrow ! > > sob., 15 sie 2020, 19:05 użytkownik Kaxil Naik > napisał: > > > Hello Apache Airflow Community, > > > > This is a call for the vote to release Apache Airflow version 1.10.12. > > > > The release

Re: Simplifying the KubernetesExecutor

lti-executor and pod > templates should be 2.1. > > T. > > On Wed, Aug 12, 2020 at 5:13 PM Daniel Imberman > wrote: > >> (Also funny enough we could use a lot of the existing infrastructure int >> 1.10 to create that migration script. Would just need to take the python &

Re: Simplifying the KubernetesExecutor

, 2020 at 8:09 AM, Daniel Imberman wrote: 100% agreed on timing. I think 2.0 should be for the breaking aspect (losing the configurations) and then 2.1/2.2 we can start adding on the new features. I also like the idea of a migration tool. We can make a script that takes your airflow.cfg

Re: Simplifying the KubernetesExecutor

, 2020 at 4:24 PM Daniel Imberman wrote: > Hello, fellow Airflowers! I hope you are all well in these trying times. > > > With the recent launch of Airflow 2.0 preparation, it now seems like a > good time to review the project's state and where we can fit in some > breaking changes

Simplifying the KubernetesExecutor

Hello, fellow Airflowers! I hope you are all well in these trying times. With the recent launch of Airflow 2.0 preparation, it now seems like a good time to review the project's state and where we can fit in some breaking changes that will improve the project for the future. When we first

Re: New Committers (Leah Cole & Ry Walker)

Congratulations both of you! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.14.6=email_footer_2] On Mon, Jul 20, 2020 at 7:26 AM, Tomasz Urbaszek wrote: Big congrats to you Leah and Ry! T. On Mon, Jul 20, 2020 at 4:18 PM Kaxil Naik wrote: > Hello Airflow Community,

Re: Separate Repo vs MonoRepo for Dockerfile & Helm Chart

/git-cherry-pick-or-merge-specific-directory-from-another-branch via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.14.6=email_footer_2] On Sun, Jul 5, 2020 at 9:57 AM, Daniel Imberman wrote: I can’t agree with this enough :). I think writing a few bots to separate out sections

Re: [VOTE] Release Apache Airflow 1.10.11 based on 1.10.11rc1

@kaxil could we post release candidates to our docker hub? Would make it easier to test via the helm chart. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.14.6=email_footer_2] On Thu, Jul 2, 2020 at 7:59 AM, Kaxil Naik wrote: Hello Airflow Community, This is a call for

Re: Separate Repo vs MonoRepo for Dockerfile & Helm Chart

, 2020 at 9:26 AM Jarek Potiuk wrote: > > On Thu, Jul 2, 2020 at 3:16 AM Daniel Imberman > wrote: > > I’m fine with keeping it as three separate repos but merging testing > > somehow (e.g. the source code chart would pull the helm/docker chart into > > .build) bu

Re: Separate Repo vs MonoRepo for Dockerfile & Helm Chart

I’m fine with keeping it as three separate repos but merging testing somehow (e.g. the source code chart would pull the helm/docker chart into .build) but we need to do it in a way that doesn’t make testing too difficult. So for example: How do I test/integration test a change that involves a

Re: [DISCUSS] Enable 'Black' for Auto Code Formatting

I think that once we add integration tests (which I’ll do now that we have helm for k8s) and we no longer have to back port this is a great idea. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.14.6=email_footer_2] On Wed, Jul 1, 2020 at 4:17 PM, Kaxil Naik wrote: That

Re: [UPDATE] AIP-31 .output update

+1 (binding) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.14.6=email_footer_2] On Wed, Jun 17, 2020 at 12:13 AM, Tomasz Urbaszek wrote: +1 (binding) On Wed, Jun 17, 2020 at 3:39 AM 蒋晓峰 wrote: > +1(not binding) > > On Wed, Jun 17, 2020 at 3:03 AM Gerard Casas Saez >

Re: [AIP-34] Rewrite SubDagOperator

> and scheduling. It will be the same as not using SubDag. > > > > > > > 2. Still have the benefits of modularized and reusable dag code > > and > > > > > > > declare dependencies between them. And with the new > > SubDagOperator > > > > (see > > >

Re: [AIP-34] Rewrite SubDagOperator

om one to another. > > > I wonder if we could essentially write in the ability to set > dependencies to all starter-tasks for that DAG. > Can you elaborate on *the ability to set dependencies to all starter-tasks > for that DAG*? > > Bin > > On Fri, Jun 12, 2020 at 10:28

Re: [AIP-34] Rewrite SubDagOperator

I hadn’t thought about using the `>>` operator to tie dags together but I think that sounds pretty great! I wonder if we could essentially write in the ability to set dependencies to all starter-tasks for that DAG. I’m personally ok with SubDag being a mostly UI concept. It doesn’t need to

Re: Unsubscribe from dev-list

Not a worry Fokko! Mental health is crucial right now. We look forward to seeing you in these conversations again soon :) On Mon, Jun 1, 2020 at 9:13 AM, Driesprong, Fokko wrote: Hi all, I've decided to take a break from the dev-list. There is a lot of action on the list, which is great.

Re: Our CI architecture as a model for GA <> DockerHub integration for ASF projects

Cool!! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.14.6=email_footer_2] On Thu, May 28, 2020 at 12:20 AM, Jarek Potiuk wrote: Hello everyone, I just wanted you to know that our way of integration GitHub Actions with the DockerHub has become one of the model examples

Re: Proposal: Change TI from having execution date to dag_run_id (in API at least)

+1 for sure! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.50=10.14.6=email_footer_2] On Thu, May 14, 2020 at 1:18 PM, Dan Davydov wrote: +1 but in the future I think better would be /dags/{dag_id}/dagRuns/{execution_date}/{run_number}. That would give an automatic ordering

Re: [DISCUSS] DockerHub vs. Github Image registry for "master"/"v1-10-test" builds

Yeah, I'm not worried about DDOS as long as the URL is stored in a secret/doesn't show up in the github action UI. On Tue, Apr 21, 2020 at 6:29 AM Ash Berlin-Taylor wrote: > I'm still not quite sure what problem are we solving here either...? > What is broken with the current/already merged

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc2

+1 (binding) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.32=10.14.6=email_footer_2] On Fri, Apr 3, 2020 at 11:10 AM, Kaxil Naik wrote: +1 (binding) On Fri, Apr 3, 2020 at 6:55 PM Kaxil Naik wrote: > Hello Airflow Community, > > This is a call for the vote to release

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

@jarek when you’re running on breeze how do you actually run airflow s.t. I can see the UI/run DAGs? It seems like it downloads everything, but it doesn’t seem like a full airflow deployment. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.32=10.14.6=email_footer_2] On Fri, Apr

Re: [VOTE] AIP-32: Airflow REST API

Looking forward to it! +1! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.32=10.14.6=email_footer_2] On Fri, Apr 3, 2020 at 4:06 AM, Kaxil Naik wrote: I had the same concern with the Auth model (Had commented on the AIP on confluence:

Re: Helm Chart for Airflow

comments and hope to merge it very soon. J. On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk < jarek.pot...@polidea.com [jarek.pot...@polidea.com] > wrote: Absolutely! Please do :). The more eyes on this the better! On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman < daniel.imber...@gmail.com

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

@jarek once we merge the helm chart we can add a breeze command to do the same with the k8sexecutor on a kubernetes cluster :) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.32=10.14.6=email_footer_2] On Fri, Apr 3, 2020 at 6:41 AM, Jarek Potiuk wrote: Yep. And the breeze

Re: Let's talk Airflow 2.0

the task_instance table as a marker to group these tasks together when rendered in the UI. Let me know how you guys think. Thanks Bin On Thu, Apr 2, 2020 at 9:55 AM Daniel Imberman wrote: > Hello all, > > I've been reviewing This wiki page > <https://cwiki.apache.org/confluence/disp

Re: Let's talk Airflow 2.0

id slow or flakey UI tests. > This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've been taking time to learn cypress recently, and once the airflow helm chart is merged I think merging a set of integration/behavior/UI tests is crucial What does everyone think? Open to sug

Re: Let's talk Airflow 2.0

taneously. It is not easy to upgrade two complex systems at the same time. However, if we do this, some users will have to do it. Older versions can be hidden behind the feature gate. We can also add deprecation warnings. > > R > > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, > wrote: > &

Re: Let's talk Airflow 2.0

Core changes without rewriting the PRs. Regards, Kaxil On Tue, Mar 24, 2020 at 2:54 PM Daniel Imberman wrote: > Hi Robin, > > > > I feel some of the stuff for instance Schedular HA could wait for a > point > > > release of version 2 (although maybe this a lot

Re: New committer: Jiajie Zhong

Woo!! Congratulations Jiajie! Well deserved :) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.32=10.14.6=email_footer_2] On Tue, Mar 31, 2020 at 10:29 AM, Jarek Potiuk wrote: The Project Management Committee (PMC) for Apache Airflow has invited Jiajie Zhong to become a

Re: How about switching from Celery to RQ?

I mean… we’re not planning it (kind of an “if it’s not broke don’t fix it” situation), but I don’t think we’re super set on Celery. Would you be interested in making an AIP to discuss potential benefits? On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen , wrote: > Hi there, > > Looks

Re: Helm Chart for Airflow

ove forward all of that and have a PR that we > can start running tests on and replacing the resources with helm > chart. > > J. > > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman > wrote: > > > > @Steven glad to help you out on that. > > > > For the moment

Re: Helm Chart for Airflow

CI working to test it. Where > do we want to run it? Do you all just use Travis only? > > Steven > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman > wrote: > > > @Jarek I think with the helm chart + prod image we can go even further > > than that :). We can test C

Re: Helm Chart for Airflow

ail :) > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman > > wrote: > > > We’re not using CRDs for the tests at the moment. We just have deployment > > > files. If anything having the helm chart as a part of the airflow repo > > > could mean that the helm

Re: [VOTE] AIP-31: Airflow functional DAG definition

+1 binding. I volunteer to help implement as well :D On Mar 26, 2020, 7:15 AM -0700, Jarek Potiuk , wrote: > +1 binding > > On Thu, Mar 26, 2020 at 8:59 AM Sumit Maheshwari > wrote: > > > +1 binding > > > > On Thu, Mar 26, 2020 at 9:50 AM Chao-Han Tsai > > wrote: > > > > > +1 binding > > > > > >

Re: Helm Chart for Airflow

kerfiles > > > > > > pull this package in. These scripts are used to coordinate running > > > > > > migrations and cleaning up failed pods. > > > > > > > > > > > > > > > > I see two scripts: > > > > > >

Re: Requesting to give details on FacebookAdsHook project

Hi Chetan, Please see our documentation here https://airflow.apache.org/. Please feel free to reach out if you have further questions. On Mar 25, 2020, 10:29 AM -0700, Chetan Agarwal , wrote: > Dear Sir, > > I am Chetan Agarwal, third-year Computer Science and Engineering > Student at National

  1   2   >