Re: [VOTE] Release Apache Airflow 2.0.0 form 2.0.0rc3

+1 binding !!! On Wed, Dec 16, 2020 at 2:42 PM Tomasz Urbaszek wrote: > +1 binding, tested it with kubernetes based deployments using > CeleryExecutor as well as locally > > Tomek > > On Wed, Dec 16, 2020 at 9:42 PM Ry Walker wrote: > >> +1 (non-binding) >> >> On Wed,

Re: [VOTE] Enable Github Discussions on Apache Airflow Github Repo

+1 (binding) On Tue, Sep 22, 2020 at 12:40 PM Kaxil Naik wrote: > @tomek - Yup we could do that > > On Tue, Sep 22, 2020 at 8:35 PM Tomasz Urbaszek > wrote: > >> +1 binding >> >> Should we change "Ask a question or get support" issue type to point >> to GH discussion? >> >> T. >> >> >> On Tue,

Re: [DISCUSS] Removing Pickling from Airflow 2.0

I'm getting bad flashbacks of fighting with pickles early on in the history of the project. I've learned since then to stay away. Almost all solutions that involve pickles are bad solutions. Beyond but related to the security implication are the issues of pickle entanglement, not really knowing

Re: Discuss: should we allow HTML emails on dev@ list

+1 how about +1000² to "let's make html email a default thing in all the lists" On Mon, Sep 14, 2020 at 12:14 PM Felix Uellendall wrote: > +1 > > On Mon, Sep 14, 2020 at 20:14, Kamil Breguła > wrote: > > > +1 (binding) > > > > On Mon, Sep 14, 2020, 19:52 Tao Feng wrote: > > > >> +1 > >> > >>

Re: Consider using stale bot for issues

About labeling on Github, I wrote a bot for Superset that enables anyone to apply labels through commenting using emojis. It does other things like auto-labeling organization on an allow-list. We could teach it new tricks top. Generalizing it or porting it to work in other communities should be

Re: Import style in Airflow codebase

+1 On Wed, Sep 9, 2020 at 10:36 AM Vikram Koka wrote: > +1 on absolute import. Honestly, a huge fan of doing it as an absolute vs. > relative. > > On Wed, Sep 9, 2020 at 4:09 AM Kaxil Naik wrote: > > > No strong opinion but absolute import seems better from a user's > > perspective. > > > > On

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 (binding) On Wed, Aug 26, 2020 at 12:34 AM Kevin Yang wrote: > +1 (binding) > > On Tue, Aug 25, 2020 at 2:47 PM Kamil Breguła > wrote: > > > +1 (binding) > > > > On Tue, Aug 25, 2020, 23:35 Daniel Imberman > > wrote: > > > > > +1 binding > > > > > > via Newton Mail > > > [ > > > > > >

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

INTHEWILD.md is as close to a standard as it gets for this. I'd advocate to move the list there and have a link to it in the README. Here's the Superset one for reference: https://github.com/apache/incubator-superset/blob/master/INTHEWILD.md Personally I like a short clean README with links to

Re: [Discuss] Improving AIP process documentation

For reference, here's SIP-0 (Superset Improvement Proposal 0) which defines the SIP process for Apache Superset: https://github.com/apache/incubator-superset/issues/5602 We also have a GitHub issue template for it:

Re: Faster builds on CI + increased stability + easier to reproduce CI problems

Great work! Investments in CI pay dividends to the whole community. On Sat, Aug 22, 2020 at 8:12 AM Jarek Potiuk wrote: > Hello everyone, > > Just wanted to let you know that we merged last week quite an overhaul of > the CI architecture we have in Github Actions. > > TL;DR; It should be

Re: [PROPOSAL][AIP-36 DAG Versioning]

; > > The worker has completed the execution of Task B and is just about > to > > > > > complete the execution of Task B. > > > > > > > > > > The 2nd version of DAG is deployed: Task A -> Task D > > > > > Now Scheduler queued

Re: [PROPOSAL][AIP-36 DAG Versioning]

gt; > would be to render independent, individual tree views for each DAG > version > > (side by side), and doing best effort aligning the tasks across blocks > and > > "linking" tasks with lines across blocks when necessary. > > > Agreed, the plan is to

Re: [PROPOSAL][AIP-36 DAG Versioning]

Some notes and ideas: *DAG Fingerprinting: *this can be tricky, especially in regards to dynamic DAGs, where in some cases each parsing of the DAG can result in a different fingerprint. I think DAG and tasks attributes are left out from the proposal that should be considered as part of the

Re: [AIP-34] Rewrite SubDagOperator

+1, proposal looks good. The original intention was really to have tasks groups and a zoom-in/out in the UI. The original reasoning was to reuse the DAG object since it is a group of tasks, but as highlighted here it does create underlying confusions since a DAG is much more than just a group of

Re: [VOTE] Naming of the transfer operators/Hooks

+1 for [1] XToYOperator On Sat, May 30, 2020 at 7:33 AM Felix Uellendall wrote: > +1 for [1] XToYOperator > > Best Regards, > Felix > > Sent from ProtonMail Mobile > > On Sat, May 30, 2020 at 13:58, Kaxil Naik wrote: > > > +1 for [1] XToYOperator > > > > On Sat, May 30, 2020, 12:56 Tomasz

Re: What's coming in Airflow 2.0 =- this Wednesday at NYC Online Meetup

I was bummed about not being able to make live today, but found that the video is available already and was able to watch it just now. https://www.crowdcast.io/e/whats-coming-airflow-2 Amazing work presenters and committers! It's fantastic to see all of this coming together. Max On Mon, May 11,

Re: [VOTE] AIP-15: Support Multiple-Schedulers for HA & Better Scheduling Performance

+1 (binding) Solid work! On Tue, Mar 17, 2020 at 10:43 PM Jarek Potiuk wrote: > +1 (binding) > > On Tue, Mar 17, 2020 at 11:16 PM Kaxil Naik wrote: > > > > +1 (binding) > > > > On Tue, Mar 17, 2020 at 10:06 PM Deng Xiaodong > wrote: > > > > > +1 (binding). > > > > > > Thanks for proceeding

Re: [VOTE] Switch from using Jira to Github Issues

+1 binding On Mon, Mar 16, 2020, 7:45 AM Jarek Potiuk wrote: > +1 binding. > > Re: contention: I think we can split reviewers by components. I think the > assignment to components was done recently so duplicates will be rare. I am > happy to be one of the reviewers. > > On Mon, Mar 16, 2020 at

Re: Airflow growth graphic?

until first comment on issues, time until first review on PRs, ...) * ... It should be useful for most communities, I'll share the link when I post Max On Wed, Feb 26, 2020 at 12:35 PM Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > > https://star-history.t9t.io/#apache/a

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

Hey, I wanted to echo the awesomeness once more, but also bring up the question as to whether any of this work may make it harder to distribute / HA the scheduler down the line (?) I almost started analyzing the code and thought it'd just be easier to ask the authors. Max On Wed, Feb 26, 2020

Re: Airflow growth graphic?

https://star-history.t9t.io/#apache/airflow/oozie/luigi/pinball/dbt/dagster/prefect [image: Screen Shot 2020-02-26 at 12.34.01 PM.png] On Wed, Feb 26, 2020 at 12:06 PM Alex Tronchin-James 949-412-7220 < alex.n.ja...@gmail.com> wrote: > Hello and happy Wednesday, Airflow community mind! > > I saw

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

Nice! On Tue, Feb 25, 2020 at 12:11 AM Robin Edwards wrote: > This is brilliant work, thank you! Looking forward to watching my RDS > metrics when this gets deployed :-) > > On Tue, 25 Feb 2020, 07:08 Driesprong, Fokko, > wrote: > > > Sweet work Kamil and others! I'll try to go through them

Re: Airflow and Machine Learning

I'd have a lot of thoughts to unpack here, but top of mind is a deeper integration with [jupyter] notebooks and/or hosted notebooks-type systems. Notebooks [with papermill ] can be parameterized predictably, and notebook files provide rich log outputs

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

+1 On Mon, Feb 17, 2020 at 7:32 AM Daniel Imberman wrote: > +1 on my end! > > via Newton Mail > [ > https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.32=10.14.6=email_footer_2 > ] > On Mon, Feb 17, 2020 at 12:30 AM, Driesprong, Fokko > > wrote: > I like this as well. It will hopefully also reduce

Re: [DISCUSS] Packaging DAG/operator dependencies in wheels

This reminds me of the "DagFetcher" idea. Basically a new abstraction that can fetch a DAG object from anywhere and run a task. In theory you could extend it to do "zip on s3", "pex on GFS", "docker on artifactory" or whatever makes sense to your organization. In the proposal I wrote about using a

Re: [VOTE] Add Probot Integrations to Airflow Github Repo

+1! I forgot whether I shared this before, but I wrote a probot for Apache Superset that does a few nice things, and we can grow over time to do more things without having to beg Apache Infra for anything (currently only them have admin rights and can register apps on our repos)

Re: Scheduler Stuck when using LocalExecutor and KubernetesPodOperator

Friend don't let friends use LocalExecutor in production. LocalExecutor is essentially a subprocess pool running in-process. When I wrote it originally I never thought it would ever be used in production. Celery / CeleryExecutor is more reasonable as Celery is a proper process/thread pool that's

Re: [DISCUSS] Using shared memory for inter-task communication

If memory is shared across tasks, they are by definition not idempotent, which can be troublesome. What if you have a chain of 3 tasks and the last one failed while operating on the memory that came from task number 2? The whole chain may have to be re-executed, which to me sounds like it's really

Re: [VOTE] Accept new Airflow website contribution

+1 (binding) this is amazing! it's hard to believe that the project gained so much traction despite the website that we had before. On Thu, Nov 21, 2019 at 5:01 PM Deng Xiaodong wrote: > +1 (binding) for the new website. It will be very value-adding. Thanks! > > XD > > On Fri, Nov 22, 2019 at

Re: [DISCUSS] Airflow Summits dates and locations

What mailing list should I send this to? Should I assume that they have any context about the summits? On Thu, Nov 21, 2019 at 3:02 PM Aizhamal Nurmamat kyzy wrote: > Updated the proposal with the exact dates and locations [1] > > +Maxime Beauchemin could you please send the &

Re: [DISCUSS] Airflow Summits dates and locations

+1, the proposed dates and location seems reasonable to me. My heart would pick Paris but London is more rational :) Max On Wed, Nov 20, 2019 at 3:46 PM Aizhamal Nurmamat kyzy wrote: > Hi all, > > I am sharing with you a list of some industry events that are happening > from April to October

Re: [Discuss] Airflow Summits 2020

it. I also added names of volunteers who >> > wants >> > > to help to organize each summit, please add your names and affiliation >> > if I >> > > missed someone. My suggestion is that we don't have more than 2 >> > organizers >> > >

Re: Donating code to add Common Workflow Language import to Airflow

After all the exploration of this topic here in this thread, I'm a pretty hard -1 on this one. I think CWL and CWL-Airflow are great projects, but they can't rely on the Airflow community to evolve/maintain/package this integration. Personally I think that generally and *within reason* (winking

Re: Donating code to add Common Workflow Language import to Airflow

The big question is why can't it just be on its own Github repository and in its own PyPI package? Why does it have to be packaged with our PyPI package or live in the Airflow repo? Max On Wed, Nov 13, 2019 at 12:42 PM Andrey Kartashov wrote: > I don't quite get what this example should to

Re: Drop Python 3.5 support?

+1 On Tue, Nov 12, 2019, 1:00 PM Bolke de Bruin wrote: > Hi All, > > Can we drop python 3.5 support and switch to 3.6 as a minimum? > > Cheers > Bolke >

[upcoming event] Data Orchestration Summit

Hey, Quick last minute note about a somewhat-Airlfow-related event coming up: the Data Orchestration Summit (@ Computer History Museum in Mountain View) this Thursday. I'm scheduled to be on a panel about creating open source projects and

Re: Donating code to add Common Workflow Language import to Airflow

As someone who has spent a lot of time acting as a maintainer, a code "donation" seems like dangerous gift to accept. Personally I like the idea of an ecosystem of packages (and repos) managed and maintained by their specialist. That way they can have their own CI, their own release processes and

Re: Announcing SIG-Knative/ The Monthly Knative Executor Meetup/Call-for-contributers

Nice! On Wed, Oct 16, 2019 at 1:19 PM Ash Berlin-Taylor wrote: > We meant October 30th (the date in the calendar is correct) > > -a > > > On 16 Oct 2019, at 18:57, Daniel Imberman > wrote: > > > > Fixing the calendar link :) > > >

Re: Airflow 1.10.5 Python 2.7 compatability

Fokko, surprised to see you use 3.5 . I thought most people went straight to 3.6 as soon as it came out, as there are very little reasons to be on the 3 to 3.5 range anymore I'd say we should go straight to 3.6+ for Airflow as we deprecate 2.7 https://pythonclock.org/ On Mon, Sep 16, 2019 at

Re: Setting to add choice of schedule at end or schedule at start of interval

> >>> range of 1 day but then if server down for couple days, this could be > > >>> caught up in one task and if you backfill it could be up to 30-day > > >> batches. > > >>> > > >>> Perhaps there is an abstraction tha

Re: Setting to add choice of schedule at end or schedule at start of interval

How about an alternative approach that would introduce 2 new keyword arguments that are clear (something like, but maybe better than `period_start_dttm`, `period_end_dttm`) and leave `execution_date` unchanged, but plan it's deprecation. As a first step `execution_date` would be inferred from the

Airflow maintenance blog post

It's common to have these maintenance DAGs, but it's great to have people share theirs and blog about it. Thanks to Robert Sanders at Clairvoyant for this post and code repo https://blog.clairvoyantsoft.com/automated-maintenance-for-apache-airflow-8d844f32737d

Re: [IE] Re: [VOTE] Change the Airflow logo

Option 1 ++ On Wed, Aug 21, 2019 at 9:52 AM Tao Feng wrote: > Option 1 > > On Wed, Aug 21, 2019 at 8:26 AM Leah Cole > wrote: > > > Option 1 > > > > On Wed, Aug 21, 2019 at 7:01 AM Sunil Varma Chiluvuri < > > sunilvarma.chiluv...@equifax.com> wrote: > > > > > Option 1 > > > > > > On Wed, Aug

Re: Airflow Dynamic tasks

e: 3.93859595 > > > > Parsing in time of execution, when scheduler submits the DAGs: > > DagBag parsing time: 99.820316 > > > > Delay between the task run inside a single DAG grow from 30 sec to > 10 min, > > then it drop

Re: [ANNOUNCE] Please welcome new Airflow committer Chao-han Tsai

We'll deserved! Welcome aboard! On Mon, Aug 19, 2019, 9:54 AM Aizhamal Nurmamat kyzy wrote: > Congratulations, Chao-Han! Thank you for your contributions! > > On Fri, Aug 16, 2019 at 7:58 PM Jarek Potiuk > wrote: > > > Congrats Chao-Han! > > > > On Fri, Aug 16, 2019 at 6:32 PM Kevin Yang

Re: [DISCUSS] Tweaks to the Airflow logo

+1! (as the "designer" of the original) :) Please make sure to share the vector files in the repo as well Also don't forget to upload here https://www.apache.org/logos/ On Tue, Aug 13, 2019 at 12:12 PM Felix Uellendall wrote: > +1 looks so much cleaner and more modern :) > > Felix > > Am

Blog post: Upgrading & Scaling Airflow at Robinhood

Thanks to Abhishek Ray @ Robinhood for this great post. I felt like I had to share it here https://robinhood.engineering/upgrading-scaling-airflow-at-robinhood-5b625dfaa2ee Max

Re: Airflow DAG Serialisation

Great to see this happening! On Fri, Jul 26, 2019 at 8:54 AM Jarek Potiuk wrote: > Great! That's definitely one of the most painful aspects of Airflow. Happy > to help/comment/take part in the discussions and later in the > implementation. > > On Fri, Jul 26, 2019 at 4:48 PM Deng Xiaodong

Re: Running Pylint/Flake/Mypy/Docs/Licence checks locally

`pre-commit` is pretty great if someone wants to set it up. https://pre-commit.com/ Here's what our configuration looks like for Superset: https://github.com/apache/incubator-superset/blob/master/.pre-commit-config.yaml It's super fast too as it only executes for the files that have been touched

Re: [PROPOSE] Introduce and encourage pre-commit hooks framework to Airflow developer workflow

+1 On Mon, Jul 1, 2019 at 11:20 PM Kaxil Naik wrote: > +1 We have been using this on some of our Astronomer repositories as well > and have been happy with it. > > Regards, > Kaxil > > On Tue, Jul 2, 2019, 11:46 Jarek Potiuk wrote: > > > TL;DR: I would like to make a proposal to add (easily

Re: Latest Sphinx (2.1.2) causes master to fail

< > > https://poetry.eustace.io/>)? How does this PR compare? > > > > -ash > > > > > > > On 24 Jun 2019, at 22:02, Maxime Beauchemin < > maximebeauche...@gmail.com> > > wrote: > > > > > > This Superset PR may be relevant a

Re: Latest Sphinx (2.1.2) causes master to fail

This Superset PR may be relevant and useful for Airflow too (still WiP): https://github.com/apache/incubator-superset/pull/7762 Basically it allows your project to define a "requirements/" folder with a "requirements.json" as a single source of truth for all deps, including the details of what

Re: [VOTE] AIP-16: CLI: Use nested commands instead of flags

+1 On Tue, Jun 11, 2019 at 12:29 PM Bas Harenslak < basharens...@godatadriven.com> wrote: > +1 (binding) > > > On 11 Jun 2019, at 18:59, Tao Feng fengta...@gmail.com>> wrote: > > +1 (binding) > > On Tue, Jun 11, 2019 at 4:15 AM Ash Berlin-Taylor a...@apache.org>> wrote: > > Hi Airflowers, > >

Re: Use -x when cherry-picking to v1-10 branches

Always, always use `-x` when cherry-picking! On Mon, Apr 22, 2019 at 1:28 PM Jarek Potiuk wrote: > Hello Everyone (committers especially). > > I have a proposal to improve slightly the cherry-picking process between > master and v1-10-branches: We could use `-x` flag when cherry-picking. This >

Re: Is `airflow backfill` disfunctional?

eers, Fokko > > Op za 13 apr. 2019 om 23:26 schreef Maxime Beauchemin < > maximebeauche...@gmail.com>: > > > +1, backfilling, and related "subdag surgeries" are core to a data > > engineer's job, and great tooling around this is super important. > Backfill

Re: [VOTE] AIP-6 Apply Pylint to Airflow

pylint and black are super solid, no questions there afaic Max On Mon, Apr 15, 2019 at 11:48 AM m...@maximilianroos.com wrote: > Hi there, > > I haven't been active in the airflow community so this should be weighed > appropriately. > > I'm a core dev of a couple of other libraries (xarray,

Re: [2.0 spring cleaning] Require unique conn_id

People may rely on this feature for [poor man's] load balancing though, I forgot what the exact use case was but used this at Airbnb at some point. Maybe the solution is to make the UI/UX/log output much more clear around this. Making the CLI log more clear should be really easy to do, web server

Re: [2.0 spring cleaning] Deprecate adding Operators and Hooks via plugins?

I'd say it could be great to deprecate the whole plugin system and use Python's "entry points" instead. I just didn't know that was an option and the standard way to do this when I originally wrote it... The current plugin system is a minefield for circular dependencies... On Sat, Apr 13, 2019 at

Re: Airflow 1.10.3 has been released!

Infinite kudos! Thanks Ash! On Fri, Apr 12, 2019 at 1:33 PM Kaxil Naik wrote: > Thanks Ash and all the contributors. > > On Fri, Apr 12, 2019, 20:38 Robin Edwards wrote: > > > Yes thanks everyone for their hard work :-) > > > > R > > > > On Fri, 12 Apr 2019 at 18:38, Feng Lu wrote: > > > > >

Re: [2.0 spring cleaning] Rename and re-icon the refresh button

Sounds to me like this is just a UI improvement, not a deprecation / spring cleanup. A simple PR should insure this fix gets in the next version. On Fri, Apr 12, 2019 at 8:49 AM James Meickle wrote: > https://issues.apache.org/jira/browse/AIRFLOW-3816 > > To quote: > > There's a "Refresh"

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

art & end datetime of a DAG run. > >> * Other: > >> * Removal of variables should be done in major version and > >> deprecation warnings should be added. > >> > >> So how about the following: > >> - We start by putting deprecation warnings

Re: [VOTE] AIP-6 Apply Pylint to Airflow

+1 (binding) Also check out black to auto-pep8! https://github.com/ambv/black On Thu, Apr 11, 2019 at 5:12 PM Tao Feng wrote: > +1 > > On Thu, Apr 11, 2019 at 4:27 PM Beau Barker > wrote: > > > +1 non binding > > > > Pylint is extremely strict so your may want to be selective about the > >

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

ation warnings on tables, latest_date, > > end_date and END_DATE and remove them in Airflow 2.0. > > - We add a lineage_enabled config which is false by default and thus > > inlets/outlets aren’t provided, unless set to true. > > - We continue discussion about yesterday* and tomorrow* > &

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

Making backwards incompatible changes that require altering the thousands (millions?!) of DAGs in the wild will alienate the community and prevent many from orchestrating an upgrade. Upgrading hundreds of DAGs and Airflow atomically would be hard and dangerous. To mitigate this, changes to the

Re: [VOTE] AIP-10: Multi-layered and multi-stage official Airflow image

+1 binding Max On Sun, Apr 7, 2019 at 5:23 PM Jiajie Zhong wrote: > +1 non binding > > Best wish. > -- Jiajie > > From: Chao-Han Tsai > Sent: Sunday, April 7, 2019 0:45 > To: dev@airflow.apache.org > Subject: Re: [VOTE] AIP-10: Multi-layered and multi-stage

Re: Potential AIP: Selenium/User testing

Side note: I'd recommend using Cypress for that. We've had a good experience using it on Superset https://www.cypress.io/ Max On Tue, Apr 2, 2019 at 11:23 AM Daniel Imberman (BLOOMBERG/ SAN FRAN) < dimber...@bloomberg.net> wrote: > Hello fellow airflowers! > > I've noticed a few times on the

FAB - New REST API in the works

Hey! I wanted to point out that there's awesome work taking place in FAB around a new REST API provided by the framework, and ways to extend it. Daniel Gaspar (cced) is working on this currently, and looking for input on design / implementation. Check it out and chime in on the PR

Re: [VOTE] Accept AIP-3: Drop support for Python 2

+1 (binding) On Sun, Mar 24, 2019 at 8:13 AM Jiajie Zhong wrote: > +1 (binding) > > could make Airflow easy to maintenance. > > > Best wish. > -- jiajie > > From: Felix Uellendall > Sent: Sunday, March 24, 2019 19:43 > To: dev@airflow.apache.org > Subject: Re:

Re: Multiple Schedulers - "scheduler_lock"

.wikipedia.org/wiki/Birthday_problem> > >> > >> > >>> On 2 Mar 2019, at 3:39 PM, Tao Feng wrote: > >>> > >>> Does the proposal use master-slave architecture(leader scheduler vs > slave > >>> scheduler)? > >>> > &

Re: Multiple Schedulers - "scheduler_lock"

Forgot to mention: the intention was to use the lock, but I never personally got to do the second phase which would consist of skipping the DAG if the lock is on, and expire the lock eventually based on a config setting. Max On Fri, Mar 1, 2019 at 1:57 PM Maxime Beauchemin wrote: > My origi

Re: Multiple Schedulers - "scheduler_lock"

My original intention with the lock was preventing "double-triggering" of task (triggering refers to the scheduler putting the message in the queue). Airflow now has good "double-firing-prevention" of tasks (firing happens when the worker receives the message and starts the task), even if the

Re: [DISCUSS] AIP-12 Persist DAG into DB

asily, if someone wants a > > full brain dump on this I can share what I know > > What was the preference for using Pickle over Docker/PEX for serialization? > I think we discussed this a long time ago with Paul but I forget the > rationale and it would be good to have the informati

Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?

gt; > >> I’m glad yarn wasn’t the only option - it would have meant I’d have > never > >> been in a position to use Airflow! (Many of our workflows don’t touch > >> EMR/Hadoop, and running Celery is much more of a known element to a > python > >> deve

Re: Short Airflow user survey

+1, this is great and we should do it periodically! On Mon, Feb 25, 2019 at 10:42 AM Dan Davydov wrote: > This is very interesting and useful, big thanks for conducting the survey! > > On Mon, Feb 25, 2019 at 12:24 PM Ash Berlin-Taylor wrote: > > > Thanks for all those who answered, there's

Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?

>From memory, I think MesosExecutor depends on pickling to get DAG definitions to workers, which we should also deprecate. About CeleryExecutor, we never had the intention to make it the recommended option for production early on. The intent back in 2014 was to write a YarnExecutor quickly (that

Re: [PROPOSAL] Add a landing page for Apache Airflow

+1, also I hear Gatsby is great. I did some research on this a while back but never was able to commit the time It'd be great to move off readthedocs and onto something more modern-looking, but the RST auto-code-api-documentation magic is valuable and hard to reproduce elsewhere. There's probably

Re: AIP-8 Split Hooks/Operators into Separate Packages

should agree on a plan. Refactoring the hooks and operators out, a set at a time, seems like a really good start. Max On Thu, Jan 10, 2019 at 8:44 AM Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > That's not what I meant. If I apply what I meant to your example we'd have &

Re: AIP-8 Split Hooks/Operators into Separate Packages

inconvenient. > > > Sent with ProtonMail Secure Email. > > ‐‐‐ Original Message ‐‐‐ > On Wednesday, January 9, 2019 9:29 PM, Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > > > If there's a strict policy of having a single hook and a single operator &g

Re: AIP-8 Split Hooks/Operators into Separate Packages

mespace, and for example Google could > >>> choose to release a airflow-gcp-operators package now and tell people > to > >>> |from gcp.airflow.operators import SomeNewOperator|​.​ > >>> > >>> My view on this currently is -1 as I don't see it solving any problem > >

Re: AIP-8 Split Hooks/Operators into Separate Packages

Something to think about is how data transfer operators like the MysqlToHiveOperator usually rely on 2 hooks. With a package-specific approach that may mean something like an `airflow-hive`, `airflow-mysql` and `airflow-mysql-hive` packages, where the `airflow-mysql-hive` package depends on the

Re: [DISCUSS] Deprecate KnownEvent/KnownEventType

+1 On Wed, Jan 2, 2019 at 1:00 PM Dan Davydov wrote: > +1 to removing > > On Wed, Jan 2, 2019 at 10:48 PM Driesprong, Fokko > wrote: > > > Hi all, > > > > Recently I've opened up a PR > > to remove the > > KnownEvent and KnownEventType