Re: [DICUSS][AIP-40] Deferrable ("Async") Operators

Thanks Jarek - some replies to those questions: 1) In general, I'm envisioning Triggers as a thing that are generally abstracted away from DAG authors - instead, they would come in a provider package (or core Airflow) and so we would be expecting the same higher level of quality and testing. That

Re: [DICUSS][AIP-40] Deferrable ("Async") Operators

Sorry I missed the original thread, it is very interesting :). I love the Trigger concept, and I have two comments/questions, mostly ones that might make the concept more "fool-proof" for even casual users, who might not understand the intrinsic details of Triggers and not understand how they impa

Re: [DICUSS][AIP-40] Deferrable ("Async") Operators

Not that soon - December 2021 is the EOL for Python 3.6, so I while it could be only 3.7+ feature, but warnings would still be nice. I don't think we discussed either timeline or scope for 2.1 or 2.2 but I think having them sooner both before December might be a good idea :) J. On Mon, Apr 26, 2

Re: [DICUSS][AIP-40] Deferrable ("Async") Operators

@ash since we’re dropping 3.6 support soon anyways, would we even need this warning? Based on the POC it seems that this involves core changes, so I imagine if this is only 2.2.X compatible then ending py3.6 support for 2.2.X would be sufficient? (unless truly ending 3.6 support would need to wa

Re: [DICUSS][AIP-40] Deferrable ("Async") Operators

Yes - the prototype currently has a `airflow.utils.asyncio` module that tries to adapt things for 3.6, which I will likely just replace with a series of very polite errors about needing 3.7+. Andrew On Mon, Apr 26, 2021 at 9:50 AM Ash Berlin-Taylor wrote: > Cool! > > I see that you are using as

Re: [DICUSS][AIP-40] Deferrable ("Async") Operators

Cool! I see that you are using async code for it in the draft PR, and I remember you saying somewhere that Python 3.6 (which is still supported) doesn't have great support for this. Should we have an explicit check for Py 3.6 and refuse to allow triggers to be run there? (and not let `trigge

[VIRTUAL MEETUP REMINDER] April 29th Agenda

👉 Meetup agenda for April 29, 10am PST (-8 GMT) 👈 For this edition, we will have liv

Re: [LAZY CONSENSUS] Policy for supporting K8S versions

Cool. So if no-one else objects till the end of week I will update the description to include K8S and make it clear that such Python/K8S 'dropping' will happen with minor version bump of Airflow.. J. On Mon, Apr 26, 2021 at 3:36 PM Kaxil Naik wrote: > Hi all, > > Dropping support for a Python v

Re: [DICUSS][AIP-40] Deferrable ("Async") Operators

Thanks Andrew, that answers my questions. If we don't have any other questions by the end of the week we should start a VOTE. Regards, Kaxil On Tue, Apr 20, 2021 at 2:24 AM Andrew Godwin wrote: > Thanks Kaxil - notes on those two things: > > - A timeout is probably a reasonable thing to have

Re: [LAZY CONSENSUS] Policy for supporting K8S versions

Hi all, Dropping support for a Python version in my opinion is not a breaking change mainly because it does not affect current users. We will simply change python_requires field in setup.cfg: https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires. i.e. if we c

Re: Configuring DagFileProcessorManager Parallelism

You *should *probably see a spike in your CPU on increasing max_threads. https://github.com/apache/airflow/blob/1.10.12/airflow/utils/dag_processing.py#L1244 Regarding the timeout in your DAG, it is difficult to say without checking your DAG code, do you code outside of DAG and Task/Operator obje

Configuring DagFileProcessorManager Parallelism

Hello everyone, Me and my team require some inputs on configuring max_threads of airflow scheduler. *Deployment setup:* We are running airflow on Kubernetes with CeleryExecutor We currently have thousands of dags and are running with max_threads configuration of 40. Airflow version: 1.10.12 We h