Re: [VOTE] Airflow Providers prepared on January 30, 2024

+1 (binding) On Fri, Feb 2, 2024 at 1:28 PM Ephraim Anierobi wrote: > +1 (binding) > > On 2024/01/30 16:42:07 Elad Kalif wrote: > > Hey all, > > > > I have just cut an ad-hoc release for the microsoft.azure provider > package. > > This email is calling a vote on the release, > > which will last

Re: Idea for Discussion: custom TI dependencies

I thought that it would allow users to code their own rules and provide them to the tasks somehow. If this is restricted to admin or deployment managers through plugins or other mechanism, I guess there is no issue here. Thanks for answering my concerns. On Fri 2 Feb 2024 at 22:53, Constance Mart

Re: Idea for Discussion: custom TI dependencies

Not missing anything. It is mainly used for deferrable operators, but feels like an acceptable place to run user-defined/non-Airflow code portion of this, and carry out the custom dependency checks. Assuming of course that these checks are meant to check external conditions and will never need acce

Re: Idea for Discussion: custom TI dependencies

Thanks both for your inputs! Hi Pierre, I think the key difference here is: by doing this, we are not allowing Airflow “users” to run their code in scheduler. We are only allowing Airflow “Admins” to deploy a plugin to run in scheduler, just the same as dag_policy/task_policy/task_instance_mu

Re: Idea for Discussion: custom TI dependencies

Naive question: Instead of running the code on the scheduler - could the condition check be delegated to the triggerer? On Fri, Feb 2, 2024 at 2:33 PM Pierre Jeambrun wrote: > But maybe it’s time to reconsider that :), curious to see what others > think. > > On Fri 2 Feb 2024 at 20:30, Pierre Je

[VOTE] Add the ability to report slack messages that don't meet code of conduct

Hey All, I'm breaking this out into another conversation because I think it warrants its own decision before we move forward. Last week, I proposed we add a code of conduct for Airflow Slack and in-person meetups. Thread: In the

Re: Idea for Discussion: custom TI dependencies

But maybe it’s time to reconsider that :), curious to see what others think. On Fri 2 Feb 2024 at 20:30, Pierre Jeambrun wrote: > I like the idea and I understand that it might help in some use cases. > > The first concern that I have is that it would allow user code to run in > the scheduler, i

Re: Idea for Discussion: custom TI dependencies

I like the idea and I understand that it might help in some use cases. The first concern that I have is that it would allow user code to run in the scheduler, if I understand correctly. This would have big implications in terms of security and how our security model works. (For instance the schedu

Idea for Discussion: custom TI dependencies

Hi folks, I’m writing to share my thought regarding the possibility of supporting “custom TI dependencies”. Currently we maintain the dependency check rules under “airflow.ti_deps.deps". They cover the dependency checks like if there are available pool slot/if the concurrency allows/TI trigger

Re: [VOTE] AIP 61 - Hybrid Executors

+1 (non-binding) This will be an awesome feature!! -- Rajesh On 2024-02-02, 12:44 AM, "Mehta, Shubham" mailto:shu...@amazon.com.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and

Re: [VOTE] Airflow Providers prepared on January 30, 2024

+1 (binding) On 2024/01/30 16:42:07 Elad Kalif wrote: > Hey all, > > I have just cut an ad-hoc release for the microsoft.azure provider package. > This email is calling a vote on the release, > which will last for 72 hours - which means that it will end on February 02, > 2024 16:40 PM UTC and unt