RE: [DISCUSS] Experimental Prod ARM64 image for 2.3.0 ?

Would be great if we can get an official ARM image out soon. Our M1 users can really benefit from this. Running Airflow in emulated mode is simply unusable at the moment. In the long run, I think there is a very high chance that ARM will take over x86. So it's not a bad idea to plant the seed earl

RE: [DISCUSS] Experimental Prod ARM64 image for 2.3.0 ?

Would be great if we can get an official ARM image out soon. Our M1 users can really benefit from this. Running Airflow in emulated mode is simply unusable at the moment. In the long run, I think there is a very high chance that ARM will take over x86. So it's not a bad idea to plant the seed earl

Re: Airflow Client for Java

just work out of the box. Once the client automation script is merged into the main airflow repo, you can send a PR to add the generated code to the airflow-client-java repo. [1]: https://github.com/apache/airflow/blob/main/clients/gen/python.sh Thanks, QP Hou On Thu, Jul 29, 2021 at 1:41 PM

Re: Roadmap ideas for Airflow 2.2 and beyond

I second Jarek's point on making Airflow's components and workloads more isolated. Another big pain point for us is cross DAG scheduling and dependency management. Thanks, QP Hou On Tue, Jun 15, 2021 at 5:45 AM Jarek Potiuk wrote: > > I think those are the most important "

Re: Airflow Airflow Client 2.0.0 is released

Good job Sumit for releasing the first official Airflow client, that's a big milestone :D One minor suggestion, would be good to update the installation instruction in readme to use the pypi package and also link to the client github repo from pypi :) Thanks, QP Hou On Fri, Apr 23, 2021 at

Re: Releasing Apache Airflow's Python Client

Thanks Sumit for kicking off this discussion! I am in favor of approach #1 as well. Airflow software version can change due to none API related changes, but the client should only care about the public API contract. I believe this is also the same approach that the k8s python client took as well?

Re: Renaming master branches to main

+1 on the rename. Thanks, QP Hou On Wed, Mar 10, 2021 at 2:24 AM Ash Berlin-Taylor wrote: > > Hello everyone, > > I'm planning on slowly working through all our active Git repos and changing > the new default branch over to be main > (https://github.com/apache/airfl

Re: Scoping out a new feature for 2.1: improving schedule_interval

On Wed, Jan 20, 2021 at 10:22 AM Daniel Imberman wrote: > > I love the idea of allowing users to create their own scheduling > objects/scheduling python functions. They could either live in the scheduler > or as a seperate process that trips some value in the DB when it is “true”. > Would be gr

Re: New PMC Member: Xiaodong Deng

late to the party, congrats to XD! On Tue, Dec 29, 2020 at 4:23 PM Kaxil Naik wrote: > > Hello Airflow Community, > > The Project Management Committee (PMC) for Apache Airflow > has invited Xiaodong Deng to become a part of PMC Member and we are pleased > to announce that he has kindly accepted i

Re: A Visual Editor for Airflow pipelines

On Tue, Nov 3, 2020 at 12:05 PM Alan K Chin wrote: > @Jarek - It sounds like git-sync is or rather should be the default way users > add/modify DAGs. With that said, have you had any experience with customers > syncing their dags to other forms of dag storage (S3 etc.) and what the > outcomes w

Re: New Airflow Committer: Ryan Hamilton

Congrats Ryan and wellcome! Great to see dedicated UI engineers to help modernize Airflow UI :D On Wed, Oct 28, 2020 at 10:01 AM Kaxil Naik wrote: > > Hello Airflow Community, > > The Project Management Committee (PMC) for Apache Airflow > has invited Ryan Hamilton to become a committer and we ar

Re: Airflow + Hacktoberfest 2020

On Thu, Aug 20, 2020 at 9:44 PM Jarek Potiuk wrote: > > @QP - for the workshops, it's not much but the time :). Not even a lot of > preparation is needed (by the people who actively commit the code to > Airflow). It's basically answering people's questions - when the attendees > get to the "hands-

Re: Airflow + Hacktoberfest 2020

+1 on Hacktoberfest. On Thu, Aug 20, 2020 at 6:08 AM Jarek Potiuk wrote: > Anyone from the community willing to help :)? What are the expected commitments if one wants to help with organizing the workshops?

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

Yeah, i am also curious to know more about the reason why we want to nuke the experimental api code soon instead of just marking it as deprecated. As for getting more insights into remote mode cli usage, would it make sense to make it part of the airflow user survey? Thanks, QP Hou On Thu, Aug

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

cing CLI that's light weight, fast and remote only. It doesn't even need to be written in python to make it easier to distribute as a single binary. Thanks, QP Hou On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor wrote: > -1 from me without a firm plan how we will replace

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

+1 for replacing the existing remote mode client with the open api based client. IMO, we don't really have other options here because the experimental API will be deprecated in the future. For OpenAPI based Airflow REST clients, the current plan is to maintain all the code gen automation within th

Re: API & CLI Permission Checking

Hi Ace, Yes, adding permission is definitely something the community is working on. https://github.com/apache/airflow/issues/8112 is a good place to track progress on this work and participate in the design discussion. Thanks, QP Hou On Wed, Aug 5, 2020 at 11:32 PM Ace Haidrey wrote: >

Re: New PMCs: Daniel Imberman, Tomek Turbaszek & Kamil Breguła

Congratulations to Daniel, Tomek and Kamil, well deserved! Thanks, QP Hou On Mon, Jul 20, 2020 at 7:21 AM Kaxil Naik wrote: > Hello Airflow Community, > > Most of you might already be aware of this news but let me officially > announce it. > > The Project Management Committe

Re: New Committers (Leah Cole & Ry Walker)

Congrats to both Leah and Ry! the family is growing fast :) Thanks, QP Hou On Mon, Jul 20, 2020 at 7:18 AM Kaxil Naik wrote: > Hello Airflow Community, > > The Project Management Committee (PMC) for Apache Airflow > has invited *Leah Cole* & *Ry Walker* to become a comm

Re: new committer: QP Hou

Thank you all for the kind words, really looking forward to more collaboration with the community as a committer :) Thanks, QP Hou On Wed, Jul 15, 2020 at 11:17 AM Kamil Breguła wrote: > Congrats! > > On Wed, Jul 15, 2020, 20:01 Felix Uellendall > wrote: > > > Congra

Re: API spec: format of dagrun conf parameter: JSON-encoded string, or JSON object

I wrote up a quick a go example that uses runtime reflection to work with object design based off Kamil's java example: https://github.com/houqp/airflow-api-clients/blob/master/out/object/go/model_dag_run_test.go. Code wise, it's indeed a little bit more verbose compared to deserializing/serializi

Re: API spec: format of dagrun conf parameter: JSON-encoded string, or JSON object

On Sat, May 30, 2020 at 2:31 AM Jarek Potiuk wrote: > > Ash - QP: also - as Kamil mentioned he already generated multiple clients > in https://github.com/mik-laj/airflow-api-clients/tree/master/out/object > so if you have time , maybe you can also take a look in parallel. This is a good idea. I t

Re: API spec: format of dagrun conf parameter: JSON-encoded string, or JSON object

please see replies inline. On Fri, May 29, 2020 at 1:14 PM Jarek Potiuk wrote: > This is fine if "additionalProperties" are used. But when you go with the > fully freeform - > the message will be the same as I understand "Model ,,, not generated since > it's a free-form object > Then it becomes

Re: API spec: format of dagrun conf parameter: JSON-encoded string, or JSON object

On Fri, May 29, 2020 at 1:27 PM Kamil Breguła wrote: > > We move the problem to a place where we have enough data to handle it. > The user has information about whether he can indicate a better type > based on many parameters e.g. dag_id or execution_date. I imagine that > the user may want to con

Re: API spec: format of dagrun conf parameter: JSON-encoded string, or JSON object

On Fri, May 29, 2020 at 7:18 AM Kamil Breguła wrote: > I have the biggest concerns about reading data, not about sending data > so in my opinion we should think about how we want the response from > the server to look like? > > GET //dags/{DAG_ID}/dagRuns/ > > { > "dag_runs": [ > {"conf": {"A":"B"

Re: API spec: format of dagrun conf parameter: JSON-encoded string, or JSON object

While I understand the difficulty of dealing with nested json without predefined schemas, I feel like returning it as a string only delays the problem to a later stage since the user will still need to parse that string into a strongly typed data structure in order to read the values. I don't have

Re: API spec questions

I would recommend option III. IMHO, hierarchy based endpoints are for CURD operations on a single entity. The flat endpoint, like /taskinstance, is for read-only multi-facet search queries. For example: * to create a resource Foo: "POST /api/parent/{parent_id}/foo" * to update a resource Foo: "P

Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection balancing HA )

+1 (non binding) On Fri, Apr 24, 2020 at 3:31 AM Ash Berlin-Taylor wrote: > > As we discussed in the thread on Spec thread for our new API, the idea > came up of making conn_id unique in Airflow > >

Re: API spec questions

It looks like we need to first make a decision on whether we want to enforce uniqueness for conn_id and implement connection load balance properly in another way. Also +1 on making audit log read only from API point of view. Thanks, QP Hou On Thu, Apr 9, 2020 at 9:47 AM Ash Berlin-Taylor wrote