Re: [DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

Yes per my commit linked above, it would be the yaml file that is shown in the webserver code view (because the fileloc field points to that). I've been doing something similar to what Damian said, but with the difference that I've to generate the YAMLs programmatically based on parameters receive

Re: Issue form for Bugs/features in GitHub

> So we should be able to use it. Yes, we can (tried here https://github.com/turbaszek/airflow/issues/new/choose). I doubted it because I saw it a few months ago and it didn't work everywhere. +1 for using it! Tomek On Sat, 21 Aug 2021 at 00:29, Vikram Koka wrote: > +1 from me. > I would love

Re: Issue form for Bugs/features in GitHub

+1 from me. I would love to get some templates in place for pre-classification of issues. On Fri, Aug 20, 2021 at 2:01 PM Jarek Potiuk wrote: > I believe Beta is just "subject to change" but generally available. I > think the "public preview" is something that needs to be enabled for an > organi

Re: [DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

Agree with Ash here. It's OK to present different view of the "source" of the DAG once we parsed the Python code. This can be done and it could be as easy as a) adding a field to dag to point to a "definition file" if the DAGs are produced by parsing files from source folder b) API call/parameter

Re: Issue form for Bugs/features in GitHub

I believe Beta is just "subject to change" but generally available. I think the "public preview" is something that needs to be enabled for an organisation. So we should be able to use it. I really think we can use this to off-load a little more work to our users in terms of "classifying" the issue

RE: [DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

Changing the code view to show the YAML is now "relatively" easy to achieve, at least from the webserver point of view, as since 2.0 it doesn't read the files on disk, but from the DB. There's a lot of details, but changing the way these DagCode rows are written could be achievable whilst stil

RE: [DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

I’d personally find this very useful. There’s usually extra information I have about the DAG, and the current “docs_md” is usually not nearly sufficient enough as it’s poorly placed so if I start adding a lot of info it gets in the way of the regular UI. Also last I tested the markdown formattin

Re: [DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

On the topic of pointing the code view to yaml, would we alternatively consider adding a view on the UI that would allow arbitrary text content? This could be accomplished by adding an optional parameter to the dag object that allowed you to pass text (or a filepath) that would then go through a re

RE: [DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

FYI this is what I did on one of my past projects for Airflow. The users wanted to write their DAGs as YAML files so my “DAG file” was a Python script that read the YAML files and converted them to DAGs. It was very easy to do and worked because of the flexibility of Airflow. The one thing that

Re: [DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

Airflow DAGS are Python code.This is a very basic assumption - which is not likely to change. Ever. And we are working on making it even more powerful. Writing DAGs in yaml/json makes them less powerful and less flexible. This is fine if you want to build on top of airflow and build a more declara

[Register] Allyship workshop for open source communities

Hi everyone, I am helping to organize an allyship training for open source contributors sponsored by Google's OSPO. Few folks have expressed interest in this workshop, so I am extending the invite to all of you. Workshop date: Thursday, September 16th, 2021 at 8:30am PST or 5:30 PM CEST. Please r

Re: Issue form for Bugs/features in GitHub

According to this link the feature is in beta, isn’t it? Maybe it will work out of the box but GitHub often enrolls repos/orgs for betas. T.

Re: Issue form for Bugs/features in GitHub

+1 I think we just need to add a YAML file with the details as explained in https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-issue-forms

[DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

Have we considered allowing dags in json/yaml formats before? I came up with a rather straightforward way to address parametrized and dynamic DAGs in Airflow, which I think makes dynamic dags work at scale. *Background / Current limitations:* 1. Dynamic DAG generation using single-file methods

Re: Issue form for Bugs/features in GitHub

This works for any type of issues, KEDA is also using it for a long time: https://github.com/kedacore/keda/issues/new/choose But from what I remember that was a beta feature enabled on a repo/org basis, has it changed? If yes then I think we should enable it. Tomek On Thu, 19 Aug 2021 at 19:16,