Hello, Can you create a ticket in Github and describe your expectations? This will allow you to discuss this topic more easily. https://github.com/apache/airflow/issues/new/choose
First of all, you should read the contribution guide, which contains a very detailed description of how to propose changes to the project. https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst In the next step, you should look at the airflow.cli package, which contains all the code related to CLI. https://github.com/apache/airflow/tree/master/airflow/cli You will probably need to create a new root-level command and subcommand., i.e. `airflow kubernetes preview` or something similar. Best regards, Kamil On Thu, Mar 26, 2020 at 11:32 PM Javier Llorente Mañas <[email protected]> wrote: > > Hi, > > Yes, I missed this feature in the Airflow CLI. For me, the ideal scenario > is to have something similar to the helm install --debug --dry-run command ( > https://helm.sh/docs/chart_template_guide/debugging/). So the same that for > a helm chart you generate multiple k8s templates, for an airflow dag you > can generate multiple k8s pod templates if the same dag uses multiple > KubernetesPodOperator, all of them with some naming convention may be the > name of the dag + name of the task + _debug. Also, it can be interesting to > generate all the templates for all the dags present in a path. > > I was thinking to add this "library" to the Airflow CLI directly, but I > don't know the code internals so well. I will like to contribute to the > project but some help will be appreciated. > > On the other hand, I found an issue. To replicate in 100% the templates you > need to set the same variables in your production cluster than in your > local environment, well this happened in my case testing with my company > dags. So whenever you type this command maybe you need to run some extra > airflow commands to set vars. I create this small CLI to run it locally > with these arguments > https://github.com/Javier162380/AirflowKuberentesDebugger/blob/master/airflow_k8s_operator/cli.py > . > > What do you think? > > Cheers > > Javier > > > > > El jue., 26 mar. 2020 a las 23:04, Kamil Breguła (<[email protected]>) > escribió: > > > Hello, > > > > The idea is fantastic. I like it very much and it will facilitate the > > work with Kubernethes. I'm just afraid that this tool will not be > > available when I need it. Did you think to add this tool to CLI of > > Airflow? This is the best place to share useful tools. > > > > When I missed the DAG preview in CLI, I added them to CLI. > > > > https://airflow.readthedocs.io/en/latest/usage-cli.html#exporting-dags-structure-to-images > > > > Now I'm working on previewing the status of tasks after the dag execution. > > https://github.com/apache/airflow/pull/7776 > > > > Best regards, > > Kamil > > > > On Thu, Mar 26, 2020 at 10:22 PM Javier Llorente Mañas > > <[email protected]> wrote: > > > > > > Hi all! it's Javier. > > > > > > I am a Data Engineer, I have been working with Airflow and it's > > Kubernetes > > > integration for almost one year and a half. It's great and it has helped > > a > > > lot to me and the data engineering team I have been working with. > > > > > > Although I and my colleagues find a recurring issue sometimes whenever we > > > were creating new dags using the KubernetesPodOperator as sometimes the > > pod > > > created was not as we expected. As an example arguments of the container > > > were not parsing as we expected or Secrets or Kubernetes resources were > > not > > > referenced as we want. > > > > > > I just created this library > > > https://github.com/Javier162380/AirflowKuberentesDebugger. > > > > > > The idea is that it can be a simple interface that can generate k8s pod > > > YAML files before deploying dags into a production environment so we can > > > test if the dag is going to generate all the k8s pods as we want or > > > something is wrong. The idea is to have a kind of helm debugger for > > > Airflow. Also, it can be really useful to recover historical data for > > > recurrent dags. just changing the pod resources and the container > > arguments > > > or entry points. > > > > > > All your feedback is appreciated. > > > > > > Cheers, > > > > > > Javier > >
