Hi, Yes, I missed this feature in the Airflow CLI. For me, the ideal scenario is to have something similar to the helm install --debug --dry-run command ( https://helm.sh/docs/chart_template_guide/debugging/). So the same that for a helm chart you generate multiple k8s templates, for an airflow dag you can generate multiple k8s pod templates if the same dag uses multiple KubernetesPodOperator, all of them with some naming convention may be the name of the dag + name of the task + _debug. Also, it can be interesting to generate all the templates for all the dags present in a path.
I was thinking to add this "library" to the Airflow CLI directly, but I don't know the code internals so well. I will like to contribute to the project but some help will be appreciated. On the other hand, I found an issue. To replicate in 100% the templates you need to set the same variables in your production cluster than in your local environment, well this happened in my case testing with my company dags. So whenever you type this command maybe you need to run some extra airflow commands to set vars. I create this small CLI to run it locally with these arguments https://github.com/Javier162380/AirflowKuberentesDebugger/blob/master/airflow_k8s_operator/cli.py . What do you think? Cheers Javier El jue., 26 mar. 2020 a las 23:04, Kamil Breguła (<[email protected]>) escribió: > Hello, > > The idea is fantastic. I like it very much and it will facilitate the > work with Kubernethes. I'm just afraid that this tool will not be > available when I need it. Did you think to add this tool to CLI of > Airflow? This is the best place to share useful tools. > > When I missed the DAG preview in CLI, I added them to CLI. > > https://airflow.readthedocs.io/en/latest/usage-cli.html#exporting-dags-structure-to-images > > Now I'm working on previewing the status of tasks after the dag execution. > https://github.com/apache/airflow/pull/7776 > > Best regards, > Kamil > > On Thu, Mar 26, 2020 at 10:22 PM Javier Llorente Mañas > <[email protected]> wrote: > > > > Hi all! it's Javier. > > > > I am a Data Engineer, I have been working with Airflow and it's > Kubernetes > > integration for almost one year and a half. It's great and it has helped > a > > lot to me and the data engineering team I have been working with. > > > > Although I and my colleagues find a recurring issue sometimes whenever we > > were creating new dags using the KubernetesPodOperator as sometimes the > pod > > created was not as we expected. As an example arguments of the container > > were not parsing as we expected or Secrets or Kubernetes resources were > not > > referenced as we want. > > > > I just created this library > > https://github.com/Javier162380/AirflowKuberentesDebugger. > > > > The idea is that it can be a simple interface that can generate k8s pod > > YAML files before deploying dags into a production environment so we can > > test if the dag is going to generate all the k8s pods as we want or > > something is wrong. The idea is to have a kind of helm debugger for > > Airflow. Also, it can be really useful to recover historical data for > > recurrent dags. just changing the pod resources and the container > arguments > > or entry points. > > > > All your feedback is appreciated. > > > > Cheers, > > > > Javier >
