Hello, The idea is fantastic. I like it very much and it will facilitate the work with Kubernethes. I'm just afraid that this tool will not be available when I need it. Did you think to add this tool to CLI of Airflow? This is the best place to share useful tools.
When I missed the DAG preview in CLI, I added them to CLI. https://airflow.readthedocs.io/en/latest/usage-cli.html#exporting-dags-structure-to-images Now I'm working on previewing the status of tasks after the dag execution. https://github.com/apache/airflow/pull/7776 Best regards, Kamil On Thu, Mar 26, 2020 at 10:22 PM Javier Llorente Mañas <javierllorent...@gmail.com> wrote: > > Hi all! it's Javier. > > I am a Data Engineer, I have been working with Airflow and it's Kubernetes > integration for almost one year and a half. It's great and it has helped a > lot to me and the data engineering team I have been working with. > > Although I and my colleagues find a recurring issue sometimes whenever we > were creating new dags using the KubernetesPodOperator as sometimes the pod > created was not as we expected. As an example arguments of the container > were not parsing as we expected or Secrets or Kubernetes resources were not > referenced as we want. > > I just created this library > https://github.com/Javier162380/AirflowKuberentesDebugger. > > The idea is that it can be a simple interface that can generate k8s pod > YAML files before deploying dags into a production environment so we can > test if the dag is going to generate all the k8s pods as we want or > something is wrong. The idea is to have a kind of helm debugger for > Airflow. Also, it can be really useful to recover historical data for > recurrent dags. just changing the pod resources and the container arguments > or entry points. > > All your feedback is appreciated. > > Cheers, > > Javier
