Hi, Thank you! For the detail explanation.
I will read everything and create a ticket with all the expectations. Cheers, Javier El jue., 26 mar. 2020 a las 23:46, Kamil Breguła (<kamil.breg...@polidea.com>) escribió: > Hello, > > Can you create a ticket in Github and describe your expectations? This > will allow you to discuss this topic more easily. > https://github.com/apache/airflow/issues/new/choose > > First of all, you should read the contribution guide, which contains a > very detailed description of how to propose changes to the project. > https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst > > In the next step, you should look at the airflow.cli package, which > contains all the code related to CLI. > https://github.com/apache/airflow/tree/master/airflow/cli > You will probably need to create a new root-level command and > subcommand., i.e. `airflow kubernetes preview` or something similar. > > Best regards, > Kamil > > On Thu, Mar 26, 2020 at 11:32 PM Javier Llorente Mañas > <javierllorent...@gmail.com> wrote: > > > > Hi, > > > > Yes, I missed this feature in the Airflow CLI. For me, the ideal scenario > > is to have something similar to the helm install --debug --dry-run > command ( > > https://helm.sh/docs/chart_template_guide/debugging/). So the same that > for > > a helm chart you generate multiple k8s templates, for an airflow dag you > > can generate multiple k8s pod templates if the same dag uses multiple > > KubernetesPodOperator, all of them with some naming convention may be the > > name of the dag + name of the task + _debug. Also, it can be interesting > to > > generate all the templates for all the dags present in a path. > > > > I was thinking to add this "library" to the Airflow CLI directly, but I > > don't know the code internals so well. I will like to contribute to the > > project but some help will be appreciated. > > > > On the other hand, I found an issue. To replicate in 100% the templates > you > > need to set the same variables in your production cluster than in your > > local environment, well this happened in my case testing with my company > > dags. So whenever you type this command maybe you need to run some extra > > airflow commands to set vars. I create this small CLI to run it locally > > with these arguments > > > https://github.com/Javier162380/AirflowKuberentesDebugger/blob/master/airflow_k8s_operator/cli.py > > . > > > > What do you think? > > > > Cheers > > > > Javier > > > > > > > > > > El jue., 26 mar. 2020 a las 23:04, Kamil Breguła (< > kamil.breg...@polidea.com>) > > escribió: > > > > > Hello, > > > > > > The idea is fantastic. I like it very much and it will facilitate the > > > work with Kubernethes. I'm just afraid that this tool will not be > > > available when I need it. Did you think to add this tool to CLI of > > > Airflow? This is the best place to share useful tools. > > > > > > When I missed the DAG preview in CLI, I added them to CLI. > > > > > > > https://airflow.readthedocs.io/en/latest/usage-cli.html#exporting-dags-structure-to-images > > > > > > Now I'm working on previewing the status of tasks after the dag > execution. > > > https://github.com/apache/airflow/pull/7776 > > > > > > Best regards, > > > Kamil > > > > > > On Thu, Mar 26, 2020 at 10:22 PM Javier Llorente Mañas > > > <javierllorent...@gmail.com> wrote: > > > > > > > > Hi all! it's Javier. > > > > > > > > I am a Data Engineer, I have been working with Airflow and it's > > > Kubernetes > > > > integration for almost one year and a half. It's great and it has > helped > > > a > > > > lot to me and the data engineering team I have been working with. > > > > > > > > Although I and my colleagues find a recurring issue sometimes > whenever we > > > > were creating new dags using the KubernetesPodOperator as sometimes > the > > > pod > > > > created was not as we expected. As an example arguments of the > container > > > > were not parsing as we expected or Secrets or Kubernetes resources > were > > > not > > > > referenced as we want. > > > > > > > > I just created this library > > > > https://github.com/Javier162380/AirflowKuberentesDebugger. > > > > > > > > The idea is that it can be a simple interface that can generate k8s > pod > > > > YAML files before deploying dags into a production environment so we > can > > > > test if the dag is going to generate all the k8s pods as we want or > > > > something is wrong. The idea is to have a kind of helm debugger for > > > > Airflow. Also, it can be really useful to recover historical data for > > > > recurrent dags. just changing the pod resources and the container > > > arguments > > > > or entry points. > > > > > > > > All your feedback is appreciated. > > > > > > > > Cheers, > > > > > > > > Javier > > > >
