potiuk commented on issue #36963: URL: https://github.com/apache/airflow/issues/36963#issuecomment-1909141017
> We currently use KubernetesPodOperator for dbt. > The intuitive expectations was that the logs would be similar and we were surprised they are not. Airflow is collectively created by > 2800 people. I am sometimes surprise the opposite, how coherent a small group of maintainers make those people (like you) to do things. And one of the reason is that they - like you notice differences and are willing to contribute back (like you) for a fantastic piece of software they can use absolutely for free and implement and improve things as partially a way to thank all those 2800 and make it even more fit-for-purpose. > Where in the code does KubernetesPodOperator poll and stream logs? If you can point me to [in the code](https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/pod.py) and provide commentary, I might be able to adjust the CR operator. You will find a few classes and code here: https://github.com/apache/airflow/blob/390eacb01a710239e69f71d0c52134b7e87d17d4/airflow/providers/cncf/kubernetes/utils/pod_manager.py Commentary is quite straightforward. Not sure if you can do it in similar way, becuase Kuberntes POD API to read logs is quite a bit complex and there is a lot of code to provide resilience, avoid duplication and generally retrieve the logs from running POD in an efficient way, taking into account the weaknesses of the API and fact that there could be multiple containers, in the pod, each with its own logs for example. But if I were to point you to important part of code, what is important is ONE line. https://github.com/apache/airflow/blob/390eacb01a710239e69f71d0c52134b7e87d17d4/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L457C33-L457C89 When yo know what next log line you should log just ... log it . The key complexity is that the task run locally from airflow side needs to retrieve the logs from Cloud Run while they are coming and log them one-by-one. Do it in a streaming way ideally so that they show up as they are coming and handle it in an efficient way. > Not having the logs available in Airflow significantly diminishes airflow's fit-for-purpose for the task. That's why we ask you (or anyone who will grab the `good first issue`) to make it more fit-for-the-purpose. Generally this is how Open source software is created. Someone sees a need to improve it and ... improves it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
