potiuk commented on issue #36963:
URL: https://github.com/apache/airflow/issues/36963#issuecomment-1909141017

   > We currently use KubernetesPodOperator for dbt.
   > The intuitive expectations was that the logs would be similar and we were 
surprised they are not.
   
   Airflow is collectively created by > 2800 people. I am sometimes surprise 
the opposite, how coherent a small group of maintainers make those people (like 
you) to do things. And one of the reason is that they - like you notice 
differences and are willing to contribute back (like you) for a fantastic piece 
of software they can use absolutely for free and implement and improve things 
as partially a way to thank all those 2800 and make it even more 
fit-for-purpose.
   
   > Where in the code does KubernetesPodOperator poll and stream logs? If you 
can point me to [in the 
code](https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/pod.py)
 and provide commentary, I might be able to adjust the CR operator.
   
   You will find a few classes and code here:
   
https://github.com/apache/airflow/blob/390eacb01a710239e69f71d0c52134b7e87d17d4/airflow/providers/cncf/kubernetes/utils/pod_manager.py
   
   Commentary is quite straightforward. Not sure if you can do  it in similar 
way, becuase Kuberntes POD API to read logs is quite a bit complex and there is 
a lot of code to provide resilience, avoid duplication and generally retrieve 
the logs from running POD in an efficient way, taking into account the 
weaknesses of the API and fact that there could be multiple containers, in the 
pod, each with its own logs for example.
   
   But if I were to point you to important part of code, what is important is 
ONE line.
   
   
https://github.com/apache/airflow/blob/390eacb01a710239e69f71d0c52134b7e87d17d4/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L457C33-L457C89
   
   When yo know what next log line you should log just ... log it . The key 
complexity is that the task run locally from airflow side needs to retrieve the 
logs from Cloud Run while they are coming and log them one-by-one. Do it in a 
streaming way ideally so that they show up as they are coming and handle it in 
an efficient way.
   
   > Not having the logs available in Airflow significantly diminishes 
airflow's fit-for-purpose for the task.
   
   That's why we ask you (or anyone who will grab the `good first issue`) to 
make it more fit-for-the-purpose. Generally this is how Open source software is 
created. Someone sees a need to improve it and ... improves it. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to