James Meickle created AIRFLOW-3305: -------------------------------------- Summary: KubernetesPodOperator has a race condition for log output Key: AIRFLOW-3305 URL: https://issues.apache.org/jira/browse/AIRFLOW-3305 Project: Apache Airflow Issue Type: Bug Components: kubernetes Affects Versions: 1.10.0 Reporter: James Meickle
The KubernetesPodOperator follows logs from the container in the pod that it launches: [https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py#L96] This is set to "follow" mode, which streams logs. However, it is possible (but not guaranteed) for the pod's container to have started before the log stream call reaches the cluster. In this case, re-running the same task may result in very different-looking logs, with no notification that there was any truncation. This is a confusing experience for operators who are not familiar with Kubernetes. My recommendation is to remove "tail_lines" which should have the effect of fetching all previous logs when streaming starts: https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CoreV1Api.md#read_namespaced_pod_log -- This message was sent by Atlassian JIRA (v7.6.3#76005)