snjypl commented on issue #21087: URL: https://github.com/apache/airflow/issues/21087#issuecomment-1120491871
my understanding is that: 1.) there is a bug with the kubernetes python client, it is not updating the watcher resource_version from the BOOKMARK events. [#1796](https://github.com/kubernetes-client/python/pull/1796) tries to fix it. 2.) BOOKMARK is not enabled by default, the client needs to request it by sending `allow_watch_bookmarks=True` in the request. by default it is false. currently KubernetesJobWatcher is not sending it. so it is not using the BOOKMARK feature. https://github.com/apache/airflow/blob/8f181c10344bd319ac5f6aeb102ee3c06e1f1637/airflow/executors/kubernetes_executor.py#L129 https://github.com/apache/airflow/blob/8f181c10344bd319ac5f6aeb102ee3c06e1f1637/airflow/executors/kubernetes_executor.py#L137-L140 3.) also we will need to add a condition here, to check the event's type and update the last_resource_version https://github.com/apache/airflow/blob/8f181c10344bd319ac5f6aeb102ee3c06e1f1637/airflow/executors/kubernetes_executor.py#L145-L149 i think, it will be nice to create a WIP pull request for it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
