potiuk commented on issue #17490: URL: https://github.com/apache/airflow/issues/17490#issuecomment-894757117
Do take a look at the current implementation and some of the fixes over the last months. I think many of the log problems have been fixed already (and I'd say if there are others - fixing those should not be difficult. From what you explain about template and Jinja i think only the possibility of using multiple template files is not out-of-the-box however this could likely be easily achieved (maybe even possible today) with Jinja include mechanism. I think you do not have to import nor use any of the k8s imports if you use KPO in your dag and use pod_template_file. Those 'old ways' are still there as optional parameters but from what I know you can skip them all. Regarding Kube Job - i'd rather defer to others who were involved there @dimberman ? Job is a bit higher abstraction level than Pod, and in a way you could achieve what Job does with Airflow itself, however yeah, i see the point why you might want to use parallelism in some cases and see those all parallel running pods as single Airflow task. I am not sure if the way how KPO handles the pod template, but i think it should be essentially possible to have one operator to run either Job or Pod. Or maybe even extracting some kind of common KTO (BaseKubernetesTemplateOperator) from KPO and implement KJO + kPO as children. Or maybe it can be handled easily with one operator and choosing whether to run job or pod. I would love to hear those more involved in KPO :). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
