osintalex commented on issue #44994: URL: https://github.com/apache/airflow/issues/44994#issuecomment-2548770306
Ah found it in the source! https://github.com/apache/airflow/blob/ce4236f365c3e8e45262b04ef0ce596907043702/providers/src/airflow/providers/cncf/kubernetes/operators/job.py#L171 calls into https://github.com/apache/airflow/blob/ce4236f365c3e8e45262b04ef0ce596907043702/providers/src/airflow/providers/cncf/kubernetes/operators/pod.py#L546 which then calls in here https://github.com/apache/airflow/blob/ce4236f365c3e8e45262b04ef0ce596907043702/providers/src/airflow/providers/cncf/kubernetes/operators/pod.py#L973 this is going to match more than one pods always if you have a parallelism count > 1 from what I can tell which in turn triggers this error https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/cncf/kubernetes/operators/pod.py#L532 Unless I'm missing something, I think this could be fixed by adding a condition here https://github.com/apache/airflow/blob/ce4236f365c3e8e45262b04ef0ce596907043702/providers/src/airflow/providers/cncf/kubernetes/operators/job.py#L170 where if parallelism is > 1 to skip this section and maybe log that this is happening because your job has more than one pod -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
