[
https://issues.apache.org/jira/browse/AIRFLOW-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045815#comment-17045815
]
ASF GitHub Bot commented on AIRFLOW-6843:
-----------------------------------------
kaxil commented on pull request #7523: [AIRFLOW-6843] Add delete_option_kwargs
to delete_namespaced_pod
URL: https://github.com/apache/airflow/pull/7523
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Add grace_period_seconds config option for delete_namespaced_pod kube client
> request
> ------------------------------------------------------------------------------------
>
> Key: AIRFLOW-6843
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6843
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executor-kubernetes
> Affects Versions: 1.10.9
> Reporter: Pete DeJoy
> Assignee: Pete DeJoy
> Priority: Major
>
> If you're using the Kubernetes executor, Amazon EKS deletes pods very quickly
> after the tasks complete. This can be an issue if you're scraping Airflow
> logs from a service like FluentD, as it means that the task pod gets deleted
> before FluentD can pick up logs for fast-running (or fast-failing) tasks.
> The [`kube_client_request_args` |#kube-client-request-args]environment
> variable is passed to the [`delete_namespaced_pod`|#L443] client request, but
> passing the necessary [`grace_period_seconds`|#delete_namespaced_pod] as a
> [`post_param`|#L108] on that object causes other client requests to fail, as
> that `kube_client_request_args` is passed as a kwarg to all client requests,
> even those that don't have a `post_params` option for `grace_period_seconds`.
> In order to provide a config option that fixes this issue, there needs to be
> a new configurable environment variable
> `AIRFLOW__KUBERNETES____DELETE_POD_GRACE_PERIOD_SECONDS` option that we pass
> to the [`delete_namespaced_pod`|#L443]] client request as its own argument.
> This will be defaulted to 0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
