This is an automated email from the ASF dual-hosted git repository.
alinsran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/apisix-ingress-controller.git
The following commit(s) were added to refs/heads/master by this push:
new d283536d docs: add best practice docs to avoid race condition bw
kubelet and apisix (#2045)
d283536d is described below
commit d283536d30b482edff861879315b7dfe3fe1b730
Author: Ashish Tiwari <[email protected]>
AuthorDate: Fri Nov 17 12:39:43 2023 +0530
docs: add best practice docs to avoid race condition bw kubelet and apisix
(#2045)
Signed-off-by: Ashish Tiwari <[email protected]>
Co-authored-by: Traky Deng <[email protected]>
---
...ow-to-avoid-race-condition-when-scaling-down.md | 63 ++++++++++++++++++++++
1 file changed, 63 insertions(+)
diff --git
a/docs/en/latest/tutorials/how-to-avoid-race-condition-when-scaling-down.md
b/docs/en/latest/tutorials/how-to-avoid-race-condition-when-scaling-down.md
new file mode 100644
index 00000000..3916df0a
--- /dev/null
+++ b/docs/en/latest/tutorials/how-to-avoid-race-condition-when-scaling-down.md
@@ -0,0 +1,63 @@
+---
+title: Best practice to avoid race condition between Kubelet and APISIX
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Prerequisite
+
+While scaling down the number of pods in your upstream, few of the pods are
marked as Terminating. On the node where the Pod is running: as soon as the
kubelet sees that a Pod has been marked as terminating (a graceful shutdown
duration has been set), the kubelet begins the local Pod shutdown process. At
the same time as the kubelet is starting graceful shutdown of the Pod, the
control plane evaluates whether to remove that shutting-down Pod from
EndpointSlice (and Endpoints) objects, wh [...]
+Pods that shut down slowly should not continue to serve regular traffic and
should start terminating and finish processing open connections.
+
+## The race condition
+
+APISIX ingress controller also watches for pod updates(for upstream pods). And
updates the APISIX instance with new upstreams and removes the older upstreams.
But this process might take time depending on the latency between Ingress
controller and APISIX instance. This latency can be large if APISIX and Ingress
controller have a number of network hops between them or it can be low if
you're running APISIX ingress controller in [composite mode](../composite.md)
where both ingress controll [...]
+
+## PreStop hook
+
+If one of the Pod's containers has defined a `PreStop` hook and the
`terminationGracePeriodSeconds` in the Pod spec is not set to 0, the kubelet
runs that hook inside of the container. The default
`terminationGracePeriodSeconds` setting is 30 seconds.
+
+## Solution
+
+You can add delay to the termination of the upstream pod by some seconds (for
eg: 5 seconds) to make sure that the pod is not terminated before the APISIX
instance gets updated.
+
+:::note
+
+If the preStop hook needs longer to complete than the default grace period
allows, you must modify `terminationGracePeriodSeconds` to suit this. This is
30 seconds by default. For more information, refer to the [Kubernetes
docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination).
+
+:::
+
+Below is the example usage of one such Pod configuration which will act as
upstream for APISIX.
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: example-pod
+spec:
+ containers:
+ - name: web-server
+ image: web-server:latest
+ lifecycle:
+ preStop:
+ exec:
+ command: ["/bin/sh", "-c", "sleep 5"]
+
+```
