Hi,
I had a pod with a emptyDir volume in my K8s cluster, and after I deleted
it, I found kubelet is extremely busy and full of the logs like below:
...
{"log":"I0220 09:27:51.719130 10489 reconciler.go:189] UnmountVolume
operation started for volume
\"kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~769416713.deleting~867770554.deleting~965559302.deleting~800760242.deleting~372961852\"
(spec.Name:
\"workspace-volume.deleting~769416713.deleting~867770554.deleting~965559302.deleting~800760242.deleting~372961852\")
from pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
\"ec37a6d5-f747-11e6-a0a1-525400f0cb63\").\n","stream":"stderr","time":
"2017-02-20T09:27:51.719197Z"}
{"log":"I0220 09:27:51.719947 10489 reconciler.go:189] UnmountVolume
operation started for volume
\"kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~606230688.deleting~546142499.deleting~376333434\"
(spec.Name:
\"workspace-volume.deleting~606230688.deleting~546142499.deleting~376333434\")
from pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
\"ec37a6d5-f747-11e6-a0a1-525400f0cb63\").\n","stream":"stderr","time":
"2017-02-20T09:27:51.719999Z"}
{"log":"I0220 09:27:51.726037 10489 reconciler.go:189] UnmountVolume
operation started for volume
\"kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~649260757.deleting~373982717.deleting~216249632.deleting~529121280.deleting~485139494\"
(spec.Name:
\"workspace-volume.deleting~649260757.deleting~373982717.deleting~216249632.deleting~529121280.deleting~485139494\")
from pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
\"ec37a6d5-f747-11e6-a0a1-525400f0cb63\").\n","stream":"stderr","time":
"2017-02-20T09:27:51.726124Z"}
{"log":"E0220 09:27:51.796315 10489 nestedpendingoperations.go:262]
Operation for
\"\\\"kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550\\\"
(\\\"ec37a6d5-f747-11e6-a0a1-525400f0cb63\\\")\" failed. No retries
permitted until 2017-02-20 09:27:52.796254285 +0000 UTC
(durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume
\"kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550\"
(volume.spec.Name:
\"workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550\")
pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
\"ec37a6d5-f747-11e6-a0a1-525400f0cb63\") with: rename
/var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/kubernetes.io~empty-dir/workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550
/var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/kubernetes.io~empty-dir/workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550.deleting~598879704:
file exists\n","stream":"stderr","time":"2017-02-20T09:27:51.796444Z"}
{"log":"E0220 09:27:51.798120 10489 nestedpendingoperations.go:262]
Operation for
\"\\\"kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785\\\"
(\\\"ec37a6d5-f747-11e6-a0a1-525400f0cb63\\\")\" failed. No retries
permitted until 2017-02-20 09:27:52.798062088 +0000 UTC
(durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume
\"kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785\"
(volume.spec.Name:
\"workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785\")
pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
\"ec37a6d5-f747-11e6-a0a1-525400f0cb63\") with: rename
/var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/kubernetes.io~empty-dir/workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785
/var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/kubernetes.io~empty-dir/workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785.deleting~823906998:
file exists\n","stream":"stderr","time":"2017-02-20T09:27:51.79826Z"}
...
And
under
/var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/kubernetes.io~empty-dir,
there are a lot of sub-directories like below:
...
drwx------ 2 root root 4096 Feb 20 03:54 workspace-volume.deleting~
266046491.deleting~882101527.deleting~227464818.deleting~661307513.deleting~
444186630.deleting~968923478
drwx------ 2 root root 4096 Feb 20 03:49 workspace-volume.deleting~
266046491.deleting~882101527.deleting~227464818.deleting~661307513.deleting~
617656523
drwx------ 2 root root 4096 Feb 20 03:53 workspace-volume.deleting~
266046491.deleting~882101527.deleting~227464818.deleting~661307513.deleting~
617656523.deleting~082024766
drwx------ 2 root root 4096 Feb 20 03:46 workspace-volume.deleting~
266046491.deleting~882101527.deleting~227464818.deleting~865803623
drwx------ 2 root root 4096 Feb 20 03:51 workspace-volume.deleting~
266046491.deleting~882101527.deleting~227464818.deleting~865803623.deleting~
116034617
drwx------ 2 root root 4096 Feb 20 04:12 workspace-volume.deleting~
266046491.deleting~882101527.deleting~227464818.deleting~865803623.deleting~
116034617.deleting~883988680
drwx------ 2 root root 4096 Feb 20 03:49 workspace-volume.deleting~
266046491.deleting~882101527.deleting~227464818.deleting~865803623.deleting~
146048683
drwx------ 2 root root 4096 Feb 20 03:53 workspace-volume.deleting~
266046491.deleting~882101527.deleting~227464818.deleting~865803623.deleting~
146048683.deleting~806072952
drwx------ 2 root root 4096 Feb 20 03:50 workspace-volume.deleting~
266046491.deleting~882101527.deleting~227464818.deleting~865803623.deleting~
225146442
...
And even worse, I can not create any new pod on this node anymore due to
the error below:
Events:
FirstSeen LastSeen Count From
SubObjectPath Type Reason Message
--------- -------- ----- ----
------------- -------- ------ -------
25m 25m 1 {default-scheduler }
Normal Scheduled Successfully assigned zzz-372425889-zz36j
to 9.37.234.90
23m 1m 11 {kubelet 9.37.234.90}
Warning FailedMount Unable to mount volumes for pod
"zzz-372425889-zz36j_default(9264fc15-f74b-11e6-a0a1-525400f0cb63)":
timeout expired waiting for volumes to attach/mount for pod "default"/
"zzz-372425889-zz36j". list of unattached/unmounted volumes=[default-token-
f27kf]
23m 1m 11 {kubelet 9.37.234.90}
Warning FailedSync Error syncing pod, skipping: timeout
expired waiting for volumes to attach/mount for pod "default"/
"zzz-372425889-zz36j". list of unattached/unmounted volumes=[default-token-
f27kf]
Any ideas?
--
You received this message because you are subscribed to the Google Groups
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
