It seems caused by the changes in Golang:
https://github.com/golang/go/commit/321c312d8246dec6889f5fe334b6193c320baf0e
On Monday, February 20, 2017 at 5:30:46 PM UTC+8, Qian Zhang wrote:
>
> Hi,
>
> I had a pod with a emptyDir volume in my K8s cluster, and after I deleted
> it, I found kubelet is extremely busy and full of the logs like below:
> ...
> {"log":"I0220 09:27:51.719130 10489 reconciler.go:189] UnmountVolume
> operation started for volume \"
> kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~769416713.deleting~867770554.deleting~965559302.deleting~800760242.deleting~372961852\
>
> <http://kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~769416713.deleting~867770554.deleting~965559302.deleting~800760242.deleting~372961852%5C>"
>
> (spec.Name:
> \"workspace-volume.deleting~769416713.deleting~867770554.deleting~965559302.deleting~800760242.deleting~372961852\")
>
> from pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
> \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\").\n","stream":"stderr","time":
> "2017-02-20T09:27:51.719197Z"}
> {"log":"I0220 09:27:51.719947 10489 reconciler.go:189] UnmountVolume
> operation started for volume \"
> kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~606230688.deleting~546142499.deleting~376333434\
>
> <http://kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~606230688.deleting~546142499.deleting~376333434%5C>"
>
> (spec.Name:
> \"workspace-volume.deleting~606230688.deleting~546142499.deleting~376333434\")
>
> from pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
> \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\").\n","stream":"stderr","time":
> "2017-02-20T09:27:51.719999Z"}
> {"log":"I0220 09:27:51.726037 10489 reconciler.go:189] UnmountVolume
> operation started for volume \"
> kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~649260757.deleting~373982717.deleting~216249632.deleting~529121280.deleting~485139494\
>
> <http://kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~649260757.deleting~373982717.deleting~216249632.deleting~529121280.deleting~485139494%5C>"
>
> (spec.Name:
> \"workspace-volume.deleting~649260757.deleting~373982717.deleting~216249632.deleting~529121280.deleting~485139494\")
>
> from pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
> \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\").\n","stream":"stderr","time":
> "2017-02-20T09:27:51.726124Z"}
> {"log":"E0220 09:27:51.796315 10489 nestedpendingoperations.go:262]
> Operation for \"\\\"
> kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550\\\
>
> <http://kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550%5C%5C%5C>"
>
> (\\\"ec37a6d5-f747-11e6-a0a1-525400f0cb63\\\")\" failed. No retries
> permitted until 2017-02-20 09:27:52.796254285 +0000 UTC
> (durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume \"
> kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550\
>
> <http://kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550%5C>"
>
> (volume.spec.Name:
> \"workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550\")
>
> pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
> \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\") with: rename
> /var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/
> kubernetes.io~empty-dir/workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550
>
> /var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/
> kubernetes.io~empty-dir/workspace-volume.deleting~721599186.deleting~533635678.deleting~661378369.deleting~498307714.deleting~012586550.deleting~598879704:
>
> file exists\n","stream":"stderr","time":"2017-02-20T09:27:51.796444Z"}
> {"log":"E0220 09:27:51.798120 10489 nestedpendingoperations.go:262]
> Operation for \"\\\"
> kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785\\\
>
> <http://kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785%5C%5C%5C>"
>
> (\\\"ec37a6d5-f747-11e6-a0a1-525400f0cb63\\\")\" failed. No retries
> permitted until 2017-02-20 09:27:52.798062088 +0000 UTC
> (durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume \"
> kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785\
>
> <http://kubernetes.io/empty-dir/ec37a6d5-f747-11e6-a0a1-525400f0cb63-workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785%5C>"
>
> (volume.spec.Name:
> \"workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785\")
>
> pod \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\" (UID:
> \"ec37a6d5-f747-11e6-a0a1-525400f0cb63\") with: rename
> /var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/
> kubernetes.io~empty-dir/workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785
>
> /var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/
> kubernetes.io~empty-dir/workspace-volume.deleting~266046491.deleting~198997082.deleting~876327544.deleting~714923576.deleting~553982785.deleting~823906998:
>
> file exists\n","stream":"stderr","time":"2017-02-20T09:27:51.79826Z"}
> ...
>
> And
> under /var/lib/kubelet/pods/ec37a6d5-f747-11e6-a0a1-525400f0cb63/volumes/
> kubernetes.io~empty-dir, there are a lot of sub-directories like below:
> ...
> drwx------ 2 root root 4096 Feb 20 03:54 workspace-volume.deleting~
> 266046491.deleting~882101527.deleting~227464818.deleting~
> 661307513.deleting~444186630.deleting~968923478
> drwx------ 2 root root 4096 Feb 20 03:49 workspace-volume.deleting~
> 266046491.deleting~882101527.deleting~227464818.deleting~
> 661307513.deleting~617656523
> drwx------ 2 root root 4096 Feb 20 03:53 workspace-volume.deleting~
> 266046491.deleting~882101527.deleting~227464818.deleting~
> 661307513.deleting~617656523.deleting~082024766
> drwx------ 2 root root 4096 Feb 20 03:46 workspace-volume.deleting~
> 266046491.deleting~882101527.deleting~227464818.deleting~865803623
> drwx------ 2 root root 4096 Feb 20 03:51 workspace-volume.deleting~
> 266046491.deleting~882101527.deleting~227464818.deleting~
> 865803623.deleting~116034617
> drwx------ 2 root root 4096 Feb 20 04:12 workspace-volume.deleting~
> 266046491.deleting~882101527.deleting~227464818.deleting~
> 865803623.deleting~116034617.deleting~883988680
> drwx------ 2 root root 4096 Feb 20 03:49 workspace-volume.deleting~
> 266046491.deleting~882101527.deleting~227464818.deleting~
> 865803623.deleting~146048683
> drwx------ 2 root root 4096 Feb 20 03:53 workspace-volume.deleting~
> 266046491.deleting~882101527.deleting~227464818.deleting~
> 865803623.deleting~146048683.deleting~806072952
> drwx------ 2 root root 4096 Feb 20 03:50 workspace-volume.deleting~
> 266046491.deleting~882101527.deleting~227464818.deleting~
> 865803623.deleting~225146442
> ...
>
> And even worse, I can not create any new pod on this node anymore due to
> the error below:
> Events:
> FirstSeen LastSeen Count From
> SubObjectPath Type Reason Message
> --------- -------- ----- ----
> ------------- -------- ------ -------
> 25m 25m 1 {default-scheduler }
> Normal Scheduled Successfully assigned zzz-372425889
> -zz36j to 9.37.234.90
> 23m 1m 11 {kubelet 9.37.234.90}
> Warning FailedMount Unable to mount volumes for pod
> "zzz-372425889-zz36j_default(9264fc15-f74b-11e6-a0a1-525400f0cb63)":
> timeout expired waiting for volumes to attach/mount for pod "default"/
> "zzz-372425889-zz36j". list of unattached/unmounted volumes=[default-token
> -f27kf]
> 23m 1m 11 {kubelet 9.37.234.90}
> Warning FailedSync Error syncing pod, skipping:
> timeout expired waiting for volumes to attach/mount for pod "default"/
> "zzz-372425889-zz36j". list of unattached/unmounted volumes=[default-token
> -f27kf]
>
> Any ideas?
>
>
--
You received this message because you are subscribed to the Google Groups
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
