So the errors recur continuously for a given pod once they start happening?
On Tue, Sep 5, 2017 at 5:07 PM, Patrick Tescher <patr...@outtherelabs.com> wrote: > No patches have been applied since we upgraded to 3.6.0 over a week ago. > The errors just popped up for a few different pods in different namespaces. > The only thing we did today was launch a stateful set in a new namespace. > Those pods were not the ones throwing this error. > > > On Sep 5, 2017, at 1:19 PM, Clayton Coleman <ccole...@redhat.com> wrote: > > Were any patches applied to the system? Some of these are normal if they > happen for a brief period of time. Are you seeing these errors > continuously for the same pod over and over? > > On Tue, Sep 5, 2017 at 3:23 PM, Patrick Tescher <patr...@outtherelabs.com> > wrote: > >> This morning our cluster started experiencing an odd error on multiple >> nodes. Pods are stuck in the terminating phase. In our node log I see the >> following: >> >> Sep 5 19:17:22 ip-10-0-1-184 origin-node: E0905 19:17:22.043257 112306 >> nestedpendingoperations.go:262] Operation for "\" >> kubernetes.io/secret/182285ee-9267-11e7-b7be-06415eb17bbf >> -default-token-f18hx\ >> <http://kubernetes.io/secret/182285ee-9267-11e7-b7be-06415eb17bbf-default-token-f18hx%5C>" >> (\"182285ee-9267-11e7-b7be-06415eb17bbf\")" failed. No retries permitted >> until 2017-09-05 19:17:22.543230782 +0000 UTC (durationBeforeRetry 500ms). >> Error: UnmountVolume.TearDown failed for volume " >> kubernetes.io/secret/182285ee-9267-11e7-b7be-06415eb17bbf- >> default-token-f18hx" (volume.spec.Name <http://volume.spec.name/>: >> "default-token-f18hx") pod "182285ee-9267-11e7-b7be-06415eb17bbf" (UID: >> "182285ee-9267-11e7-b7be-06415eb17bbf") with: remove >> /var/lib/origin/openshift.local.volumes/pods/182285ee-9267- >> 11e7-b7be-06415eb17bbf/volumes/kubernetes.io~secret/default-token-f18hx: >> device or resource busy >> >> That path is not mounted (running mount does not list it) and running >> fuser -v on that directory does not show anything. Trying to rmdir results >> in a similar error: >> >> sudo rmdir var/lib/origin/openshift.local.volumes/pods/182285ee-9267- >> 11e7-b7be-06415eb17bbf/volumes/kubernetes.io~secret/default-token-f18hx >> rmdir: failed to remove ‘var/lib/origin/openshift.loca >> l.volumes/pods/182285ee-9267-11e7-b7be-06415eb17bbf/volumes/kubernetes.io >> ~secret/default-token-f18hx’: No such file or directory >> >> Is anyone else getting this error? >> >> >> _______________________________________________ >> dev mailing list >> dev@lists.openshift.redhat.com >> http://lists.openshift.redhat.com/openshiftmm/listinfo/dev >> >> > >
_______________________________________________ dev mailing list dev@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
