Oops, actually our thresholds are: --image-gc-high-threshold=80 --image-gc-low-threshold=70
On 6 October 2016 at 14:55, Andrew Smith <m...@andrewmichaelsmith.com> wrote: > Hi > > We're running kubernetes 1.2. We are finding some cases where nodes fill > up with images. When I check the logs I can see ImageManager attempting to > remove images (sometimes failing) and the disk slowly but surely completely > fill. > > In today's particular instance I can see there are some images that are > weeks old and not in use by any containers. However, when I check the > ImageManager logs, I see stuff like: > > Oct 06 11:22:32 featuretest-12 kubelet[4206]: I1006 11:22:32.436162 > 4206 image_manager.go:282] [ImageManager]: Removing image " > 5b0bde439b3f53bee6e341cd07caba7b3db7eb7863e871294ee0fa8b43c11e63" to free > 276839339 bytes > Oct 06 11:22:34 featuretest-12 kubelet[4206]: E1006 11:22:34.386336 > 4206 kubelet.go:956] Image garbage collection failed: API error (409): > Conflict, cannot delete 5b0bde439b3f because the running container > e79a884a0ba1 is using it, stop it and use -f to force > > Which seems pretty odd - there's plenty of images there that could be > deleted and aren't being used - but it's picked one that is being used. > > After doing this dance for a bit we hit 100% usage an the box falls over. > Our threshold is 80%. > > Has anyone experiencing anything similar? I know we are on an older > version and intend to upgrade but I've failed to find any relevant bugs on > github. > > Thanks > -- > Andy Smith > http://andrewmichaelsmith.com | @bingleybeep > -- Andy Smith http://andrewmichaelsmith.com | @bingleybeep -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
