+Vivek
I don't know the exact bug here, but one possibility is that since you've disabled the auto-extend, that it's leaving the metadata size at it's default? There is indeed a relationship between the two and I did a lot of testing around it to make sure we were not going to hit what you've hit :-/ At that time however, we did not have the auto-extend feature. So I think you may have found a situation where we need METADATA_SIZE=N but you said you set that at 2%? Can you paste the output of "docker info" and "lsblk" please? >From April 2015 when I last looked at this, the tests ran were to start 1000 httpd containers and measure the thinpool used size. Here: So, for 1000 rhel7+httpd containers, on-disk it was using about 3GB (left-Y-axis). And the metadata consumed was about 50MB (right-Y-axis). You can see we concluded to set metadata @ 0.1% of the data partition size, which in the case: https://github.com/projectatomic/docker-storage-setup/blob/master/docker-storage-setup.sh#L220 I am wondering what your docker info will say. On Fri, Jul 22, 2016 at 8:08 AM, Andrew Smith <[email protected]> wrote: > Hi > > We've been using docker-storage-setup ( > https://github.com/projectatomic/docker-storage-setup) to set up our > CentOS LVM volumes. > > Because our block devices are fairly small and our containers don't write > data inside them (and we hit issues with resizes being very slow) we have a > static setup: > > AUTO_EXTEND_POOL=False > DATA_SIZE=98%VG > > The intention is to just create the lv as big as we can up front, because > we don't need that space for anything else. We leave 2% for metadata. > > However - we're running in to a problem, where our metadata volume is > hitting 100% before out data volume gets close (sometimes, 60%). We're > deploying kubernetes on these boxes, which garbage collects based on data > usage not metadata usage, so we are often hitting the 100% and taking out > boxes. > > What I'm struggling to understand is how much metadata a docker image or > docker container uses. If I can find that out I can figure out a good > data:metadata ratio. Is there a way for me to discover this? > > Or, as I suspect, am I missing something here? > > Thanks > -- > Andy Smith > http://andrewmichaelsmith.com | @bingleybeep > > -- -- Jeremy Eder
