JF Vincent created CLOUDSTACK-7703: -------------------------------------- Summary: Cloudstack server endless loop when trying to create a volume while storage pool is full Key: CLOUDSTACK-7703 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7703 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: Management Server Affects Versions: 4.3.0 Environment: Centos 6.5 Reporter: JF Vincent Priority: Critical
When trying to create a VM, and thus a volume for it and the primary storage is full (over 90%), the managament server enter in and endless loop (extract below) and we have to restart it to exit this loop. 2014-10-14 11:39:20,701 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c) No suitable pools found for volume: Vol[5436|vm=5855|DATADISK] under cluster: 2 2014-10-14 11:39:20,702 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c) No suitable pools found 2014-10-14 11:39:20,702 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c) No suitable storagePools found under this Cluster: 2 2014-10-14 11:39:20,705 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Could not find suitable Deployment Destination for this VM under any clusters, returning. 2014-10-14 11:39:20,705 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Searching all possible resources under this Zone: 2 2014-10-14 11:39:20,705 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 2 2014-10-14 11:39:20,707 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Removing from the clusterId list these clusters from avoid set: [] 2014-10-14 11:39:20,714 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Checking resources in Cluster: 2 under Pod: 2 2014-10-14 11:39:20,714 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Looking for hosts in dc: 2 pod:2 cluster:2 2014-10-14 11:39:20,716 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) FirstFitAllocator has 3 hosts to check for allocation: [Host[-79-Routing], Host[-89-Routing], Host[-77-Routing]] 2014-10-14 11:39:20,717 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Found 3 hosts for allocation after prioritization: [Host[-79-Routing], Host[-89-Routing], Host[-77-Routing]] 2014-10-14 11:39:20,717 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Looking for speed=500Mhz, Ram=500 2014-10-14 11:39:20,720 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host: 79 has cpu capability (cpu:8, speed:2399) to support requested CPU: 1 and requested speed: 500 2014-10-14 11:39:20,720 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Checking if host: 79 has enough capacity for requested CPU: 500 and requested RAM: 524288000 , cpuOverprovisioningFactor: 4.0 2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Hosts's actual total CPU: 19192 and CPU after applying overprovisioning: 76768 2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free CPU: 57268 , Requested CPU: 500 2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free RAM: 93916725248 , Requested RAM: 524288000 2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host has enough CPU and RAM available 2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 79, used: 19000, reserved: 500, actual total: 19192, total with overprovisioning: 76768; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true 2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 79, used: 17192452096, reserved: 524288000, total: 111633465344; requested mem: 524288000,alloc_from_last_host?:false ,considerReservedCapacity?: true 2014-10-14 11:39:20,721 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Found a suitable host, adding to list: 79 2014-10-14 11:39:20,724 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host: 89 has cpu capability (cpu:8, speed:2399) to support requested CPU: 1 and requested speed: 500 2014-10-14 11:39:20,724 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Checking if host: 89 has enough capacity for requested CPU: 500 and requested RAM: 524288000 , cpuOverprovisioningFactor: 4.0 2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Hosts's actual total CPU: 19192 and CPU after applying overprovisioning: 76768 2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free CPU: 66268 , Requested CPU: 500 2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free RAM: 101013487616 , Requested RAM: 524288000 2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host has enough CPU and RAM available 2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 89, used: 10500, reserved: 0, actual total: 19192, total with overprovisioning: 76768; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true 2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 89, used: 10619977728, reserved: 0, total: 111633465344; requested mem: 524288000,alloc_from_last_host?:false ,considerReservedCapacity?: true 2014-10-14 11:39:20,725 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Found a suitable host, adding to list: 89 2014-10-14 11:39:20,727 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host: 77 has cpu capability (cpu:8, speed:2399) to support requested CPU: 1 and requested speed: 500 2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Checking if host: 77 has enough capacity for requested CPU: 500 and requested RAM: 524288000 , cpuOverprovisioningFactor: 4.0 2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Hosts's actual total CPU: 19192 and CPU after applying overprovisioning: 76768 2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free CPU: 59768 , Requested CPU: 500 2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free RAM: 95586058240 , Requested RAM: 524288000 2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host has enough CPU and RAM available 2014-10-14 11:39:20,729 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can alloc CPU from host: 77, used: 11500, reserved: 5500, actual total: 19192, total with overprovisioning: 76768; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true 2014-10-14 11:39:20,729 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can alloc MEM from host: 77, used: 10280239104, reserved: 5767168000, total: 111633465344; requested mem: 524288000,alloc_from_last_host?:false ,considerReservedCapacity?: true 2014-10-14 11:39:20,729 DEBUG [allocator.impl.FirstFitAllocator] (Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Found a suitable host, adding to list: 77 -- This message was sent by Atlassian JIRA (v6.3.4#6332)