There is an issue with local storage as well. Looks like there was change in the storage_pool table where the available_bytes column was changed to used_bytes but in the code available bytes was still passed for used bytes for local storage.
> -----Original Message----- > From: Indra Pramana [mailto:in...@sg.or.id] > Sent: Monday, July 15, 2013 10:29 AM > To: users@cloudstack.apache.org > Cc: guangj...@gmail.com; Wido den Hollander > Subject: Re: Wrong storage capacity issue reported > > Hi Prasanna, > > Good day to you, and thank you for your e-mail. > > See my reply inline below. > > > On Mon, Jul 15, 2013 at 12:38 PM, Prasanna Santhanam <t...@apache.org> > wrote: > > > On Mon, Jul 15, 2013 at 11:58:47AM +0800, Indra Pramana wrote: > > > Dear all, > > > > > > In addition to my previous e-mail, I just realised that the wrong > > capacity > > > usage information is only applicable to the Ceph RBD primary > > > storage. I > > did > > > a check manually on the "storage_pool" table on the "cloud" MySQL > > database: > > > > > > > > +-----+-----------------+--------------------------------------+-------------------+---- > --+----------------+--------+------------+-------------------+----------------+----------- > ----------------+------------------------------------------------+------------------------+- > --------------------+---------------------+-------------+-------------+-------------------- > -+-------+ > > > | id | name | uuid | > > > pool_type | port | data_center_id | pod_id | cluster_id | > > > available_bytes | capacity_bytes | host_address | > > > user_info | path | > > > created | removed | update_time | status | > > > storage_provider_id | scope | > > > > > +-----+-----------------+--------------------------------------+-------------------+---- > --+----------------+--------+------------+-------------------+----------------+----------- > ----------------+------------------------------------------------+------------------------+- > --------------------+---------------------+-------------+-------------+-------------------- > -+-------+ > > > | 209 | sc-image | bab81ce8-d53f-3a7d-b8f6-841702f65c89 | > > > RBD | 6789 | 6 | 6 | 6 | > > > 38283921137336466 | 6013522722816 | ceph-mon.xxx.com | admin:xxx | > > > sc1 | 2013-07-13 08:58:27 | NULL | > > > NULL | Up | NULL | NULL | > > > > > > The "available_bytes" column is wrong. > > > > > > My issue is similar to the one reported by Guangjian Liu here, but > > > it > > seems > > > that so far there's no solutions available? > > > > > > > > http://mail-archives.apache.org/mod_mbox/cloudstack- > dev/201304.mbox/%3 > > > CCAKryD0b_QtjhtsFjk8McEQzDH6frOxe6EkJmFrcMDN54e5ZH9A@mail.gmail. > com%3E > > > > It might be a bug on ceph but the thread you referenced was resolved. > > See here: http://markmail.org/message/mkm2fqyawmwpufsc > > > > Thanks for the email thread. However, the conversation doesn't mention on > how was the problem resolved. > > I tried to execute the same command which have suggested by Wido on the > email thread: > > === > root@hv-kvm-02:~# virsh pool-list > Name State Autostart > ----------------------------------------- > 87ba6ca3-1b46-3f36-b138-2fd3bffb3d71 active no > bab81ce8-d53f-3a7d-b8f6-841702f65c89 active no > ff06ae2a-ff27-4ff6-87b3-c2f942cf76d6 active no > > root@hv-kvm-02:~# virsh pool-info bab81ce8-d53f-3a7d-b8f6-841702f65c89 > Name: bab81ce8-d53f-3a7d-b8f6-841702f65c89 > UUID: bab81ce8-d53f-3a7d-b8f6-841702f65c89 > State: running > Persistent: no > Autostart: no > Capacity: 5.47 TiB > Allocation: *34819.02 TiB* <-- wrong information > Available: 5.47 TiB > === > > If it still persists, can you please file a bug on JIRA? > > > > Any specific instructions on how to file the bug? > > > > > Is it safe for me to update the database record manually (using the > > UPDATE > > > MySQL command) to reflect the actual usage of the disk? > > > > Even if you do whatever is reporting the stats to CS about Ceph's > > storage usage, will overwrite those values. > > > > Noted, thanks. So it's an issue on the Ceph RBD side rather than on > CloudStack side? Updating the Cloudstack database record manually will not > help? > > In any case, I tried to update the "available_bytes" and "capacity_bytes" > record on the "storage_pool" table of "cloud" database manually to reflect > the actual size of the RBD image, which is 3 TB: > > === > indra@cs-mgmt-01:~$ rbd --image sc-image -p sc1 info rbd image 'sc-image': > size 3072 GB in 786432 objects > order 22 (4096 KB objects) > block_name_prefix: rb.0.1825.238e1f29 > format: 1 > > mysql> UPDATE storage_pool SET available_bytes=3072000000, > capacity_bytes=3072000000 WHERE id=209; > Query OK, 1 row affected (0.03 sec) > Rows matched: 1 Changed: 1 Warnings: 0 === > > I tried to re-run the creation of the VM instance again, and it's still > failed even > though the error message is a bit different. Not too sure where the > "usedBytes" value is coming from? > > usedBytes: 38283921137336466 > > === > 2013-07-15 12:10:46,232 DEBUG [cloud.storage.StorageManagerImpl] > (Job-Executor-5:job-170) Checking pool 209 for storage, totalSize: > 3072000000, usedBytes: 38283921137336466, usedPct: 1.246221391189338E7, > disable threshold: 0.85 > 2013-07-15 12:10:46,232 DEBUG [cloud.storage.StorageManagerImpl] > (Job-Executor-5:job-170) Insufficient space on pool: 209 since its usage > percentage: 1.246221391189338E7 has crossed the > pool.storage.capacity.disablethreshold: 0.85 > 2013-07-15 12:10:46,232 DEBUG > [storage.allocator.FirstFitStoragePoolAllocator] (Job-Executor-5:job-170) > FirstFitStoragePoolAllocator returning 0 suitable storage pools > 2013-07-15 12:10:46,232 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-5:job-170) No suitable pools found for volume: > Vol[231|vm=227|DATADISK] under cluster: 6 > 2013-07-15 12:10:46,233 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-5:job-170) No suitable pools found > 2013-07-15 12:10:46,233 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-5:job-170) No suitable storagePools found under this Cluster: > 6 > 2013-07-15 12:10:46,233 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-5:job-170) Could not find suitable Deployment Destination for > this VM under any clusters, returning. > 2013-07-15 12:10:46,235 DEBUG [cloud.vm.UserVmManagerImpl] > (Job-Executor-5:job-170) Destroying vm VM[User|Indra-Test-6] as it failed to > create on Host with Id:null > 2013-07-15 12:10:46,358 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-5:job-170) VM state transitted from :Stopped to Error with > event: OperationFailedToErrorvm's original host id: null new host id: null > host > id before state transition: null > 2013-07-15 12:10:47,052 INFO [user.vm.DeployVMCmd] > (Job-Executor-5:job-170) > com.cloud.exception.InsufficientServerCapacityException: Unable to create > a deployment for VM[User|Indra-Test-6]Scope=interface > com.cloud.dc.DataCenter; id=6 > 2013-07-15 12:10:47,052 INFO [user.vm.DeployVMCmd] > (Job-Executor-5:job-170) Unable to create a deployment for VM[User|Indra- > Test-6] === > > > > > > > > > > > Looking forward to your reply, thank you. > > > > > > Cheers. > > > > > > > > > > > > > > > On Mon, Jul 15, 2013 at 11:41 AM, Indra Pramana <in...@sg.or.id> wrote: > > > > > > > Dear all, > > > > > > > > I am using CloudStack 4.1.0. Just managed to get it setup over the > > > > weekend. System VMs have been created and both (SSVM and CPVM) > are > > running > > > > fine. The default CentOS template has also been downloaded and > > > > ready > > to use. > > > > > > > > However, I am not able to launch my first VM instance because the > > storage > > > > capacity checker is reporting wrong usage information. I have a > > > > total > > of 13 > > > > TB of primary storage and the capacity checker is reporting my > > > > usage is > > > > 36,510 TB (274879.22%), which is not supposed to be the case. > > > > > > > > === > > > > 2013-07-15 11:27:31,632 DEBUG [cloud.alert.AlertManagerImpl] > > > > (CapacityChecker:null) System Alert: Low Available Storage in > > > > cluster > > > > Cluster-01 pod Pod-01 of availability zone 01 > > > > 2013-07-15 11:27:31,632 DEBUG [cloud.alert.AlertManagerImpl] > > > > (CapacityChecker:null) Available storage space is low, total: > > > > 13282342 > > MB, > > > > used: 36510398411 MB (274879.22%) > > > > === > > > > > > > > As a result, VM instance creation fails since it's not able to > > > > find available storage pool. > > > > > > > > === > > > > 2013-07-15 11:15:28,313 DEBUG [cloud.storage.StorageManagerImpl] > > > > (Job-Executor-3:job-168) Checking pool: 208 for volume allocation > > > > [Vol[227|vm=225|ROOT]], maxSize : 15828044742656, > totalAllocatedSize : > > > > 1769538048, askingSize : 8589934592, allocated disable threshold: > > > > 0.85 > > > > 2013-07-15 11:15:28,313 DEBUG > > > > [storage.allocator.AbstractStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > Checking if storage pool is suitable, name: sc-image ,poolId: 209 > > > > 2013-07-15 11:15:28,313 DEBUG > > > > [storage.allocator.AbstractStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > Is localStorageAllocationNeeded? false > > > > 2013-07-15 11:15:28,313 DEBUG > > > > [storage.allocator.AbstractStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > Is storage pool shared? true > > > > 2013-07-15 11:15:28,317 DEBUG [cloud.storage.StorageManagerImpl] > > > > (Job-Executor-3:job-168) Checking pool 209 for storage, totalSize: > > > > 6013522722816, usedBytes: 38283921137336466, usedPct: > > 6366.305226067051, > > > > disable threshold: 0.85 > > > > 2013-07-15 11:15:28,317 DEBUG [cloud.storage.StorageManagerImpl] > > > > (Job-Executor-3:job-168) Insufficient space on pool: 209 since its > > usage > > > > percentage: 6366.305226067051 has crossed the > > > > pool.storage.capacity.disablethreshold: 0.85 > > > > 2013-07-15 11:15:28,317 DEBUG > > > > [storage.allocator.FirstFitStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > FirstFitStoragePoolAllocator returning 1 suitable storage pools > > > > 2013-07-15 11:15:28,317 DEBUG [cloud.deploy.FirstFitPlanner] > > > > (Job-Executor-3:job-168) Checking suitable pools for volume (Id, Type): > > > > (228,DATADISK) > > > > 2013-07-15 11:15:28,317 DEBUG [cloud.deploy.FirstFitPlanner] > > > > (Job-Executor-3:job-168) We need to allocate new storagepool for > > > > this > > volume > > > > 2013-07-15 11:15:28,319 DEBUG [cloud.deploy.FirstFitPlanner] > > > > (Job-Executor-3:job-168) Calling StoragePoolAllocators to find > > > > suitable pools > > > > 2013-07-15 11:15:28,319 DEBUG > > > > [storage.allocator.FirstFitStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > Looking for pools in dc: 6 pod:6 cluster:6 having tags:[rbd] > > > > 2013-07-15 11:15:28,322 DEBUG > > > > [storage.allocator.FirstFitStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > FirstFitStoragePoolAllocator has 1 pools to check for allocation > > > > 2013-07-15 11:15:28,322 DEBUG > > > > [storage.allocator.AbstractStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > Checking if storage pool is suitable, name: sc-image ,poolId: 209 > > > > 2013-07-15 11:15:28,322 DEBUG > > > > [storage.allocator.AbstractStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > Is localStorageAllocationNeeded? false > > > > 2013-07-15 11:15:28,322 DEBUG > > > > [storage.allocator.AbstractStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > Is storage pool shared? true > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.storage.StorageManagerImpl] > > > > (Job-Executor-3:job-168) Checking pool 209 for storage, totalSize: > > > > 6013522722816, usedBytes: 38283921137336466, usedPct: > > 6366.305226067051, > > > > disable threshold: 0.85 > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.storage.StorageManagerImpl] > > > > (Job-Executor-3:job-168) Insufficient space on pool: 209 since its > > usage > > > > percentage: 6366.305226067051 has crossed the > > > > pool.storage.capacity.disablethreshold: 0.85 > > > > 2013-07-15 11:15:28,326 DEBUG > > > > [storage.allocator.FirstFitStoragePoolAllocator] > > (Job-Executor-3:job-168) > > > > FirstFitStoragePoolAllocator returning 0 suitable storage pools > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > > (Job-Executor-3:job-168) No suitable pools found for volume: > > > > Vol[228|vm=225|DATADISK] under cluster: 6 > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > > (Job-Executor-3:job-168) No suitable pools found > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > > (Job-Executor-3:job-168) No suitable storagePools found under this > > Cluster: > > > > 6 > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > > (Job-Executor-3:job-168) Could not find suitable Deployment > > Destination for > > > > this VM under any clusters, returning. > > > > 2013-07-15 11:15:28,332 DEBUG [cloud.vm.UserVmManagerImpl] > > > > (Job-Executor-3:job-168) Destroying vm VM[User|Indra-Test-3] as it > > failed > > > > to create on Host with Id:null > > > > 2013-07-15 11:15:28,498 DEBUG [cloud.capacity.CapacityManagerImpl] > > > > (Job-Executor-3:job-168) VM state transitted from :Stopped to > > > > Error > > with > > > > event: OperationFailedToErrorvm's original host id: null new host id: > > null > > > > host id before state transition: null > > > > 2013-07-15 11:15:29,125 INFO [user.vm.DeployVMCmd] > > > > (Job-Executor-3:job-168) > > > > com.cloud.exception.InsufficientServerCapacityException: Unable to > > create a > > > > deployment for VM[User|Indra-Test-3]Scope=interface > > > > com.cloud.dc.DataCenter; id=6 > > > > === > > > > > > > > Anyone can advise on how to resolve this problem? > > > > > > > > Looking forward to your reply, thank you. > > > > > > > > Cheers. > > > > > > > > -- > > Prasanna., > > > > ------------------------ > > Powered by BigRock.com > > > >
