hi Koushik, Good day to you, and thank you for your e-mail.
May I know which local storage you are referring to? Looking forward to your reply, thank you. Cheers. On Mon, Jul 15, 2013 at 4:28 PM, Koushik Das <koushik....@citrix.com> wrote: > There is an issue with local storage as well. Looks like there was change > in the storage_pool table where the available_bytes column was changed to > used_bytes but in the code available bytes was still passed for used bytes > for local storage. > > > -----Original Message----- > > From: Indra Pramana [mailto:in...@sg.or.id] > > Sent: Monday, July 15, 2013 10:29 AM > > To: users@cloudstack.apache.org > > Cc: guangj...@gmail.com; Wido den Hollander > > Subject: Re: Wrong storage capacity issue reported > > > > Hi Prasanna, > > > > Good day to you, and thank you for your e-mail. > > > > See my reply inline below. > > > > > > On Mon, Jul 15, 2013 at 12:38 PM, Prasanna Santhanam <t...@apache.org> > > wrote: > > > > > On Mon, Jul 15, 2013 at 11:58:47AM +0800, Indra Pramana wrote: > > > > Dear all, > > > > > > > > In addition to my previous e-mail, I just realised that the wrong > > > capacity > > > > usage information is only applicable to the Ceph RBD primary > > > > storage. I > > > did > > > > a check manually on the "storage_pool" table on the "cloud" MySQL > > > database: > > > > > > > > > > > > +-----+-----------------+--------------------------------------+-------------------+---- > > > --+----------------+--------+------------+-------------------+----------------+----------- > > > ----------------+------------------------------------------------+------------------------+- > > > --------------------+---------------------+-------------+-------------+-------------------- > > -+-------+ > > > > | id | name | uuid | > > > > pool_type | port | data_center_id | pod_id | cluster_id | > > > > available_bytes | capacity_bytes | host_address | > > > > user_info | path > | > > > > created | removed | update_time | status > | > > > > storage_provider_id | scope | > > > > > > > > +-----+-----------------+--------------------------------------+-------------------+---- > > > --+----------------+--------+------------+-------------------+----------------+----------- > > > ----------------+------------------------------------------------+------------------------+- > > > --------------------+---------------------+-------------+-------------+-------------------- > > -+-------+ > > > > | 209 | sc-image | bab81ce8-d53f-3a7d-b8f6-841702f65c89 | > > > > RBD | 6789 | 6 | 6 | 6 | > > > > 38283921137336466 | 6013522722816 | ceph-mon.xxx.com | admin:xxx | > > > > sc1 | 2013-07-13 08:58:27 | NULL | > > > > NULL | Up | NULL | NULL | > > > > > > > > The "available_bytes" column is wrong. > > > > > > > > My issue is similar to the one reported by Guangjian Liu here, but > > > > it > > > seems > > > > that so far there's no solutions available? > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/cloudstack- > > dev/201304.mbox/%3 > > > > > CCAKryD0b_QtjhtsFjk8McEQzDH6frOxe6EkJmFrcMDN54e5ZH9A@mail.gmail. > > com%3E > > > > > > It might be a bug on ceph but the thread you referenced was resolved. > > > See here: http://markmail.org/message/mkm2fqyawmwpufsc > > > > > > > Thanks for the email thread. However, the conversation doesn't mention on > > how was the problem resolved. > > > > I tried to execute the same command which have suggested by Wido on the > > email thread: > > > > === > > root@hv-kvm-02:~# virsh pool-list > > Name State Autostart > > ----------------------------------------- > > 87ba6ca3-1b46-3f36-b138-2fd3bffb3d71 active no > > bab81ce8-d53f-3a7d-b8f6-841702f65c89 active no > > ff06ae2a-ff27-4ff6-87b3-c2f942cf76d6 active no > > > > root@hv-kvm-02:~# virsh pool-info bab81ce8-d53f-3a7d-b8f6-841702f65c89 > > Name: bab81ce8-d53f-3a7d-b8f6-841702f65c89 > > UUID: bab81ce8-d53f-3a7d-b8f6-841702f65c89 > > State: running > > Persistent: no > > Autostart: no > > Capacity: 5.47 TiB > > Allocation: *34819.02 TiB* <-- wrong information > > Available: 5.47 TiB > > === > > > > If it still persists, can you please file a bug on JIRA? > > > > > > > Any specific instructions on how to file the bug? > > > > > > > > Is it safe for me to update the database record manually (using the > > > UPDATE > > > > MySQL command) to reflect the actual usage of the disk? > > > > > > Even if you do whatever is reporting the stats to CS about Ceph's > > > storage usage, will overwrite those values. > > > > > > > Noted, thanks. So it's an issue on the Ceph RBD side rather than on > > CloudStack side? Updating the Cloudstack database record manually will > not > > help? > > > > In any case, I tried to update the "available_bytes" and "capacity_bytes" > > record on the "storage_pool" table of "cloud" database manually to > reflect > > the actual size of the RBD image, which is 3 TB: > > > > === > > indra@cs-mgmt-01:~$ rbd --image sc-image -p sc1 info rbd image > 'sc-image': > > size 3072 GB in 786432 objects > > order 22 (4096 KB objects) > > block_name_prefix: rb.0.1825.238e1f29 > > format: 1 > > > > mysql> UPDATE storage_pool SET available_bytes=3072000000, > > capacity_bytes=3072000000 WHERE id=209; > > Query OK, 1 row affected (0.03 sec) > > Rows matched: 1 Changed: 1 Warnings: 0 === > > > > I tried to re-run the creation of the VM instance again, and it's still > failed even > > though the error message is a bit different. Not too sure where the > > "usedBytes" value is coming from? > > > > usedBytes: 38283921137336466 > > > > === > > 2013-07-15 12:10:46,232 DEBUG [cloud.storage.StorageManagerImpl] > > (Job-Executor-5:job-170) Checking pool 209 for storage, totalSize: > > 3072000000, usedBytes: 38283921137336466, usedPct: 1.246221391189338E7, > > disable threshold: 0.85 > > 2013-07-15 12:10:46,232 DEBUG [cloud.storage.StorageManagerImpl] > > (Job-Executor-5:job-170) Insufficient space on pool: 209 since its usage > > percentage: 1.246221391189338E7 has crossed the > > pool.storage.capacity.disablethreshold: 0.85 > > 2013-07-15 12:10:46,232 DEBUG > > [storage.allocator.FirstFitStoragePoolAllocator] (Job-Executor-5:job-170) > > FirstFitStoragePoolAllocator returning 0 suitable storage pools > > 2013-07-15 12:10:46,232 DEBUG [cloud.deploy.FirstFitPlanner] > > (Job-Executor-5:job-170) No suitable pools found for volume: > > Vol[231|vm=227|DATADISK] under cluster: 6 > > 2013-07-15 12:10:46,233 DEBUG [cloud.deploy.FirstFitPlanner] > > (Job-Executor-5:job-170) No suitable pools found > > 2013-07-15 12:10:46,233 DEBUG [cloud.deploy.FirstFitPlanner] > > (Job-Executor-5:job-170) No suitable storagePools found under this > Cluster: > > 6 > > 2013-07-15 12:10:46,233 DEBUG [cloud.deploy.FirstFitPlanner] > > (Job-Executor-5:job-170) Could not find suitable Deployment Destination > for > > this VM under any clusters, returning. > > 2013-07-15 12:10:46,235 DEBUG [cloud.vm.UserVmManagerImpl] > > (Job-Executor-5:job-170) Destroying vm VM[User|Indra-Test-6] as it > failed to > > create on Host with Id:null > > 2013-07-15 12:10:46,358 DEBUG [cloud.capacity.CapacityManagerImpl] > > (Job-Executor-5:job-170) VM state transitted from :Stopped to Error with > > event: OperationFailedToErrorvm's original host id: null new host id: > null host > > id before state transition: null > > 2013-07-15 12:10:47,052 INFO [user.vm.DeployVMCmd] > > (Job-Executor-5:job-170) > > com.cloud.exception.InsufficientServerCapacityException: Unable to create > > a deployment for VM[User|Indra-Test-6]Scope=interface > > com.cloud.dc.DataCenter; id=6 > > 2013-07-15 12:10:47,052 INFO [user.vm.DeployVMCmd] > > (Job-Executor-5:job-170) Unable to create a deployment for VM[User|Indra- > > Test-6] === > > > > > > > > > > > > > > > > > Looking forward to your reply, thank you. > > > > > > > > Cheers. > > > > > > > > > > > > > > > > > > > > On Mon, Jul 15, 2013 at 11:41 AM, Indra Pramana <in...@sg.or.id> > wrote: > > > > > > > > > Dear all, > > > > > > > > > > I am using CloudStack 4.1.0. Just managed to get it setup over the > > > > > weekend. System VMs have been created and both (SSVM and CPVM) > > are > > > running > > > > > fine. The default CentOS template has also been downloaded and > > > > > ready > > > to use. > > > > > > > > > > However, I am not able to launch my first VM instance because the > > > storage > > > > > capacity checker is reporting wrong usage information. I have a > > > > > total > > > of 13 > > > > > TB of primary storage and the capacity checker is reporting my > > > > > usage is > > > > > 36,510 TB (274879.22%), which is not supposed to be the case. > > > > > > > > > > === > > > > > 2013-07-15 11:27:31,632 DEBUG [cloud.alert.AlertManagerImpl] > > > > > (CapacityChecker:null) System Alert: Low Available Storage in > > > > > cluster > > > > > Cluster-01 pod Pod-01 of availability zone 01 > > > > > 2013-07-15 11:27:31,632 DEBUG [cloud.alert.AlertManagerImpl] > > > > > (CapacityChecker:null) Available storage space is low, total: > > > > > 13282342 > > > MB, > > > > > used: 36510398411 MB (274879.22%) > > > > > === > > > > > > > > > > As a result, VM instance creation fails since it's not able to > > > > > find available storage pool. > > > > > > > > > > === > > > > > 2013-07-15 11:15:28,313 DEBUG [cloud.storage.StorageManagerImpl] > > > > > (Job-Executor-3:job-168) Checking pool: 208 for volume allocation > > > > > [Vol[227|vm=225|ROOT]], maxSize : 15828044742656, > > totalAllocatedSize : > > > > > 1769538048, askingSize : 8589934592, allocated disable threshold: > > > > > 0.85 > > > > > 2013-07-15 11:15:28,313 DEBUG > > > > > [storage.allocator.AbstractStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > Checking if storage pool is suitable, name: sc-image ,poolId: 209 > > > > > 2013-07-15 11:15:28,313 DEBUG > > > > > [storage.allocator.AbstractStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > Is localStorageAllocationNeeded? false > > > > > 2013-07-15 11:15:28,313 DEBUG > > > > > [storage.allocator.AbstractStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > Is storage pool shared? true > > > > > 2013-07-15 11:15:28,317 DEBUG [cloud.storage.StorageManagerImpl] > > > > > (Job-Executor-3:job-168) Checking pool 209 for storage, totalSize: > > > > > 6013522722816, usedBytes: 38283921137336466, usedPct: > > > 6366.305226067051, > > > > > disable threshold: 0.85 > > > > > 2013-07-15 11:15:28,317 DEBUG [cloud.storage.StorageManagerImpl] > > > > > (Job-Executor-3:job-168) Insufficient space on pool: 209 since its > > > usage > > > > > percentage: 6366.305226067051 has crossed the > > > > > pool.storage.capacity.disablethreshold: 0.85 > > > > > 2013-07-15 11:15:28,317 DEBUG > > > > > [storage.allocator.FirstFitStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > FirstFitStoragePoolAllocator returning 1 suitable storage pools > > > > > 2013-07-15 11:15:28,317 DEBUG [cloud.deploy.FirstFitPlanner] > > > > > (Job-Executor-3:job-168) Checking suitable pools for volume (Id, > Type): > > > > > (228,DATADISK) > > > > > 2013-07-15 11:15:28,317 DEBUG [cloud.deploy.FirstFitPlanner] > > > > > (Job-Executor-3:job-168) We need to allocate new storagepool for > > > > > this > > > volume > > > > > 2013-07-15 11:15:28,319 DEBUG [cloud.deploy.FirstFitPlanner] > > > > > (Job-Executor-3:job-168) Calling StoragePoolAllocators to find > > > > > suitable pools > > > > > 2013-07-15 11:15:28,319 DEBUG > > > > > [storage.allocator.FirstFitStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > Looking for pools in dc: 6 pod:6 cluster:6 having tags:[rbd] > > > > > 2013-07-15 11:15:28,322 DEBUG > > > > > [storage.allocator.FirstFitStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > FirstFitStoragePoolAllocator has 1 pools to check for allocation > > > > > 2013-07-15 11:15:28,322 DEBUG > > > > > [storage.allocator.AbstractStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > Checking if storage pool is suitable, name: sc-image ,poolId: 209 > > > > > 2013-07-15 11:15:28,322 DEBUG > > > > > [storage.allocator.AbstractStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > Is localStorageAllocationNeeded? false > > > > > 2013-07-15 11:15:28,322 DEBUG > > > > > [storage.allocator.AbstractStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > Is storage pool shared? true > > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.storage.StorageManagerImpl] > > > > > (Job-Executor-3:job-168) Checking pool 209 for storage, totalSize: > > > > > 6013522722816, usedBytes: 38283921137336466, usedPct: > > > 6366.305226067051, > > > > > disable threshold: 0.85 > > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.storage.StorageManagerImpl] > > > > > (Job-Executor-3:job-168) Insufficient space on pool: 209 since its > > > usage > > > > > percentage: 6366.305226067051 has crossed the > > > > > pool.storage.capacity.disablethreshold: 0.85 > > > > > 2013-07-15 11:15:28,326 DEBUG > > > > > [storage.allocator.FirstFitStoragePoolAllocator] > > > (Job-Executor-3:job-168) > > > > > FirstFitStoragePoolAllocator returning 0 suitable storage pools > > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > > > (Job-Executor-3:job-168) No suitable pools found for volume: > > > > > Vol[228|vm=225|DATADISK] under cluster: 6 > > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > > > (Job-Executor-3:job-168) No suitable pools found > > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > > > (Job-Executor-3:job-168) No suitable storagePools found under this > > > Cluster: > > > > > 6 > > > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > > > (Job-Executor-3:job-168) Could not find suitable Deployment > > > Destination for > > > > > this VM under any clusters, returning. > > > > > 2013-07-15 11:15:28,332 DEBUG [cloud.vm.UserVmManagerImpl] > > > > > (Job-Executor-3:job-168) Destroying vm VM[User|Indra-Test-3] as it > > > failed > > > > > to create on Host with Id:null > > > > > 2013-07-15 11:15:28,498 DEBUG [cloud.capacity.CapacityManagerImpl] > > > > > (Job-Executor-3:job-168) VM state transitted from :Stopped to > > > > > Error > > > with > > > > > event: OperationFailedToErrorvm's original host id: null new host > id: > > > null > > > > > host id before state transition: null > > > > > 2013-07-15 11:15:29,125 INFO [user.vm.DeployVMCmd] > > > > > (Job-Executor-3:job-168) > > > > > com.cloud.exception.InsufficientServerCapacityException: Unable to > > > create a > > > > > deployment for VM[User|Indra-Test-3]Scope=interface > > > > > com.cloud.dc.DataCenter; id=6 > > > > > === > > > > > > > > > > Anyone can advise on how to resolve this problem? > > > > > > > > > > Looking forward to your reply, thank you. > > > > > > > > > > Cheers. > > > > > > > > > > > -- > > > Prasanna., > > > > > > ------------------------ > > > Powered by BigRock.com > > > > > > >
