Hello CloudStack community,
Could you please take a look to following issue. The similar was
described it that topic
http://comments.gmane.org/gmane.comp.apache.cloudstack.user/13848 but
looks like wasn't resolved.
In my environment I have:
CloudStack v. 4.2.1
VMware v. 5.1
Recently I have found that CloudStack UI provide following information
for SATA storage:
Disk Total 5.00 TB
Disk Allocated 7.26 TB
I have looked to VMware side and found absolutely different data:
Capacity: 5.00 TB
Provisioned Space: 4.27 TB
As you can see CloudStack show allocated disk space on 3 TB more.
My Global Settings that could be helpful for investigation:
storage.overprovisioning.factor = 4
storage.cleanup.enabled = true
storage.cleanup.interval = 86400
storage.stats.interval = 60000
capacity.check.period = 60000
expunge.delay = 60
expunge.interval = 86400
expunge.workers = 10
I have looked to CloudStack database and found that some Volumes state
in 'Destroy' but also I have found a lot of in 'Expunged':
mysql> select v.state, sum(v.size) from volumes v, vm_instance vm,
vm_root_disk_tags d where v.instance_id=vm.id and d.vm_id=vm.id and
d.root_disk_tag like 'sata' group by v.state;
+-----------+---------------+
| state | sum(v.size) |
+-----------+---------------+
| Destroy | 664646189056 |
| Expunged | 5267777388544 |
| Expunging | 2147483648 |
| Ready | 3833258311680 |
+-----------+---------------+
4 rows in set (0.00 sec)
And a lot of VMs that have state=Expunged (in volume table) still
present in VMware and on storage. For example all these VMs still
present in VMware:
mysql> select vm.name,vm.instance_name,vm.state AS
VM_state,v.id,v.size,v.state AS VOLUME_state,v.removed,d.root_disk_tag
from volumes v, vm_instance vm, vm_root_disk_tags d where
v.instance_id=vm.id and d.vm_id=vm.id and v.state like 'Expunged' and
d.root_disk_tag like 'sata'; (Names have been
changed)
+----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
| name | instance_name | VM_state | id |
size | VOLUME_state | removed | root_disk_tag |
+----------+---------------+-----------+-----------+--------------+--------------+---------------------+---------------+
| name1 | i-43-2147-VM | Expunging | 2195 | 53687091200 |
Expunged | 2014-10-16 08:12:04 | sata |
| name2 | i-17-2123-VM | Expunging | 2173 | 10737418240 |
Expunged | 2014-10-15 19:14:30 | sata |
| name3 | i-43-2052-VM | Expunging | 2098 | 10737418240 |
Expunged | 2014-10-07 11:57:49 | sata |
| name4 | i-191-1954-VM | Expunging | 2000 | 42949672960 |
Expunged | 2014-10-01 15:15:04 | sata |
| name5 | i-132-1966-VM | Expunging | 2012 | 21474836480 |
Expunged | 2014-09-29 20:33:12 | sata |
| name6 | i-132-1967-VM | Expunging | 2013 | 21474836480 |
Expunged | 2014-09-29 20:33:12 | sata |
| name7 | i-132-1962-VM | Expunging | 2008 | 10737418240 |
Expunged | 2014-09-29 20:33:11 | sata |
| name8 | i-132-1964-VM | Expunging | 2010 | 10737418240 |
Expunged | 2014-09-29 20:33:11 | sata |
Not all VMs with 'Expunged' state still present in VMware but a lot of
from them still weren't removed!
mysql> select
id,uuid,name,status,capacity_bytes,capacity_iops,tag,disk_used_capacity,disk_reserved_capacity
from storage_pool_view;
+----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
| id | uuid | name | status | capacity_bytes |
capacity_iops | tag | disk_used_capacity | disk_reserved_capacity |
+----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
| 2 | 096b3b6e-481d-33b4-9b34-7178612a2535 | SATA | Up |
5497289703424 | NULL | sata | 7913594810761 | 0 |
+----+--------------------------------------------------------------+---------------+-------------+----------------+---------------+------+--------------------+------------------------+
5 rows in set (0.02 sec)
Also I have found that Expunged volumes have 'chain_info' field:
mysql> select name,removed,state,chain_info from volumes where id=2112
\G
*************************** 1. row ***************************
name: ROOT-2066
removed: 2014-10-09 11:57:52
state: Expunged
chain_info: {"diskDeviceBusName":"scsi0:0","diskChain":["[SATA1]
i-238-2066-VM/ROOT-2066.vmdk"]}
1 row in set (0.00 sec)
BUT Destroyed VMs don't have it:
mysql> select name,removed,state,chain_info from volumes where id=2192
\G
*************************** 1. row ***************************
name: ROOT-2143
removed: 2014-10-15 18:42:48
state: Destroy
chain_info: NULL
1 row in set (0.00 sec)
So, the questions are:
- How CloudStack calculate allocated disk size for storage? Is it use
disk size from 'Expunged' volumes?
- Can I manually push CloudStack to remove 'Expunged' volumes from
VMware?
I haven't created bug in Jira but I can if it's needed.
Thank you!
--
Best regards,
Denis
