Hi all, in the meantime we have found out that obviously not only the usage records are missing for migrated volumes but even the event records. It looks as if a volume is migrated, for the new volume record/id no events are written to the database.
Example, a new volume record after migration: mysql> select * from cloud.volumes where id=1213\G *************************** 1. row *************************** * id: 1213* account_id: 78 domain_id: 1 pool_id: 8 last_pool_id: 5 instance_id: 581 device_id: 1 name: xxxxx * uuid: a1a3668b-14cf-490e-bcbe-ef0d4f225574* size: 214748364800 folder: cloudstack01 path: e886bea4-4c94-43a3-9b17-e712c130f1a7 pod_id: 2 data_center_id: 2 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 6 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2015-06-10 06:45:53 attached: 2014-09-24 07:29:45 updated: 2015-06-10 08:53:12 removed: NULL * state: Ready* chain_info: NULL update_count: 2 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: RAW min_iops: NULL max_iops: NULL hv_ss_reserve: NULL 1 row in set (0.00 sec) ==> No events in the database: mysql> select * from cloud.event where description like '% *1213* %'\G Empty set (0.00 sec) Thus no usage records: mysql> select * from cloud_usage.usage_event where resource_id=*1213*\G Empty set (0.00 sec) mysql> select * from cloud.usage_event where resource_id=*1213*\G Empty set (0.01 sec) ---------------------------------------------- Example, volume record before migration (has now NULL for uuid): mysql> select * from cloud.volumes where id=747\G *************************** 1. row *************************** * id: 747* account_id: 78 domain_id: 1 pool_id: 5 last_pool_id: NULL instance_id: 581 device_id: 1 name: xxxxxx * uuid: NULL* size: 214748364800 folder: NULL path: f303f233-29ac-4b3f-9524-16c7e901dbdc pod_id: NULL data_center_id: 2 iscsi_name: NULL host_ip: NULL volume_type: DATADISK pool_type: NULL disk_offering_id: 6 template_id: NULL first_snapshot_backup_uuid: NULL recreatable: 0 created: 2014-08-27 06:49:25 attached: 2014-09-24 07:29:45 updated: 2015-06-10 08:53:15 removed: 2015-06-10 08:53:15 * state: Expunged* chain_info: NULL update_count: 8 disk_type: NULL vm_snapshot_chain_size: NULL iso_id: NULL display_volume: 1 format: QCOW2 min_iops: NULL max_iops: NULL hv_ss_reserve: NULL 1 row in set (0.00 sec) Events exist: mysql> select * from cloud.event where description like '% *747* %'\G *************************** 1. row *************************** id: 24165 uuid: 6669a7b2-ab27-4fdb-b21c-5e6e0c2df003 type: VOLUME.MIGRATE state: Scheduled description: Attempting to migrate volume Id: *747* to storage pool Id: 8 user_id: 60 account_id: 78 domain_id: 9 created: 2015-06-10 06:45:51 level: INFO start_id: 0 parameters: NULL archived: 0 display: 1 1 row in set (0.00 sec) And thus usage records exist: mysql> select * from cloud_usage.usage_event where resource_id=*747*\G *************************** 1. row *************************** id: 3449 type: VOLUME.CREATE account_id: 78 created: 2014-08-27 06:49:25 zone_id: 2 resource_id: *747* resource_name: bts014-data01 offering_id: 6 template_id: NULL size: 214748364800 resource_type: NULL processed: 1 virtual_size: NULL 1 row in set (0.01 sec) If some knows where and how to debug this, any ideas are very welcome. thx Norbert Am 24.06.2015 um 17:16 schrieb Norbert Klein: > Hi Praveen, > > we have come a little bit further with our research. > To answer your questions, I have investigated this case for data > volumes so far. > > The usage records which don't have the usageid point to (old) volume > table records of migrated volumes (via the volume id in the > description field). > These records have NULL in their uuid and the state is Expunged. It > looks like CS creates a new record for a mirgrated volume and > keeps the old one but sets uuid to NULL. Unfortunately there are no > new usage records for these > new volume records. This leads to the situation that volumes cannot be > charged any more after migration. > > Here is an example: > > I copied from the CS web page a virtualmachine name, a data volume > name and the uuid of the volume. > > In the database I find two volumes with this name, one has NULL for > uuid, the other one has a valid uuid: > > mysql> select * from cloud.volumes where name like 'xxxxx-data01'\G > *************************** 1. row *************************** > *id: 747* > account_id: 78 > domain_id: 1 > pool_id: 5 > last_pool_id: NULL > instance_id: 581 > device_id: 1 > name: xxxxx-data01 > * uuid: NULL* > size: 214748364800 > folder: NULL > path: f303f233-29ac-4b3f-9524-16c7e901dbdc > pod_id: NULL > data_center_id: 2 > iscsi_name: NULL > host_ip: NULL > volume_type: DATADISK > pool_type: NULL > disk_offering_id: 6 > template_id: NULL > first_snapshot_backup_uuid: NULL > recreatable: 0 > created: 2014-08-27 06:49:25 > attached: 2014-09-24 07:29:45 > updated: 2015-06-10 08:53:15 > removed: 2015-06-10 08:53:15 > state: Expunged > chain_info: NULL > update_count: 8 > disk_type: NULL > vm_snapshot_chain_size: NULL > iso_id: NULL > display_volume: 1 > format: QCOW2 > min_iops: NULL > max_iops: NULL > hv_ss_reserve: NULL > *************************** 2. row *************************** > *id: 1213* > account_id: 78 > domain_id: 1 > pool_id: 8 > last_pool_id: 5 > instance_id: 581 > device_id: 1 > name: xxxxx-data01 > * uuid: a1a3668b-14cf-490e-bcbe-ef0d4f225574* > size: 214748364800 > folder: cloudstack01 > path: e886bea4-4c94-43a3-9b17-e712c130f1a7 > pod_id: 2 > data_center_id: 2 > iscsi_name: NULL > host_ip: NULL > volume_type: DATADISK > pool_type: NULL > disk_offering_id: 6 > template_id: NULL > first_snapshot_backup_uuid: NULL > recreatable: 0 > created: 2015-06-10 06:45:53 > attached: 2014-09-24 07:29:45 > updated: 2015-06-10 08:53:12 > removed: NULL > state: Ready > chain_info: NULL > update_count: 2 > disk_type: NULL > vm_snapshot_chain_size: NULL > iso_id: NULL > display_volume: 1 > format: RAW > min_iops: NULL > max_iops: NULL > hv_ss_reserve: NULL > 2 rows in set (0.00 sec) > > > If I use the table ID of the first record (747) and search for usage > records I find all old records and all miss the usageid: > > 🐵 > list usagerecords type=6 > projectid=4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e startdate=2015-06-01 > enddate=2015-06-24 | grep -A 10 'Volume Id: *747*' > description = Volume Id: 747 usage time (DiskOffering: 6) > domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06 > enddate = 2015-06-23'T'23:59:59+00:00 > offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > rawusage = 24 > size = 214748364800 > startdate = 2015-06-23'T'00:00:00+00:00 > usage = 24 Hrs > usagetype = 6 > -- > description = Volume Id: 747 usage time (DiskOffering: 6) > domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06 > enddate = 2015-06-22'T'23:59:59+00:00 > offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > rawusage = 24 > size = 214748364800 > startdate = 2015-06-22'T'00:00:00+00:00 > usage = 24 Hrs > usagetype = 6 > -- > description = Volume Id: 747 usage time (DiskOffering: 6) > domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06 > enddate = 2015-06-21'T'23:59:59+00:00 > offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > rawusage = 24 > size = 214748364800 > startdate = 2015-06-21'T'00:00:00+00:00 > usage = 24 Hrs > usagetype = 6 > -- > description = Volume Id: 747 usage time (DiskOffering: 6) > domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06 > enddate = 2015-06-20'T'23:59:59+00:00 > offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > rawusage = 24 > size = 214748364800 > startdate = 2015-06-20'T'00:00:00+00:00 > usage = 24 Hrs > usagetype = 6 > -- > description = Volume Id: 747 usage time (DiskOffering: 6) > domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06 > enddate = 2015-06-19'T'23:59:59+00:00 > offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > rawusage = 24 > size = 214748364800 > startdate = 2015-06-19'T'00:00:00+00:00 > usage = 24 Hrs > usagetype = 6 > -- > description = Volume Id: 747 usage time (DiskOffering: 6) > domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06 > enddate = 2015-06-18'T'23:59:59+00:00 > offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > rawusage = 24 > size = 214748364800 > startdate = 2015-06-18'T'00:00:00+00:00 > usage = 24 Hrs > usagetype = 6 > -- > description = Volume Id: 747 usage time (DiskOffering: 6) > domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06 > enddate = 2015-06-17'T'23:59:59+00:00 > offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > rawusage = 24 > size = 214748364800 > startdate = 2015-06-17'T'00:00:00+00:00 > usage = 24 Hrs > usagetype = 6 > -- > description = Volume Id: 747 usage time (DiskOffering: 6) > domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06 > enddate = 2015-06-16'T'23:59:59+00:00 > offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > rawusage = 24 > size = 214748364800 > startdate = 2015-06-16'T'00:00:00+00:00 > usage = 24 Hrs > usagetype = 6 > -- > description = Volume Id: 747 usage time (DiskOffering: 6) > domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06 > enddate = 2015-06-15'T'23:59:59+00:00 > offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > rawusage = 24 > size = 214748364800 > startdate = 2015-06-15'T'00:00:00+00:00 > usage = 24 Hrs > usagetype = 6 > > > Now the real problem comes: > > We have to charge the migrated volumes of course and therefor I > searched for the corresponding usage records. First > I tried to find the id of the table record (*1213*) and then I tried > to find the uuid. Both have no result, so billing is not possible: > 🐵 > list usagerecords type=6 > projectid=4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e startdate=2015-06-01 > enddate=2015-06-24 | grep -A 10 'Volume Id: *1213*' > 🐵 > list usagerecords type=6 > projectid=4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e startdate=2015-06-01 > enddate=2015-06-24 | grep 'a1a3668b-14cf-490e-bcbe-ef0d4f225574' > 🐵 > > > But this volume is available through the API nevertheless as long as > you use the volume name: > 🐵 > list volumes listall=true > projectid=4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e name=xxxxx-data01 > count = 1 > volume: > id = a1a3668b-14cf-490e-bcbe-ef0d4f225574 > name = xxxxx-data01 > attached = 2014-09-24T09:29:45+0200 > created = 2015-06-10T08:45:53+0200 > destroyed = False > deviceid = 1 > diskofferingdisplaytext = Custom Disk > diskofferingid = 1b618634-73cf-4a95-99ba-1f6e536a6051 > diskofferingname = Custom > displayvolume = True > domain = ROOT > domainid = a7b69499-a912-11e3-a7c1-5254005b13af > hypervisor = KVM > isextractable = True > path = e886bea4-4c94-43a3-9b17-e712c130f1a7 > project = xxxxx > projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e > quiescevm = False > size = 214748364800 > state = Ready > storage = ceph-cloudstack01 > storageid = ccfacebe-19c3-38f4-a161-2bd9e4ac70a9 > storagetype = shared > tags: > type = DATADISK > virtualmachineid = 2360e425-1171-4ad7-a6b3-eec403486b19 > vmdisplayname = xxxxx > vmname = xxxxx > vmstate = Running > zoneid = 10992277-e637-4795-ab02-be57cca0c66a > zonename = EU-GER-01 > > > How can we get usage records of migrated volumes? > If this is a database inconsistancy, or even a bug, is there a way to > repair it? > > thx > Norbert > > > -----Ursprüngliche Nachricht----- > Von: Praveen B [mailto:pbprave...@gmail.com] > Gesendet: Dienstag, 23. Juni 2015 08:53 > An: users@cloudstack.apache.org > Betreff: Re: usageid missing > > Hi Ingo, > > We haven't seen this type of issue anytime before. Need to analyze > more on the issue. > > Can you upload output of usage records somewhere and provide me the > URL link to access it? > > There could be multiple volumes in CloudStack > 1) Root volumes 2) Data Volumes 3) Volumes created from Snapshots > 4) Uploaded volumes...etc. > > You need to find out if there is any specific pattern for the missing > usage id volumes. That will help us to understand the issue behaviour. > > Thaks, > Praveen Kumar, > Founder,PSI Cloud Solutions, > psiclouds.com > > On Tue, Jun 23, 2015 at 2:51 AM, Jochim, Ingo <ingo.joc...@bautzen-it.de> > wrote: > > > I'm using the usagerecords to get usage of volumes. Unfortunately for > > most of my usagerecords of volumes the usageid is missing. > > For those I have a usageid I can use this to the the volume (it's the > > same as the volume id) > > > > Why are there so many usageid's missing? > > > > We're running 4.4. > > > > Thanks and regards, > > Ingo > > > > -- > > This email was Virus checked by UTM 9. http://www.sophos.com > > > > -- > This email was Virus checked by UTM 9. http://www.sophos.com > -- www.infosecprojects.net <http://www.infosecprojects.net/>