Hi Hean, Looks like no data is being processed at all, the usage server takes the raw data from usage_event table and parses it into helper tables (such as usage_network). The data from the usage_event at the cloud_usage database is copied from the usage_event table on the cloud database. Can you check the records from those tables and see if there is a discrepancy? Also, can you verify if the usage db.properties is configured properly?
Regards, Nicolas Vazquez ________________________________ From: Hean Seng <heans...@gmail.com> Sent: Saturday, October 31, 2020 5:41 AM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: listUsageRecords Update frequency I think selec the wrong id, but still is return empty set: mysql> select * from usage_network where account_id=59; Empty set (0.00 sec) root@mgmt01 usage]# more usage.log | grep "accountId: 59" 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId: 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr: 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId: 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr: 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId: 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr: 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId: 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr: 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId: 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr: 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId: 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr: 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId: 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr: 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId: 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr: 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 2020-10-31 02:00:00,854 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId: 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 2020-10-31 02 nicolas.vazq...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue On Sat, Oct 31, 2020 at 2:57 AM Hean Seng <heans...@gmail.com> wrote: > Hi Nicolas > > The cloudstacl_usage is running . confirm from log > > 20-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > (Usage-Job-1:null) (logid:) For 5 it is already more than 29 days, will > send statement in next cycle > > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > (Usage-Job-1:null) (logid:) For 10 it is already more than 29 days, will > send statement in next cycle > > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > (Usage-Job-1:null) (logid:) For 11 it is already more than 59 days, will > send statement in next cycle > > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > (Usage-Job-1:null) (logid:) For 12 it is already more than 29 days, will > send statement in next cycle > > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > (Usage-Job-1:null) (logid:) For 13 it is already more than 58 days, will > send statement in next cycle > > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > (Usage-Job-1:null) (logid:) For 14 it is already more than 29 days, will > send statement in next cycle > > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl] > (Usage-Job-1:null) (logid:) For 23 it is already more than 29 days, will > send statement in next cycle > > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl] > (Usage-Job-1:null) (logid:) For 24 it is already more than 29 days, will > send statement in next cycle > > 2020-10-30 18:45:11,107 INFO [cloud.usage.UsageManagerImpl] > (Usage-Job-1:null) (logid:) usage job complete > > ^C > > It looks like it did not write to usage table as you say: > > mysql> select * from usage_network where > account_id="a40e9892-3051-41b6-baf2-fabe3af06c92" ; > > Empty set, 1 warning (0.00 sec) > > > > > Where is process to update the usage helper? > > > > > On Sat, Oct 31, 2020 at 1:36 AM Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > >> Hi Hean, >> >> The listVirtualMachineMetrics show the "current" usage of the virtual >> machine, and the usage records display the aggregated information (hourly, >> daily) per account. If the records are not updating can you please check >> the usage server logs if there is any error when the usage jobs run? Can >> you also check if data is being added on the helper table for network usage >> (usage_network table)? The information from this table is summarized into >> the resulting usage records when the usage jobs run. >> >> >> Regards, >> >> Nicolas Vazquez >> >> ________________________________ >> From: Hean Seng <heans...@gmail.com> >> Sent: Friday, October 30, 2020 4:18 AM >> To: users@cloudstack.apache.org <users@cloudstack.apache.org> >> Subject: Re: listUsageRecords Update frequency >> >> Hi Nicolas >> >> Thanks for answering. >> >> Some questions puzzle me, what is different between listUsageRecords and >> listVirtualMachineMetrics , in terms of Network Read and Write data. >> (other >> then one is per account basis, and another one is per VM basis) , the >> data >> in listVirtualMachineMetrics almost instant update . >> >> Is listUsageReocrds (type 4 and type 5) only keep metric with Public IP , >> and not calculated the SharedNetwork ? >> >> I facing issue of AdvanceNetwork withSG, and listVirtualMachineMetrics >> network sent and receive update almost instant, and listUsageRecords never >> update , I do not know where to check if there is error when updating this >> Account. Or if the account have setting/options to choose if process >> usage >> by Cloudstack-Usage ? >> >> >> >> >> >> >> On Fri, Oct 30, 2020 at 5:55 AM Nicolas Vazquez < >> nicolas.vazq...@shapeblue.com> wrote: >> >> > Yes, the usage jobs information is stored in the cloud_usage database on >> > the usage_job table. >> > >> > The start_date and end_date columns indicate which usage range was >> parsed >> > and the start_millis and end_millis the jobs execution start and end >> time >> > in milliseconds, which can be converted to date. >> > >> > >> > Regards, >> > >> > Nicolas Vazquez >> > >> > ________________________________ >> > From: Hean Seng <heans...@gmail.com> >> > Sent: Thursday, October 29, 2020 4:58 PM >> > To: users@cloudstack.apache.org <users@cloudstack.apache.org> >> > Subject: Re: listUsageRecords Update frequency >> > >> > Hi NIC >> > >> > Do know where can confirm the last run for usage server ? >> > >> > It seems my counter is not running , my setup is AdvanceZone Security >> > Group. >> > >> > >> > >> > >> > >> > >> > >> > On Thu, Oct 29, 2020 at 4:34 AM Nicolas Vazquez < >> > nicolas.vazq...@shapeblue.com> wrote: >> > >> > > Hi Hean, >> > > >> > > You are looking at the right configuration, if you set >> > > 'usage.stats.job.aggregation.range' = 60 then you only need to restart >> > the >> > > usage server to propagate the changes. >> > > >> > > Also please check the 'usage.stats.job.exec.time' configuration, you >> can >> > > set that exec time to a previous time in the day so after the usage >> > server >> > > restarts, the usage jobs start running, and then the interval remains >> > > hourly (by the previous configuration) >> > > >> > > >> > > Regards, >> > > >> > > Nicolas Vazquez >> > > >> > > ________________________________ >> > > From: Hean Seng <heans...@gmail.com> >> > > Sent: Wednesday, October 28, 2020 3:10 PM >> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> >> > > Subject: listUsageRecords Update frequency >> > > >> > > Hi >> > > >> > > I notice the usage record update once in a day . >> > > >> > > Is that possible to update the Usage data more frequently? May be >> hourly >> > > etc. >> > > >> > > I try to update usage.stats.job.aggregation.range but seem that is >> not I >> > > am looking for, it only affect startdate and enddate of the record, >> but >> > > not update usage record interval. >> > > >> > > >> > > -- >> > > Regards, >> > > Hean Seng >> > > >> > > nicolas.vazq...@shapeblue.com >> > > www.shapeblue.com<http://www.shapeblue.com> >> > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK >> > > @shapeblue >> > > >> > > >> > > >> > > >> > >> > -- >> > Regards, >> > Hean Seng >> > >> > nicolas.vazq...@shapeblue.com >> > www.shapeblue.com<http://www.shapeblue.com> >> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK >> > @shapeblue >> > >> > >> > >> > >> >> -- >> Regards, >> Hean Seng >> >> nicolas.vazq...@shapeblue.com >> www.shapeblue.com<http://www.shapeblue.com> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK >> @shapeblue >> >> >> >> > > -- > Regards, > Hean Seng > -- Regards, Hean Seng
