Hi Nicolas, I try ing to get VM to download something, and I notice the there is nothing update to user_statistics table . Do you know which prcess is suppose respobsible to update the user statistic table. I saw the listvmmetrcs, and the data is is updated correctly , while there is nohting update to user_statistics table.
On Wed, Nov 18, 2020 at 11:27 AM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Hi Hean, > > Apologies, in this case it looks like data is obtained from > user_statistics table as Andrija mentioned in the other thread. > > > Regards, > > Nicolas Vazquez > > ________________________________ > From: Hean Seng <heans...@gmail.com> > Sent: Friday, November 13, 2020 3:24 PM > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > Subject: Re: listUsageRecords Update frequency > > is that any parameter that In network offering etc that do not let it count > the data sent and receive ? > > > nicolas.vazq...@shapeblue.com > www.shapeblue.com > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > @shapeblue > > > > On Sat, Nov 14, 2020 at 2:22 AM Hean Seng <heans...@gmail.com> wrote: > > > Hi Nicolas, > > > > i checked both, the data is tally , it has following type in > usage_event: > > > > | NET.IPASSIGN | > > > > | VOLUME.CREATE | > > > > | VM.CREATE | > > > > | NETWORK.OFFERING.ASSIGN | > > > > | VM.START | > > > > | SG.ASSIGN | > > > > | VMSNAPSHOT.CREATE | > > > > | VMSNAPSHOT.ON_PRIMARY > > > > > > > > I check the ListUsageRecord , it has usage type of 3, 6, 13 m , just > > do not have type of type4 and 5 which for network bytes > > > > > > > > On Tue, Nov 3, 2020 at 10:03 PM Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > >> Hi Hean, > >> > >> Looks like no data is being processed at all, the usage server takes the > >> raw data from usage_event table and parses it into helper tables (such > as > >> usage_network). The data from the usage_event at the cloud_usage > database > >> is copied from the usage_event table on the cloud database. Can you > check > >> the records from those tables and see if there is a discrepancy? Also, > can > >> you verify if the usage db.properties is configured properly? > >> > >> > >> Regards, > >> > >> Nicolas Vazquez > >> > >> ________________________________ > >> From: Hean Seng <heans...@gmail.com> > >> Sent: Saturday, October 31, 2020 5:41 AM > >> To: users@cloudstack.apache.org <users@cloudstack.apache.org> > >> Subject: Re: listUsageRecords Update frequency > >> > >> I think selec the wrong id, but still is return empty set: > >> > >> > >> mysql> select * from usage_network where account_id=59; > >> > >> Empty set (0.00 sec) > >> > >> root@mgmt01 usage]# more usage.log | grep "accountId: 59" > >> > >> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) getting current accounted bytes for... > >> accountId: > >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 > >> > >> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 > in > >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; > >> abr: > >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 > >> > >> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) getting current accounted bytes for... > >> accountId: > >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 > >> > >> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 > in > >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; > >> abr: > >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 > >> > >> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) getting current accounted bytes for... > >> accountId: > >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 > >> > >> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 > in > >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; > >> abr: > >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 > >> > >> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) getting current accounted bytes for... > >> accountId: > >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 > >> > >> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 > in > >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; > >> abr: > >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 > >> > >> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) getting current accounted bytes for... > >> accountId: > >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 > >> > >> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 > in > >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; > >> abr: > >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 > >> > >> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) getting current accounted bytes for... > >> accountId: > >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 > >> > >> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 > in > >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; > >> abr: > >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 > >> > >> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) getting current accounted bytes for... > >> accountId: > >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 > >> > >> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 > in > >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; > >> abr: > >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 > >> > >> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) getting current accounted bytes for... > >> accountId: > >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 > >> > >> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 > in > >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; > >> abr: > >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0 > >> > >> 2020-10-31 02:00:00,854 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) getting current accounted bytes for... > >> accountId: > >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0 > >> > >> 2020-10-31 02 > >> > >> > >> > >> > >> nicolas.vazq...@shapeblue.com > >> www.shapeblue.com<http://www.shapeblue.com> > >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > >> @shapeblue > >> > >> > >> > >> On Sat, Oct 31, 2020 at 2:57 AM Hean Seng <heans...@gmail.com> wrote: > >> > >> > Hi Nicolas > >> > > >> > The cloudstacl_usage is running . confirm from log > >> > > >> > 20-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > >> > (Usage-Job-1:null) (logid:) For 5 it is already more than 29 days, > will > >> > send statement in next cycle > >> > > >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > >> > (Usage-Job-1:null) (logid:) For 10 it is already more than 29 days, > will > >> > send statement in next cycle > >> > > >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > >> > (Usage-Job-1:null) (logid:) For 11 it is already more than 59 days, > will > >> > send statement in next cycle > >> > > >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > >> > (Usage-Job-1:null) (logid:) For 12 it is already more than 29 days, > will > >> > send statement in next cycle > >> > > >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > >> > (Usage-Job-1:null) (logid:) For 13 it is already more than 58 days, > will > >> > send statement in next cycle > >> > > >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl] > >> > (Usage-Job-1:null) (logid:) For 14 it is already more than 29 days, > will > >> > send statement in next cycle > >> > > >> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl] > >> > (Usage-Job-1:null) (logid:) For 23 it is already more than 29 days, > will > >> > send statement in next cycle > >> > > >> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl] > >> > (Usage-Job-1:null) (logid:) For 24 it is already more than 29 days, > will > >> > send statement in next cycle > >> > > >> > 2020-10-30 18:45:11,107 INFO [cloud.usage.UsageManagerImpl] > >> > (Usage-Job-1:null) (logid:) usage job complete > >> > > >> > ^C > >> > > >> > It looks like it did not write to usage table as you say: > >> > > >> > mysql> select * from usage_network where > >> > account_id="a40e9892-3051-41b6-baf2-fabe3af06c92" ; > >> > > >> > Empty set, 1 warning (0.00 sec) > >> > > >> > > >> > > >> > > >> > Where is process to update the usage helper? > >> > > >> > > >> > > >> > > >> > On Sat, Oct 31, 2020 at 1:36 AM Nicolas Vazquez < > >> > nicolas.vazq...@shapeblue.com> wrote: > >> > > >> >> Hi Hean, > >> >> > >> >> The listVirtualMachineMetrics show the "current" usage of the virtual > >> >> machine, and the usage records display the aggregated information > >> (hourly, > >> >> daily) per account. If the records are not updating can you please > >> check > >> >> the usage server logs if there is any error when the usage jobs run? > >> Can > >> >> you also check if data is being added on the helper table for network > >> usage > >> >> (usage_network table)? The information from this table is summarized > >> into > >> >> the resulting usage records when the usage jobs run. > >> >> > >> >> > >> >> Regards, > >> >> > >> >> Nicolas Vazquez > >> >> > >> >> ________________________________ > >> >> From: Hean Seng <heans...@gmail.com> > >> >> Sent: Friday, October 30, 2020 4:18 AM > >> >> To: users@cloudstack.apache.org <users@cloudstack.apache.org> > >> >> Subject: Re: listUsageRecords Update frequency > >> >> > >> >> Hi Nicolas > >> >> > >> >> Thanks for answering. > >> >> > >> >> Some questions puzzle me, what is different between listUsageRecords > >> and > >> >> listVirtualMachineMetrics , in terms of Network Read and Write data. > >> >> (other > >> >> then one is per account basis, and another one is per VM basis) , > the > >> >> data > >> >> in listVirtualMachineMetrics almost instant update . > >> >> > >> >> Is listUsageReocrds (type 4 and type 5) only keep metric with Public > >> IP , > >> >> and not calculated the SharedNetwork ? > >> >> > >> >> I facing issue of AdvanceNetwork withSG, and > listVirtualMachineMetrics > >> >> network sent and receive update almost instant, and listUsageRecords > >> never > >> >> update , I do not know where to check if there is error when updating > >> this > >> >> Account. Or if the account have setting/options to choose if process > >> >> usage > >> >> by Cloudstack-Usage ? > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> On Fri, Oct 30, 2020 at 5:55 AM Nicolas Vazquez < > >> >> nicolas.vazq...@shapeblue.com> wrote: > >> >> > >> >> > Yes, the usage jobs information is stored in the cloud_usage > >> database on > >> >> > the usage_job table. > >> >> > > >> >> > The start_date and end_date columns indicate which usage range was > >> >> parsed > >> >> > and the start_millis and end_millis the jobs execution start and > end > >> >> time > >> >> > in milliseconds, which can be converted to date. > >> >> > > >> >> > > >> >> > Regards, > >> >> > > >> >> > Nicolas Vazquez > >> >> > > >> >> > ________________________________ > >> >> > From: Hean Seng <heans...@gmail.com> > >> >> > Sent: Thursday, October 29, 2020 4:58 PM > >> >> > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > >> >> > Subject: Re: listUsageRecords Update frequency > >> >> > > >> >> > Hi NIC > >> >> > > >> >> > Do know where can confirm the last run for usage server ? > >> >> > > >> >> > It seems my counter is not running , my setup is AdvanceZone > Security > >> >> > Group. > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > On Thu, Oct 29, 2020 at 4:34 AM Nicolas Vazquez < > >> >> > nicolas.vazq...@shapeblue.com> wrote: > >> >> > > >> >> > > Hi Hean, > >> >> > > > >> >> > > You are looking at the right configuration, if you set > >> >> > > 'usage.stats.job.aggregation.range' = 60 then you only need to > >> restart > >> >> > the > >> >> > > usage server to propagate the changes. > >> >> > > > >> >> > > Also please check the 'usage.stats.job.exec.time' configuration, > >> you > >> >> can > >> >> > > set that exec time to a previous time in the day so after the > usage > >> >> > server > >> >> > > restarts, the usage jobs start running, and then the interval > >> remains > >> >> > > hourly (by the previous configuration) > >> >> > > > >> >> > > > >> >> > > Regards, > >> >> > > > >> >> > > Nicolas Vazquez > >> >> > > > >> >> > > ________________________________ > >> >> > > From: Hean Seng <heans...@gmail.com> > >> >> > > Sent: Wednesday, October 28, 2020 3:10 PM > >> >> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > >> >> > > Subject: listUsageRecords Update frequency > >> >> > > > >> >> > > Hi > >> >> > > > >> >> > > I notice the usage record update once in a day . > >> >> > > > >> >> > > Is that possible to update the Usage data more frequently? May be > >> >> hourly > >> >> > > etc. > >> >> > > > >> >> > > I try to update usage.stats.job.aggregation.range but seem that > is > >> >> not I > >> >> > > am looking for, it only affect startdate and enddate of the > >> record, > >> >> but > >> >> > > not update usage record interval. > >> >> > > > >> >> > > > >> >> > > -- > >> >> > > Regards, > >> >> > > Hean Seng > >> >> > > > >> >> > > nicolas.vazq...@shapeblue.com > >> >> > > www.shapeblue.com<http://www.shapeblue.com> > >> >> > > 3 London Bridge Street, 3rd floor, News Building, London SE1 > >> 9SGUK > >> >> > > @shapeblue > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > >> >> > -- > >> >> > Regards, > >> >> > Hean Seng > >> >> > > >> >> > nicolas.vazq...@shapeblue.com > >> >> > www.shapeblue.com<http://www.shapeblue.com> > >> >> > 3 London Bridge Street, 3rd floor, News Building, London SE1 > 9SGUK > >> >> > @shapeblue > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > >> >> -- > >> >> Regards, > >> >> Hean Seng > >> >> > >> >> nicolas.vazq...@shapeblue.com > >> >> www.shapeblue.com<http://www.shapeblue.com> > >> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > >> >> @shapeblue > >> >> > >> >> > >> >> > >> >> > >> > > >> > -- > >> > Regards, > >> > Hean Seng > >> > > >> > >> > >> -- > >> Regards, > >> Hean Seng > >> > > > > > > -- > > Regards, > > Hean Seng > > > > > -- > Regards, > Hean Seng > -- Regards, Hean Seng
