vivek, *take care with this and make a backup of the cloud_usage.usage_job first* check in the usage_job table to see if there is a record for a not yet completed job. get the pid from that job and check if that is the same as the currently running usage server. If not, stop the usage server, remove the record and restart the usage server.
if the pids match you'll need to investigate further. On Sun, Jul 10, 2022 at 8:50 AM Vivek Kumar <vivek.ku...@indiqus.com.invalid> wrote: > Hello Folks, > > Can someone please help me on this ? > > > Vivek Kumar > Sr. Manager - Cloud & DevOps > TechOps | Indiqus Technologies > > vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> > www.indiqus.com <https://www.indiqus.com/> > > > > > > On 30-Mar-2022, at 1:01 AM, Vivek Kumar <vivek.ku...@indiqus.com> wrote: > > > > Hello Guys, > > > > I have a ACS 4.15 deployment which was upgraded from 4.11 in past. We > have recently observed that cloud_usage is not being generated since the > moment we had upgrade. Here is the details and logs from usage.log. Can > someone help me out what else do we need to change ot check. We have > restarted both management and usage service multiple times. > > > > 1- Global settings - > > > > mysql> select * from configuration where name like '%usage%' and > component = 'management-server'; > > > +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+ > > | category | instance | component | name > | value | description > | default_value | > updated | scope | is_dynamic | > > > +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+ > > | Usage | DEFAUL. | management-server | enable.usage.server > | true | Flag for enabling usage > | true | > NULL | NULL | 0 | > > | Advanced | DEFAULT | management-server | publish.usage.events > | true | enable or disable publishing of usage events on the event > bus | true | > NULL | NULL | 0 | > > | Usage | DEFAULT | management-server | usage.aggregation.timezone > | UTC-3 | The timezone to use for usage stats aggregation > | GMT | > NULL | NULL | 0 | > > | Usage | DEFAULT | management-server | usage.execution.timezone > | UTC-3 | The timezone to use for usage job execution time > | NULL | > NULL | NULL | 0 | > > | Usage | DEFAULT | management-server | usage.sanity.check.interval > | NULL | Interval (in days) to check sanity of usage data > | NULL | > NULL | NULL | 0 | > > | Usage | DEFAULT | management-server | > usage.stats.job.aggregation.range | 60 | The range of time for > aggregating the user statistics specified in minutes (e.g. 1440 for daily, > 60 for hourly. | 1440 | NULL | NULL | 0 | > > | Usage | DEFAULT | management-server | usage.stats.job.exec.time > | 00:15 | The time at which the usage statistics aggregation job will > run as an HH24:MM time, e.g. 00:30 to run at 12:30am. | 00:15 | > NULL | NULL | 0 | > > > +----------+----------+-------------------+-----------------------------------+-------+————————————————————————————————————————————————————————— > > > > 2- Rpm installed - > > > > > > > > [root@acs-mgmt-01 ~]# rpm -qa| grep "cloudstack" > > cloudstack-management-4.15.1.0-shapeblue0.el7.x86_64 > > cloudstack-mysql-ha-4.11.2.0-shapeblue.el7.x86_64 > > cloudstack-agent-4.15.1.0-shapeblue0.el7.x86_64 > > cloudstack-usage-4.15.1.0-shapeblue0.el7.x86_64 > > cloudstack-common-4.15.1.0-shapeblue0.el7.x86_64 > > > > > > 3- I am getting below logs in all usage.log - > > > > > > 2022-03-29 13:21:01,102 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f > > 2022-03-29 13:21:01,102 DEBUG [cloud.usage.UsageManagerImpl] (main:null) > (logid:) Checking to see if usage.vmops.pid exists. > > 2022-03-29 13:21:01,103 INFO [cloud.usage.UsageManagerImpl] (main:null) > (logid:) Implementation Version is 4.15.1.0 > > 2022-03-29 13:21:45,059 DEBUG [utils.crypt.EncryptionSecretKeyChecker] > (main:null) (logid:) Encryption Type: file > > 2022-03-29 13:21:45,239 INFO [db.Transaction.Transaction] (main:null) > (logid:) Is Data Base High Availiability enabled? Ans : true > > 2022-03-29 13:21:45,916 INFO [db.Transaction.Transaction] (main:null) > (logid:) The slaves configured for Cloud Data base is/are : 198.18.8.8 > > 2022-03-29 13:21:46,344 DEBUG [utils.db.DriverLoader] (main:null) > (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver > > 2022-03-29 13:21:46,719 DEBUG [utils.db.DriverLoader] (main:null) > (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded. > > 2022-03-29 13:21:46,810 DEBUG [utils.db.DriverLoader] (main:null) > (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded. > > 2022-03-29 13:21:55,270 INFO [utils.component.ComponentContext] > (main:null) (logid:) Setup Spring Application context > > 2022-03-29 13:21:56,947 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_bfbc13ef > > 2022-03-29 13:21:56,948 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_4d85f5d9 > > 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c9f19855 > > 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_b10c0263 > > 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_408a2306 > > 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_89749699 > > 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_a36f3910 > > 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_ae855734 > > 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_cb1733ab > > 2022-03-29 13:21:56,954 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_8d96e26a > > 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_a05d1c55 > > 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6f0406e6 > > 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_dff13e2a > > 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_f9d661d9 > > 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_43e0fb59 > > 2022-03-29 13:21:56,959 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_4cc635cb > > 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_48492877 > > 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_ae87651c > > 2022-03-29 13:21:56,961 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_a9feedbc > > 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_66484b16 > > 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_f129d147 > > 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_78d12e3d > > 2022-03-29 13:21:56,963 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.ExternalPublicIpStatisticsDaoImpl_EnhancerByCloudStack_6e93cdb5 > > 2022-03-29 13:21:56,964 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_13ca285f > > 2022-03-29 13:21:56,969 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_66889b3f > > 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_ee19cc42 > > 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_dfca8d86 > > 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_b91503da > > 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_acd6e53e > > 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_4d93146e > > 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_8eb3b09e > > 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_70285a6 > > 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3fddafd8 > > 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_c9f85d24 > > 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_4705c1b6 > > 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_79306495 > > 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6bfb9717 > > 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_b35444be > > 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_63681c89 > > 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_b7b85ce8 > > 2022-03-29 13:21:56,975 INFO [utils.component.ComponentContext] > (main:null) (logid:) Configuring > com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f > > 2022-03-29 13:21:56,975 DEBUG [cloud.usage.UsageManagerImpl] (main:null) > (logid:) Checking to see if usage.vmops.pid exists. > > 2022-03-29 13:21:56,975 INFO [cloud.usage.UsageManagerImpl] (main:null) > (logid:) Implementation Version is 4.15.1.0 > > > > > > > > Regards, > > Vivek Kumar > > > > > > > > > -- > This message is intended only for the use of the individual or entity to > which it is addressed and may contain confidential and/or privileged > information. If you are not the intended recipient, please delete the > original message and any copy of it from your computer system. You are > hereby notified that any dissemination, distribution or copying of this > communication is strictly prohibited unless proper authorization has been > obtained for such action. If you have received this communication in > error, > please notify the sender immediately. Although IndiQus attempts to sweep > e-mail and attachments for viruses, it does not guarantee that both are > virus-free and accepts no liability for any damage sustained as a result > of > viruses. > -- Daan
