this is another matter entirely Vivek. I'm guessing for each of those you have jobs with the same start_date/end_date that are successful. only <null> matters. A new job should be inserted when you start a new usageserver. So that you have an empty set for that is the issue. The usage server didn´t insert it's record (and can´t reach the DB) this is where you should look for a remedy.
On Fri, Jul 15, 2022 at 6:05 AM Vivek Kumar <vivek.ku...@indiqus.com.invalid> wrote: > Hello Daan. > > So we have multiple entries where success is zero - > > mysql> select * from usage_job where success is null order by id desc > limit 10; > Empty set (0.00 sec) > > mysql> select * from usage_job where success = 0 order by id desc limit 10; > > +------+------------------------+------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+ > | id | host | pid | job_type | scheduled | > start_millis | end_millis | exec_time | start_date | end_date > | success | heartbeat | > > +------+------------------------+------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+ > | 2003 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1630022400000 | 1630022399999 | 6 | 2021-08-27 00:00:00 | > 2021-08-26 23:59:59 | 0 | 2021-08-27 02:52:15 | > | 2002 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1630022400000 | 1630022399999 | 5 | 2021-08-27 00:00:00 | > 2021-08-26 23:59:59 | 0 | 2021-08-27 02:52:15 | > | 2000 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 | > 2021-08-25 23:59:59 | 0 | 2021-08-26 02:52:26 | > | 1999 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 | > 2021-08-25 23:59:59 | 0 | 2021-08-26 02:52:25 | > | 1998 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1629936000000 | 1629935999999 | 1 | 2021-08-26 00:00:00 | > 2021-08-25 23:59:59 | 0 | 2021-08-26 02:52:25 | > | 1996 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1629849600000 | 1629849599999 | 2 | 2021-08-25 00:00:00 | > 2021-08-24 23:59:59 | 0 | 2021-08-25 02:51:45 | > | 1995 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1629849600000 | 1629849599999 | 1 | 2021-08-25 00:00:00 | > 2021-08-24 23:59:59 | 0 | 2021-08-25 02:51:45 | > | 1993 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1629763200000 | 1629763199999 | 2 | 2021-08-24 00:00:00 | > 2021-08-23 23:59:59 | 0 | 2021-08-24 02:51:46 | > | 1992 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1629763200000 | 1629763199999 | 1 | 2021-08-24 00:00:00 | > 2021-08-23 23:59:59 | 0 | 2021-08-24 02:51:46 | > | 1990 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > 1629676800000 | 1629676799999 | 1 | 2021-08-23 00:00:00 | > 2021-08-22 23:59:59 | 0 | 2021-08-23 02:51:43 | > > +------+------------------------+------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+ > 10 rows in set (0.00 sec) > > —— > > > select count(id) from usage_job where success = 0; > +-----------+ > | count(id) | > +-----------+ > | 999 | > +-----------+ > 1 row in set (0.00 sec) > > > Vivek Kumar > Sr. Manager - Cloud & DevOps > TechOps | Indiqus Technologies > > vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> > www.indiqus.com <https://www.indiqus.com/> > > > > > > On 14-Jul-2022, at 3:03 PM, Daan Hoogland <daan.hoogl...@gmail.com> > wrote: > > > > On Wed, Jul 13, 2022 at 3:51 AM Vivek Kumar <vivek.ku...@indiqus.com.invalid > <mailto:vivek.ku...@indiqus.com.invalid>> > > wrote: > > > >> Hello Daan, > >> > >> Thanks for the response. > >> > >> So we have deployed a replica of our management servers and databases on > >> a complete isolated environment, Tried a work around which you have > >> suggested but PID is not getting updated in the usage_job. Usage.logs > gets > >> stuck on below logs - We have also tried to upgrade on 4.16 on the > replica > >> setup but same situation with 4.16 as well. > >> > >> > >> 2022-07-04 10:23:11,977 INFO [cloud.usage.UsageManagerImpl] (main:null) > >> (logid:) Implementation Version is 4.16.1.0 > >> 2022-07-11 04:26:23,905 DEBUG [utils.crypt.EncryptionSecretKeyChecker] > >> (main:null) (logid:) Encryption Type: file > >> 2022-07-11 04:26:23,927 INFO [db.Transaction.Transaction] (main:null) > >> (logid:) Is Data Base High Availiability enabled? Ans : true > >> 2022-07-11 04:26:23,981 INFO [db.Transaction.Transaction] (main:null) > >> (logid:) The replicas configured for Cloud Data base is/are : 198.18.8.8 > >> 2022-07-11 04:26:24,014 DEBUG [utils.db.DriverLoader] (main:null) > (logid:) > >> Successfully loaded DB driver com.mysql.cj.jdbc.Driver > >> 2022-07-11 04:26:24,063 DEBUG [utils.db.DriverLoader] (main:null) > (logid:) > >> DB driver com.mysql.cj.jdbc.Driver was already loaded. > >> 2022-07-11 04:26:24,070 DEBUG [utils.db.DriverLoader] (main:null) > (logid:) > >> DB driver com.mysql.cj.jdbc.Driver was already loaded. > >> 2022-07-11 04:26:25,172 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Setup Spring Application context > >> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_b5109665 > >> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_42da784f > >> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_bf461acb > >> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_a66084d9 > >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_35dea57c > >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_7ec9190f > >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_98c3bb86 > >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_a3d9d9aa > >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_c06bb621 > >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_82eb64e0 > >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_95b19ecb > >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6458895c > >> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_d545c0a0 > >> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_ef2ae44f > >> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_39357dcf > >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_421ab841 > >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_3d9daaed > >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_a3dbe792 > >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_9f537032 > >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_5b9ccd8c > >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_e67e53bd > >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_6e25b0b3 > >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_91eaad5 > >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_5bdd1db5 > >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_e36e4eb8 > >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_d51f0ffc > >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_ae698650 > >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_a22b67b4 > >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_42e796e4 > >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_84083314 > >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_fc57081c > >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3532324e > >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_bf4cdf9a > >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_3c5a442c > >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_6e84e70b > >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6150198d > >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_a8a8c734 > >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> > org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_58bc9eff > >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ad0cdf5e > >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] > >> (main:null) (logid:) Configuring > >> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_87f6b7d5 > >> 2022-07-11 04:26:25,625 DEBUG [cloud.usage.UsageManagerImpl] (main:null) > >> (logid:) Checking to see if usage.vmops.pid exists. > >> 2022-07-11 04:26:25,625 INFO [cloud.usage.UsageManagerImpl] (main:null) > >> (logid:) Implementation Version is 4.16.1.0 > >> > > > > I see no errors here. > > > >> > >> > >> > >> — Also we have seen in the usage_job - there are multiple jobs which are > >> showing as ‘zero’ in success, So my question is - Is last job has to be > set > >> to success true to all the jobs has to be success true ? > >> > >> > >> mysql> select > >> > id,host,pid,job_type,scheduled,start_millis,end_millis,exec_time,start_date,end_date,success > >> from usage_job order by id desc limit 10; > >> > >> > +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+ > >> | id | host | pid | job_type | scheduled | > >> start_millis | end_millis | exec_time | start_date | end_date > >> | success | > >> > >> > +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+ > >> | 2004 | acs-mgmt-02/198.18.8.6 | 30073 | 0 | 0 | > >> 1630022400000 | 1630108799999 | 194785 | 2021-08-27 00:00:00 | > >> 2021-08-27 23:59:59 | 1 | > >> | 2003 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > >> 1630022400000 | 1630022399999 | 6 | 2021-08-27 00:00:00 | > >> 2021-08-26 23:59:59. | 0 | > >> | 2002 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > >> 1630022400000 | 1630022399999 | 5 | 2021-08-27 00:00:00 | > >> 2021-08-26 23:59:59. | 0 | > >> | 2001 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > >> 1629936000000 | 1630022399999 | 189715 | 2021-08-26 00:00:00 | > >> 2021-08-26 23:59:59. | 1 | > >> | 2000 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > >> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 | > >> 2021-08-25 23:59:59. | 0 | > >> | 1999 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > >> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 | > >> 2021-08-25 23:59:59. | 0 | > >> | 1998 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > >> 1629936000000 | 1629935999999 | 1 | 2021-08-26 00:00:00 | > >> 2021-08-25 23:59:59. | 0 | > >> | 1997 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > >> 1629849600000 | 1629935999999 | 200641 | 2021-08-25 00:00:00 | > >> 2021-08-25 23:59:59. | 1 | > >> | 1996 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > >> 1629849600000 | 1629849599999 | 2 | 2021-08-25 00:00:00 | > >> 2021-08-24 23:59:59. | 0 | > >> | 1995 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | > >> 1629849600000 | 1629849599999 | 1 | 2021-08-25 00:00:00 | > >> 2021-08-24 23:59:59. | 0 | > >> > >> > +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+ > >> > > > > What I would expect, if the usage server is running is to see a reord > where > > none of start_millis, end_millis and success are filled. Your DB shows > > multiple jobs with the same time stamps which is wrong. As if multiple > > processes with the same PID are running, which should be impossible on > any > > *nis-like system. > > A record like > > |2005|acs-mgmt-02/198.18.8.6|30073|| 0| 0| 0| 0| 0| > > [NULL]| [NULL] | [NULL]| <heartbeat-timestamp>| > > is expected. > > can you try > >> select * from usage_job where success is null order by id desc limit 10; > > > > > >> > >> — I have also observed while checking the usage_service - It’s saying > >> something related to mysql jdbc driver. However same driver is working > >> everywhere - > >> > >> > >> # rpm -qa | grep “connector" > >> > >> mysql-connector-python-1.1.6-1.el7.noarch > >> mysql-connector-java-5.1.25-3.el7.noarch > >> > >> — usage service status > >> > >> [root@acs-mgmt-01 ~]# systemctl status cloudstack-usage > >> ● cloudstack-usage.service - CloudStack Usage Server > >> Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service; > >> enabled; vendor preset: disabled) > >> Active: active (running) since Mon 2022-07-11 04:26:20 -03; 4s ago > >> Docs: http://www.cloudstack.org/ > >> Main PID: 1079 (java) > >> Tasks: 24 > >> CGroup: /system.slice/cloudstack-usage.service > >> └─1079 /usr/bin/java -Dpid=1079 -Xms256m -Xmx2048m -cp > >> > /usr/share/cloudstack-usage/*:/usr/share/cloudstack-usage/lib/*:/usr/share/cloudstack-mysql-ha/lib/*:/etc/cl... > >> > >> Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: Defaulting to no-operation > >> (NOP) logger implementation > >> Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: See > >> http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. > >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: An illegal reflective > >> access operation has occurred > >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Illegal reflective access > >> by net.sf.cglib.core.ReflectUtils$1 > >> (file:/usr/share/cloudstack-usage/lib/cglib-nodep-3.3....tionDomain) > >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Please consider reporting > >> this to the maintainers of net.sf.cglib.core.ReflectUtils$1 > >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Use --illegal-access=warn > >> to enable warnings of further illegal reflective access operations > >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: All illegal access > >> operations will be denied in a future release > >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,927 INFO Transaction:1027 > >> - Is Data Base High Availiability enabled? Ans : true > >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,981 INFO Transaction:1070 > >> - The replicas configured for Cloud Data base is/are : 198.18.8.8 > >> Jul 11 04:26:24 acs-mgmt-01 sh[1079]: Loading class > >> `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is > >> `com.mysql.cj.jdbc.Driver'. The driver is ...nnecessary. > >> > >> > >> — I am also concerned about the time zone, is there any specific time > >> zones which needs to be placed in global setting - I have tried putting > UTC > >> and other near Time Zone as well but nothing worked. > >> > >> mysql> select name,value from configuration where name like '%usage%' > and > >> component = 'management-server'; > >> +-----------------------------------+----------------+ > >> | name | value | > >> +-----------------------------------+----------------+ > >> | enable.usage.server | true | > >> | publish.usage.events | true | > >> | usage.aggregation.timezone | America/Cuiaba | > >> | usage.execution.timezone | America/Cuiaba | > >> | usage.sanity.check.interval | 1 | > >> | usage.stats.job.aggregation.range | 60 | > >> | usage.stats.job.exec.time | 00:10 | > >> +-----------------------------------+----------------+ > >> > > > > Time zones need to be defined on your system to be used in mysql. you'll > > have to look into your zones dir (see google for howto) 'System', > > 'Europe/Amsterdam' or '+01:00' should work depending on your system > > configuration. > > > > > > > >> > >> > >> > >> > >> Vivek Kumar > >> Sr. Manager - Cloud & DevOps > >> TechOps | Indiqus Technologies > >> > >> vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> <mailto: > vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com>> > >> www.indiqus.com <http://www.indiqus.com/> <https://www.indiqus.com/ < > https://www.indiqus.com/>> > >> > >> > >> > >> > >>> On 11-Jul-2022, at 12:15 PM, Daan Hoogland <daan.hoogl...@gmail.com > <mailto:daan.hoogl...@gmail.com>> > >> wrote: > >>> > >>> vivek, > >>> *take care with this and make a backup of the cloud_usage.usage_job > >> first* > >>> check in the usage_job table to see if there is a record for a not yet > >>> completed job. > >>> get the pid from that job and check if that is the same as the > currently > >>> running usage server. > >>> If not, stop the usage server, remove the record and restart the usage > >>> server. > >>> > >>> if the pids match you'll need to investigate further. > >>> > >>> On Sun, Jul 10, 2022 at 8:50 AM Vivek Kumar <vivek.ku...@indiqus.com > >> .invalid> > >>> wrote: > >>> > >>>> Hello Folks, > >>>> > >>>> Can someone please help me on this ? > >>>> > >>>> > >>>> Vivek Kumar > >>>> Sr. Manager - Cloud & DevOps > >>>> TechOps | Indiqus Technologies > >>>> > >>>> vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> > >>>> www.indiqus.com <https://www.indiqus.com/> > >>>> > >>>> > >>>> > >>>> > >>>>> On 30-Mar-2022, at 1:01 AM, Vivek Kumar <vivek.ku...@indiqus.com> > >> wrote: > >>>>> > >>>>> Hello Guys, > >>>>> > >>>>> I have a ACS 4.15 deployment which was upgraded from 4.11 in past. We > >>>> have recently observed that cloud_usage is not being generated since > the > >>>> moment we had upgrade. Here is the details and logs from usage.log. > Can > >>>> someone help me out what else do we need to change ot check. We have > >>>> restarted both management and usage service multiple times. > >>>>> > >>>>> 1- Global settings - > >>>>> > >>>>> mysql> select * from configuration where name like '%usage%' and > >>>> component = 'management-server'; > >>>>> > >>>> > >> > +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+ > >>>>> | category | instance | component | name > >>>> | value | description > >>>> | default_value | > >>>> updated | scope | is_dynamic | > >>>>> > >>>> > >> > +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+ > >>>>> | Usage | DEFAUL. | management-server | enable.usage.server > >>>> | true | Flag for enabling usage > >>>> | true | > >>>> NULL | NULL | 0 | > >>>>> | Advanced | DEFAULT | management-server | publish.usage.events > >>>> | true | enable or disable publishing of usage events on the event > >>>> bus | true > >> | > >>>> NULL | NULL | 0 | > >>>>> | Usage | DEFAULT | management-server | usage.aggregation.timezone > >>>> | UTC-3 | The timezone to use for usage stats aggregation > >>>> | GMT | > >>>> NULL | NULL | 0 | > >>>>> | Usage | DEFAULT | management-server | usage.execution.timezone > >>>> | UTC-3 | The timezone to use for usage job execution time > >>>> | NULL | > >>>> NULL | NULL | 0 | > >>>>> | Usage | DEFAULT | management-server | usage.sanity.check.interval > >>>> | NULL | Interval (in days) to check sanity of usage data > >>>> | NULL | > >>>> NULL | NULL | 0 | > >>>>> | Usage | DEFAULT | management-server | > >>>> usage.stats.job.aggregation.range | 60 | The range of time for > >>>> aggregating the user statistics specified in minutes (e.g. 1440 for > >> daily, > >>>> 60 for hourly. | 1440 | NULL | NULL | 0 | > >>>>> | Usage | DEFAULT | management-server | usage.stats.job.exec.time > >>>> | 00:15 | The time at which the usage statistics aggregation job > >> will > >>>> run as an HH24:MM time, e.g. 00:30 to run at 12:30am. | 00:15 | > >>>> NULL | NULL | 0 | > >>>>> > >>>> > >> > +----------+----------+-------------------+-----------------------------------+-------+————————————————————————————————————————————————————————— > >>>>> > >>>>> 2- Rpm installed - > >>>>> > >>>>> > >>>>> > >>>>> [root@acs-mgmt-01 ~]# rpm -qa| grep "cloudstack" > >>>>> cloudstack-management-4.15.1.0-shapeblue0.el7.x86_64 > >>>>> cloudstack-mysql-ha-4.11.2.0-shapeblue.el7.x86_64 > >>>>> cloudstack-agent-4.15.1.0-shapeblue0.el7.x86_64 > >>>>> cloudstack-usage-4.15.1.0-shapeblue0.el7.x86_64 > >>>>> cloudstack-common-4.15.1.0-shapeblue0.el7.x86_64 > >>>>> > >>>>> > >>>>> 3- I am getting below logs in all usage.log - > >>>>> > >>>>> > >>>>> 2022-03-29 13:21:01,102 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f > >>>>> 2022-03-29 13:21:01,102 DEBUG [cloud.usage.UsageManagerImpl] > >> (main:null) > >>>> (logid:) Checking to see if usage.vmops.pid exists. > >>>>> 2022-03-29 13:21:01,103 INFO [cloud.usage.UsageManagerImpl] > >> (main:null) > >>>> (logid:) Implementation Version is 4.15.1.0 > >>>>> 2022-03-29 13:21:45,059 DEBUG > [utils.crypt.EncryptionSecretKeyChecker] > >>>> (main:null) (logid:) Encryption Type: file > >>>>> 2022-03-29 13:21:45,239 INFO [db.Transaction.Transaction] (main:null) > >>>> (logid:) Is Data Base High Availiability enabled? Ans : true > >>>>> 2022-03-29 13:21:45,916 INFO [db.Transaction.Transaction] (main:null) > >>>> (logid:) The slaves configured for Cloud Data base is/are : 198.18.8.8 > >>>>> 2022-03-29 13:21:46,344 DEBUG [utils.db.DriverLoader] (main:null) > >>>> (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver > >>>>> 2022-03-29 13:21:46,719 DEBUG [utils.db.DriverLoader] (main:null) > >>>> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded. > >>>>> 2022-03-29 13:21:46,810 DEBUG [utils.db.DriverLoader] (main:null) > >>>> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded. > >>>>> 2022-03-29 13:21:55,270 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Setup Spring Application context > >>>>> 2022-03-29 13:21:56,947 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_bfbc13ef > >>>>> 2022-03-29 13:21:56,948 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_4d85f5d9 > >>>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c9f19855 > >>>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_b10c0263 > >>>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_408a2306 > >>>>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_89749699 > >>>>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_a36f3910 > >>>>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_ae855734 > >>>>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_cb1733ab > >>>>> 2022-03-29 13:21:56,954 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_8d96e26a > >>>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_a05d1c55 > >>>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6f0406e6 > >>>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_dff13e2a > >>>>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_f9d661d9 > >>>>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_43e0fb59 > >>>>> 2022-03-29 13:21:56,959 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_4cc635cb > >>>>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_48492877 > >>>>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_ae87651c > >>>>> 2022-03-29 13:21:56,961 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_a9feedbc > >>>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_66484b16 > >>>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_f129d147 > >>>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_78d12e3d > >>>>> 2022-03-29 13:21:56,963 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > com.cloud.usage.dao.ExternalPublicIpStatisticsDaoImpl_EnhancerByCloudStack_6e93cdb5 > >>>>> 2022-03-29 13:21:56,964 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_13ca285f > >>>>> 2022-03-29 13:21:56,969 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_66889b3f > >>>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_ee19cc42 > >>>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_dfca8d86 > >>>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_b91503da > >>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_acd6e53e > >>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_4d93146e > >>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_8eb3b09e > >>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_70285a6 > >>>>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3fddafd8 > >>>>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_c9f85d24 > >>>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_4705c1b6 > >>>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_79306495 > >>>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6bfb9717 > >>>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_b35444be > >>>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> > >> > org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_63681c89 > >>>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_b7b85ce8 > >>>>> 2022-03-29 13:21:56,975 INFO [utils.component.ComponentContext] > >>>> (main:null) (logid:) Configuring > >>>> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f > >>>>> 2022-03-29 13:21:56,975 DEBUG [cloud.usage.UsageManagerImpl] > >> (main:null) > >>>> (logid:) Checking to see if usage.vmops.pid exists. > >>>>> 2022-03-29 13:21:56,975 INFO [cloud.usage.UsageManagerImpl] > >> (main:null) > >>>> (logid:) Implementation Version is 4.15.1.0 > >>>>> > >>>>> > >>>>> > >>>>> Regards, > >>>>> Vivek Kumar > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> This message is intended only for the use of the individual or entity > to > >>>> which it is addressed and may contain confidential and/or privileged > >>>> information. If you are not the intended recipient, please delete the > >>>> original message and any copy of it from your computer system. You are > >>>> hereby notified that any dissemination, distribution or copying of > this > >>>> communication is strictly prohibited unless proper authorization has > >> been > >>>> obtained for such action. If you have received this communication in > >>>> error, > >>>> please notify the sender immediately. Although IndiQus attempts to > sweep > >>>> e-mail and attachments for viruses, it does not guarantee that both > are > >>>> virus-free and accepts no liability for any damage sustained as a > result > >>>> of > >>>> viruses. > >>>> > >>> > >>> > >>> -- > >>> Daan > >> > >> > >> -- > >> This message is intended only for the use of the individual or entity to > >> which it is addressed and may contain confidential and/or privileged > >> information. If you are not the intended recipient, please delete the > >> original message and any copy of it from your computer system. You are > >> hereby notified that any dissemination, distribution or copying of this > >> communication is strictly prohibited unless proper authorization has > been > >> obtained for such action. If you have received this communication in > >> error, > >> please notify the sender immediately. Although IndiQus attempts to sweep > >> e-mail and attachments for viruses, it does not guarantee that both are > >> virus-free and accepts no liability for any damage sustained as a result > >> of > >> viruses. > >> > > > > > > -- > > Daan > > > -- > This message is intended only for the use of the individual or entity to > which it is addressed and may contain confidential and/or privileged > information. If you are not the intended recipient, please delete the > original message and any copy of it from your computer system. You are > hereby notified that any dissemination, distribution or copying of this > communication is strictly prohibited unless proper authorization has been > obtained for such action. If you have received this communication in > error, > please notify the sender immediately. Although IndiQus attempts to sweep > e-mail and attachments for viruses, it does not guarantee that both are > virus-free and accepts no liability for any damage sustained as a result > of > viruses. > -- Daan
