sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5031208910
We had the same issue on several stage and prod environments. Her are our
findings and solution to it.
You see that start_date of the usage_job is stuck.
`SELECT * FROM ( SELECT * FROM cloud_usage.usage_job ORDER BY id DESC LIMIT
10 ) sub ORDER BY id ASC;
+-------+------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| id | host | pid | job_type | scheduled
| start_millis | end_millis | exec_time | start_date | end_date
| success | heartbeat |
+-------+------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| 52246 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0
| 1781690455000 | 1784365199999 | 25095596 | 2026-06-17 10:00:55 | 2026-07-18
08:59:59 | 1 | 2026-07-18 16:13:25 |
| 52247 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0
| 1781690455000 | 1784390399999 | 25288347 | 2026-06-17 10:00:55 | 2026-07-18
15:59:59 | 1 | 2026-07-18 23:14:25 |
| 52248 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0
| 1781690455000 | 1784415599999 | 25566185 | 2026-06-17 10:00:55 | 2026-07-18
22:59:59 | 1 | 2026-07-19 06:21:25 |
| 52249 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0
| 1781690455000 | 1784440799999 | 25793951 | 2026-06-17 10:00:55 | 2026-07-19
05:59:59 | 1 | 2026-07-19 13:31:25 |
| 52250 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0
| 1781690455000 | 1784465999999 | 26110859 | 2026-06-17 10:00:55 | 2026-07-19
12:59:59 | 1 | 2026-07-19 20:46:25 |
| 52251 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0
| 1781690455000 | 1784491199999 | 26317753 | 2026-06-17 10:00:55 | 2026-07-19
19:59:59 | 1 | 2026-07-20 04:04:25 |
| 52252 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0
| 1781690455000 | 1784519999999 | 26624851 | 2026-06-17 10:00:55 | 2026-07-20
03:59:59 | 1 | 2026-07-20 11:28:25 |
| 52256 | mgmt-stage1.proio.local/127.0.1.1 | 916185 | 0 | 0
| 0 | 0 | 0 | NULL | NULL
| NULL | 2026-07-20 14:59:10 |
+-------+------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+`
The reason it is stuck is that we had unprocessed usage_events.
`SELECT id, type, resource_id, created, processed FROM usage_event WHERE
processed = 0;
+--------+---------------------+-------------+---------------------+-----------+
| id | type | resource_id | created |
processed |
+--------+---------------------+-------------+---------------------+-----------+
| 481831 | BACKUP.USAGE.METRIC | 40 | 2026-06-17 10:00:55 |
0 |
| 481903 | VOLUME.CREATE | 1897 | 2026-06-17 10:26:15 |
0 |
+--------+---------------------+-------------+---------------------+-----------+`
To get usage back on track and using the correct start_date, we needed to
set processed to 1.
`UPDATE usage_event SET processed = 1 WHERE processed = 0`
The next usage run was working as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
