winterhazel commented on PR #10848: URL: https://github.com/apache/cloudstack/pull/10848#issuecomment-2938156862
> Can you update, @winterhazel ? @DaanHoogland I had a look into the code, and found that one situation in which it happens is when the last open job is a "single" job (a job that was created via the `generateUsageRecords` API) that was assigned by the API either to an Usage server process that is not running anymore, or to none. The take over mechanism does not consider these jobs, so they remain open indefinitely, blocking Usage from executing. One way to reproduce: 1. Stop Usage and ensure that there are no open jobs left 2. Call `generateUsageRecords` 3. Start Usage Doing this will result in the Usage job never executing. I think that the ideal approach here would be adapting the take over mechanism to consider "single" jobs as well, but I am also ok with this patch for the moment. It does allow Usage to execute when the blocking job was assigned to a dead process from the same host, which I think will consist of most production scenarios. However, if it was assigned to a host that does not run Usage anymore, or not assigned at all, Usage still won't run. Thus, I will implement my suggestion to also cover these cases when I get some time. > The strange thing that we noticed was multiple open jobs for different hosts and PIDs being pending Do you still have access to these entries to confirm whether it was the same situation that I found? Was there a "single" open job (`job_type=1` and `end_millis=0`)? > @sureshanaparti , I think @winterhazel has a point here. Can you comment on this scenario? On a note, regarding these concurrency problems that I mentioned, I checked that they can already happen on the current version. This patch will not make them worse. The way Usage handles the jobs should be addressed instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
