> On 8 Aug 2022, at 09:52, Jirka Simon <[email protected]> wrote: > > Hi Benny, Andrei. > > do you have any news? I deleted frozen task and locked snapshot and disk on > Friday and during the weekend everything worked fine. (there were 3 snapshots > and backups from Friday to now.)
Deleting zombie task from Postgresql didn’t help. > > Thank you > > Jirka > > On 8/5/22 11:53, Benny Zlotnik wrote: >> Jirka, I suppose your issue is different than Anderi's no? >> >> In your log I see the command has failed: >> 2022-08-05 10:26:57,741+02 INFO >> [org.ovirt.engine.core.bll.VirtJobCallback] >> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) >> [8ff04bca-b5a2-4ba3-892 >> 5-1037c5a2850e] Command CreateLiveSnapshotForVm id: >> '2f94c337-487e-40f7-b8f8-df69225d5c79': execution was completed, the >> command status is 'FAILED' >> 2022-08-05 10:26:58,742+02 ERROR >> [org.ovirt.engine.core.bll.snapshots.CreateLiveSnapshotForVmCommand] >> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-35 >> ) [8ff04bca-b5a2-4ba3-8925-1037c5a2850e] Ending command >> 'org.ovirt.engine.core.bll.snapshots.CreateLiveSnapshotForVmCommand' >> with failure. >> 2022-08-05 10:26:59,796+02 INFO >> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] >> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-31) >> [8 >> ff04bca-b5a2-4ba3-8925-1037c5a2850e] Command 'CreateSnapshotForVm' id: >> '93f98c9c-2936-4615-b12e-c295b90eace0' child commands >> '[2f94c337-487e-40f7-b8f8-df69225d5c79, 1b3abc1e >> -f0c1-40f7-b52e-f7df52c52601]' executions were completed, status 'FAILED' >> 2022-08-05 10:27:00,826+02 ERROR >> [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] >> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-4) >> [8f >> f04bca-b5a2-4ba3-8925-1037c5a2850e] Ending command >> 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with >> failure. >> 2022-08-05 10:27:00,829+02 ERROR >> [org.ovirt.engine.core.bll.snapshots.CreateSnapshotDiskCommand] >> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-4) >> [8ff >> 04bca-b5a2-4ba3-8925-1037c5a2850e] Ending command >> 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotDiskCommand' with >> failure. >> >> Generally, live snapshot has a default 30 minute timeout to complete >> and it looks like it was reached in your case, I don't see a command >> that's still running >> >> On Fri, Aug 5, 2022 at 12:36 PM Jirka Simon <[email protected]> wrote: >>> Hi Benny , Andrei, >>> >>> >>> here are my logs >>> >>> >>> affected job was " waiting for job '6e73cda8-85d0-471c-b090-4c89719a97f4' >>> on host 'ovirt3.corp.sldev.cz' " >>> >>> thank you >>> >>> >>> Jirka >>> >>> On 8/5/22 11:00, Andrei Verovski wrote: >>> >>> Hi, >>> >>> I have e-mailed vdsm log to Benny. >>> Unfortunately, yesterday I deleted engine.log before restarting engine to >>> have more shorter log. >>> >>> Jirka, can you e-mail engine.log and vdsm logs (SPM at the time and the host >>> running the VM) to Benny? >>> >>> >>> On 5 Aug 2022, at 11:42, Jirka Simon <[email protected]> wrote: >>> >>> Hi Benny, so we have the same problem last 2 days. >>> >>> there are two affected VMs (furtunately not more) and we cant start any >>> snapshot for these VMS. >>> >>> it looks that the snapshot haven't start yet , then to unlock objects >>> (snapshot and disk) shouldn't damage disk (i tried it on both VMS), then i >>> restarted them and VMS run again. I made clone from one of them ant the >>> clone works properly. >>> >>> >>> Jirka >>> >>> On 8/5/22 10:34, Benny Zlotnik wrote: >>> >>> So based on your logs the lock you are seeing is a memory lock, >>> unlock_entity.sh can't really help with these. >>> Also, the job table is used mainly for presentation so removing an >>> entry from will not help. >>> >>> Do you have the logs from when this snapshot operation started, you >>> can use the correlation id (28353fa0-5e36-4fe8-8609-e74cd1da6d36) to >>> search? Also, do you have the vdsm logs (SPM at the time and the host >>> running the VM), same correlation id can be used for this as well >>> >>> The table that's used to coordinate this is command_entities, so in >>> theory removing the entries with this correlation id can help, but I'd >>> like to see what led to this first >>> >>> On Fri, Aug 5, 2022 at 8:37 AM Andrei Verovski <[email protected]> wrote: >>> >>> Hi, Benny, >>> >>> I have sent log on your mailbox, its too big to post here on mailing list. >>> >>> Looks like ghost task is still running, anything else need to be removed >>> from Postgres DB? >>> >>> BTW, frozen dead snapshot is in invalid state, is there any way to get rid >>> of it? I think it's actually exists, but due to invalid state its not >>> possible to do anything with it. >>> >>> >>> —— >>> >>> Log file still shows zombie task: >>> >>> 2022-08-04 20:57:30,145+03 INFO >>> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] >>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) >>> [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm' (id: >>> '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command id: >>> 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f' type:'CreateLiveSnapshotForVm' to >>> complete >>> 2022-08-04 20:57:40,176+03 INFO >>> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] >>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-85) >>> [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm' (id: >>> '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command id: >>> 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f' type:'CreateLiveSnapshotForVm' to >>> complete >>> 2022-08-04 20:57:50,252+03 INFO >>> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] >>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-66) >>> [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm' (id: >>> '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command id: >>> 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f' type:'CreateLiveSnapshotForVm' to >>> complete >>> >>> >>> On 4 Aug 2022, at 19:06, Benny Zlotnik <[email protected]> wrote: >>> >>> can you share the logs after restarting ovirt-engine? >>> >>> On Thu, Aug 4, 2022 at 4:58 PM Andrei Verovski <[email protected]> wrote: >>> >>> Hi, >>> >>> >>> Creating snapshot of one of the VM vailed, and zombie tasks was killed with: >>> >>> su postgres >>> psql -d engine -U postgres >>> select * from job order by start_time desc; >>> >>> select DeleteJob('UUID_FROZEN_TASK_ID’); >>> >>> >>> However, VM remains in locked state (with lock sign left-below red “DOWN” >>> arrow in status column of web interface. >>> >>> I run: >>> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all >>> >>> then rebooted engine VM, still no luck. Can’t do anything with that VM. >>> >>> Please advise how to fix. >>> Thanks in advance. >>> _______________________________________________ >>> Users mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/[email protected]/message/P2TVMLHC53JWCLDJNK6UXLZ7ZAUOSYFJ/ >>> >>> _______________________________________________ >>> Users mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/[email protected]/message/ILYSLIM7UNRU3B3G43IFM3VMSBGZNRDU/ _______________________________________________ Users mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/FQ553VPGHGQTEMNKC55XOTILM4KN75KK/

