Hi, Benny,
3 records found.
engine=# select command_id,root_command_id from command_entities where
root_command_id = '2f8b32d8-fd3c-46c9-90e9-4863d63c0530';
command_id | root_command_id
--------------------------------------+--------------------------------------
2f8b32d8-fd3c-46c9-90e9-4863d63c0530 | 2f8b32d8-fd3c-46c9-90e9-4863d63c0530
3c09e1ce-1e49-4d03-82dd-2844fb9dc39f | 2f8b32d8-fd3c-46c9-90e9-4863d63c0530
ed816f9d-e25c-4b58-8c8f-fd0393abda2f | 2f8b32d8-fd3c-46c9-90e9-4863d63c0530
Now this ?
select DeleteJob('2f8b32d8-fd3c-46c9-90e9-4863d63c0530');
> On 8 Aug 2022, at 16:27, Benny Zlotnik <[email protected]> wrote:
>
> you can do:
> select command_id,root_command_id from command_entities where
> root_command_id = '2f8b32d8-fd3c-46c9-90e9-4863d63c0530’;
>
> On Mon, Aug 8, 2022 at 4:19 PM Andrei Verovski <[email protected]> wrote:
>>
>> Hi, Benny,
>>
>>
>> select * from command_entities where root_command_id =
>> 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f’;
>> -> 0 rows
>>
>> select * from command_entities where root_command_id =
>> '2f8b32d8-fd3c-46c9-90e9-4863d63c0530’;
>> -> Huge page of smth, have to page out with “more”, is it possible to
>> truncate it to meaningful value, e.g. just number of rows?
>>
>> I can restore this VM from backup copy, bit since it is unmanageable, I
>> can’t even remove it.
>>
>>
>>> On 8 Aug 2022, at 13:07, Benny Zlotnik <[email protected]> wrote:
>>>
>>> you can look up the relevant command by command_id in the
>>> command_entities table, in your case it would be
>>> 2f8b32d8-fd3c-46c9-90e9-4863d63c0530 and
>>> ed816f9d-e25c-4b58-8c8f-fd0393abda2f, there might be more as the log
>>> is trimmed, so I suggest to look it up with
>>> select *
>>> from command_entities
>>> where root_command_id = '2f8b32d8-fd3c-46c9-90e9-4863d63c0530';
>>>
>>> Then delete the relevant entries and restart ovirt-engine (as they
>>> might still be present in the cache)
>>>
>>> But before doing that, is the command still running? Async commands
>>> like create snapshot are failed automatically after 50 hours and I
>>> believe it has already passed
>>> Also, this manual operation is very intrusive and might have
>>> unexpected consequences so make sure you have backups.
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Aug 8, 2022 at 9:33 AM Andrei Verovski <[email protected]> wrote:
>>>>
>>>> HI,
>>>>
>>>> OK, how to properly remove this lock? Right now VM is locked and is
>>>> unmanageable at all in any way.
>>>> I suppose its with some SQL commands in Postgres.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>> On 5 Aug 2022, at 11:34, Benny Zlotnik <[email protected]> wrote:
>>>>>
>>>>> So based on your logs the lock you are seeing is a memory lock,
>>>>> unlock_entity.sh can't really help with these.
>>>>> Also, the job table is used mainly for presentation so removing an
>>>>> entry from will not help.
>>>>>
>>>>> Do you have the logs from when this snapshot operation started, you
>>>>> can use the correlation id (28353fa0-5e36-4fe8-8609-e74cd1da6d36) to
>>>>> search? Also, do you have the vdsm logs (SPM at the time and the host
>>>>> running the VM), same correlation id can be used for this as well
>>>>>
>>>>> The table that's used to coordinate this is command_entities, so in
>>>>> theory removing the entries with this correlation id can help, but I'd
>>>>> like to see what led to this first
>>>>>
>>>>> On Fri, Aug 5, 2022 at 8:37 AM Andrei Verovski <[email protected]>
>>>>> wrote:
>>>>>>
>>>>>> Hi, Benny,
>>>>>>
>>>>>> I have sent log on your mailbox, its too big to post here on mailing
>>>>>> list.
>>>>>>
>>>>>> Looks like ghost task is still running, anything else need to be removed
>>>>>> from Postgres DB?
>>>>>>
>>>>>> BTW, frozen dead snapshot is in invalid state, is there any way to get
>>>>>> rid of it? I think it's actually exists, but due to invalid state its
>>>>>> not possible to do anything with it.
>>>>>>
>>>>>>
>>>>>> ——
>>>>>>
>>>>>> Log file still shows zombie task:
>>>>>>
>>>>>> 2022-08-04 20:57:30,145+03 INFO
>>>>>> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback]
>>>>>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24)
>>>>>> [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm'
>>>>>> (id: '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command
>>>>>> id: 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f'
>>>>>> type:'CreateLiveSnapshotForVm' to complete
>>>>>> 2022-08-04 20:57:40,176+03 INFO
>>>>>> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback]
>>>>>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-85)
>>>>>> [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm'
>>>>>> (id: '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command
>>>>>> id: 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f'
>>>>>> type:'CreateLiveSnapshotForVm' to complete
>>>>>> 2022-08-04 20:57:50,252+03 INFO
>>>>>> [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback]
>>>>>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-66)
>>>>>> [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm'
>>>>>> (id: '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command
>>>>>> id: 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f'
>>>>>> type:'CreateLiveSnapshotForVm' to complete
>>>>>>
>>>>>>
>>>>>>> On 4 Aug 2022, at 19:06, Benny Zlotnik <[email protected]> wrote:
>>>>>>>
>>>>>>> can you share the logs after restarting ovirt-engine?
>>>>>>>
>>>>>>> On Thu, Aug 4, 2022 at 4:58 PM Andrei Verovski <[email protected]>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>> Creating snapshot of one of the VM vailed, and zombie tasks was killed
>>>>>>>> with:
>>>>>>>>
>>>>>>>> su postgres
>>>>>>>> psql -d engine -U postgres
>>>>>>>> select * from job order by start_time desc;
>>>>>>>>
>>>>>>>> select DeleteJob('UUID_FROZEN_TASK_ID’);
>>>>>>>>
>>>>>>>>
>>>>>>>> However, VM remains in locked state (with lock sign left-below red
>>>>>>>> “DOWN” arrow in status column of web interface.
>>>>>>>>
>>>>>>>> I run:
>>>>>>>> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all
>>>>>>>>
>>>>>>>> then rebooted engine VM, still no luck. Can’t do anything with that VM.
>>>>>>>>
>>>>>>>> Please advise how to fix.
>>>>>>>> Thanks in advance.
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list -- [email protected]
>>>>>>>> To unsubscribe send an email to [email protected]
>>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>>>>>>> oVirt Code of Conduct:
>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>>>>> List Archives:
>>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/P2TVMLHC53JWCLDJNK6UXLZ7ZAUOSYFJ/
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/HDSG2DRD5XPLOWUSEIRZCC6XKJI5M7S4/