Re: [ovirt-users] how to clean stuck task
oVirt upgrade to 4.1.7 will probably cleanup this stale task. However, if you want to do it before upgrading, run this command: PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all -u engine Note that unlock_entity.sh has many flags and this is just an example (should clean all stale tasks). You can find the PGPASSWORD value in the /etc/ovirt-engine/engine.conf.d/10-setup-database.conf file. As of 4.2 you won't need to supply credentials anymore [1]. Regards, Nicolás [1]: https://gerrit.ovirt.org/82615 El 2017-11-10 14:16, Gianluca Cecchi escribió: Hello, I have a task that seems stuck in webadmin gui, in the sens tha I have "Tasks(1)" listed The task is Restoring VM Snapshot Active VM before the preview of VM snaptest and the VM is powered down. Screenshot of expanded steps of task, that actually seem all completed, is here: https://drive.google.com/file/d/1bfl_gEfVotIrxGC9TDzPHPCeRub41mUa/view?usp=sharing [1] Any hint on what to do to clean things? I'm on oVirt 4.1.6.2-1.el7.centos and I would like to clean before upgrading to 4.1.7. Thanks Gianluca Links: -- [1] https://drive.google.com/file/d/1bfl_gEfVotIrxGC9TDzPHPCeRub41mUa/view?usp=sharing ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
On Fri, Nov 10, 2017 at 3:34 PM, wrote: > oVirt upgrade to 4.1.7 will probably cleanup this stale task. However, if > you want to do it before upgrading, run this command: > >PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh > -t all -u engine > > Note that unlock_entity.sh has many flags and this is just an example > (should clean all stale tasks). > > You can find the PGPASSWORD value in the > /etc/ovirt-engine/engine.conf.d/10-setup-database.conf > file. As of 4.2 you won't need to supply credentials anymore [1]. > > Regards, > > Nicolás > It seems it didn't work as expected. I got this at command line output " select fn_db_unlock_all(); INSERT 0 1 unlock all completed successfully. " But the task remains in webadmin gui and I got an alert message in alert section, of this type " /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh : System user root run manually unlock_entity script on entity [type,id] [all,] with db user engine " Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
El 2017-11-10 14:41, Gianluca Cecchi escribió: On Fri, Nov 10, 2017 at 3:34 PM, wrote: oVirt upgrade to 4.1.7 will probably cleanup this stale task. However, if you want to do it before upgrading, run this command: PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all -u engine Note that unlock_entity.sh has many flags and this is just an example (should clean all stale tasks). You can find the PGPASSWORD value in the /etc/ovirt-engine/engine.conf.d/10-setup-database.conf file. As of 4.2 you won't need to supply credentials anymore [1]. Regards, Nicolás It seems it didn't work as expected. I got this at command line output " select fn_db_unlock_all(); INSERT 0 1 unlock all completed successfully. " This is expected. But the task remains in webadmin gui and I got an alert message in alert section, of this type " /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh : System user root run manually unlock_entity script on entity [type,id] [all,] with db user engine " I've seen this behavior too. IIRC the stale cleaning was not instant, it took some time to be applied. Regards. Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
You could also go database diving. I had an issue where I tried to import a VM from my export domain and it just got hung. I tried running the unlock_entity script but it just kept failing. It sat there for months stuck, and found http://lists.ovirt.org/pipermail/users/2015-April/032346.html Of course deleting something from your database is quite permanent. I would wait and upgrade to 4.1.7, but something like the below should work. But probably not recommended Drop into postgres psql -d engine -U postgres List your tasks and grab the job_id select * from job order by start_time desc; select DeleteJob('8424f7a9-2a4c-4567-b528-45bbc1c2534f'); Where the string here is the job ID On Fri, Nov 10, 2017 at 9:48 AM, wrote: > El 2017-11-10 14:41, Gianluca Cecchi escribió: > >> On Fri, Nov 10, 2017 at 3:34 PM, wrote: >> >> oVirt upgrade to 4.1.7 will probably cleanup this stale task. >>> However, if you want to do it before upgrading, run this command: >>> >>>PGPASSWORD=... >>> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all -u >>> engine >>> >>> Note that unlock_entity.sh has many flags and this is just an >>> example (should clean all stale tasks). >>> >>> You can find the PGPASSWORD value in the >>> /etc/ovirt-engine/engine.conf.d/10-setup-database.conf file. As of >>> 4.2 you won't need to supply credentials anymore [1]. >>> >>> Regards, >>> >>> Nicolás >>> >> >> It seems it didn't work as expected. >> I got this at command line output >> >> " >> >> select fn_db_unlock_all(); >> >> >> INSERT 0 1 >> unlock all completed successfully. >> " >> >> > This is expected. > > But the task remains in webadmin gui and I got an alert message in >> alert section, of this type >> " >> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh : System user >> root run manually unlock_entity script on entity [type,id] [all,] with >> db user engine >> " >> >> > I've seen this behavior too. IIRC the stale cleaning was not instant, it > took some time to be applied. > > Regards. > > Gianluca >> > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
On Fri, Nov 10, 2017 at 3:48 PM, wrote: > >> > I've seen this behavior too. IIRC the stale cleaning was not instant, it > took some time to be applied. > > Regards. > > Gianluca >> > Confirmed. Quite soon after the command I saw that the status of the "Current" snapshot line changed from Locked (it was so since 8/11) to OK, but the task remained at least for half an hour. Now, after about one hour and a half I connected again to the web admin gui and I see 0 Tasks, so the problem has been resolved. Thanks again, Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
Hello, it seems the same lock (with the same date) has come again in the list of tasks. Now I have two things in "supposed running" tasks: - the "Restoring VM Snapshot Active VM before the preview of VM snaptest" of 8/11 - the "Creating VM Snapshot 13-11-2017-16-30-Danilo for VM apex" (now running since half an hour for a vm that has only one 90Gb disk on SAN FC storage domain) It was not me to run the "create snapshot" command today, so I'm not sure if it was the snapshot creation job responsible of letting come out again the former hung task... or if it was already there before... Any further hint? How can I manually verify the status of the snapshot creation task? Gianluca On Fri, Nov 10, 2017 at 5:01 PM, Gianluca Cecchi wrote: > On Fri, Nov 10, 2017 at 3:48 PM, wrote: > >> >>> >> I've seen this behavior too. IIRC the stale cleaning was not instant, it >> took some time to be applied. >> >> Regards. >> >> Gianluca >>> >> > Confirmed. > Quite soon after the command I saw that the status of the "Current" > snapshot line changed from Locked (it was so since 8/11) to OK, but the > task remained at least for half an hour. > Now, after about one hour and a half I connected again to the web admin > gui and I see 0 Tasks, so the problem has been resolved. > > Thanks again, > Gianluca > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
Is the VM showing as locked? You could try to manually unlocking it. I did this in a test environment a while back with some success after a vm was stuck in a "locked" state. Of course the gui and engine should handle most of this for you, manually mucking around the internal DB can cause some pretty serious issues if you are not careful... sudo su postgres psql -d engine -U postgres SELECT vm_guid, vm_name FROM vm_static WHERE vm_name='*VM_Name_Here*'; This should return a string such as: "0ec20854-e1ca-4e49-be87-a6cd36d40c18" Reset the lock: update vm_dynamic SET status=0 where vm_guid=' 0ec20854-e1ca-4e49-be87-a6cd36d40c18'; On Mon, Nov 13, 2017 at 11:01 AM, Gianluca Cecchi wrote: > Hello, > > it seems the same lock (with the same date) has come again in the list of > tasks. > Now I have two things in "supposed running" tasks: > - the "Restoring VM Snapshot Active VM before the preview of VM snaptest" > of 8/11 > - the "Creating VM Snapshot 13-11-2017-16-30-Danilo for VM apex" (now > running since half an hour for a vm that has only one 90Gb disk on SAN FC > storage domain) > > It was not me to run the "create snapshot" command today, so I'm not sure > if it was the snapshot creation job responsible of letting come out again > the former hung task... or if it was already there before... > Any further hint? > How can I manually verify the status of the snapshot creation task? > > Gianluca > > > On Fri, Nov 10, 2017 at 5:01 PM, Gianluca Cecchi < > gianluca.cec...@gmail.com> wrote: > >> On Fri, Nov 10, 2017 at 3:48 PM, wrote: >> >>> >>> I've seen this behavior too. IIRC the stale cleaning was not instant, it >>> took some time to be applied. >>> >>> Regards. >>> >>> Gianluca >>> >> Confirmed. >> Quite soon after the command I saw that the status of the "Current" >> snapshot line changed from Locked (it was so since 8/11) to OK, but the >> task remained at least for half an hour. >> Now, after about one hour and a half I connected again to the web admin >> gui and I see 0 Tasks, so the problem has been resolved. >> >> Thanks again, >> Gianluca >> >> > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
On Mon, Nov 13, 2017 at 5:28 PM, Wesley Stewart wrote: > Is the VM showing as locked? You could try to manually unlocking it. I > did this in a test environment a while back with some success after a vm > was stuck in a "locked" state. Of course the gui and engine should handle > most of this for you, manually mucking around the internal DB can cause > some pretty serious issues if you are not careful... > > sudo su postgres > > psql -d engine -U postgres > > SELECT vm_guid, vm_name FROM vm_static WHERE vm_name='*VM_Name_Here*'; > This should return a string such as: "0ec20854-e1ca-4e49-be87- > a6cd36d40c18" > > Reset the lock: > update vm_dynamic SET status=0 where vm_guid='0ec20854-e1ca-4e49- > be87-a6cd36d40c18'; > > > > I'd prefer to avoid direct manage of the rdbms... As it happened on Friday, after running again the command PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all -u engine now the gui after some time shows no active task but the vm still shows as locked and in the event pane I only have the message Snapshot '13-11-2017-16-30-Danilo' creation for VM 'apex' was initiated by user1@internal-authz. but no message about its correct completion This is my disks' subtab situation for the vm, where it seems all ok: https://drive.google.com/file/d/1tF3VdA8gzMVvtn2x4HTKOmltRmUR7cTG/view?usp=sharing And this the snapshots' subtab where it seems all ok: https://drive.google.com/file/d/1qbEhdS6eOZL-f8y5xCnuc_4_HsYM4mAa/view?usp=sharing But the VM has the lock symbol in side of itself. Any guidance appreciated, both to solve the problem and to understand if the snapshot is consistent, as a user should to changes to the VM with a chance to have to come back Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
On Mon, Nov 13, 2017 at 7:01 PM, Gianluca Cecchi wrote: > > > > I'd prefer to avoid direct manage of the rdbms... > > As it happened on Friday, after running again the command > > PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t > all -u engine > > now the gui after some time shows no active task > > but the vm still shows as locked and in the event pane I only have the > message > Snapshot '13-11-2017-16-30-Danilo' creation for VM 'apex' was initiated by > user1@internal-authz. > but no message about its correct completion > > This is my disks' subtab situation for the vm, where it seems all ok: > https://drive.google.com/file/d/1tF3VdA8gzMVvtn2x4HTKOmltRmUR7 > cTG/view?usp=sharing > > And this the snapshots' subtab where it seems all ok: > https://drive.google.com/file/d/1qbEhdS6eOZL-f8y5xCnuc_4_ > HsYM4mAa/view?usp=sharing > > But the VM has the lock symbol in side of itself. > Any guidance appreciated, both to solve the problem and to understand if > the snapshot is consistent, as a user should to changes to the VM with a > chance to have to come back > > Gianluca > I have just verified that if I run a new task (in my case has been the removal of the snaptest VM), suddenly I see from 0 to 3 tasks, with the deletion task completing with success, while the 2 stale tasks appearing again in the task pane where they disappeared see here: https://drive.google.com/file/d/1xcbJsI02iSEAHYEiFYkRWvz9SW_ElsIY/view?usp=sharing ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
For cleaning these "Tasks" entries (which are actually Jobs in the DB), you will need to use taskcleaner.sh. The unlock_entity.sh is for removing locks on entities. Before running the task cleaner, you need to be sure you actually don't have any tasks running in the SPM. Regarding the snapshot creation on 'apex", can you provide the logs around the time of the operation? On Mon, Nov 13, 2017 at 8:06 PM, Gianluca Cecchi wrote: > On Mon, Nov 13, 2017 at 7:01 PM, Gianluca Cecchi < > gianluca.cec...@gmail.com> wrote: > >> >> >> >> I'd prefer to avoid direct manage of the rdbms... >> >> As it happened on Friday, after running again the command >> >> PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t >> all -u engine >> >> now the gui after some time shows no active task >> >> but the vm still shows as locked and in the event pane I only have the >> message >> Snapshot '13-11-2017-16-30-Danilo' creation for VM 'apex' was initiated >> by user1@internal-authz. >> but no message about its correct completion >> >> This is my disks' subtab situation for the vm, where it seems all ok: >> https://drive.google.com/file/d/1tF3VdA8gzMVvtn2x4HTKOmltRmU >> R7cTG/view?usp=sharing >> >> And this the snapshots' subtab where it seems all ok: >> https://drive.google.com/file/d/1qbEhdS6eOZL-f8y5xCnuc_4_HsY >> M4mAa/view?usp=sharing >> >> But the VM has the lock symbol in side of itself. >> Any guidance appreciated, both to solve the problem and to understand if >> the snapshot is consistent, as a user should to changes to the VM with a >> chance to have to come back >> >> Gianluca >> > > I have just verified that if I run a new task (in my case has been the > removal of the snaptest VM), suddenly I see from 0 to 3 tasks, with the > deletion task completing with success, while the 2 stale tasks appearing > again in the task pane where they disappeared > see here: > https://drive.google.com/file/d/1xcbJsI02iSEAHYEiFYkRWvz9SW_ > ElsIY/view?usp=sharing > > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
On Mon, Nov 13, 2017 at 8:53 PM, Fred Rolland wrote: > For cleaning these "Tasks" entries (which are actually Jobs in the DB), > you will need to use taskcleaner.sh. > The unlock_entity.sh is for removing locks on entities. > ok. Thanks for clarifying the concept. But from the inline help of taskcleaner.sh it is not so clear to me which option makes it display vs remove... Perhaps without options it only display? or with "-T" it first displays asking if I want to remove "all comands that have running tasks"? How can I only display? > Before running the task cleaner, you need to be sure you actually don't > have any tasks running in the SPM. > How can I check this? Is still valid this command I found on a previous (around 2014) thread and executed on the SPM (as shown in web admin portal) host right now? [root@ov300 vdsm]# vdsClient -s 0 getAllTasksStatuses {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} [root@ov300 vdsm]# Does it mean no task? > Regarding the snapshot creation on 'apex", can you provide the logs around > the time of the operation? > engine.log is here: https://drive.google.com/file/d/1nD0p90ZvwphrEafgKZkCCquNu_ vVY8p8/view?usp=sharing vdsm log of the SPM host in .xz format here: https://drive.google.com/file/d/1Zzk04W6400wPh1sM8O4WyFDAGQaqa UZK/view?usp=sharing Thanks, Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] how to clean stuck task
In the mean time, as I had to give an answer for the snapshotted VM, I decided to follow one of the suggestions to run engine-setup and so also to pass my engine from 4.1.6 to 4.1.7. And indeed the 2 stale tasks have been cleaned. The lock symbol has gone away from the apex VM too. Probably the steps solving the problems were these during engine-setup: [ INFO ] Cleaning async tasks and compensations [ INFO ] Unlocking existing entities Does this mean that in general I can also run engine-setup without upgrading at all? Is the clean part run also in that case or only during effective upgrades? I initiated a clone of the taken snapshot on apex VM and it seems to go correctly and in task pane I see only that task and no more. In SPM now I have indeed [root@ov300 ~]# vdsClient -s 0 getAllTasksStatuses {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {'20fa401f-b6f8-43f5-b0fd-6767d46e2335': {'message': 'running job 1 of 1', 'code': 0, 'taskID': '20fa401f-b6f8-43f5-b0fd-6767d46e2335', 'taskResult': '', 'taskState': 'running'}}} [root@ov300 ~]# It should take about half an hour to complete and I will see. Anyway in my opinion it would be nice to have some more in deep documentation about how to run taskcleaner.sh or simply officially say to leave it to developers if this is the intended case (or to Red Hat support in case of RHEV usage) Cheers, Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users