Well, issue was solved by installing latest async update 4.2.5...
Probably helped this part of upgrade process...

...
[ INFO  ] Cleaning async tasks and compensations
[ INFO  ] Unlocking existing entities
...


Cheers,

Jiri


On 08/10/2018 11:03 AM, Jiří Sléžka wrote:
> On 08/10/2018 10:52 AM, Raz Tamir wrote:
>>
>>
>> On Fri, Aug 10, 2018 at 11:30 AM, Jiří Sléžka <jiri.sle...@slu.cz
>> <mailto:jiri.sle...@slu.cz>> wrote:
>>
>>     Hi,
>>
>>     On 08/09/2018 11:25 PM, Raz Tamir wrote:
>>     > You can try using vdsm-client to check if there are running tasks on 
>> the
>>     > host using this command:
>>     > 
>>     >     vdsm-client Host getAllTasksInfo
>>
>>     thanks for support, I tried this on affected host
>>
>>     [root@blade01 ~]# vdsm-client Host getAllTasksInfo
>>     vdsm-client: Command Host.getAllTasksInfo with args {} failed:
>>     (code=654, message=Not SPM: ())
>>
>>     so I moved into SPM host (blade03 at this time)
>>
>>     [root@blade03 ~]# vdsm-client Host getAllTasksInfo
>>     {}
>>
>>     I tried also
>>
>>     [root@blade01 ~]# vdsm-client Host getAllTasks
>>     {}
>>     [root@blade01 ~]# vdsm-client Host getJobs
>>     {}
>>
>>     [root@blade03 ~]# vdsm-client Host getAllTasks
>>     {}
>>     [root@blade03 ~]# vdsm-client Host getJobs
>>     {}
>>
>>     It looks like there are no tasks/jobs running on hosts. It looks like
>>     something is stucked in engine db.
>>
>>     At the beginning of this issue, there were some race condition where
>>     blade01 had two running tasks/jobs which cannot be finished (network
>>     configuration and activating of host). I deleted this jobs manually in
>>     db via
>>
>>     SELECT * FROM job ORDER BY start_time DESC;
>>     SELECT DeleteJob('...job_id...');
>>     SELECT DeleteJob('...job_id...');
>>
>>     Maybe it was not wise... but what can I do now to solve this? :-)
>>
>> Yes it can be bad to mess with the DB.
>> My best suugestion at the momet is to try and re-add this host.
> 
> I would like to but I cannot even remove this host (Cannot remove Host.
> Related operation is currently in progress. Please try again later.) :-(
> But maybe I can try to add it as brand new one?
> 
> But I am curious what checks do manager and why we cannot see the
> tasks/jobs/locks/anything in db...
> 
> Cheers, Jiri
> 
> 
> 
>>  
>>
>>
>>     Cheers, Jiri
>>
>>
>>     > 
>>     > If there are running tasks, you can stop them:
>>     > 
>>     >     vdsm-client Task stop taskID=xxx-yyy
>>     > 
>>     > If there are finished tasks, you can clear them:
>>     > 
>>     >     vdsm-client Task clear taskID=xxx-yyy
>>     > 
>>     > 
>>     > On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.sle...@slu.cz 
>> <mailto:jiri.sle...@slu.cz>
>>     > <mailto:jiri.sle...@slu.cz <mailto:jiri.sle...@slu.cz>>> wrote:
>>     >
>>     >     Hello,
>>     >
>>     >     still no luck with solving this issue.
>>     >
>>     >     I cannot even remove this host.
>>     >
>>     >     engine log is now spammed with this messages
>>     >
>>     >     2018-08-09 15:03:08,410+02 INFO
>>     >     [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>>     >     (EE-ManagedThreadFactory-engine-Thread-1057)
>>     >     [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock
>>     and wait
>>     >     lock
>>     >   
>>      
>> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
>>     >     sharedLocks=''}'
>>     >     2018-08-09 15:03:08,446+02 INFO
>>     >     [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>>     >     (EE-ManagedThreadFactory-engine-Thread-1057)
>>     >     [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock
>>     and wait
>>     >     lock
>>     >   
>>      
>> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
>>     >     sharedLocks=''}'
>>     >     2018-08-09 15:03:08,451+02 INFO
>>     >     [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>>     >     (EE-ManagedThreadFactory-engine-Thread-1057)
>>     >     [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock
>>     and wait
>>     >     lock
>>     >   
>>      
>> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
>>     >     sharedLocks=''}'
>>     >     2018-08-09 15:03:08,486+02 INFO
>>     >     [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>>     >     (EE-ManagedThreadFactory-engine-Thread-1057)
>>     >     [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock
>>     and wait
>>     >     lock
>>     >   
>>      
>> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
>>     >     sharedLocks=''}'
>>     >
>>     >     btw. oVirt 4.2.5.2-1.el7/CentOS7
>>     >
>>     >     Any tips how to resolve this and at least remove this host
>>     from ovirt?
>>     >
>>     >     Cheers, Jiri
>>     >
>>     >
>>     >     On 08/06/2018 12:52 PM, Gobinda Das wrote:
>>     >     > Can you please post vdsm log which is inside
>>     /var/log/vdsm/vdsm.log ?
>>     >     >
>>     >     > On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka
>>     <jiri.sle...@slu.cz <mailto:jiri.sle...@slu.cz>
>>     >     <mailto:jiri.sle...@slu.cz <mailto:jiri.sle...@slu.cz>>
>>     >     > <mailto:jiri.sle...@slu.cz <mailto:jiri.sle...@slu.cz>
>>     <mailto:jiri.sle...@slu.cz <mailto:jiri.sle...@slu.cz>>>> wrote:
>>     >     >
>>     >     >     Hi,
>>     >     >
>>     >     >     no one can help?
>>     >     >
>>     >     >     I still cannot activate this host - error is "Cannot
>>     activate
>>     >     Host.
>>     >     >     Related operation is currently in progress. Please try again
>>     >     later."
>>     >     >
>>     >     >     I believe relevant log entrieas are
>>     >     >
>>     >     >     2018-08-06 12:15:50,398+02 INFO
>>     >     >     [org.ovirt.engine.core.bll.ActivateVdsCommand] (default
>>     task-426)
>>     >     >     [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire
>>     Lock
>>     >     to object
>>     >     >   
>>     >   
>>       
>> 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
>>     >     >     sharedLocks=''}'
>>     >     >     2018-08-06 12:15:50,398+02 WARN
>>     >     >     [org.ovirt.engine.core.bll.ActivateVdsCommand] (default
>>     task-426)
>>     >     >     [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action
>>     >     >     'ActivateVds' failed for user ***my_username***. Reasons:
>>     >     >   
>>     >   
>>       VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
>>     >     >
>>     >     >
>>     >     >     when I tried to "reinstall host" I got entry in Events
>>     "Failed
>>     >     to update
>>     >     >     Host blade01" and correlation id. Relevant log entries are
>>     >     >
>>     >     >
>>     >     >     cat /var/log/ovirt-engine/engine.log | grep
>>     >     >     d2b332da-9050-419c-8188-8fdda5e9d807
>>     >     >
>>     >     >     2018-08-06 12:08:57,451+02 INFO
>>     >     >     [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand]
>>     (default
>>     >     >     task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running
>>     command:
>>     >     >     InstallVdsCommand internal: false. Entities affected :  ID:
>>     >     >     786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group
>>     >     >     EDIT_HOST_CONFIGURATION with role type ADMIN
>>     >     >     2018-08-06 12:08:57,476+02 INFO
>>     >     >   
>>      [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
>>     >     >     (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
>>     >     Failed to
>>     >     >     Acquire Lock to object
>>     >     >   
>>     >   
>>       
>> 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
>>     >     >     sharedLocks=''}'
>>     >     >     2018-08-06 12:08:57,476+02 WARN
>>     >     >   
>>      [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
>>     >     >     (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
>>     >     Validation of
>>     >     >     action 'InstallVdsInternal' failed for user
>>     ***my_username***.
>>     >     Reasons:
>>     >     >     ACTION_TYPE_FAILED_OBJECT_LOCKED
>>     >     >     2018-08-06 12:08:57,479+02 ERROR
>>     >     >     [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand]
>>     (default
>>     >     >     task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
>>     >     Installation/upgrade of
>>     >     >     Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01'
>>     failed:
>>     >     Cannot
>>     >     >     ${action} ${type}. Related operation is currently in
>>     progress.
>>     >     Please
>>     >     >     try again later.
>>     >     >     2018-08-06 12:08:57,482+02 INFO
>>     >     >     [org.ovirt.engine.core.bll.CommandCompensator] (default
>>     task-425)
>>     >     >     [d2b332da-9050-419c-8188-8fdda5e9d807] Command
>>     >     >     [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating
>>     >     >     DELETED_OR_UPDATED_ENTITY of
>>     >     >     org.ovirt.engine.core.common.businessentities.VdsStatic;
>>     snapshot:
>>     >     >     id=786646cd-c9ef-49a8-8aea-56e858dcf202.
>>     >     >     2018-08-06 12:08:57,486+02 ERROR
>>     >     >     [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand]
>>     (default
>>     >     >     task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction
>>     >     rolled-back
>>     >     >     for command
>>     >     'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'.
>>     >     >     2018-08-06 12:08:57,491+02 ERROR
>>     >     >   
>>     >   
>>       [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>     >     >     (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
>>     >     EVENT_ID:
>>     >     >     USER_FAILED_UPDATE_VDS(105), Failed to update Host
>>     blade01 (User:
>>     >     >     ***my_username***).
>>     >     >
>>     >     >     How can I unlock this entries?
>>     >     >
>>     >     >     Thanks in advance,
>>     >     >
>>     >     >     Jiri Slezka
>>     >     >
>>     >     >
>>     >     >
>>     >     >     On 08/03/2018 02:07 PM, Jiří Sléžka wrote:
>>     >     >     > Hi,
>>     >     >     >
>>     >     >     > I have one interesting issue. I was asked by colleague to
>>     >     repair his
>>     >     >     > cluster which is part of our oVirt installation. He
>>     tried to
>>     >     >     upgrade his
>>     >     >     > host to gain 4.2 compatibility level but it fails in some
>>     >     cases. Never
>>     >     >     > mind, I checked it, repaired, installed upgrades and
>>     >     activate his
>>     >     >     hosts
>>     >     >     > with one exception.
>>     >     >     >
>>     >     >     > One of hosts was stucked in finishing two tasks. One was
>>     >     changing
>>     >     >     > network configuration, the second was activating of
>>     this host. I
>>     >     >     have to
>>     >     >     > manually clear this tasks in database by
>>     >     >     >
>>     >     >     > SELECT * FROM job ORDER BY start_time DESC;
>>     >     >     > SELECT DeleteJob('...job_id...');
>>     >     >     > SELECT DeleteJob('...job_id...');
>>     >     >     >
>>     >     >     > Now this tasks are gone but still cannot activate this
>>     host.
>>     >     >     >
>>     >     >     >
>>     >     >     > Error while executing action:
>>     >     >     >
>>     >     >     > blade01:
>>     >     >     >
>>     >     >     >     Cannot activate Host. Related operation is
>>     currently in
>>     >     progress.
>>     >     >     > Please try again later.
>>     >     >     >
>>     >     >     > Host is updatet (latest 4.2 repo), rebooted, looks fine...
>>     >     >     >
>>     >     >     > In tasks in UI are no tasks found (the same in db)...
>>     >     >     >
>>     >     >     > What can I do now?
>>     >     >     >
>>     >     >     > Cheers, Jiri
>>     >     >     >
>>     >     >     >
>>     >     >     >
>>     >     >     >
>>     >     >     > _______________________________________________
>>     >     >     > Users mailing list -- users@ovirt.org
>>     <mailto:users@ovirt.org>
>>     >     <mailto:users@ovirt.org <mailto:users@ovirt.org>>
>>     <mailto:users@ovirt.org <mailto:users@ovirt.org>
>>     >     <mailto:users@ovirt.org <mailto:users@ovirt.org>>>
>>     >     >     > To unsubscribe send an email to users-le...@ovirt.org 
>> <mailto:users-le...@ovirt.org>
>>     >     <mailto:users-le...@ovirt.org <mailto:users-le...@ovirt.org>>
>>     >     >     <mailto:users-le...@ovirt.org
>>     <mailto:users-le...@ovirt.org> <mailto:users-le...@ovirt.org
>>     <mailto:users-le...@ovirt.org>>>
>>     >     >     > Privacy Statement: 
>> https://www.ovirt.org/site/privacy-policy/
>>     <https://www.ovirt.org/site/privacy-policy/>
>>     >     >     <https://www.ovirt.org/site/privacy-policy/
>>     <https://www.ovirt.org/site/privacy-policy/>>
>>     >     >     > oVirt Code of Conduct:
>>     >     >     https://www.ovirt.org/community/about/community-guidelines/
>>     <https://www.ovirt.org/community/about/community-guidelines/>
>>     >     >     <https://www.ovirt.org/community/about/community-guidelines/
>>     <https://www.ovirt.org/community/about/community-guidelines/>>
>>     >     >     > List Archives:
>>     >     >   
>>     >      
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/
>>     
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/>
>>     >     >   
>>     >      
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/
>>     
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/>>
>>     >     >     >
>>     >     >
>>     >     >
>>     >     >
>>     >     >     _______________________________________________
>>     >     >     Users mailing list -- users@ovirt.org 
>> <mailto:users@ovirt.org> <mailto:users@ovirt.org
>>     <mailto:users@ovirt.org>>
>>     >     <mailto:users@ovirt.org <mailto:users@ovirt.org>
>>     <mailto:users@ovirt.org <mailto:users@ovirt.org>>>
>>     >     >     To unsubscribe send an email to users-le...@ovirt.org 
>> <mailto:users-le...@ovirt.org>
>>     >     <mailto:users-le...@ovirt.org <mailto:users-le...@ovirt.org>>
>>     >     >     <mailto:users-le...@ovirt.org
>>     <mailto:users-le...@ovirt.org> <mailto:users-le...@ovirt.org
>>     <mailto:users-le...@ovirt.org>>>
>>     >     >     Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>     <https://www.ovirt.org/site/privacy-policy/>
>>     >     >     <https://www.ovirt.org/site/privacy-policy/
>>     <https://www.ovirt.org/site/privacy-policy/>>
>>     >     >     oVirt Code of Conduct:
>>     >     >     https://www.ovirt.org/community/about/community-guidelines/
>>     <https://www.ovirt.org/community/about/community-guidelines/>
>>     >     >     <https://www.ovirt.org/community/about/community-guidelines/
>>     <https://www.ovirt.org/community/about/community-guidelines/>>
>>     >     >     List Archives:
>>     >     >   
>>     >      
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/
>>     
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/>
>>     >     >   
>>     >      
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/
>>     
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/>>
>>     >     >
>>     >     >
>>     >     >
>>     >     >
>>     >     > --
>>     >     > Thanks,
>>     >     > Gobinda
>>     > 
>>     > 
>>     >     _______________________________________________
>>     >     Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> 
>> <mailto:users@ovirt.org
>>     <mailto:users@ovirt.org>>
>>     >     To unsubscribe send an email to users-le...@ovirt.org 
>> <mailto:users-le...@ovirt.org>
>>     >     <mailto:users-le...@ovirt.org <mailto:users-le...@ovirt.org>>
>>     >     Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>     <https://www.ovirt.org/site/privacy-policy/>
>>     >     oVirt Code of Conduct:
>>     >     https://www.ovirt.org/community/about/community-guidelines/
>>     <https://www.ovirt.org/community/about/community-guidelines/>
>>     >     List Archives:
>>     >   
>>      
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/
>>     
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/>
>>     >
>>
>>
>>
>>
>>
>> -- 
>>
>>
>> Raz Tamir
>> Manager, RHV QE
> 
> 
> 
> 
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/HGS5KF22YQD3XSNXHSQARGZHUVDXXZAQ/
> 


Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OOTD2GXB7XXQVQE67CNGU2HN5TCZPJUZ/

Reply via email to