Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire exclusive lock

2014-04-13 Thread Arik Hadas 
Hi Jeremiah,

Thanks for providing the relevant information from the engine log.

A fix was just merged to 3.4 branch.
For more info about the bug and the solution see: 
https://bugzilla.redhat.com/1086280

Note that in the meantime you can unlock the VM by restarting the engine as Roy 
mentioned.

Regards,
Arik

- Original Message -
> 
> 
> - Forwarded Message -
> > From: "Jeremiah Jahn" 
> > To: "Yair Zaslavsky" 
> > Cc: users@ovirt.org
> > Sent: Thursday, April 10, 2014 5:48:24 AM
> > Subject: Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire
> > exclusive lock
> > 
> > here's the start of the problem. If you really want the WHOLE log I
> > can do that too, but this is where I ask it to migrate, and where it
> > fails. After which it just keeps repeating that it can't get the lock.
> > 
> > 
> > 
> > 2014-04-09 14:18:48,050 INFO
> > [org.ovirt.engine.core.bll.MigrateVmCommand]
> > (DefaultQuartzScheduler_Worker-33) Lock freed to object EngineLock
> > [exclusiveLocks= key:
> > 0af0aaf6-85cf-4c67-9744-68a63601ab85 value: VM
> > , sharedLocks= ]
> > 2014-04-09 14:19:32,177 INFO
> > [org.ovirt.engine.core.bll.MigrateVmCommand] (ajp--127.0.0.1-8702-1)
> > [4350ffd] Lock Acquired to object EngineLock [exclusiveLocks= key:
> > b0108933-deb2-4fa0-ae74-e10cefbb0cea value: VM
> > , sharedLocks= ]
> > 2014-04-09 14:19:32,199 INFO
> > [org.ovirt.engine.core.bll.MigrateVmCommand]
> > (org.ovirt.thread.pool-6-thread-16) [4350ffd] Running command:
> > MigrateVmCommand internal: false. Entities affected :  ID:
> > b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM
> > 2014-04-09 14:19:32,203 INFO
> > [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit]
> > (org.ovirt.thread.pool-6-thread-16) [4350ffd] Started HA reservation
> > scoring method
> > 2014-04-09 14:19:32,208 INFO
> > [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> > (org.ovirt.thread.pool-6-thread-16) [4350ffd] START,
> > MigrateVDSCommand(HostName = kvm01.virt, HostId =
> > fa455012-b929-4805-a5c1-50e6c788039a,
> > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
> > dstVdsId=ce6b5b29-778f-4812-97b5-950f3d3b6c71,
> > dstHost=10.10.4.34:54321, migrationMethod=ONLINE,
> > tunnelMigration=false, migrationDowntime=0), log id: 745cd0ba
> > 2014-04-09 14:19:32,209 INFO
> > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> > (org.ovirt.thread.pool-6-thread-16) [4350ffd]
> > VdsBroker::migrate::Entered
> > (vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
> > dstHost=10.10.4.34:54321,  method=online
> > 2014-04-09 14:19:32,210 INFO
> > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> > (org.ovirt.thread.pool-6-thread-16) [4350ffd] START,
> > MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId =
> > fa455012-b929-4805-a5c1-50e6c788039a,
> > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
> > dstVdsId=ce6b5b29-778f-4812-97b5-950f3d3b6c71,
> > dstHost=10.10.4.34:54321, migrationMethod=ONLINE,
> > tunnelMigration=false, migrationDowntime=0), log id: 5de39b66
> > 2014-04-09 14:19:32,224 INFO
> > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> > (org.ovirt.thread.pool-6-thread-16) [4350ffd] FINISH,
> > MigrateBrokerVDSCommand, log id: 5de39b66
> > 2014-04-09 14:19:32,228 INFO
> > [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> > (org.ovirt.thread.pool-6-thread-16) [4350ffd] FINISH,
> > MigrateVDSCommand, return: MigratingFrom, log id: 745cd0ba
> > 2014-04-09 14:19:32,233 INFO
> > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> > (org.ovirt.thread.pool-6-thread-16) [4350ffd] Correlation ID: 4350ffd,
> > Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom
> > Event ID: -1, Message: Migration started (VM: web.judici, Source:
> > kvm01.virt, Destination: kvm04, User: admin).
> > 2014-04-09 14:19:33,465 INFO
> > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> > (DefaultQuartzScheduler_Worker-62) RefreshVmList vm id
> > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm04
> > ignoring it in the refresh until migration is done
> > 2014-04-09 14:19:34,268 INFO
> > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> > (DefaultQuartzScheduler_Worker-74) VM web.judici
> > b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up
> > 2014-04-09 14:19:34,269 I

Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire exclusive lock

2014-04-10 Thread Roy Golan 

On 04/10/2014 02:18 AM, Jeremiah Jahn wrote:

I can't start it, I can't migrate it. I tried to migrate it before,
but the machine was stuck in a read only state. The migration failed
because the machine it was being migrated to was also in a read only
state. somewhere  in the process the lock obviously got lost, and I
can't get it back...

what version of ovirt are you using?

if you need you VM asap then a restart to ovirt engine will free it back.

please attach the full log if you can




2014-04-09 18:11:16,675 INFO  [org.ovirt.engine.core.bll.RunVmCommand]
(ajp--127.0.0.1-8702-3) [58b40832] Failed to Acquire Lock to object
EngineLock [exclusiveLocks= key: b0108933-deb2-4fa0-ae74-e10cefbb0cea
value: VM
, sharedLocks= ]


2014-04-09 18:11:16,676 WARN  [org.ovirt.engine.core.bll.RunVmCommand]
(ajp--127.0.0.1-8702-3) [58b40832] CanDoAction of action RunVm failed.
Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED,$VmName
web.judici
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire exclusive lock

2014-04-09 Thread Jeremiah Jahn 
spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,
maxVCpus=160, clientIp=10.10.4.242,
devices=[Ljava.lang.Object;@1fbf5420, vmName=web.judici,
cpuType=Penryn}], log id: 7ef5612e
2014-04-09 14:25:00,988 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-59) Received a console Device without
an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea
devices, skipping device: {alias=console0, specParams={},
device=console, type=console,
deviceId=05fba965-133e-4810-a87c-1688be8beb5a}
2014-04-09 14:25:01,030 ERROR
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-59) Rerun vm
b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt
2014-04-09 14:25:01,038 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) START,
MigrateStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 47013f27
2014-04-09 14:25:01,045 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) Failed in MigrateStatusVDS method
2014-04-09 14:25:01,045 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) Command
org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
return value
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=47,
mMessage=Migration canceled]]
2014-04-09 14:25:01,046 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) HostName = kvm01.virt
2014-04-09 14:25:01,046 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) Command
MigrateStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed.
Exception: VDSErrorException: VDSGenericException: VDSErrorException:
Failed to MigrateStatusVDS, error = Migration canceled, code = 47
2014-04-09 14:25:01,048 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) FINISH, MigrateStatusVDSCommand,
log id: 47013f27
2014-04-09 14:25:01,055 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-37) Correlation ID: 4350ffd, Job ID:
433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event
ID: -1, Message: Migration failed due to Error: Migration not in
progress. Trying to migrate to another Host (VM: web.judici, Source:
kvm01.virt, Destination: kvm03.virt).
2014-04-09 14:25:01,061 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-37) Correlation ID: 4350ffd, Job ID:
433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event
ID: -1, Message: Migration failed due to Error: Migration not in
progress (VM: web.judici, Source: kvm01.virt, Destination:
kvm03.virt).
2014-04-09 14:34:08,665 INFO
[org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand]
(org.ovirt.thread.pool-6-thread-43) [4045389e] Running command:
MaintenanceNumberOfVdssCommand internal: false. Entities affected :
ID: fa455012-b929-4805-a5c1-50e6c788039a Type: VDS
2014-04-09 14:34:08,667 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-43) [4045389e] START,
SetVdsStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a, status=PreparingForMaintenance,
nonOperationalReason=NONE, stopSpmFailureLogged=true), log id:
64965c26
2014-04-09 14:34:08,678 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-43) [4045389e] FINISH,
SetVdsStatusVDSCommand, log id: 64965c26
2014-04-09 14:34:08,725 INFO
[org.ovirt.engine.core.bll.MaintenanceVdsCommand]
(org.ovirt.thread.pool-6-thread-43) [4045389e] Running command:
MaintenanceVdsCommand internal: true. Entities affected :  ID:
fa455012-b929-4805-a5c1-50e6c788039a Type: VDS
2014-04-09 14:34:08,744 INFO
[org.ovirt.engine.core.bll.InternalMigrateVmCommand]
(org.ovirt.thread.pool-6-thread-43) [35f06602] Failed to Acquire Lock
to object EngineLock [exclusiveLocks= key:
b0108933-deb2-4fa0-ae74-e10cefbb0cea value: VM
, sharedLocks= ]

On Wed, Apr 9, 2014 at 6:33 PM, Yair Zaslavsky  wrote:
> Can you please attach full engine.log?
>
> Many thanks,
> Yair
>
>
> - Original Message -
>> From: "Jeremiah Jahn" 
>> To: users@ovirt.org
>> Sent: Thursday, April 10, 2014 2:18:48 AM
>> Subject: [ovirt-users] help.. vm trapped in limbo aka can't acquire   
>> exclusive lock
>>
>> I can't start it, I can't migrate it. I tried to migrate it before,
>> but the machine was stuck in a read only state. The migration failed
>> because

Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire exclusive lock

2014-04-09 Thread Yair Zaslavsky 
Can you please attach full engine.log?

Many thanks,
Yair


- Original Message -
> From: "Jeremiah Jahn" 
> To: users@ovirt.org
> Sent: Thursday, April 10, 2014 2:18:48 AM
> Subject: [ovirt-users] help.. vm trapped in limbo aka can't acquire   
> exclusive lock
> 
> I can't start it, I can't migrate it. I tried to migrate it before,
> but the machine was stuck in a read only state. The migration failed
> because the machine it was being migrated to was also in a read only
> state. somewhere  in the process the lock obviously got lost, and I
> can't get it back...
> 
> 
> 
> 2014-04-09 18:11:16,675 INFO  [org.ovirt.engine.core.bll.RunVmCommand]
> (ajp--127.0.0.1-8702-3) [58b40832] Failed to Acquire Lock to object
> EngineLock [exclusiveLocks= key: b0108933-deb2-4fa0-ae74-e10cefbb0cea
> value: VM
> , sharedLocks= ]
> 
> 
> 2014-04-09 18:11:16,676 WARN  [org.ovirt.engine.core.bll.RunVmCommand]
> (ajp--127.0.0.1-8702-3) [58b40832] CanDoAction of action RunVm failed.
> Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED,$VmName
> web.judici
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] help.. vm trapped in limbo aka can't acquire exclusive lock

2014-04-09 Thread Jeremiah Jahn 
I can't start it, I can't migrate it. I tried to migrate it before,
but the machine was stuck in a read only state. The migration failed
because the machine it was being migrated to was also in a read only
state. somewhere  in the process the lock obviously got lost, and I
can't get it back...



2014-04-09 18:11:16,675 INFO  [org.ovirt.engine.core.bll.RunVmCommand]
(ajp--127.0.0.1-8702-3) [58b40832] Failed to Acquire Lock to object
EngineLock [exclusiveLocks= key: b0108933-deb2-4fa0-ae74-e10cefbb0cea
value: VM
, sharedLocks= ]


2014-04-09 18:11:16,676 WARN  [org.ovirt.engine.core.bll.RunVmCommand]
(ajp--127.0.0.1-8702-3) [58b40832] CanDoAction of action RunVm failed.
Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED,$VmName
web.judici
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users