Re: [Users] Migration issues with ovirt 3.3
On Wed, Oct 09, 2013 at 04:52:20PM +0200, Gianluca Cecchi wrote: > On Wed, Oct 9, 2013 at 3:43 PM, Dan Kenigsberg wrote: > > On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote: > >> On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote: > >> > >> > > >> >> > >> >> But migration still fails > >> >> > >> > > >> > It seems like an unrelated failure. I do not know what's blocking > >> > migration traffic. Could you see if libvirtd.log and qemu logs at source > >> > and destinaiton have clues? > >> > > >> > >> It seems that on VM.log under qemu of desdt host I have: > >> ... > >> -incoming tcp:[::]:49153: Failed to bind socket: Address already in use > > > > Is that port really taken (`ss -ntp` should tell by whom)? > > yeah ! > It seems gluster uses it on both sides Since libvirt has been using this port range first, would you open a bug on gluster to avoid it? Dan. (prays that Vdsm is not expected to edit libvirt's migration port range on installation) ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Wed, Oct 9, 2013 at 3:43 PM, Dan Kenigsberg wrote:
> On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
>> On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
>>
>> >
>> >>
>> >> But migration still fails
>> >>
>> >
>> > It seems like an unrelated failure. I do not know what's blocking
>> > migration traffic. Could you see if libvirtd.log and qemu logs at source
>> > and destinaiton have clues?
>> >
>>
>> It seems that on VM.log under qemu of desdt host I have:
>> ...
>> -incoming tcp:[::]:49153: Failed to bind socket: Address already in use
>
> Is that port really taken (`ss -ntp` should tell by whom)?
yeah !
It seems gluster uses it on both sides
On destination
[root@f18ovn01 qemu]# ss -ntp |egrep "State|49153"
State Recv-Q Send-QLocal Address:Port Peer Address:Port
ESTAB 0 0 192.168.3.1:975
192.168.3.1:49153 users:(("glusterfs",31166,7))
ESTAB 0 0 192.168.3.1:49153
192.168.3.3:972users:(("glusterfsd",18615,14))
ESTAB 0 0 192.168.3.1:49153
192.168.3.1:965users:(("glusterfsd",18615,13))
ESTAB 0 0 192.168.3.1:963
192.168.3.3:49153 users:(("glusterfs",31152,17))
ESTAB 0 0 192.168.3.1:49153
192.168.3.1:975users:(("glusterfsd",18615,9))
ESTAB 0 0 192.168.3.1:49153
192.168.3.3:966users:(("glusterfsd",18615,15))
ESTAB 0 0 192.168.3.1:965
192.168.3.1:49153 users:(("glusterfs",31152,7))
ESTAB 0 0 192.168.3.1:960
192.168.3.3:49153 users:(("glusterfs",31166,11))
...
[root@f18ovn01 qemu]# ps -ef|grep 31166
root 14950 10958 0 16:50 pts/000:00:00 grep --color=auto 31166
root 31166 1 0 Oct07 ?00:00:04 /usr/sbin/glusterfs -s
localhost --volfile-id gluster/glustershd -p
/var/lib/glusterd/glustershd/run/glustershd.pid -l
/var/log/glusterfs/glustershd.log -S
/var/run/626066f6d74e376808c27ad679a1e85c.socket --xlator-option
*replicate*.node-uuid=ebaf2f1a-65a8-409a-b911-6e631a5f182f
[root@f18ovn01 qemu]# lsof -Pp 31166|grep 49153
glusterfs 31166 root7u IPv44703891 0t0
TCP f18ovn01.mydomain:975->f18ovn01.mydomain:49153 (ESTABLISHED)
glusterfs 31166 root 11u IPv44780563 0t0
TCP f18ovn01.mydomain:960->f18ovn03.mydomain:49153 (ESTABLISHED)
not so good indeed ;-)
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Wed, Oct 09, 2013 at 03:41:58PM +0200, Gianluca Cecchi wrote: > Il 09/ott/2013 15:39 "Gianluca Cecchi" ha > scritto: > > > > > > Il 09/ott/2013 15:35 "Dan Kenigsberg" ha scritto: > > > > > > > > On Wed, Oct 09, 2013 at 02:27:16PM +0200, Gianluca Cecchi wrote: > > > > On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote: > > > > > > > > >> > > > > > >> > so now I'm able to start VM without having to select run once and > > > > >> > attaching a cd iso > > > > >> > (note that is only valid for newly created VMs though) > > > > >> > > > > >> Yes. Old VMs are trashed with the bogus address reproted by the > buggy > > > > >> Vdsm. Can someone from engine supply a script to clear all device > > > > >> addresses from the VM database table? > > > > >> > > > > > > > > > > > > > > > you can use this line (assuming engine as db and user), make sure > only 'bad' vms return: > > > > > psql -U engine -d engine -c "select distinct vm_name from > vm_static, vm_device where vm_guid=vm_id and device='cdrom' and address > ilike '%pci%';" > > > > > > > > > > if so, you can run this to clear the address field for them, so > they could run again: > > > > > psql -U engine -d engine -c "update vm_device set address='' where > device='cdrom' and address ilike '%pci%';" > > > > > > > > > > > > > I wanted to test this but for some reason it seems actually it solved > itself. > > > > I first ran the query and already had no value: > > > > engine=# select distinct vm_name from vm_static, vm_device where > > > > vm_guid=vm_id and device='cdrom' and address ilike '%pci%'; > > > > vm_name > > > > - > > > > (0 rows) > > > > > > > > (overall my VMs are > > > > engine=# select distinct vm_name from vm_static; > > > > vm_name > > > > --- > > > > Blank > > > > c6s > > > > c8again32 > > > > (3 rows) > > > > > > Which of these 3 is the one that was started up with an empty cdrom on a > > > vanilla ovirt-3.3.0 vdsm? The script is expect to show only those. > > > > It is c6s > > Note that the query suggested by Omer had 0 rows. > My further query with 3 results was to show you all rows from vm_static at > the moment Yeah, understood. I have no idea how your c6s cleaned itslef up. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
> On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
>
> >
> >>
> >> But migration still fails
> >>
> >
> > It seems like an unrelated failure. I do not know what's blocking
> > migration traffic. Could you see if libvirtd.log and qemu logs at source
> > and destinaiton have clues?
> >
>
> It seems that on VM.log under qemu of desdt host I have:
> ...
> -incoming tcp:[::]:49153: Failed to bind socket: Address already in use
Is that port really taken (`ss -ntp` should tell by whom)?
>
>
> See all:
> - In libvirtd.log of source host
> 2013-10-07 23:20:54.471+: 1209: debug :
> qemuMonitorOpenInternal:751 : QEMU_MONITOR_NEW: mon=0x7fc66412e820
> refs=2 fd=30
> 2013-10-07 23:20:54.472+: 1209: warning :
> qemuDomainObjEnterMonitorInternal:1136 : This thread seems to be the
> async job owner; entering monitor without asking for a nested job is
> dangerous
> 2013-10-07 23:20:54.472+: 1209: debug :
> qemuMonitorSetCapabilities:1145 : mon=0x7fc66412e820
> 2013-10-07 23:20:54.472+: 1209: debug : qemuMonitorSend:887 :
> QEMU_MONITOR_SEND_MSG: mon=0x7fc66412e820
> msg={"execute":"qmp_capabilities","id":"libvirt-1"}
> fd=-1
> 2013-10-07 23:20:54.769+: 1199: error : qemuMonitorIORead:505 :
> Unable to read from monitor: Connection reset by peer
> 2013-10-07 23:20:54.769+: 1199: debug : qemuMonitorIO:638 : Error
> on monitor Unable to read from monitor: Connection reset by peer
> 2013-10-07 23:20:54.769+: 1199: debug : qemuMonitorIO:672 :
> Triggering error callback
> 2013-10-07 23:20:54.769+: 1199: debug :
> qemuProcessHandleMonitorError:351 : Received error on 0x7fc664124fb0
> 'c8again32'
> 2013-10-07 23:20:54.769+: 1209: debug : qemuMonitorSend:899 : Send
> command resulted in error Unable to read from monitor: Connection
> reset by peer
> 2013-10-07 23:20:54.770+: 1199: debug : qemuMonitorIO:638 : Error
> on monitor Unable to read from monitor: Connection reset by peer
> 2013-10-07 23:20:54.770+: 1209: debug : virFileMakePathHelper:1283
> : path=/var/run/libvirt/qemu mode=0777
> 2013-10-07 23:20:54.770+: 1199: debug : qemuMonitorIO:661 :
> Triggering EOF callback
> 2013-10-07 23:20:54.770+: 1199: debug :
> qemuProcessHandleMonitorEOF:294 : Received EOF on 0x7fc664124fb0
> 'c8again32'
> 2013-10-07 23:20:54.770+: 1209: debug : qemuProcessStop:3992 :
> Shutting down VM 'c8again32' pid=18053 flags=0
> 2013-10-07 23:20:54.771+: 1209: error :
> virNWFilterDHCPSnoopEnd:2135 : internal error ifname "vnet0" not in
> key map
> 2013-10-07 23:20:54.782+: 1209: debug : virCommandRunAsync:2251 :
> About to run /bin/sh -c 'IPT="/usr/sbin/iptables"
> $IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
> vnet0 -g FO-vnet0
> $IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
> $IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
> $IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
> $IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
> $IPT -F FO-vnet0
> $IPT -X FO-vnet0
> $IPT -F FI-vnet0
> $IPT -X FI-vnet0
> $IPT -F HI-vnet0
> $IPT -X HI-vnet0
> IPT="/usr/sbin/ip6tables"
> $IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
> vnet0 -g FO-vnet0
> $IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
> $IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
> $IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
> $IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
> $IPT -F FO-vnet0
> $IPT -X FO-vnet0
> $IPT -F FI-vnet0
> $IPT -X FI-vnet0
> $IPT -F HI-vnet0
> $IPT -X HI-vnet0
> EBT="/usr/sbin/ebtables"
> $EBT -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0
> $EBT -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0
> EBT="/usr/sbin/ebtables"
> collect_chains()
> {
> for tmp2 in $*; do
> for tmp in $($EBT -t nat -L $tmp2 | \
> sed -n "/Bridge chain/,\$ s/.*-j \\([IO]-.*\\)/\\1/p");
> do
> echo $tmp
> collect_chains $tmp
> done
> done
> }
> rm_chains()
> {
> for tmp in $*; do $EBT -t nat -F $tmp; done
> for tmp in $*; do $EBT -t nat -X $tmp; done
> }
> tmp='\''
> '\''
> IFS='\'' '\'''\'' '\''$tmp
> chains="$(collect_chains libvirt-I-vnet0 libvirt-O-vnet0)"
> $EBT -t nat -F libvirt-I-vnet0
> $EBT -t nat -F libvirt-O-vnet0
> rm_chains $chains
> $EBT -t nat -F libvirt-I-vnet0
> $EBT -t nat -X libvirt-I-vnet0
> $EBT -t nat -F libvirt-O-vnet0
> $EBT -t nat -X libvirt-O-vnet0
> '
> 2013-10-07 23:20:54.784+: 1209: debug : virCommandRunAsync:2256 :
> Command result 0, with PID 18076
> 2013-10-07 23:20:54.863+: 1209: debug : virCommandRun:2125 :
> Result exit status 255, stdout: '' stderr: 'iptables v1.4.18: goto
> 'FO-vnet0' is not a chain
>
> Try `iptables -h' or 'iptables --help' for more information.
> iptables v1.4.18: goto 'FO-vnet0' is not a chain
>
> Try `iptables -h' or 'iptables --help' for more information.
> iptables v1.4.18: goto 'FI
Re: [Users] Migration issues with ovirt 3.3
Il 09/ott/2013 15:39 "Gianluca Cecchi" ha scritto: > > > Il 09/ott/2013 15:35 "Dan Kenigsberg" ha scritto: > > > > > On Wed, Oct 09, 2013 at 02:27:16PM +0200, Gianluca Cecchi wrote: > > > On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote: > > > > > > >> > > > > >> > so now I'm able to start VM without having to select run once and > > > >> > attaching a cd iso > > > >> > (note that is only valid for newly created VMs though) > > > >> > > > >> Yes. Old VMs are trashed with the bogus address reproted by the buggy > > > >> Vdsm. Can someone from engine supply a script to clear all device > > > >> addresses from the VM database table? > > > >> > > > > > > > > > > > > you can use this line (assuming engine as db and user), make sure only 'bad' vms return: > > > > psql -U engine -d engine -c "select distinct vm_name from vm_static, vm_device where vm_guid=vm_id and device='cdrom' and address ilike '%pci%';" > > > > > > > > if so, you can run this to clear the address field for them, so they could run again: > > > > psql -U engine -d engine -c "update vm_device set address='' where device='cdrom' and address ilike '%pci%';" > > > > > > > > > > I wanted to test this but for some reason it seems actually it solved itself. > > > I first ran the query and already had no value: > > > engine=# select distinct vm_name from vm_static, vm_device where > > > vm_guid=vm_id and device='cdrom' and address ilike '%pci%'; > > > vm_name > > > - > > > (0 rows) > > > > > > (overall my VMs are > > > engine=# select distinct vm_name from vm_static; > > > vm_name > > > --- > > > Blank > > > c6s > > > c8again32 > > > (3 rows) > > > > Which of these 3 is the one that was started up with an empty cdrom on a > > vanilla ovirt-3.3.0 vdsm? The script is expect to show only those. > > It is c6s Note that the query suggested by Omer had 0 rows. My further query with 3 results was to show you all rows from vm_static at the moment ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
Il 09/ott/2013 15:35 "Dan Kenigsberg" ha scritto: > > On Wed, Oct 09, 2013 at 02:27:16PM +0200, Gianluca Cecchi wrote: > > On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote: > > > > >> > > > >> > so now I'm able to start VM without having to select run once and > > >> > attaching a cd iso > > >> > (note that is only valid for newly created VMs though) > > >> > > >> Yes. Old VMs are trashed with the bogus address reproted by the buggy > > >> Vdsm. Can someone from engine supply a script to clear all device > > >> addresses from the VM database table? > > >> > > > > > > > > > you can use this line (assuming engine as db and user), make sure only 'bad' vms return: > > > psql -U engine -d engine -c "select distinct vm_name from vm_static, vm_device where vm_guid=vm_id and device='cdrom' and address ilike '%pci%';" > > > > > > if so, you can run this to clear the address field for them, so they could run again: > > > psql -U engine -d engine -c "update vm_device set address='' where device='cdrom' and address ilike '%pci%';" > > > > > > > I wanted to test this but for some reason it seems actually it solved itself. > > I first ran the query and already had no value: > > engine=# select distinct vm_name from vm_static, vm_device where > > vm_guid=vm_id and device='cdrom' and address ilike '%pci%'; > > vm_name > > - > > (0 rows) > > > > (overall my VMs are > > engine=# select distinct vm_name from vm_static; > > vm_name > > --- > > Blank > > c6s > > c8again32 > > (3 rows) > > Which of these 3 is the one that was started up with an empty cdrom on a > vanilla ovirt-3.3.0 vdsm? The script is expect to show only those. It is c6s ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Wed, Oct 09, 2013 at 02:27:16PM +0200, Gianluca Cecchi wrote: > On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote: > > >> > > >> > so now I'm able to start VM without having to select run once and > >> > attaching a cd iso > >> > (note that is only valid for newly created VMs though) > >> > >> Yes. Old VMs are trashed with the bogus address reproted by the buggy > >> Vdsm. Can someone from engine supply a script to clear all device > >> addresses from the VM database table? > >> > > > > > > you can use this line (assuming engine as db and user), make sure only > > 'bad' vms return: > > psql -U engine -d engine -c "select distinct vm_name from vm_static, > > vm_device where vm_guid=vm_id and device='cdrom' and address ilike '%pci%';" > > > > if so, you can run this to clear the address field for them, so they could > > run again: > > psql -U engine -d engine -c "update vm_device set address='' where > > device='cdrom' and address ilike '%pci%';" > > > > I wanted to test this but for some reason it seems actually it solved itself. > I first ran the query and already had no value: > engine=# select distinct vm_name from vm_static, vm_device where > vm_guid=vm_id and device='cdrom' and address ilike '%pci%'; > vm_name > - > (0 rows) > > (overall my VMs are > engine=# select distinct vm_name from vm_static; > vm_name > --- > Blank > c6s > c8again32 > (3 rows) Which of these 3 is the one that was started up with an empty cdrom on a vanilla ovirt-3.3.0 vdsm? The script is expect to show only those. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
>
>>
>> But migration still fails
>>
>
> It seems like an unrelated failure. I do not know what's blocking
> migration traffic. Could you see if libvirtd.log and qemu logs at source
> and destinaiton have clues?
>
It seems that on VM.log under qemu of desdt host I have:
...
-incoming tcp:[::]:49153: Failed to bind socket: Address already in use
See all:
- In libvirtd.log of source host
2013-10-07 23:20:54.471+: 1209: debug :
qemuMonitorOpenInternal:751 : QEMU_MONITOR_NEW: mon=0x7fc66412e820
refs=2 fd=30
2013-10-07 23:20:54.472+: 1209: warning :
qemuDomainObjEnterMonitorInternal:1136 : This thread seems to be the
async job owner; entering monitor without asking for a nested job is
dangerous
2013-10-07 23:20:54.472+: 1209: debug :
qemuMonitorSetCapabilities:1145 : mon=0x7fc66412e820
2013-10-07 23:20:54.472+: 1209: debug : qemuMonitorSend:887 :
QEMU_MONITOR_SEND_MSG: mon=0x7fc66412e820
msg={"execute":"qmp_capabilities","id":"libvirt-1"}
fd=-1
2013-10-07 23:20:54.769+: 1199: error : qemuMonitorIORead:505 :
Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.769+: 1199: debug : qemuMonitorIO:638 : Error
on monitor Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.769+: 1199: debug : qemuMonitorIO:672 :
Triggering error callback
2013-10-07 23:20:54.769+: 1199: debug :
qemuProcessHandleMonitorError:351 : Received error on 0x7fc664124fb0
'c8again32'
2013-10-07 23:20:54.769+: 1209: debug : qemuMonitorSend:899 : Send
command resulted in error Unable to read from monitor: Connection
reset by peer
2013-10-07 23:20:54.770+: 1199: debug : qemuMonitorIO:638 : Error
on monitor Unable to read from monitor: Connection reset by peer
2013-10-07 23:20:54.770+: 1209: debug : virFileMakePathHelper:1283
: path=/var/run/libvirt/qemu mode=0777
2013-10-07 23:20:54.770+: 1199: debug : qemuMonitorIO:661 :
Triggering EOF callback
2013-10-07 23:20:54.770+: 1199: debug :
qemuProcessHandleMonitorEOF:294 : Received EOF on 0x7fc664124fb0
'c8again32'
2013-10-07 23:20:54.770+: 1209: debug : qemuProcessStop:3992 :
Shutting down VM 'c8again32' pid=18053 flags=0
2013-10-07 23:20:54.771+: 1209: error :
virNWFilterDHCPSnoopEnd:2135 : internal error ifname "vnet0" not in
key map
2013-10-07 23:20:54.782+: 1209: debug : virCommandRunAsync:2251 :
About to run /bin/sh -c 'IPT="/usr/sbin/iptables"
$IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
vnet0 -g FO-vnet0
$IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
$IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
$IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
$IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
$IPT -F FO-vnet0
$IPT -X FO-vnet0
$IPT -F FI-vnet0
$IPT -X FI-vnet0
$IPT -F HI-vnet0
$IPT -X HI-vnet0
IPT="/usr/sbin/ip6tables"
$IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
vnet0 -g FO-vnet0
$IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
$IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
$IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
$IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
$IPT -F FO-vnet0
$IPT -X FO-vnet0
$IPT -F FI-vnet0
$IPT -X FI-vnet0
$IPT -F HI-vnet0
$IPT -X HI-vnet0
EBT="/usr/sbin/ebtables"
$EBT -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0
$EBT -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0
EBT="/usr/sbin/ebtables"
collect_chains()
{
for tmp2 in $*; do
for tmp in $($EBT -t nat -L $tmp2 | \
sed -n "/Bridge chain/,\$ s/.*-j \\([IO]-.*\\)/\\1/p");
do
echo $tmp
collect_chains $tmp
done
done
}
rm_chains()
{
for tmp in $*; do $EBT -t nat -F $tmp; done
for tmp in $*; do $EBT -t nat -X $tmp; done
}
tmp='\''
'\''
IFS='\'' '\'''\'' '\''$tmp
chains="$(collect_chains libvirt-I-vnet0 libvirt-O-vnet0)"
$EBT -t nat -F libvirt-I-vnet0
$EBT -t nat -F libvirt-O-vnet0
rm_chains $chains
$EBT -t nat -F libvirt-I-vnet0
$EBT -t nat -X libvirt-I-vnet0
$EBT -t nat -F libvirt-O-vnet0
$EBT -t nat -X libvirt-O-vnet0
'
2013-10-07 23:20:54.784+: 1209: debug : virCommandRunAsync:2256 :
Command result 0, with PID 18076
2013-10-07 23:20:54.863+: 1209: debug : virCommandRun:2125 :
Result exit status 255, stdout: '' stderr: 'iptables v1.4.18: goto
'FO-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'FO-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'FI-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.18: goto 'HI-vnet0' is not a chain
Try `iptables -h' or 'iptables --help' for more information.
iptables: Bad rule (does a matching rule exist in that chain?).
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match
Re: [Users] Migration issues with ovirt 3.3
On Tue, Oct 8, 2013 at 12:27 PM, Omer Frenkel wrote: >> > >> > so now I'm able to start VM without having to select run once and >> > attaching a cd iso >> > (note that is only valid for newly created VMs though) >> >> Yes. Old VMs are trashed with the bogus address reproted by the buggy >> Vdsm. Can someone from engine supply a script to clear all device >> addresses from the VM database table? >> > > > you can use this line (assuming engine as db and user), make sure only 'bad' > vms return: > psql -U engine -d engine -c "select distinct vm_name from vm_static, > vm_device where vm_guid=vm_id and device='cdrom' and address ilike '%pci%';" > > if so, you can run this to clear the address field for them, so they could > run again: > psql -U engine -d engine -c "update vm_device set address='' where > device='cdrom' and address ilike '%pci%';" > I wanted to test this but for some reason it seems actually it solved itself. I first ran the query and already had no value: engine=# select distinct vm_name from vm_static, vm_device where vm_guid=vm_id and device='cdrom' and address ilike '%pci%'; vm_name - (0 rows) (overall my VMs are engine=# select distinct vm_name from vm_static; vm_name --- Blank c6s c8again32 (3 rows) ) Then I tried to start the powered off VM that gave problems and it started ok. Tried to run shutdown inside it and power on again and it worked too. As soon as I updated the bugzilla with my comment, I restarted vdsmd on both nodes but I wasn't able to run it... strange... any other thing that could have resolved by itself. Things done in sequence related to that VM were: modify vm.py on both nodes and restart vdsmd while VM was powered off verify that power on gave the error verify that run once worked as before shutdown and power off VM after two days (no activity on my side at all) I wanted to try the DB workaround to solve this VM status but apparently the column was already empty and VM able to start normally Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
- Original Message -
> From: "Dan Kenigsberg"
> To: "Gianluca Cecchi"
> Cc: "users" , "Michal Skrivanek"
> Sent: Tuesday, October 8, 2013 11:40:25 AM
> Subject: Re: [Users] Migration issues with ovirt 3.3
>
> On Tue, Oct 08, 2013 at 01:44:47AM +0200, Gianluca Cecchi wrote:
> > On Mon, Oct 7, 2013 at 2:59 AM, Dan Kenigsberg wrote:
> >
> > >
> > > Would you please test if http://gerrit.ovirt.org/19906 solves the
> > > issue? (I haven't. Too late at night.)
> > >
> > > Regards,
> > > Dan.
> >
> > I can confirm that it resolves
> > https://bugzilla.redhat.com/show_bug.cgi?id=1007980
> >
> > so now I'm able to start VM without having to select run once and
> > attaching a cd iso
> > (note that is only valid for newly created VMs though)
>
> Yes. Old VMs are trashed with the bogus address reproted by the buggy
> Vdsm. Can someone from engine supply a script to clear all device
> addresses from the VM database table?
>
you can use this line (assuming engine as db and user), make sure only 'bad'
vms return:
psql -U engine -d engine -c "select distinct vm_name from vm_static, vm_device
where vm_guid=vm_id and device='cdrom' and address ilike '%pci%';"
if so, you can run this to clear the address field for them, so they could run
again:
psql -U engine -d engine -c "update vm_device set address='' where
device='cdrom' and address ilike '%pci%';"
> >
> > But migration still fails
> >
>
> It seems like an unrelated failure. I do not know what's blocking
> migration traffic. Could you see if libvirtd.log and qemu logs at source
> and destinaiton have clues?
>
>
>
> > Thread-4644::DEBUG::2013-10-08
> > 01:20:53,933::vm::360::vm.Vm::(_startUnderlyingMigration)
> > vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration to
> > qemu+tls://10.4.4.58/system with miguri tcp://10.4.4.58
>
> > Thread-4645::DEBUG::2013-10-08 01:20:53,934::vm::718::vm.Vm::(run)
> > vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread
> > started
> > Thread-4646::DEBUG::2013-10-08 01:20:53,935::vm::756::vm.Vm::(run)
> > vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration
> > monitor thread
> > Thread-4648::DEBUG::2013-10-08
> > 01:20:54,321::BindingXMLRPC::979::vds::(wrapper) client
> > [10.4.4.60]::call volumesList with () {}
> > Thread-4648::DEBUG::2013-10-08
> > 01:20:54,349::BindingXMLRPC::986::vds::(wrapper) return volumesList
> > with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'gviso':
> > {'transportType': ['TCP'], 'uuid':
> > 'c8cbcac7-1d40-4cee-837d-bb97467fb2bd', 'bricks':
> > ['f18ovn01.mydomain:/gluster/ISO_GLUSTER/brick1',
> > 'f18ovn03.mydomain:/gluster/ISO_GLUSTER/brick1'], 'volumeName':
> > 'gviso', 'volumeType': 'REPLICATE', 'replicaCount': '2', 'brickCount':
> > '2', 'distCount': '2', 'volumeStatus': 'ONLINE', 'stripeCount': '1',
> > 'options': {'storage.owner-gid': '36', 'storage.owner-uid': '36',
> > 'server.allow-insecure': 'on'}}, 'gvdata': {'transportType': ['TCP'],
> > 'uuid': 'ed71a4c2-6205-4aad-9aab-85da086d5ba3', 'bricks':
> > ['f18ovn01.mydomain:/gluster/DATA_GLUSTER/brick1',
> > 'f18ovn03.mydomain:/gluster/DATA_GLUSTER/brick1'], 'volumeName':
> > 'gvdata', 'volumeType': 'REPLICATE', 'replicaCount': '2',
> > 'brickCount': '2', 'distCount': '2', 'volumeStatus': 'ONLINE',
> > 'stripeCount': '1', 'options': {'server.allow-insecure': 'on',
> > 'storage.owner-uid': '36', 'storage.owner-gid': '36'
> > Thread-4644::ERROR::2013-10-08
> > 01:20:54,873::libvirtconnection::94::libvirtconnection::(wrapper)
> > connection to libvirt broken. ecode: 38 edom: 7
> > Thread-4644::ERROR::2013-10-08
> > 01:20:54,873::libvirtconnection::96::libvirtconnection::(wrapper)
> > taking calling process down.
> > MainThread::DEBUG::2013-10-08
> > 01:20:54,874::vdsm::45::vds::(sigtermHandl
Re: [Users] Migration issues with ovirt 3.3
On Tue, Oct 08, 2013 at 01:44:47AM +0200, Gianluca Cecchi wrote:
> On Mon, Oct 7, 2013 at 2:59 AM, Dan Kenigsberg wrote:
>
> >
> > Would you please test if http://gerrit.ovirt.org/19906 solves the
> > issue? (I haven't. Too late at night.)
> >
> > Regards,
> > Dan.
>
> I can confirm that it resolves
> https://bugzilla.redhat.com/show_bug.cgi?id=1007980
>
> so now I'm able to start VM without having to select run once and
> attaching a cd iso
> (note that is only valid for newly created VMs though)
Yes. Old VMs are trashed with the bogus address reproted by the buggy
Vdsm. Can someone from engine supply a script to clear all device
addresses from the VM database table?
>
> But migration still fails
>
It seems like an unrelated failure. I do not know what's blocking
migration traffic. Could you see if libvirtd.log and qemu logs at source
and destinaiton have clues?
> Thread-4644::DEBUG::2013-10-08
> 01:20:53,933::vm::360::vm.Vm::(_startUnderlyingMigration)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration to
> qemu+tls://10.4.4.58/system with miguri tcp://10.4.4.58
> Thread-4645::DEBUG::2013-10-08 01:20:53,934::vm::718::vm.Vm::(run)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread
> started
> Thread-4646::DEBUG::2013-10-08 01:20:53,935::vm::756::vm.Vm::(run)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::starting migration
> monitor thread
> Thread-4648::DEBUG::2013-10-08
> 01:20:54,321::BindingXMLRPC::979::vds::(wrapper) client
> [10.4.4.60]::call volumesList with () {}
> Thread-4648::DEBUG::2013-10-08
> 01:20:54,349::BindingXMLRPC::986::vds::(wrapper) return volumesList
> with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'gviso':
> {'transportType': ['TCP'], 'uuid':
> 'c8cbcac7-1d40-4cee-837d-bb97467fb2bd', 'bricks':
> ['f18ovn01.mydomain:/gluster/ISO_GLUSTER/brick1',
> 'f18ovn03.mydomain:/gluster/ISO_GLUSTER/brick1'], 'volumeName':
> 'gviso', 'volumeType': 'REPLICATE', 'replicaCount': '2', 'brickCount':
> '2', 'distCount': '2', 'volumeStatus': 'ONLINE', 'stripeCount': '1',
> 'options': {'storage.owner-gid': '36', 'storage.owner-uid': '36',
> 'server.allow-insecure': 'on'}}, 'gvdata': {'transportType': ['TCP'],
> 'uuid': 'ed71a4c2-6205-4aad-9aab-85da086d5ba3', 'bricks':
> ['f18ovn01.mydomain:/gluster/DATA_GLUSTER/brick1',
> 'f18ovn03.mydomain:/gluster/DATA_GLUSTER/brick1'], 'volumeName':
> 'gvdata', 'volumeType': 'REPLICATE', 'replicaCount': '2',
> 'brickCount': '2', 'distCount': '2', 'volumeStatus': 'ONLINE',
> 'stripeCount': '1', 'options': {'server.allow-insecure': 'on',
> 'storage.owner-uid': '36', 'storage.owner-gid': '36'
> Thread-4644::ERROR::2013-10-08
> 01:20:54,873::libvirtconnection::94::libvirtconnection::(wrapper)
> connection to libvirt broken. ecode: 38 edom: 7
> Thread-4644::ERROR::2013-10-08
> 01:20:54,873::libvirtconnection::96::libvirtconnection::(wrapper)
> taking calling process down.
> MainThread::DEBUG::2013-10-08
> 01:20:54,874::vdsm::45::vds::(sigtermHandler) Received signal 15
> Thread-4644::DEBUG::2013-10-08 01:20:54,874::vm::733::vm.Vm::(cancel)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::canceling migration
> downtime thread
> Thread-4644::DEBUG::2013-10-08 01:20:54,875::vm::803::vm.Vm::(stop)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::stopping migration
> monitor thread
> Thread-4645::DEBUG::2013-10-08 01:20:54,875::vm::730::vm.Vm::(run)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::migration downtime thread
> exiting
> Thread-4644::ERROR::2013-10-08
> 01:20:54,875::vm::244::vm.Vm::(_recover)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Cannot recv data:
> Input/output error
> Thread-4644::ERROR::2013-10-08 01:20:55,008::vm::324::vm.Vm::(run)
> vmId=`d54660a2-45ed-41ae-ab99-a6f93ebbdbb1`::Failed to migrate
> Traceback (most recent call last):
> File "/usr/share/vdsm/vm.py", line 311, in run
> self._startUnderlyingMigration()
> File "/usr/share/vdsm/vm.py", line 388, in _startUnderlyingMigration
> None, maxBandwidth)
> File "/usr/share/vdsm/vm.py", line 826, in f
> ret = attr(*args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py",
> line 76, in wrapper
> ret = f(*args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1253, in
> migrateToURI2
> if ret == -1: raise libvirtError ('virDomainMigrateToURI2()
> failed', dom=self)
> libvirtError: Cannot recv data: Input/output error
>
> Chain INPUT (policy ACCEPT)
> target prot opt source destination
> ACCEPT all -- 192.168.3.3 0.0.0.0/0
> ACCEPT all -- 0.0.0.0/00.0.0.0/0state
> RELATED,ESTABLISHED
> ACCEPT all -- 0.0.0.0/00.0.0.0/0
> ACCEPT tcp -- 0.0.0.0/00.0.0.0/0tcp dpt:54321
> ...
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Mon, Oct 7, 2013 at 2:59 AM, Dan Kenigsberg wrote:
>
> Would you please test if http://gerrit.ovirt.org/19906 solves the
> issue? (I haven't. Too late at night.)
>
> Regards,
> Dan.
I can confirm that it resolves
https://bugzilla.redhat.com/show_bug.cgi?id=1007980
so now I'm able to start VM without having to select run once and
attaching a cd iso
(note that is only valid for newly created VMs though)
But migration still fails
On engine:
2013-10-08 01:20:51,459 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-15) START, GlusterV
olumesListVDSCommand(HostName = f18ovn03, HostId =
b67bcfd4-f868-49d5-8704-4936ee922249), log id: 4425a532
2013-10-08 01:20:51,547 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-15) FINISH, Gluster
VolumesListVDSCommand, return:
{ed71a4c2-6205-4aad-9aab-85da086d5ba3=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8defb13f
,
c8cbcac7-1d40-4cee-837d-bb97467fb2bd=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@91be2b8a},
log id: 4425a532
2013-10-08 01:20:51,697 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-39) RefreshVmList vm id d54660a2-45
ed-41ae-ab99-a6f93ebbdbb1 is migrating to vds f18ovn01 ignoring it in
the refresh until migration is done
2013-10-08 01:20:54,765 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand]
(pool-6-thread-38) START, GlusterServicesListVDSC
ommand(HostName = f18ovn01, HostId =
80188ccc-83b2-4bc8-9385-8d07f7458a3c), log id: 41ed75a4
2013-10-08 01:20:54,766 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand]
(pool-6-thread-36) START, GlusterServicesListVDSC
ommand(HostName = f18ovn03, HostId =
b67bcfd4-f868-49d5-8704-4936ee922249), log id: 7eea38bc
2013-10-08 01:20:56,618 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-2) START, GlusterVo
lumesListVDSCommand(HostName = f18ovn01, HostId =
80188ccc-83b2-4bc8-9385-8d07f7458a3c), log id: 34d0827b
2013-10-08 01:20:56,722 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand]
(DefaultQuartzScheduler_Worker-84) Command ListVDS execution failed.
Exception: VDSNetworkException: java.net.SocketException: Connection
reset
2013-10-08 01:20:56,725 ERROR
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand]
(pool-6-thread-36) Command GlusterServicesListVDS execution failed.
Exception: VDSNetworkException: java.net.SocketException: Connection
reset
2013-10-08 01:20:56,725 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand]
(pool-6-thread-36) FINISH, GlusterServicesListVDSCommand, log id:
7eea38bc
2013-10-08 01:20:56,727 WARN
[org.ovirt.engine.core.vdsbroker.VdsManager]
(DefaultQuartzScheduler_Worker-84) Failed to refresh VDS , vds =
b67bcfd4-f868-49d5-8704-4936ee922249 : f18ovn03, VDS Network Error,
continuing.
java.net.SocketException: Connection reset
2013-10-08 01:20:58,583 INFO
[org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand]
(DefaultQuartzScheduler_Worker-48) Running command:
SetStoragePoolStatusCommand internal: true. Entities affected : ID:
eb679feb-4da2-4fd0-a185-abbe459ffa70 Type: StoragePool
2013-10-08 01:20:58,603 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-48) Correlation ID: 16f779a3, Call
Stack: null, Custom Event ID: -1, Message: Invalid status on Data
Center Gluster. Setting status to Non Responsive.
2013-10-08 01:20:58,701 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-48) Command
org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand
return value
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654,
mMessage=Not SPM: ()]]
2013-10-08 01:20:58,714 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-48) HostName = f18ovn03
2013-10-08 01:20:58,714 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-48) Command HSMGetAllTasksStatusesVDS
execution failed. Exception: IRSNonOperationalException:
IRSGenericException: IRSErrorException: IRSNonOperationalException:
Not SPM: ()
2013-10-08 01:20:59,356 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand]
(DefaultQuartzScheduler_Worker-41) Command ListVDS execution failed.
Exception: VDSNetworkException: java.net.SocketException: Connection
reset
On host where VM was running vdsm.log contains:
Thread-4643::DEBUG::2013-10-08
01:20:53,700::BindingXMLRPC::979::vds::(wrapper) client
[10.4.4.60]::call vmMigrate with ({'tunneled': 'false', 'dst':
'10.4.4.58:54321', 'vmId': 'd54660a2-45ed-41ae-ab99-a6f93ebbdbb1',
'method': 'online', 'src': '10.4.4.59'},) {} flowID [7c55a4a9]
Thread-464
Re: [Users] Migration issues with ovirt 3.3
On Fri, Oct 04, 2013 at 02:55:42PM -0300, emi...@gmail.com wrote: > Dan, I've made a whole new installation of oVirt 3.3 using the stable > repos, I've updated libvirt to 1.1.2-1 and configured a glusterFS cluster. > > I've just tried the live migration of the Windows Host, here i send you the > vdsm, libvir and qemu logs Here goes. vm.Vm._getUnderlyingDriveInfo() matches the list of disks provided by Engine with the list of disks reported back by libvirt. After a match has been found, the libvirt-assigned address of each disk is reported to Engine. _getUnderlyingDriveInfo is awfully naive. If the cdrom is empty, the function matches it with the gluster volume, and overides the cdrom's address with the gluster volume's. Would you please test if http://gerrit.ovirt.org/19906 solves the issue? (I haven't. Too late at night.) Regards, Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Mon, Oct 07, 2013 at 12:25:30AM +0200, Gianluca Cecchi wrote: > I've done some further tests and comparisons Is this related to emitor's bug? For emitor, when a VM does not start it happens when libvirt recongnizes that its devices have bad addresses. It never gets to qemu command line phase. > > In this F19 based + ovirt stable repo the difference in qemu command > line between the working run once with cd enabled of this CentOS 6.4 > x86_64 and the normal run that doesn't proceed is: > > run once > -drive > file=/rhev/data-center/mnt/f18engine.mydomain:_var_lib_exports_iso/cc790a86-72e2-4aa2-a0b6-700756c399c3/images/----/CentOS-6.4-x86_64-bin-DVD1.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2 > > run > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > > > A working normal run of a WinXP 32bit VM in another oVirt 3.2.3 environment is > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > > so it is the same as in 3.3. > > Possible problems: > > - 64bit OS > I can try tomorrow to create a CentOS 6.4 32 bit and compare results > > - qemu default emulated pc version > > Some comparisons > > a) f18 host (AIO) with oVirt 3.2.3 > > qemu-kvm-1.2.2-14.fc18.x86_64 > > [root@tekkaman polimi]# qemu-kvm -machine ? > Supported machines are: > none empty machine > pc Standard PC (alias of pc-1.2) > pc-1.2 Standard PC (default) > pc-1.1 Standard PC > pc-1.0 Standard PC > pc-0.15 Standard PC > pc-0.14 Standard PC > pc-0.13 Standard PC > pc-0.12 Standard PC > pc-0.11 Standard PC, qemu 0.11 > pc-0.10 Standard PC, qemu 0.10 > isapcISA-only PC > > When I start WinXP (32bit of course) I see this command line: > /usr/bin/qemu-kvm -name winxp -S -M pc-0.14 -cpu Opteron_G3 > > b) f19 host in oVirt 3.3 > > qemu-kvm-1.4.2-11.fc19.x86_64 > > [root@f18ovn03 ~]# qemu-kvm -machine ? > Supported machines are: > none empty machine > pc Standard PC (i440FX + PIIX, 1996) (alias of > pc-i440fx-1.4) > pc-i440fx-1.4Standard PC (i440FX + PIIX, 1996) (default) > pc-1.3 Standard PC > pc-1.2 Standard PC > pc-1.1 Standard PC > pc-1.0 Standard PC > pc-0.15 Standard PC > pc-0.14 Standard PC > pc-0.13 Standard PC > pc-0.12 Standard PC > pc-0.11 Standard PC, qemu 0.11 > pc-0.10 Standard PC, qemu 0.10 > isapcISA-only PC > q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-1.4) > pc-q35-1.4 Standard PC (Q35 + ICH9, 2009) > > qemu command line for a CentOS 6.4 64bit vm is: > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name c6s -S -machine > pc-1.0,accel=kvm,usb=off -cpu Opteron_G2 > > both /usr/bin/qemu-kvm and /usr/bin/qemu-system-x86_64 are in > qemu-system-x86 rpm package. > > Questions: > - given a version of oVirt, pc version in qemu generated command line > is the same for every defined VM or is there any logic depending on OS > chosen and/or bit (32 vs 64)? emulatedMachine is set by the effective clusterLevel. > - if answer to previous question is fixed pc version, is there any > particular reason to use pc-0.14 in oVirt 3.2.3 and pc-1.0 in oVirt > 3.3 and not the latest available (so pc-1.2 in 3.2.3 and pc-1.3 or > pc-i440fx-1.4 in 3.3)? We promise inter-cluster migration of VMs. Hence, 3.2.3 may use only the newest machine available at 3.2.0. > - where can I find documentation about difference between the > different pc-x.y versions? I am not aware of anything beyond the qemu source code and release notes. > - can I use a wrapper in oVirt 3.3 to start VM with a different pc-x.y > (eg pc-0.14) and see if anything changes helping debug and solve the > problem? Yes, you may write a vdsm before_vm_start hook. It should search for the 'machine' attribute and set it to whatever value you please. Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
I've done some further tests and comparisons In this F19 based + ovirt stable repo the difference in qemu command line between the working run once with cd enabled of this CentOS 6.4 x86_64 and the normal run that doesn't proceed is: run once -drive file=/rhev/data-center/mnt/f18engine.mydomain:_var_lib_exports_iso/cc790a86-72e2-4aa2-a0b6-700756c399c3/images/----/CentOS-6.4-x86_64-bin-DVD1.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2 run -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 A working normal run of a WinXP 32bit VM in another oVirt 3.2.3 environment is -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 so it is the same as in 3.3. Possible problems: - 64bit OS I can try tomorrow to create a CentOS 6.4 32 bit and compare results - qemu default emulated pc version Some comparisons a) f18 host (AIO) with oVirt 3.2.3 qemu-kvm-1.2.2-14.fc18.x86_64 [root@tekkaman polimi]# qemu-kvm -machine ? Supported machines are: none empty machine pc Standard PC (alias of pc-1.2) pc-1.2 Standard PC (default) pc-1.1 Standard PC pc-1.0 Standard PC pc-0.15 Standard PC pc-0.14 Standard PC pc-0.13 Standard PC pc-0.12 Standard PC pc-0.11 Standard PC, qemu 0.11 pc-0.10 Standard PC, qemu 0.10 isapcISA-only PC When I start WinXP (32bit of course) I see this command line: /usr/bin/qemu-kvm -name winxp -S -M pc-0.14 -cpu Opteron_G3 b) f19 host in oVirt 3.3 qemu-kvm-1.4.2-11.fc19.x86_64 [root@f18ovn03 ~]# qemu-kvm -machine ? Supported machines are: none empty machine pc Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-1.4) pc-i440fx-1.4Standard PC (i440FX + PIIX, 1996) (default) pc-1.3 Standard PC pc-1.2 Standard PC pc-1.1 Standard PC pc-1.0 Standard PC pc-0.15 Standard PC pc-0.14 Standard PC pc-0.13 Standard PC pc-0.12 Standard PC pc-0.11 Standard PC, qemu 0.11 pc-0.10 Standard PC, qemu 0.10 isapcISA-only PC q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-1.4) pc-q35-1.4 Standard PC (Q35 + ICH9, 2009) qemu command line for a CentOS 6.4 64bit vm is: /usr/bin/qemu-system-x86_64 -machine accel=kvm -name c6s -S -machine pc-1.0,accel=kvm,usb=off -cpu Opteron_G2 both /usr/bin/qemu-kvm and /usr/bin/qemu-system-x86_64 are in qemu-system-x86 rpm package. Questions: - given a version of oVirt, pc version in qemu generated command line is the same for every defined VM or is there any logic depending on OS chosen and/or bit (32 vs 64)? - if answer to previous question is fixed pc version, is there any particular reason to use pc-0.14 in oVirt 3.2.3 and pc-1.0 in oVirt 3.3 and not the latest available (so pc-1.2 in 3.2.3 and pc-1.3 or pc-i440fx-1.4 in 3.3)? - where can I find documentation about difference between the different pc-x.y versions? - can I use a wrapper in oVirt 3.3 to start VM with a different pc-x.y (eg pc-0.14) and see if anything changes helping debug and solve the problem? Thanks, Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On 4 Oct 2013, at 19:55, emi...@gmail.com wrote: > Dan, I've made a whole new installation of oVirt 3.3 using the stable repos, > I've updated libvirt to 1.1.2-1 and configured a glusterFS cluster. > > I've just tried the live migration of the Windows Host, here i send you the > vdsm, libvir and qemu logs hmm…I see both PCI address for CDROM as we've been seeing on gluster setups. So far no update from gluster AFAIK…it seems to be broken on their side somewhere… and I also see error on migrations dst which seems to be another/different disk device mess up The W2K8 create VM call in the ovirt1's log is already wrong, can you capture/include the initial create call? In the previous cases it was always the case that the VM was created(from engine) with IDE address but then after the first run it somehow changed to PCI on the first status update from vdsm Thanks, michal > > > 2013/10/4 Dan Kenigsberg > On Wed, Oct 02, 2013 at 01:36:56PM -0300, emi...@gmail.com wrote: > > I've configured the host as a NFS cluster. I've imported the VMs, the > > debian now it's migrating without problems, the pfSense now gives me an > > error (Migration failed due to Error: Desktop does not exist) and the > > Windows VM doesn't start in the new cluster after import it. > > > > I've attached the logs of the vdsm > > Thanks for providing your, but they begin too late. > > The bug is still a mystery to me. Something garbles Vdsm's internal > representation of virtual disks. And it's bad. > > I would appreciate if you could dig backward in your logs and find the > last successfull startup of a VM, say W2K8_cluster1 > (86f10137-d917-4508-a07a-3f83e6f24a95). Then, browse forward to find the > first failure (be it on next startup of VM or during migration). > > Best regards, > Dan. > > > > -- > Emiliano Tortorella > +598 98941176 > emi...@gmail.com > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Wed, Oct 02, 2013 at 01:36:56PM -0300, emi...@gmail.com wrote: > I've configured the host as a NFS cluster. I've imported the VMs, the > debian now it's migrating without problems, the pfSense now gives me an > error (Migration failed due to Error: Desktop does not exist) and the > Windows VM doesn't start in the new cluster after import it. > > I've attached the logs of the vdsm Thanks for providing your, but they begin too late. The bug is still a mystery to me. Something garbles Vdsm's internal representation of virtual disks. And it's bad. I would appreciate if you could dig backward in your logs and find the last successfull startup of a VM, say W2K8_cluster1 (86f10137-d917-4508-a07a-3f83e6f24a95). Then, browse forward to find the first failure (be it on next startup of VM or during migration). Best regards, Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On 10/04/2013 12:39 PM, Sven Kieske wrote: Hi, if I understand Gianluca correct he has setup the ManagementNode and 2 ComputeNodes with Fedora 19. CentOS 6.4 is the VM-OS (inside the VM). So this should work? yes, it should. HTH Sven On 04/10/13 11:29, Itamar Heim wrote: Me too with oVirt 3.3 setup and GlusterFS DataCenter. One dedicated engine + 2 vdsm hosts. All fedora 19 + ovirt stable. My trying to migrate VM is CentOS 6.4 fully updated centos qemu/libvirt versions don't support glusterfs storage domain yet. 6.5 would be needed. we're trying to find a solution until then. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Fri, Oct 4, 2013 at 11:39 AM, Sven Kieske wrote: > Hi, > > if I understand Gianluca correct he has setup the ManagementNode and > 2 ComputeNodes with Fedora 19. > CentOS 6.4 is the VM-OS (inside the VM). > > So this should work? > Yes, this is my environment with plain F19 systems installed and updated as of yesterday and ovirt repo stable enabled. CentOS 6.4 x86_64 was configured as a server in oVirt, but wanting to generate some significant I/O during install to test Gluster efficiency I choosed Gnome Desktop in CentOS anaconda phase. This led to 1089 packages installed. After reboot I ran "yum update" on the VM, enabled epel and installed the new ovirt-guest-agent so that I was correctly able to see ip and kernel of the VM from the webadmin GUI. But then I incurred in these problems with preliminary tests, donna if correlated or not: - migration issue - shutdown (so power off ) and power on error see: https://bugzilla.redhat.com/show_bug.cgi?id=1007980 that coud be related to both of them - gluster split brain Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
Hi, if I understand Gianluca correct he has setup the ManagementNode and 2 ComputeNodes with Fedora 19. CentOS 6.4 is the VM-OS (inside the VM). So this should work? HTH Sven On 04/10/13 11:29, Itamar Heim wrote: >> >> Me too with oVirt 3.3 setup and GlusterFS DataCenter. >> One dedicated engine + 2 vdsm hosts. All fedora 19 + ovirt stable. >> My trying to migrate VM is CentOS 6.4 fully updated > > centos qemu/libvirt versions don't support glusterfs storage domain yet. > 6.5 would be needed. we're trying to find a solution until then. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On 10/03/2013 06:12 PM, Gianluca Cecchi wrote:
On Wed, Oct 2, 2013 at 12:07 AM, Jason Brooks wrote:
I'm having this issue on my ovirt 3.3 setup (two node, one is AIO,
GlusterFS storage, both on F19) as well.
Jason
Me too with oVirt 3.3 setup and GlusterFS DataCenter.
One dedicated engine + 2 vdsm hosts. All fedora 19 + ovirt stable.
My trying to migrate VM is CentOS 6.4 fully updated
centos qemu/libvirt versions don't support glusterfs storage domain yet.
6.5 would be needed. we're trying to find a solution until then.
After trying to migrate in webadmin gui I get:
VM c6s is down. Exit message: 'iface'.
For a few moments the VM appears as down in the GUI but actually an
ssh session I had before is still alive.
Also the qemu process on source host is still there.
After a little, the VM results again as "Up" in the source node from the gui
Actually it never stopped:
[root@c6s ~]# uptime
16:36:56 up 19 min, 1 user, load average: 0.00, 0.00, 0.00
At target many errors such as
Thread-8609::ERROR::2013-10-03
16:17:13,086::task::850::TaskManager.Task::(_setError)
Task=`a102541e-fbe5-46a3-958c-e5f4026cac8c`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 857, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2123, in getAllTasksStatuses
allTasksStatus = sp.getAllTasksStatuses()
File "/usr/share/vdsm/storage/securable.py", line 66, in wrapper
raise SecureError()
SecureError
But relevant perhaps is around the time of migration (16:33-16:34)
Thread-9968::DEBUG::2013-10-03
16:33:38,811::task::1168::TaskManager.Task::(prepare)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::finished: {'taskStatus':
{'code': 0, 'message': 'Task is initializing', 'taskState': 'running',
'taskResult': '', 'taskID': '0eaac2f3-3d25-4c8c-9738-708aba290404'}}
Thread-9968::DEBUG::2013-10-03
16:33:38,811::task::579::TaskManager.Task::(_updateState)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::moving from state
preparing -> state finished
Thread-9968::DEBUG::2013-10-03
16:33:38,811::resourceManager::939::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-9968::DEBUG::2013-10-03
16:33:38,812::resourceManager::976::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-9968::DEBUG::2013-10-03
16:33:38,812::task::974::TaskManager.Task::(_decref)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::ref 0 aborting False
0eaac2f3-3d25-4c8c-9738-708aba290404::ERROR::2013-10-03
16:33:38,847::task::850::TaskManager.Task::(_setError)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 857, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 318, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/sp.py", line 272, in startSpm
self.masterDomain.acquireHostId(self.id)
File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
self._clusterLock.acquireHostId(hostId, async)
File "/usr/share/vdsm/storage/clusterlock.py", line 189, in acquireHostId
raise se.AcquireHostIdFailure(self._sdUUID, e)
AcquireHostIdFailure: Cannot acquire host id:
('d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', SanlockException(5, 'Sanlock
lockspace add failure', 'Input/output error'))
0eaac2f3-3d25-4c8c-9738-708aba290404::DEBUG::2013-10-03
16:33:38,847::task::869::TaskManager.Task::(_run)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::Task._run:
0eaac2f3-3d25-4c8c-9738-708aba290404 () {} failed - stopping task
0eaac2f3-3d25-4c8c-9738-708aba290404::DEBUG::2013-10-03
16:33:38,847::task::1194::TaskManager.Task::(stop)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::stopping in state running
(force False)
Instead at source:
Thread-10402::ERROR::2013-10-03
16:35:03,713::vm::244::vm.Vm::(_recover)
vmId=`4147e0d3-19a7-447b-9d88-2ff19365bec0`::migration destination
error: Error creating the requested VM
Thread-10402::ERROR::2013-10-03 16:35:03,740::vm::324::vm.Vm::(run)
vmId=`4147e0d3-19a7-447b-9d88-2ff19365bec0`::Failed to migrate
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 311, in run
self._startUnderlyingMigration()
File "/usr/share/vdsm/vm.py", line 347, in _startUnderlyingMigration
response['status']['message'])
RuntimeError: migration destination error: Error creating the requested VM
Thread-1161::DEBUG::2013-10-03
16:35:04,243::fileSD::238::Storage.Misc.excCmd::(getReadDelay)
'/usr/bin/dd iflag=direct
if=/rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/metadata
bs=4096 count=1' (cwd None)
Thread-1161::DEBUG::2013-10-03
16:35:04,262::fileSD::238::Storage.Misc.excCmd::(getReadDelay)
SUCCESS: = '0+1 records in\n0+1 records out\n512 bytes (512 B)
copied, 0.0015976 s
Re: [Users] Migration issues with ovirt 3.3
On Wed, Oct 2, 2013 at 12:07 AM, Jason Brooks wrote: > I'm having this issue on my ovirt 3.3 setup (two node, one is AIO, > GlusterFS storage, both on F19) as well. > > Jason I uploaded all my day logs to bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1007980 I can reproduce the problem and the migrate phase is what generates apparently files to heal in gluster volume VM is running on f18ovn03 before migrate [root@f18ovn01 vdsm]# gluster volume heal gvdata info Gathering Heal info on volume gvdata has been successful Brick 10.4.4.58:/gluster/DATA_GLUSTER/brick1 Number of entries: 0 Brick 10.4.4.59:/gluster/DATA_GLUSTER/brick1 Number of entries: 0 Start migrate that fails with the "iface" error And now you see: [root@f18ovn01 vdsm]# gluster volume heal gvdata info Gathering Heal info on volume gvdata has been successful Brick 10.4.4.58:/gluster/DATA_GLUSTER/brick1 Number of entries: 1 /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/ids Brick 10.4.4.59:/gluster/DATA_GLUSTER/brick1 Number of entries: 1 /d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/ids While on f18ovn03: [root@f18ovn03 vdsm]# gluster volume heal gvdata info Gathering Heal info on volume gvdata has been successful Brick 10.4.4.58:/gluster/DATA_GLUSTER/brick1 Number of entries: 0 Brick 10.4.4.59:/gluster/DATA_GLUSTER/brick1 Number of entries: 0 Gianluca ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Wed, Oct 2, 2013 at 12:07 AM, Jason Brooks wrote:
> I'm having this issue on my ovirt 3.3 setup (two node, one is AIO,
> GlusterFS storage, both on F19) as well.
>
> Jason
Me too with oVirt 3.3 setup and GlusterFS DataCenter.
One dedicated engine + 2 vdsm hosts. All fedora 19 + ovirt stable.
My trying to migrate VM is CentOS 6.4 fully updated
After trying to migrate in webadmin gui I get:
VM c6s is down. Exit message: 'iface'.
For a few moments the VM appears as down in the GUI but actually an
ssh session I had before is still alive.
Also the qemu process on source host is still there.
After a little, the VM results again as "Up" in the source node from the gui
Actually it never stopped:
[root@c6s ~]# uptime
16:36:56 up 19 min, 1 user, load average: 0.00, 0.00, 0.00
At target many errors such as
Thread-8609::ERROR::2013-10-03
16:17:13,086::task::850::TaskManager.Task::(_setError)
Task=`a102541e-fbe5-46a3-958c-e5f4026cac8c`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 857, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2123, in getAllTasksStatuses
allTasksStatus = sp.getAllTasksStatuses()
File "/usr/share/vdsm/storage/securable.py", line 66, in wrapper
raise SecureError()
SecureError
But relevant perhaps is around the time of migration (16:33-16:34)
Thread-9968::DEBUG::2013-10-03
16:33:38,811::task::1168::TaskManager.Task::(prepare)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::finished: {'taskStatus':
{'code': 0, 'message': 'Task is initializing', 'taskState': 'running',
'taskResult': '', 'taskID': '0eaac2f3-3d25-4c8c-9738-708aba290404'}}
Thread-9968::DEBUG::2013-10-03
16:33:38,811::task::579::TaskManager.Task::(_updateState)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::moving from state
preparing -> state finished
Thread-9968::DEBUG::2013-10-03
16:33:38,811::resourceManager::939::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-9968::DEBUG::2013-10-03
16:33:38,812::resourceManager::976::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-9968::DEBUG::2013-10-03
16:33:38,812::task::974::TaskManager.Task::(_decref)
Task=`6c1d3161-edcd-4344-a32a-4a18f75f5ba3`::ref 0 aborting False
0eaac2f3-3d25-4c8c-9738-708aba290404::ERROR::2013-10-03
16:33:38,847::task::850::TaskManager.Task::(_setError)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 857, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 318, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/sp.py", line 272, in startSpm
self.masterDomain.acquireHostId(self.id)
File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
self._clusterLock.acquireHostId(hostId, async)
File "/usr/share/vdsm/storage/clusterlock.py", line 189, in acquireHostId
raise se.AcquireHostIdFailure(self._sdUUID, e)
AcquireHostIdFailure: Cannot acquire host id:
('d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291', SanlockException(5, 'Sanlock
lockspace add failure', 'Input/output error'))
0eaac2f3-3d25-4c8c-9738-708aba290404::DEBUG::2013-10-03
16:33:38,847::task::869::TaskManager.Task::(_run)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::Task._run:
0eaac2f3-3d25-4c8c-9738-708aba290404 () {} failed - stopping task
0eaac2f3-3d25-4c8c-9738-708aba290404::DEBUG::2013-10-03
16:33:38,847::task::1194::TaskManager.Task::(stop)
Task=`0eaac2f3-3d25-4c8c-9738-708aba290404`::stopping in state running
(force False)
Instead at source:
Thread-10402::ERROR::2013-10-03
16:35:03,713::vm::244::vm.Vm::(_recover)
vmId=`4147e0d3-19a7-447b-9d88-2ff19365bec0`::migration destination
error: Error creating the requested VM
Thread-10402::ERROR::2013-10-03 16:35:03,740::vm::324::vm.Vm::(run)
vmId=`4147e0d3-19a7-447b-9d88-2ff19365bec0`::Failed to migrate
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 311, in run
self._startUnderlyingMigration()
File "/usr/share/vdsm/vm.py", line 347, in _startUnderlyingMigration
response['status']['message'])
RuntimeError: migration destination error: Error creating the requested VM
Thread-1161::DEBUG::2013-10-03
16:35:04,243::fileSD::238::Storage.Misc.excCmd::(getReadDelay)
'/usr/bin/dd iflag=direct
if=/rhev/data-center/mnt/glusterSD/f18ovn01.mydomain:gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/dom_md/metadata
bs=4096 count=1' (cwd None)
Thread-1161::DEBUG::2013-10-03
16:35:04,262::fileSD::238::Storage.Misc.excCmd::(getReadDelay)
SUCCESS: = '0+1 records in\n0+1 records out\n512 bytes (512 B)
copied, 0.0015976 s, 320 kB/s\n'; = 0
Thread-1161::INFO::2013-10-03
16:35:04,269::clusterlock::174::SANLock::(acquireHostId) Acquiring
host id for domain d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291 (id: 2)
Thread-1161::DEBUG::2013-10-03
Re: [Users] Migration issues with ovirt 3.3
I'm having this issue on my ovirt 3.3 setup (two node, one is AIO, GlusterFS storage, both on F19) as well. Jason On Tue, 2013-10-01 at 21:54 +0100, Dan Kenigsberg wrote: > On Tue, Oct 01, 2013 at 02:35:34PM -0300, emi...@gmail.com wrote: > > Dan, which are the path to those files that you're asking to me? > > > > Now the debian vm it's not working. After try the migration of the Windows > > vm, the live migration it's not working in the debian. > > The fact that now both VMs behave the same makes comparing them > fruitless. Still it may be intersting to see your vdsm.log (from source > and destination) since the first time a faulty VM was started until the > 'iface' exception. > > > > > Do you think that this issue it's related to the hosts or with the manager? > > It smells very much Vdsm-specific, but I'm only guessing. > > > i really would like to get this working and to other user this is working > > fine. Do you think if that i make a whole new install maybe this start to > > work? > > I do not suppose this would help, but I'm not in the position to exclude > exploration routes. > > > > > I'm going to test what Michal say right now with the same host, but i think > > that's going to happen the same thing. > > > > Regards! > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Tue, Oct 01, 2013 at 02:35:34PM -0300, emi...@gmail.com wrote: > Dan, which are the path to those files that you're asking to me? > > Now the debian vm it's not working. After try the migration of the Windows > vm, the live migration it's not working in the debian. The fact that now both VMs behave the same makes comparing them fruitless. Still it may be intersting to see your vdsm.log (from source and destination) since the first time a faulty VM was started until the 'iface' exception. > > Do you think that this issue it's related to the hosts or with the manager? It smells very much Vdsm-specific, but I'm only guessing. > i really would like to get this working and to other user this is working > fine. Do you think if that i make a whole new install maybe this start to > work? I do not suppose this would help, but I'm not in the position to exclude exploration routes. > > I'm going to test what Michal say right now with the same host, but i think > that's going to happen the same thing. > > Regards! ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
Dan, which are the path to those files that you're asking to me? Now the debian vm it's not working. After try the migration of the Windows vm, the live migration it's not working in the debian. Do you think that this issue it's related to the hosts or with the manager? i really would like to get this working and to other user this is working fine. Do you think if that i make a whole new install maybe this start to work? I'm going to test what Michal say right now with the same host, but i think that's going to happen the same thing. Regards! 2013/9/30 Michal Skrivanek > > On Sep 29, 2013, at 14:15 , Dan Kenigsberg wrote: > > > On Fri, Sep 27, 2013 at 03:36:22PM -0300, emi...@gmail.com wrote: > >> So Dan, could I do some other test for you?, The live migration it's > only > >> working with a Debian install, I've just finished to install a windows > >> server 2K8 and it doesn't work neither, gives the following error "VM > W2K8 > >> is down. Exit message: 'iface'." > > Do you have any non-gluster setup to give it a try? > We have this bug 1007980 which also seems to demonstrate in gluster-setup > only….don't know why yet, but perhaps this can be related…. > > Thanks, > michal > > > > > Frankly, I am out of ideas of what could be causing this very important > > bug :-( > > > > It is unrelated to hotplugging, and probably not to migration - > > something bad happens when the VM parameters are persisted to disk, or > > loaded from it. > > > > I do not see how the type of the guest could matter. Could you tell if > > there is any other difference betrween your W2K8 VM and your Debian one? > > > > Could you post here the first vmCreate command of the two VMs, and their > > libvirt domxml? > > > > Regards, > > Dan. > > > > > > -- *Emiliano Tortorella* +598 98941176 emi...@gmail.com ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Sep 29, 2013, at 14:15 , Dan Kenigsberg wrote: > On Fri, Sep 27, 2013 at 03:36:22PM -0300, emi...@gmail.com wrote: >> So Dan, could I do some other test for you?, The live migration it's only >> working with a Debian install, I've just finished to install a windows >> server 2K8 and it doesn't work neither, gives the following error "VM W2K8 >> is down. Exit message: 'iface'." Do you have any non-gluster setup to give it a try? We have this bug 1007980 which also seems to demonstrate in gluster-setup only….don't know why yet, but perhaps this can be related…. Thanks, michal > > Frankly, I am out of ideas of what could be causing this very important > bug :-( > > It is unrelated to hotplugging, and probably not to migration - > something bad happens when the VM parameters are persisted to disk, or > loaded from it. > > I do not see how the type of the guest could matter. Could you tell if > there is any other difference betrween your W2K8 VM and your Debian one? > > Could you post here the first vmCreate command of the two VMs, and their > libvirt domxml? > > Regards, > Dan. > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Fri, Sep 27, 2013 at 03:36:22PM -0300, emi...@gmail.com wrote: > So Dan, could I do some other test for you?, The live migration it's only > working with a Debian install, I've just finished to install a windows > server 2K8 and it doesn't work neither, gives the following error "VM W2K8 > is down. Exit message: 'iface'." Frankly, I am out of ideas of what could be causing this very important bug :-( It is unrelated to hotplugging, and probably not to migration - something bad happens when the VM parameters are persisted to disk, or loaded from it. I do not see how the type of the guest could matter. Could you tell if there is any other difference betrween your W2K8 VM and your Debian one? Could you post here the first vmCreate command of the two VMs, and their libvirt domxml? Regards, Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
So Dan, could I do some other test for you?, The live migration it's only working with a Debian install, I've just finished to install a windows server 2K8 and it doesn't work neither, gives the following error "VM W2K8 is down. Exit message: 'iface'." Regards! 2013/9/25 emi...@gmail.com > I've just tried to do a migration without vdsm restart and it works with > the debian-test2 VM. > > I've also tried with the pfsense1 VM but it doesn't, might it be because > this vm has an IDE type disk? > > Here goes the logs! > > > 2013/9/25 emi...@gmail.com > > No, it doesn't succeed neither. >> >> I've also noticed that in the Gluster Swift secction it appears to me >> everything as Not installed, except the memcached that appears as Down, >> could this has something to do with all these? >> >> The private chat was my bad I think, I've only answered to you in some >> point, sorry! >> >> Thanks! >> >> >> >> 2013/9/25 Dan Kenigsberg >> >>> On Wed, Sep 25, 2013 at 10:40:30AM -0300, emi...@gmail.com wrote: >>> > I've created a new debian VM. I've restarted the vdsm on the source and >>> > destination host and after try the migration now I received the >>> following >>> > message: >>> > >>> > Migration failed due to Error: Fatal error during migration (VM: >>> > debian-test2, Source: ovirt1, Destination: ovirt2). >>> > >>> > Do you think that with a clean instalation of the host could this be >>> solved? >>> No. >>> >>> We have a real bug on re-connecting to VMs on startup. >>> >>> But why are we in a private chat? Please report it as a reply to the >>> mailing list thread. >>> >>> But let me verify: if you start a VM and then migrate (no vdsm restart), >>> does migration succede? >>> >>> Dan. >>> >> >> >> >> -- >> *Emiliano Tortorella* >> +598 98941176 >> emi...@gmail.com >> > > > > -- > *Emiliano Tortorella* > +598 98941176 > emi...@gmail.com > -- *Emiliano Tortorella* +598 98941176 emi...@gmail.com ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
No, it doesn't succeed neither. I've also noticed that in the Gluster Swift secction it appears to me everything as Not installed, except the memcached that appears as Down, could this has something to do with all these? The private chat was my bad I think, I've only answered to you in some point, sorry! Thanks! 2013/9/25 Dan Kenigsberg > On Wed, Sep 25, 2013 at 10:40:30AM -0300, emi...@gmail.com wrote: > > I've created a new debian VM. I've restarted the vdsm on the source and > > destination host and after try the migration now I received the following > > message: > > > > Migration failed due to Error: Fatal error during migration (VM: > > debian-test2, Source: ovirt1, Destination: ovirt2). > > > > Do you think that with a clean instalation of the host could this be > solved? > No. > > We have a real bug on re-connecting to VMs on startup. > > But why are we in a private chat? Please report it as a reply to the > mailing list thread. > > But let me verify: if you start a VM and then migrate (no vdsm restart), > does migration succede? > > Dan. > -- *Emiliano Tortorella* +598 98941176 emi...@gmail.com ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Tue, Sep 24, 2013 at 02:41:58PM -0300, emi...@gmail.com wrote: > Thanks for your answer Dan! > > Yesterday was talking with an user in the IRC and gave me the hint to > upgrade the libvirt to the 1.1.2 after trying in his implementation the > live migration successfully. > > I've upgraded the libvirt but I'm still having the issue. I send to you the > logs that you asked to me and the information bellow: > OS Version: > Fedora - 19 - 3 > Kernel Version: > 3.11.1 - 200.fc19.x86_64 > KVM Version: > 1.4.2 - 9.fc19 > LIBVIRT Version: > libvirt-1.1.2-1.fc19 > VDSM Version: > vdsm-4.12.1-2.fc19 > SPICE Version: > 0.12.4 - 1.fc19 > iSCSI Initiator Name: > iqn.1994-05.com.redhat:d990cf85cdeb > SPM Priority: > Medium > Active VMs: > 1 > CPU Name: > Intel Westmere Family > CPU Type: > Intel(R) Xeon(R) CPU E5620 @ 2.40GHz > CPU Sockets: > 1 > CPU Cores per Socket: > 4 > CPU Threads per Core: > 2 (SMT Enabled) > Physical Memory: > 12007 MB total, 2762 MB used, 9245 MB free > Swap Size: > 15999 MB total, 0 MB used, 15999 MB free > Shared Memory: > 0% > Max free Memory for scheduling new VMs: > 15511.5 MB > Memory Page Sharing: > Inactive > Automatic Large Pages: > Always > > (Both hypervisors have the same hardware and software version) > > I'm going to keep trying some things becouse something must get messed up > because now i have a VM with Debian that doesn't start giving me the error > "Failed to run VM debian on Host ovirt1." and "Failed to run VM debian on > Host ovirt2." > > Anyway I'll wait for your answer. > Very Regards! > Emiliano Your destination Vdsm has vmId=`1f7e60c7-51cb-469a-8016-58a5837f3316`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 2022, in _startUnderlyingVm self._run() File "/usr/share/vdsm/vm.py", line 2819, in _run devices = self.buildConfDevices() File "/usr/share/vdsm/vm.py", line 1839, in buildConfDevices devices = self.getConfDevices() File "/usr/share/vdsm/vm.py", line 1806, in getConfDevices self.normalizeDrivesIndices(devices[DISK_DEVICES]) File "/usr/share/vdsm/vm.py", line 1990, in normalizeDrivesIndices if drv['iface'] not in self._usedIndices: KeyError: 'iface' Which looks just like Bug 1011472 - [vdsm] cannot recover VM upon vdsm restart after a disk has been hot plugged to it. Could it be that you have hot-plugged a disk to your VM at the source host? Somehow, Vdsm forgets to keep the 'iface' element passed from Engine for the hot-plugged disk. Dan. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Migration issues with ovirt 3.3
On Mon, Sep 23, 2013 at 12:15:04PM -0300, emi...@gmail.com wrote: > Hi, > > I'm running ovirt-engine 3.3 in a server with fedora 19, also two host with > fedora 19 running vdsm and gluster. I'm using the repositories like it > say's here: http://www.ovirt.org/OVirt_3.3_TestDay with enable the > [ovirt-beta] & [ovirt-stable] repos and disable the [ovirt-nightly] repo. > > I've configured a datacenter with glusterfs active an the two Host. I've > installed a VM and when I do a migration it fails with the message *"VM > pfSense1 is down. Exit message: 'iface'."* and the VM reboots. Could you sare vdsm.log from source and destination? > Also if i > try to make an snapshot the VM with the Save memory it fails with the > message *"VM pfSense1 is down. Exit message: Lost connection with qemu > process."* If I make an snapshot without the Save Memory checked it works. Here, beyond vdsm.log, I would ask to see /etc/libvirt/qemu/pfSense1.log to understand why qemu crashed. Please report the version of your qemu-kcm and kernel. > > I've tried to restart the libvirtd service but it's still happening. Would you make sure you upgrade to newest libvirt? We had some annoying bugs resolved by a recent upgrade there. > > Before this I've tried the cluster with NFS storage and have problems with > migration too, but the error messages were differents. Now I'm trying with > gluster because this is what i want to use. > > Could you give me any hint about this? ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
