Bug#441249: xen-hypervisor-3.0.3-1-i386-pae: "Problems using XEN when Quagga is running"
Package: xen-hypervisor-3.0.3-1-i386-pae
Version: 3.0.3-0-2
Severity: normal
If quagga is running, and I start & stop a domain, the VIF interface doesn't go
down properly (in addition to other problems).
Here's what I do:
1. Boot the computer. By default, quagga is running (zebra + ospfd).
2. xm create test7.cfg, wait until the domU starts.
3. xm shutdown test7, wait until the domU stops. (I also tried halt from
within
domU)
4. xm list - gives me "Device 0 not connected" error after shutdown is finished
5. "ifconfig vifX.0" still shows the interface.
6. xm create test7.cfg --> I get the "Hotplug scripts not working" error
message
7. /etc/init.d/quagga restart
8. "ifconfig vifX.0" doesn't show the interface anymore
9. xm create test7.cfg now works as expected.
This might also be a bug of the quagga package.
See below to see the full attempt log:
<-
xen1:~# uname -a
Linux xen1 2.6.18-5-xen-vserver-686 #1 SMP Thu Aug 30 06:32:53 UTC 2007 i686
GNU/Linux
xen1:~# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1892 2 r- 43.5
xen1:~# xm create test7.cfg
Using config file "/etc/xen/test7.cfg".
Started domain test7
xen1:~# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1892 2 r- 44.8
test7 12 128 1 -b 2.8
xen1:~# xm shutdown test7
xen1:~# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1892 2 r- 45.0
test7 12 128 1 -b 4.4
xen1:~# xm list
Error: Device 0 not connected
Usage: xm list [options] [Domain, ...]
List information about all/some domains.
-l, --long Output all VM details in SXP
--labelInclude security labels
xen1:~# ifconfig vif12.0
vif12.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet addr:AAA.BBB.CCC.DDD Bcast:AAA.BBB.CCC.DDD
Mask:255.255.255.224
inet6 addr: fe80::fcff::feff:/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:7 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:412 (412.0 b) TX bytes:1284 (1.2 KiB)
xen1:~# xm create test7.cfg
Using config file "/etc/xen/test7.cfg".
Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
xen1:~# /etc/init.d/quagga restart
Stopping Quagga daemons (prio:0): (waiting) .. ospfd (waiting) .. zebra (bgpd)
(ripd) (ripngd) (ospf6d) (isisd).
Removing all routes made by zebra.
Nothing to flush.
Loading capability module if not yet done.
Starting Quagga daemons (prio:10): zebra ospfd.
xen1:~# ifconfig vif12.0
vif12.0: error fetching interface information: Device not found
xen1:~# xm create test7.cfg
Using config file "/etc/xen/test7.cfg".
Started domain test7
xen1:~#
->
Logs:
<-
xen1:~# xm shutdown test7
xen1:~# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1892 2 r- 47.7
test7 14 128 1 r- 5.2
xen1:~# xm list
Error: Device 0 not connected
Usage: xm list [options] [Domain, ...]
List information about all/some domains.
-l, --long Output all VM details in SXP
--labelInclude security labels
xen1:~# tail -60 /var/log/xen/xend.log
[2007-09-08 00:37:40 xend 8997] DEBUG (__init__:1072) DevController: writing
{'domain': 'test7', 'handle': '0', 'script': '/etc/xen/scripts/vif-route',
'ip': '172.16.255.44', 'state': '1', 'frontend':
'/local/domain/13/device/vif/0', 'mac': '00:16:3e:37:4f:f3', 'online': '1',
'frontend-id': '13'} to /local/domain/0/backend/vif/13/0.
[2007-09-08 00:37:40 xend.XendDomainInfo 8997] DEBUG (__init__:1072) Storing VM
details: {'shadow_memory': '0', 'uuid': '31cc83aa-a269-571e-5c9b-b33d239df2e3',
'on_reboot': 'restart', 'start_time': '1189201060.77', 'on_poweroff':
'destroy',
'name': 'test7', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1',
'memory': '128', 'on_crash': 'restart', 'image': "(linux (kernel
/boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) (root '/dev/hda1 ro'))",
'maxmem': '128'}
[2007-09-08 00:37:40 xend.XendDomainInfo 8997] DEBUG (__init__:1072) Storing
domain details: {'console/ring-ref': '268947', 'console/port': '2', 'name':
'test7', 'console/limit': '1048576', 'vm':
'/vm/31cc83aa-a269-571e-5c9b-b33d239df2e3', 'domid': '13',
'cpu/0/availability': 'online', 'memory/target': '131072', 'store/ring-ref':
'268948', 'store/port': '1'}
[2007-09-08 00:37:40 xend.XendDomainInfo 8997] DEBUG (__init__:1072)
XendDomainInfo.handleShutdownWatch
[2007-09-08 00:3
Bug#441249: [Pkg-xen-devel] Bug#441249: xen-hypervisor-3.0.3-1-i386-pae: "Problems using XEN when Quagga is running"
Hi, And thanks for the quick answer. First of all I discoved a very similar bug report in CentOS: http://bugs.centos.org/view.php?id=2300 Guido Trotter kirjoitti: > Are you using the standard vif-route scripts or have you customized them? > Yes, I have customized my vif-route script. I just tried with the original vif-route script, and same happened. > Can you show us your quagga configuration too? > I've changed the hostnames and passwords, but here they are: <- xen1:~# cat /etc/quagga/ospfd.conf hostname xen1.XXX.XXX password XXX enable password XXX log file /var/log/ospfd.log interface dummy0 interface eth0 interface dummy2 router ospf ospf router-id 172.16.255.16 ospf abr-type standard ospf rfc1583compatibility network 172.16.255.16/32 area 0.0.0.0 network 172.16.255.11/28 area 0.0.0.0 passive-interface dummy0 passive-interface dummy2 redistribute static area 0.0.0.0 authentication xen1:~# cat /etc/quagga/zebra.conf log file /var/log/quagga/zebra.log hostname xen1.XXX.XXX xen1:~# -> > Can you manually do on the xen interfaces what the scripts would? How about > doing it on some other interface configured in a similar way? > I'll try this next week, see below :) > Can you try to insulate it within quagga? Does this happen if you just run > zebra > but not ospfd? > Wow... I disabled ospfd, /etc/init.d/quagga restart, and creating a domain crashed xen. Here's what happened. <- xen1:~# pico /etc/quagga/daemons xen1:~# /etc/init.d/quagga restart Stopping Quagga daemons (prio:0): (zebra) (bgpd) (ripd) (ripngd) (ospfd) (ospf6d ) (isisd). Removing all routes made by zebra. Nothing to flush. Loading capability module if not yet done. Starting Quagga daemons (prio:10): zebra. xen1:~# xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 1892 2 r-193.1 xen1:~# xm create test7.cfg Using config file "/etc/xen/test7.cfg". Started domain test7 xen1:~# Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: [ cut here ] Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: kernel BUG at drivers/xen/core/evtchn.c:481! Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: invalid opcode: [#1] Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: SMP Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: CPU:1 Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: EIP is at retrigger+0x1f/0x35 Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: eax: ebx: 0208 ecx: 0018 edx: f55f6000 Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: esi: c0324460 edi: 010a ebp: esp: ed78deb0 Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: ds: 007b es: 007b ss: 0069 Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: Process xenwatch (pid: 11[#0], ti=ed78c000 task=c03a50d0 task.ti=ed 78c000) Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: Stack: c01418b1 c0324460 010a c0324488 c0141607 e6b6a2c0 00 00 Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel:e6b6a2c0 c022077c c0220b5c c0219ceb 0010 00 00 020b Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel:020a e74703b6 c02f10e4 ee868000 00 00 0002 Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: Call Trace: Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: Code: ee 85 f6 75 96 58 5a 5b 5e 5f 5d c3 0f b7 0c 85 00 89 38 c0 8 b 15 a4 b9 2d c0 85 c9 74 1d 0f a3 8a 80 08 00 00 19 c0 85 c0 75 08 <0f> 0b e1 0 1 ce c2 2b c0 f0 0f ab 8a 00 08 00 00 b8 01 00 00 00 Message from [EMAIL PROTECTED] at Sun Sep 9 16:33:46 2007 ... xen1 kernel: EIP: [] retrigger+0x1f/0x35 SS:ESP 0069:ed78deb0 xen1:~# -> The hex codes got a bit mangled because they didn't fit in my putty window. The computer (which is at work) doesn't respond to ping anymore, and I'm at home currently, so I'll get back to this next week. I appreciate the help. Regards, Mikko -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a
Bug#441249: [Pkg-xen-devel] Bug#441249: xen-hypervisor-3.0.3-1-i386-pae: "Problems using XEN when Quagga is running"
Hi,
> Can you manually do on the xen interfaces what the scripts would? How about
> doing it on some other interface configured in a similar way?
Toying with the vif-route script, I might have found a workaround for this
issue.
If I disable the ifconfig and ip route commands from vif-route script, and bring
up vif interface by hand later on, everything seems to work.
In other works, bringing up the interface in vif-route script causes it to stay
up when I shutdown the domain.
I'm not sure though if the workaround is stable, as I again experienced a crash
like before ("kernel BUG at drivers/xen/core/evtchn.c:481!") when doing a lot
of create/shutdown commands, manually bringing up interfaces, etc. I haven't
found a reliable way to reproduce the crash yet.
I need to do more testing before I can say for sure.
What do you think of this?
> Can you try to insulate it within quagga? Does this happen if you just run
> zebra but not ospfd?
If only zebra is running - still a jammed VIF interface.
If only ospfd is running, it actually fixes the problem. (too bad ospfd is
useless without the zebra daemon)
By the way, if you want I could arrange you an SSH access to the xen1 host.
Best regards,
Mikko Korkalo
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#441249: [Pkg-xen-devel] Bug#441249: xen-hypervisor-3.0.3-1-i386-pae: "Problems using XEN when Quagga is running"
ip} netmask 255.255.255.255 up
#echo 1 >/proc/sys/net/ipv4/conf/${vif}/proxy_arp
ipcmd='add'
cmdprefix=''
;;
offline)
do_without_error ifdown ${vif}
ipcmd='del'
cmdprefix='do_without_error'
;;
esac
#if [ "${ip}" ] ; then
## If we've been given a list of IP addresses, then add routes from
dom0 to
## the guest using those addresses.
#for addr in ${ip} ; do
# ${cmdprefix} ip route ${ipcmd} ${addr} dev ${vif} src ${main_ip}
#done
#fi
#
#handle_iptable
log debug "Successful vif-route $command for $vif."
if [ "$command" == "online" ]
then
success
fi
xen1:/etc/xen/scripts# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1891 2 r-203.2
xen1:/etc/xen/scripts# xm create test7.cfg
Using config file "/etc/xen/test7.cfg".
Started domain test7
xen1:/etc/xen/scripts# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1891 2 r-204.5
test7 18 128 1 -b 2.8
xen1:/etc/xen/scripts# ip link|grep vif
23: vif18.0: mtu 1500 qdisc noop
xen1:/etc/xen/scripts# ifconfig vif18.0 172.16.255.11 netmask
255.255.255.255 up
xen1:/etc/xen/scripts# echo 1 >/proc/sys/net/ipv4/conf/vif18.0/proxy_arp
xen1:/etc/xen/scripts# ip route add 172.16.255.44 dev vif18.0
xen1:/etc/xen/scripts# ping 172.16.255.44 -c 1
PING 172.16.255.44 (172.16.255.44) 56(84) bytes of data.
64 bytes from 172.16.255.44: icmp_seq=1 ttl=64 time=782 ms
--- 172.16.255.44 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 782.787/782.787/782.787/0.000 ms
xen1:/etc/xen/scripts# ping 172.16.255.44 -c 1
PING 172.16.255.44 (172.16.255.44) 56(84) bytes of data.
64 bytes from 172.16.255.44: icmp_seq=1 ttl=64 time=0.037 ms
--- 172.16.255.44 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.037/0.037/0.037/0.000 ms
xen1:/etc/xen/scripts# xm shutdown test7
xen1:/etc/xen/scripts# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1891 2 r-205.4
xen1:/etc/xen/scripts# ip link|grep vif
xen1:/etc/xen/scripts#
-->
Oddly enough, if I run these commands from within the vif-route script
in background, with a 10 second delay before they are executed, I get
the same problem (as if something in the environment was different when
running them by hand).
Best Regards,
Mikko Korkalo
Guido Trotter kirjoitti:
> On Tue, Sep 11, 2007 at 03:01:57PM +0300, Mikko Korkalo wrote:
>
> Hi,
>
>
>>> Can you manually do on the xen interfaces what the scripts would? How about
>>> doing it on some other interface configured in a similar way?
>>>
>> Toying with the vif-route script, I might have found a workaround for this
>> issue.
>> If I disable the ifconfig and ip route commands from vif-route script, and
>> bring
>> up vif interface by hand later on, everything seems to work.
>>
>> In other works, bringing up the interface in vif-route script causes it to
>> stay
>> up when I shutdown the domain.
>>
>>
>
> This is strange, as vif-route should just be doing what you're doing manually
> later... Are you sure you're then doing exactly the same steps?
>
>
>> I'm not sure though if the workaround is stable, as I again experienced a
>> crash
>> like before ("kernel BUG at drivers/xen/core/evtchn.c:481!") when doing a lot
>> of create/shutdown commands, manually bringing up interfaces, etc. I haven't
>> found a reliable way to reproduce the crash yet.
>>
>>
>
> This is an unrelated problem... It depends from the xen patch you're using and
> should disappear if you update it!
>
>
>> If only zebra is running - still a jammed VIF interface.
>> If only ospfd is running, it actually fixes the problem. (too bad ospfd is
>> useless without the zebra daemon)
>>
>>
>
> Thanks!
>
> Guido
>
>
>
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#441249: [Pkg-xen-devel] Bug#441249: xen-hypervisor-3.0.3-1-i386-pae: "Problems using XEN when Quagga is running"
Hello, I disabled ipv6: no more problems, whatever vif-script I use. I don't need ipv6 so this workaround works fine for me. This issue is maybe unrelated to Xen. I would need to do more testing to be sure. I hope this will save some time from someone else getting the same problem. :) Here's what I did: <- xen1:~# pico /etc/modprobe.d/00disable-ipv6 xen1:~# cat /etc/modprobe.d/00disable-ipv6 alias net-pf-10 off alias ipv6 off xen1:~# reboot ... xen1:~# cp /etc/xen/scripts/vif-route.orig /etc/xen/scripts/vif-route xen1:~# xm create test7.cfg Using config file "/etc/xen/test7.cfg". Started domain test7 xen1:~# xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 1892 2 r- 9.8 test7 1 128 1 -b 2.8 xen1:~# xm shutdown test7 xen1:~# xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 1892 2 r- 10.6 xen1:~# xm create test7.cfg Using config file "/etc/xen/test7.cfg". Started domain test7 xen1:~# -----> Best regards, Mikko Korkalo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
