Bug#574358: [Pkg-libvirt-maintainers] Bug#574358: libvirt-bin: libvirtd fails to autostart iscsi pool at boot time
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido Günther wrote: On Fri, Mar 19, 2010 at 09:15:54AM +0100, Nicolas Greneche wrote: [..snip..] I don't understand why after a sleep, my ARP cache works ... It's very odd ... It is weird but looks like a networking setup issue to me rather than a libvirt issue. Your ping and tcpdump proves this. Seems like either the network interface isn't really up yet or a switch or similar is broken. Although this isn't a libvirt issue we should try to find out what happens. Can you try to hook a bridge between your eth1 and the network switch and see if the arp request leaves your host at all? Maybe you can also use the switches diagnostic software. I had to time to dig it, putting a network TAP on the wire. It seems to be a network issue because no ARP packet runs on the wire for eth1 (dedicated to iSCSI). So it's not a libvirtd problem, I think that this problem can be marked as solved. ARP works very well for the orther interface linked to a casual network. What kind of iSCSI target is this? Might it fail to generate the arp reply in time? It's an equalogic iSCSI target. I heard about jumbo frame when reseller came to introduice. Maybe there is a custom setting for MTU and this switch. Cheers, Thanks for your help Guido, Regards, -- Guido - -- Nicolas Greneche - RSSI et Sysadmin Centre de Ressources Informatiques (CRI) Doctorant au sein du projet SDS - www.sds-project.fr Mail : nicolas.greneche_(at)_univ-orleans.fr GPG : http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x5FEBD0EF Universite d'OrleansWeb : http://blog.garnett.fr Batiment 3IA - 2e etage Tel : 02 38 49 25 26 6 rue Leonard de Vinci BP 6102 45061 ORLEANS Cedex 2 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkunlgMACgkQTx/Y+1/r0O8FyQCgn26RTA2bH8bbXEhYc9DBdMzz qsoAn2M6cOmdpv35XCiQuAhHS8wSh/G6 =8cwA -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#574358: [Pkg-libvirt-maintainers] Bug#574358: libvirt-bin: libvirtd fails to autostart iscsi pool at boot time
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido Günther wrote: Hi Nicolas, On Thu, Mar 18, 2010 at 03:23:16PM +0100, Nicolas Greneche wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido Günther wrote: On Wed, Mar 17, 2010 at 04:28:41PM +0100, Nicolas Greneche wrote: It seems that libvirtd comes too early in booting process. Or the iscsi stuff to late. Could you add a Could you change: # Should-Start: hal avahi # Should-Start: hal avahi open-iscsi Did you verify that open-iscsi indeed starts before libvirtd? Done, it's OK, open-iscsi is started before libvirtd You need to run: update-rc.d libvirt-bin defaults 28 7 Done to make this active. Are you using dependency based boot? Yes, but the problem still remains. to the LSB section of /etc/init.d/libvirt-bin and see if this helps (assuming you're using dependency based boot)? Cheers, -- Guido I added a sleep in start function : fi ** sleep 60 ** rm -f /var/run/libvirtd.pid And it solved the issue. Maybe there is a cleaner way to do it ? See above. I still don't understand _what_ causes the actual problem - why do we have to wait here? What service/interface isn't ready? I don't know but I have pieces of information. I added an ifconfig command before daemon-stop-start of libvirtd and its OK : br0 Link encap:Ethernet HWaddr 00:26:b9:3f:50:30 inet addr:194.167.31.37 Bcast:194.167.31.255 Mask:255.255.255.0 inet6 addr: fe80::226:b9ff:fe3f:5030/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6 errors:0 dropped:0 overruns:0 frame:0 TX packets:11 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:390 (390.0 B) TX bytes:846 (846.0 B) eth0 Link encap:Ethernet HWaddr 00:26:b9:3f:50:30 inet6 addr: fe80::226:b9ff:fe3f:5030/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:22 errors:0 dropped:0 overruns:0 frame:0 TX packets:11 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2110 (2.0 KiB) TX bytes:926 (926.0 B) Interrupt:36 Memory:da00-da012800 eth1 Link encap:Ethernet HWaddr 00:26:b9:3f:50:31 inet addr:10.10.0.23 Bcast:10.10.0.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Interrupt:48 Memory:dc00-dc012800 loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:560 (560.0 B) TX bytes:560 (560.0 B) Next I added a ping which is OK for eth1:10.10.0.23 but not for 10.10.0.1 which refers to my target. With a tcpdump also launched before starting libvirtd i saw that the arp reply is not received by tcp/ip stack of my host in response of arp request for 10.10.0.1. I have 2 interfaces : a bridge for kvm guest (194.167.31.0/24) and a dedicated network for iSCSI (10.10.0.0/24). I added and iptables dump of my rules (iptables -L) before libvirtd start and there is no rules. If i add the entry of target in ARP cache before libvirtd starts it works like a charm (without sleeping). Could you attach your open-iscsi and pool configuration? My open-iscsi : r...@sandi:~# sed -e '/^[ ]*#/d' -e '/^$/d' /etc/iscsi/iscsid.conf node.startup = manual node.session.timeo.replacement_timeout = 120 node.conn[0].timeo.login_timeout = 15 node.conn[0].timeo.logout_timeout = 15 node.conn[0].timeo.noop_out_interval = 5 node.conn[0].timeo.noop_out_timeout = 5 node.session.err_timeo.abort_timeout = 15 node.session.err_timeo.lu_reset_timeout = 20 node.session.initial_login_retry_max = 8 node.session.cmds_max = 128 node.session.queue_depth = 32 node.session.xmit_thread_priority = -20 node.session.iscsi.InitialR2T = No node.session.iscsi.ImmediateData = Yes node.session.iscsi.FirstBurstLength = 262144 node.session.iscsi.MaxBurstLength = 16776192 node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768 node.session.iscsi.FastAbort = Yes r...@sandi:~# sed -e '/^[ ]*#/d' -e '/^$/d' /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1993-08.org.debian:01:6c3b7cdee7 No targets are activated by defaults (every node is set to node.startup = manual). My pool : r...@sandi:~# virsh pool-dumpxml equalog pool type='iscsi' nameequalog/name uuida6804c21-a610-a8bb-da64-921a7b448527/uuid capacity1099526307840/capacity allocation1099526307840
Bug#574358: [Pkg-libvirt-maintainers] Bug#574358: libvirt-bin: libvirtd fails to autostart iscsi pool at boot time
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido Günther wrote: On Wed, Mar 17, 2010 at 04:28:41PM +0100, Nicolas Greneche wrote: It seems that libvirtd comes too early in booting process. Or the iscsi stuff to late. Could you add a Could you change: # Should-Start: hal avahi # Should-Start: hal avahi open-iscsi to the LSB section of /etc/init.d/libvirt-bin and see if this helps (assuming you're using dependency based boot)? Cheers, -- Guido Hi Guido, I've tried your suggestion, it doesn't work. Whatever, I've found what was wrong. This an ARP issue. I've dumped network traffic by a tcpdump in libvirtd startup script to see what happens at network layer. The host send a who has and obtains no reply. By adding this ugly piece of code in startup (start section) script it works : if running ; then log_progress_msg already running log_end_msg 0 exit 0 fi ** arp -i eth1 -s 10.10.0.1 00:09:8A:05:7F:2C ** rm -f /var/run/libvirtd.pid start-stop-daemon --start --quiet --pidfile $PIDFILE \ --exec $DAEMON -- $libvirtd_opts Here is my /etc/network/interfaces : # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet manual auto br0 iface br0 inet static address 194.167.31.37 netmask 255.255.255.0 network 194.167.31.0 broadcast 194.167.31.255 gateway 194.167.31.242 bridge_ports eth0 bridge_fd 9 bridge_hello 12 bridge_stp off auto eth1 iface eth1 inet static address 10.10.0.23 netmask 255.255.255.0 network 10.10.0.0 broadcast 10.10.0.255 Maybe there is a parameter to add to authorize arp cache altering during boot process ? Do you have tips about it ? Many thanks for your help, - -- Nicolas Greneche - RSSI et Sysadmin Centre de Ressources Informatiques (CRI) Doctorant au sein du projet SDS - www.sds-project.fr Mail : nicolas.greneche_(at)_univ-orleans.fr GPG : http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x5FEBD0EF Universite d'OrleansWeb : http://blog.garnett.fr Batiment 3IA - 2e etage Tel : 02 38 49 25 26 6 rue Leonard de Vinci BP 6102 45061 ORLEANS Cedex 2 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuh8UwACgkQTx/Y+1/r0O/QzwCdG867f7ffaUMqQOfViDyVkMgI plsAoJuy0V+YpJdzgxailjqqGSWjHdzK =agLD -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#574358: [Pkg-libvirt-maintainers] Bug#574358: libvirt-bin: libvirtd fails to autostart iscsi pool at boot time
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido Günther wrote: On Wed, Mar 17, 2010 at 04:28:41PM +0100, Nicolas Greneche wrote: It seems that libvirtd comes too early in booting process. Or the iscsi stuff to late. Could you add a Could you change: # Should-Start: hal avahi # Should-Start: hal avahi open-iscsi to the LSB section of /etc/init.d/libvirt-bin and see if this helps (assuming you're using dependency based boot)? Cheers, -- Guido I added a sleep in start function : fi ** sleep 60 ** rm -f /var/run/libvirtd.pid And it solved the issue. Maybe there is a cleaner way to do it ? - -- Nicolas Greneche - RSSI et Sysadmin Centre de Ressources Informatiques (CRI) Doctorant au sein du projet SDS - www.sds-project.fr Mail : nicolas.greneche_(at)_univ-orleans.fr GPG : http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x5FEBD0EF Universite d'OrleansWeb : http://blog.garnett.fr Batiment 3IA - 2e etage Tel : 02 38 49 25 26 6 rue Leonard de Vinci BP 6102 45061 ORLEANS Cedex 2 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuiN1QACgkQTx/Y+1/r0O+LnACfYzGa2qDj3/x8vI54ZFsmb2m5 6ssAoI6bYEqJqw5sqGPH86B9t3Pim9Eb =GQd/ -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#574358: libvirt-bin: libvirtd fails to autostart iscsi pool at boot time
Package: libvirt-bin Version: 0.7.6-2 Severity: normal Hi, I've defined an iscsi target in a pool through XML file : pool type=iscsi nameequalog/name source host name=10.10.0.1/ device path=iqn.2001-05.com.equallogic:0-8a0906-4992c7f05-39c00114b8fc-vglog/ /source target path/dev/disk/by-path/path /target /pool At pool startup (virsh pool-start mypool) an entry is created in /dev/disk/by-path corresponding to my target. I added this XML code in my VM config to make it usable in guest : disk type='block' device='disk' driver name='qemu'/ source dev='/dev/disk/by-path/ip-10.10.0.1:3260-iscsi-iqn.2001-05.com.equallogic:0-8a0906-4992c7f05-39c00114b8fc-vglog-lun-0'/ target dev='vdc' bus='virtio'/ address type='pci' domain='0x' bus='0x00' slot='0x06' function='0x0'/ /disk The guest boots and a /dev/vdc is fully usable. Guest and iscsi pool are flagged autostart. When you reboot the host, libvirtd failed at autostarting iscsi pool with the folowing message (grep iscsi /var/log/syslog) : Mar 17 15:15:51 sandi libvirtd: 15:15:51.035: error : virRunWithHook:869 : internal error '/usr/bin/iscsiadm --mode discovery --type sendtargets --portal 10.10.0.1:3260,1' exited with non-zero status 255 and signal 0: iscsiadm: cannot make connection to 10.10.0.1:3260 (113)#012iscsiadm: connection to discovery address 10.10.0.1 failed#012iscsiadm: cannot make connection to 10.10.0.1:3260 (113)#012iscsiadm: connection to discovery address 10.10.0.1 failed#012iscsiadm: cannot make connection to 10.10.0.1:3260 (113)#012iscsiadm: connection to discovery address 10.10.0.1 failed#012iscsiadm: cannot make connection to 10.10.0.1:3260 (113)#012iscsiadm: connection to discovery address 10.10.0.1 failed#012iscsiadm: cannot make connection to 10.10.0.1:3260 (113)#012iscsiadm: connection to discovery address 10.10.0.1 failed#012iscsiadm: connection login retries (reopen_max) 5 exceeded#012iscsiadm: Could not perform SendTargets discovery.#012 As a consequence, the guest does not start : Mar 17 15:15:52 sandi libvirtd: 15:15:52.026: error : virStorageFileGetMetadata:415 : cannot open file '/dev/disk/by-path/ip-10.10.0.1:3260-iscsi-iqn.2001-05.com.equallogic:0-8a0906-4992c7f05-39c00114b8fc-vglog-lun-0': No such file or directory Mar 17 15:15:52 sandi libvirtd: 15:15:52.026: error : qemuAutostartDomain:595 : Failed to autostart VM 'gaelle': cannot open file '/dev/disk/by-path/ip-10.10.0.1:3260-iscsi-iqn.2001-05.com.equallogic:0-8a0906-4992c7f05-39c00114b8fc-vglog-lun-0': No such file or directory If you perform an /etc/init.d/libvirt-bin restart just after boot sequence (ie with the OS started and user logged as root) it operates smoothly. It seems that libvirtd comes too early in booting process. Regards, Nicolas Greneche - Sysadmin at Orleans University (France) -- System Information: Debian Release: squeeze/sid APT prefers testing APT policy: (500, 'testing') Architecture: amd64 (x86_64) Kernel: Linux 2.6.32-trunk-amd64 (SMP w/8 CPU cores) Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages libvirt-bin depends on: ii adduser 3.112 add and remove users and groups ii libavahi-client3 0.6.25-3 Avahi client library ii libavahi-common3 0.6.25-3 Avahi common library ii libc6 2.10.2-6 Embedded GNU C Library: Shared lib ii libcap-ng00.6.2-4+b1 An alternate posix capabilities li ii libdevmapper1.02. 2:1.02.39-1The Linux Kernel Device Mapper use ii libgcrypt11 1.4.5-2LGPL Crypto library - runtime libr ii libgnutls26 2.8.5-2the GNU TLS library - runtime libr ii libparted1.8-12 1.8.8.git.2009.07.19-6 The GNU Parted disk partitioning s ii libpciaccess0 0.11.0-1 Generic PCI access library for X ii libreadline6 6.1-1 GNU readline and history libraries ii libsasl2-22.1.23.dfsg1-5 Cyrus SASL - authentication abstra ii libudev0 151-2 libudev shared library ii libuuid1 2.16.2-0 Universally Unique ID library ii libvirt0 0.7.6-2library for interfacing with diffe ii libxenstore3.03.4.3~rc3-1Xenstore communications library fo ii libxml2 2.7.6.dfsg-2+b1GNOME XML library ii logrotate 3.7.8-4Log rotation utility Versions of packages libvirt-bin recommends: ii bridge-utils 1.4-5 Utilities for configuring the Linu ii dnsmasq-base 2.52-1 A small caching DNS proxy and DHCP ii iptables 1.4.6-2administration tools for packet fi ii netcat-openbsd1.89-3 TCP/IP swiss army knife ii qemu 0.11.1-2 fast processor emulator Versions of packages libvirt-bin suggests: pn
