On 29 October 2015 11:29:18 CET, hw <h...@gc-24.de> wrote: >J. Roeleveld wrote: >> On Thursday, October 15, 2015 05:46:07 PM hw wrote: >>> J. Roeleveld wrote: >>>> On Thursday, October 15, 2015 03:30:01 PM hw wrote: >>>>> Hi, >>>>> >>>>> I have a xen host with some HV guests which becomes unreachable >via >>>>> the network after apparently random amount of times. I have >already >>>>> switched the network card to see if that would make a difference, >>>>> and with the card currently installed, it worked fine for over 20 >days >>>>> until it become unreachable again. Before switching the network >card, >>>>> it would run a week or two before becoming unreachable. The >previous >>>>> card was the on-board BCM5764M which uses the tg3 driver. >>>>> >>>>> There are messages like this in the log file: >>>>> >>>>> >>>>> Oct 14 20:58:02 moonflo kernel: ------------[ cut here >]------------ >>>>> Oct 14 20:58:02 moonflo kernel: WARNING: CPU: 10 PID: 0 at >>>>> net/sched/sch_generic.c:303 dev_watchdog+0x259/0x270() Oct 14 >20:58:02 >>>>> moonflo kernel: NETDEV WATCHDOG: enp55s4 (r8169): transmit queue 0 >timed >>>>> out Oct 14 20:58:02 moonflo kernel: Modules linked in: arc4 ecb >md4 hmac >>>>> nls_utf8 cifs fscache xt_physdev br_netfilter iptable_filter >ip_tables >>>>> xen_pciback xen_gntalloc xen_gntdev bridge stp llc zfs(PO) nouveau >>>>> snd_hda_codec_realtek snd_hda_codec_generic zunicode(PO) zavl(PO) >>>>> zcommon(PO) znvpair(PO) spl(O) zlib_deflate video backlight >>>>> drm_kms_helper >>>>> ttm snd_hda_intel snd_hda_controller snd_hda_codec snd_pcm >snd_timer snd >>>>> soundcore r8169 mii xts aesni_intel glue_helper lrw gf128mul >ablk_helper >>>>> cryptd aes_x86_64 sha256_generic hid_generic usbhid uhci_hcd >usb_storage >>>>> ehci_pci ehci_hcd usbcore usb_common Oct 14 20:58:02 moonflo >kernel: CPU: >>>>> 10 PID: 0 Comm: swapper/10 Tainted: P O 4.0.5-gentoo >#3 Oct >>>>> 14 >>>>> 20:58:02 moonflo kernel: Hardware name: Hewlett-Packard HP Z800 >>>>> Workstation/0AECh, BIOS 786G5 v03.57 07/15/2013 Oct 14 20:58:02 >moonflo >>>>> kernel: ffffffff8175a77d ffff880124d43d98 ffffffff814da8d8 >>>>> 0000000000000001 Oct 14 20:58:02 moonflo kernel: ffff880124d43de8 >>>>> ffff880124d43dd8 ffffffff81088850 ffff880124d43dd8 Oct 14 20:58:02 >>>>> moonflo >>>>> kernel: 0000000000000000 ffff8800d45f2000 0000000000000001 >>>>> ffff8800d5294880 Oct 14 20:58:02 moonflo kernel: Call Trace: >>>>> Oct 14 20:58:02 moonflo kernel: <IRQ> [<ffffffff814da8d8>] >>>>> dump_stack+0x45/0x57 Oct 14 20:58:02 moonflo kernel: >>>>> [<ffffffff81088850>] >>>>> warn_slowpath_common+0x80/0xc0 Oct 14 20:58:02 moonflo kernel: >>>>> [<ffffffff810888d1>] warn_slowpath_fmt+0x41/0x50 Oct 14 20:58:02 >moonflo >>>>> kernel: [<ffffffff812b31c5>] ? >add_interrupt_randomness+0x35/0x1e0 Oct >>>>> 14 >>>>> 20:58:02 moonflo kernel: [<ffffffff8145b819>] >dev_watchdog+0x259/0x270 >>>>> Oct >>>>> 14 20:58:02 moonflo kernel: [<ffffffff8145b5c0>] ? >>>>> dev_graft_qdisc+0x80/0x80 Oct 14 20:58:02 moonflo kernel: >>>>> [<ffffffff8145b5c0>] ? dev_graft_qdisc+0x80/0x80 Oct 14 20:58:02 >moonflo >>>>> kernel: [<ffffffff810d4047>] call_timer_fn.isra.30+0x17/0x70 Oct >14 >>>>> 20:58:02 moonflo kernel: [<ffffffff810d42a6>] >>>>> run_timer_softirq+0x176/0x2b0 Oct 14 20:58:02 moonflo kernel: >>>>> [<ffffffff8108bd0a>] __do_softirq+0xda/0x1f0 Oct 14 20:58:02 >moonflo >>>>> kernel: [<ffffffff8108c04e>] irq_exit+0x7e/0xa0 Oct 14 20:58:02 >moonflo >>>>> kernel: [<ffffffff8130e075>] xen_evtchn_do_upcall+0x35/0x50 Oct >14 >>>>> 20:58:02 moonflo kernel: [<ffffffff814e1e8e>] >>>>> xen_do_hypervisor_callback+0x1e/0x40 Oct 14 20:58:02 moonflo >kernel: >>>>> <EOI> >>>>> >>>>> [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 Oct 14 >20:58:02 >>>>> >>>>> moonflo kernel: [<ffffffff810013aa>] ? >xen_hypercall_sched_op+0xa/0x20 >>>>> Oct >>>>> 14 20:58:02 moonflo kernel: [<ffffffff810459e0>] ? >>>>> xen_safe_halt+0x10/0x20 >>>>> Oct 14 20:58:02 moonflo kernel: [<ffffffff81053979>] ? >>>>> default_idle+0x9/0x10 Oct 14 20:58:02 moonflo kernel: >>>>> [<ffffffff810542da>] >>>>> ? arch_cpu_idle+0xa/0x10 Oct 14 20:58:02 moonflo kernel: >>>>> [<ffffffff810bd170>] ? cpu_startup_entry+0x190/0x2f0 Oct 14 >20:58:02 >>>>> moonflo kernel: [<ffffffff81047cd5>] ? >cpu_bringup_and_idle+0x25/0x40 >>>>> Oct >>>>> 14 20:58:02 moonflo kernel: ---[ end trace 98d961bae351244d ]--- >Oct 14 >>>>> 20:58:02 moonflo kernel: r8169 0000:37:04.0 enp55s4: link up >>>>> >>>>> >>>>> After that, there are lots of messages about the link being up, >one >>>>> message >>>>> every 12 seconds. When you unplug the network cable, you get a >message >>>>> that the link is down, and no message when you plug it in again. >>>>> >>>>> I was hoping that switching the network card (to one that uses a >>>>> different >>>>> driver) might solve the problem, and it did not. Now I can only >guess >>>>> that >>>>> the network card goes to sleep and sometimes cannot be woken up >again. >>>>> >>>>> I tried to reduce the connection speed to 100Mbit and found that >>>>> accessing >>>>> the VMs (via RDP) becomes too slow to use them. So I disabled the >power >>>>> management of the network card (through sysfs) and will have to >see if >>>>> the >>>>> problem persists. >>>>> >>>>> We'll be getting decent network cards in a couple days, but since >the >>>>> problem doesn't seem to be related to a particular >>>>> card/model/manufacturer, >>>>> that might not fix it, either. >>>>> >>>>> This problem seems to only occur on machines that operate as a xen >>>>> server. >>>>> Other machines, identical Z800s, not running xen, run just fine. >>>>> >>>>> What would you suggest? >>>> >>>> More info required: >>>> >>>> - Which version of Xen >>> >>> 4.5.1 >>> >>> Installed versions: 4.5.1^t(02:44:35 PM 07/14/2015)(-custom-cflags >-debug >>> -efi -flask -xsm) >> >> Ok, recent one. >> >>>> - Does this only occur with HVM guests? >>> >>> The host has been running only HVM guests every time it happend. >>> It was running a PV guest in between (which I had to shut down >>> because other VMs were migrated, requiring the RAM). >> >> The PV didn't have any issues? > >The whole server has the issue, not a particular VM. While the PV >guest >was running, the server didn't freeze. > >>>> - Which network-driver are you using inside the guest >>> >>> r8169, compiled as a module >>> >>> Same happened with the tg3 driver when the on-board cards were used. >>> The tg3 driver is completely disabled in the kernel config, i. e. >>> not even compiled as a module. >> >> You have network cards assigned to the guests? > >No, they are all connected via a bridge. > >I enabled STP on the bridge and the server was ok for a week, then had >to be restarted. I'm seeing lots of messages in the log: > > >Oct 28 11:14:05 moonflo kernel: brloc: topology change detected, >propagating >Oct 28 11:14:05 moonflo kernel: brloc: port 1(enp55s4) received tcn >bpdu >Oct 28 11:14:05 moonflo kernel: brloc: topology change detected, >propagating >Oct 28 11:14:05 moonflo kernel: brloc: port 1(enp55s4) received tcn >bpdu >Oct 28 11:14:05 moonflo kernel: brloc: topology change detected, >propagating >Oct 28 11:14:05 moonflo kernel: brloc: port 1(enp55s4) received tcn >bpdu >Oct 28 11:14:05 moonflo kernel: brloc: topology change detected, >propagating >Oct 28 11:14:05 moonflo kernel: brloc: port 1(enp55s4) received tcn >bpdu >Oct 28 11:14:05 moonflo kernel: brloc: topology change detected, >propagating > > >and sometimes: > >Oct 28 10:47:04 moonflo kernel: brloc: port 1(enp55s4) neighbor >8000.00:00:10:11:12:00 lost > > >Any idea what this means? > >(Google has gone on strike, and another search engine didn't give any >useful >findings ...) > > >>>> - Can you connect to the "local" console of the guest? >>> >>> Yes, the host seems to be running fine except for having no network >>> connectivity. There's a keyboard and monitor physically connected >to >>> it with which you can log in and do stuff. >> >> The HOST loses network connectivity? > >Yes. > >Apparently when it became unresponsive yesterday, it was not possible >to log in at the console, either. I wasn't there yesterday, though >I've >see that happen before. We tried to shut it down via acpid by pressing >the >power button. It didn't turn off, so it was switched off by holding the >power >button. What I can see in the log is: > > >Oct 28 14:12:33 moonflo logger[20322]: /etc/xen/scripts/block: remove >XENBUS_PATH=backend/vbd/2/768 >Oct 28 14:12:33 moonflo logger[20323]: /etc/xen/scripts/vif-bridge: >offline type_if=vif XENBUS_PATH=backend/vif/2/0 >Oct 28 14:12:33 moonflo logger[20347]: /etc/xen/scripts/vif-bridge: >brctl delif brloc vif2.0 failed >Oct 28 14:12:33 moonflo logger[20353]: /etc/xen/scripts/vif-bridge: >ifconfig vif2.0 down failed >Oct 28 14:12:33 moonflo logger[20361]: /etc/xen/scripts/vif-bridge: >Successful vif-bridge offline for vif2.0, bridge brloc. >Oct 28 14:12:33 moonflo logger[20372]: /etc/xen/scripts/vif-bridge: >remove type_if=tap XENBUS_PATH=backend/vif/2/0 >Oct 28 14:12:33 moonflo logger[20391]: /etc/xen/scripts/vif-bridge: >Successful vif-bridge remove for vif2.0-emu, bridge brloc. >Oct 28 14:15:33 moonflo shutdown[20476]: shutting down for system halt >^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Oct >28 14:17:34 moonflo syslog-ng[4611]: syslog-ng starting up; >version='3.6.2' > > >And: > > >Oct 24 11:47:42 moonflo kernel: NETDEV WATCHDOG: enp55s4 (r8169): >transmit queue 0 timed out >Oct 24 11:47:42 moonflo kernel: Modules linked in: xt_physdev >br_netfilter iptable_filter ip_tables xen_pciback xen_gntalloc >xen_gntdev bridge stp llc zfs(PO) zuni >code(PO) zavl(PO) zcommon(PO) znvpair(PO) nouveau snd_hda_codec_realtek >snd_hda_codec_generic video spl(O) backlight zlib_deflate >drm_kms_helper snd_hda_intel snd_ >hda_controller snd_hda_codec snd_pcm snd_timer r8169 snd ttm soundcore >mii xts aesni_intel glue_helper lrw gf128mul ablk_helper cryptd >aes_x86_64 sha256_generic hi >d_generic usbhid uhci_hcd usb_storage ehci_pci ehci_hcd usbcore >usb_common >Oct 24 11:47:42 moonflo kernel: CPU: 12 PID: 0 Comm: swapper/12 >Tainted: P O 4.0.5-gentoo #3 >Oct 24 11:47:42 moonflo kernel: Hardware name: Hewlett-Packard HP Z800 >Workstation/0AECh, BIOS 786G5 v03.57 07/15/2013 >Oct 24 11:47:42 moonflo kernel: ffffffff8175a77d ffff880124d83d98 >ffffffff814da8d8 0000000000000001 >Oct 24 11:47:42 moonflo kernel: ffff880124d83de8 ffff880124d83dd8 >ffffffff81088850 ffff880124d83e68 >Oct 24 11:47:42 moonflo kernel: 0000000000000000 ffff88011efd8000 >0000000000000001 ffff8800d4eb5e80 >Oct 24 11:47:42 moonflo kernel: Call Trace: >Oct 24 11:47:42 moonflo kernel: <IRQ> [<ffffffff814da8d8>] >dump_stack+0x45/0x57 >Oct 24 11:47:42 moonflo kernel: [<ffffffff81088850>] >warn_slowpath_common+0x80/0xc0 >Oct 24 11:47:42 moonflo kernel: [<ffffffff810888d1>] >warn_slowpath_fmt+0x41/0x50 >Oct 24 11:47:42 moonflo kernel: [<ffffffff812b31c5>] ? >add_interrupt_randomness+0x35/0x1e0 >Oct 24 11:47:42 moonflo kernel: [<ffffffff8145b819>] >dev_watchdog+0x259/0x270 >Oct 24 11:47:42 moonflo kernel: [<ffffffff8145b5c0>] ? >dev_graft_qdisc+0x80/0x80 >Oct 24 11:47:42 moonflo kernel: [<ffffffff8145b5c0>] ? >dev_graft_qdisc+0x80/0x80 >Oct 24 11:47:42 moonflo kernel: [<ffffffff810d4047>] >call_timer_fn.isra.30+0x17/0x70 >Oct 24 11:47:42 moonflo kernel: [<ffffffff810d42a6>] >run_timer_softirq+0x176/0x2b0 >Oct 24 11:47:42 moonflo kernel: [<ffffffff8108bd0a>] >__do_softirq+0xda/0x1f0 >Oct 24 11:47:42 moonflo kernel: [<ffffffff8108c04e>] >irq_exit+0x7e/0xa0 >Oct 24 11:47:42 moonflo kernel: [<ffffffff8130e075>] >xen_evtchn_do_upcall+0x35/0x50 >Oct 24 11:47:42 moonflo kernel: [<ffffffff814e1e8e>] >xen_do_hypervisor_callback+0x1e/0x40 >Oct 24 11:47:42 moonflo kernel: <EOI> [<ffffffff810013aa>] ? >xen_hypercall_sched_op+0xa/0x20 >Oct 24 11:47:42 moonflo kernel: [<ffffffff810013aa>] ? >xen_hypercall_sched_op+0xa/0x20 >Oct 24 11:47:42 moonflo kernel: [<ffffffff810459e0>] ? >xen_safe_halt+0x10/0x20 >Oct 24 11:47:42 moonflo kernel: [<ffffffff81053979>] ? >default_idle+0x9/0x10 >Oct 24 11:47:42 moonflo kernel: [<ffffffff810542da>] ? >arch_cpu_idle+0xa/0x10 >Oct 24 11:47:42 moonflo kernel: [<ffffffff810bd170>] ? >cpu_startup_entry+0x190/0x2f0 >Oct 24 11:47:42 moonflo kernel: [<ffffffff81047cd5>] ? >cpu_bringup_and_idle+0x25/0x40 >Oct 24 11:47:42 moonflo kernel: ---[ end trace 320b6f98f8fc070f ]--- >Oct 24 11:47:42 moonflo kernel: r8169 0000:37:04.0 enp55s4: link up > > >That was two days before it went down. After that, messages about >topology changes >are starting to appear. > >I'm not sure if I should call this "progress" ;) > >> >>> You get no answer when you ping the host while it is unreachable. >>> >>>> - If yes, does it still have no connectivity? >>> >>> It has been restarted this morning when it was found to be >unreachable. >>> >>>> I saw the same on my lab machine, which was related to: >>>> - Not using correct drivers inside HVM guests >>> >>> There are Windoze 7 guests running that have PV drivers installed. >>> One of those has formerly been running on a VMware host and was >>> migrated on Tuesday. I deinstalled the VMware tools from it. >> >> Which PV drivers? > >Xen GPL PV Driver Developers >17.09.2014 >0.11.0.373 >Univention GmbH > >> And did you ensure all VMWare related drivers were removed? >> I am not convinced uninstalling the VMWare tools is sufficient. > >What would I need to look at to make sure they are removed? > >The problem has been there before the VM that had VMWare drivers >installed was migrated to this server. So I don't think they are >causing this problem. > > >>> Since Monday, a HVM Linux system (a modified 32-bit Debian) has also >>> been migrated from the VMware host to this one. I don't know if it >>> has VMware tools installed (I guess it does because it could be shut >>> down via VMware) and how those might react now. It's working, and I >>> don't want to touch it. >>> >>> However, the problem already occured before this migration, when the >>> on-board cards were still used. >>> >>>> - Switch hardware not keeping the MAC/IP/Port lists long enough >>> >>> What might be the reason for the lists becoming too short? Too many >>> devices connected to the network? >> >> No network activity for a while. (clean installs, nothing running) >> Switch forgetting the MAC-address assigned to the VM. >> >> Connecting to the VM-console, I could ping www.google.com and then >the >> connectivity re-appeared. > >Half of the switches have been replaced last week in order to track >down >what appears to be a weird network problem. The problem is that the >RDP >clients are being randomly stalled. If it was only that, I'd suspect >this >server some more, but the internet connection goes through the same >switches >and is apprently also slowed down when the RPD clients are stalled. >They >got also randomly stalled when the RDP clients were accessing a totally >different server (the VMWare server), so this might be entirely >unrelated. > >Replacing the switches didn't fix the problem, so I'll probably put >them >back into service and replace the other half. > >>> The host has been connected to two different switches and showed the >>> problem. Previously, that was an 8-port 1Gb switch, now it's a >24-port >>> 1Gb switch. However, the 8-port switch is also connected to the >24-port >>> switch the host is now connected to. (The 24-port switch connects >it >>> "directly" to the rest of the network.) >> >> Assuming it's a managed switch, you could test this. >> Alternatively, check if you can access the VMs from the host. > >Good idea, I'll try that when it happens when I'm here. > >The network cards have arrived, Intel PRO 1000 dual port, made for IBM. >I hope I get to swap the card today. Those *really* should work. > >Hm, I could plug in two of them and give each VM and the host its own >physical card. Do you think that might help?
Quick reply from mobile. Will give a more detailed one later. Noticed you are using ZFS. Where is your swap partition located? On ZFS or? -- Joost -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
