Re: 11.1 running on HyperV hn interface hangs
On Fri, Sep 29, 2017 at 8:23 PM, Pete Frenchwrote: >> Hmm, by ntpd I think you mean ntp client? You will have to disable >> timesync if you run ntp client: >> sysctl hw.hvtimesync.sample_thresh=-1 >> sysctl hw.hvtimesync.ignore_sync=1 >> >> They interfere w/ each other. > > Oh! Does this apply to machines in Azure Hyper-V as well as on standalone > installations ? I wasn't aware of this, and have ntp on as standard (which > I wil now turn off pending a reply to this email, as sods law says that now > I am aware of it then I will get lockups over the weekend) :-) It applies to both Azure and standalone Hyper-V. In addition to the sysctls I mentioned, you can also add tunable hint.hvtimesync.0.disabled=1 to /boot/loader.conf to disable timesync, if you want to use ntp. Thanks, sephe -- Tomorrow Will Never Die ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
> Hmm, by ntpd I think you mean ntp client? You will have to disable > timesync if you run ntp client: > sysctl hw.hvtimesync.sample_thresh=-1 > sysctl hw.hvtimesync.ignore_sync=1 > > They interfere w/ each other. Oh! Does this apply to machines in Azure Hyper-V as well as on standalone installations ? I wasn't aware of this, and have ntp on as standard (which I wil now turn off pending a reply to this email, as sods law says that now I am aware of it then I will get lockups over the weekend) :-) -pete. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
On Fri, 29 Sep 2017 13:31:22 +0800 Sepherosa Ziehauwrote: > On Fri, Sep 29, 2017 at 12:36 PM, Paul Koch wrote: > > On Thu, 14 Sep 2017 09:54:56 +0800 > > Sepherosa Ziehau wrote: > > > >> If you have any updates on this, please let me know. There is still > >> time for 10.4. > > > > We are still playing around with this in the lab... > > > > Running similar setup as the customer > > Microsoft Windows Server 2012 R2 Datacentre (6.3.9600) Revision 16384 > > Hyper-V 2012 > > > > Two VM guests > > - 11.0-RELEASE > > - 11.1-p1 > > > > We can not get the Hyper-V hn interface to lock up like the customer can > > though. > > > > We can get the VMs to hang/stall regularly if the guests run ntpd - approx > > every 15 mins, but no real obvious pattern to it. Disabling ntpd fixes > > it. > > Hmm, by ntpd I think you mean ntp client? You will have to disable > timesync if you run ntp client: > sysctl hw.hvtimesync.sample_thresh=-1 > sysctl hw.hvtimesync.ignore_sync=1 > > They interfere w/ each other. > > Or do you mean the network hanging triggered by "RXBUF ack failed"? > > Thanks, > sephe Yes, ntp client running on the VM guest. After finding it was unstable, we concluded that there must be some type of interference. Best that we automatically force ntp off when our software detects it is running in Hyper-V. We haven't been able to trigger the hn to hang like our customer can with the RXBUF problem though. Different underlying hardware probably. Paul. -- Paul Koch | Founder | CEO AKIPS Network Monitor | akips.com Brisbane, Australia ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
On Fri, Sep 29, 2017 at 12:36 PM, Paul Kochwrote: > On Thu, 14 Sep 2017 09:54:56 +0800 > Sepherosa Ziehau wrote: > >> If you have any updates on this, please let me know. There is still >> time for 10.4. > > We are still playing around with this in the lab... > > Running similar setup as the customer > Microsoft Windows Server 2012 R2 Datacentre (6.3.9600) Revision 16384 > Hyper-V 2012 > > Two VM guests > - 11.0-RELEASE > - 11.1-p1 > > We can not get the Hyper-V hn interface to lock up like the customer can > though. > > We can get the VMs to hang/stall regularly if the guests run ntpd - approx > every 15 mins, but no real obvious pattern to it. Disabling ntpd fixes it. Hmm, by ntpd I think you mean ntp client? You will have to disable timesync if you run ntp client: sysctl hw.hvtimesync.sample_thresh=-1 sysctl hw.hvtimesync.ignore_sync=1 They interfere w/ each other. Or do you mean the network hanging triggered by "RXBUF ack failed"? Thanks, sephe -- Tomorrow Will Never Die ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
On Thu, 14 Sep 2017 09:54:56 +0800 Sepherosa Ziehauwrote: > If you have any updates on this, please let me know. There is still > time for 10.4. We are still playing around with this in the lab... Running similar setup as the customer Microsoft Windows Server 2012 R2 Datacentre (6.3.9600) Revision 16384 Hyper-V 2012 Two VM guests - 11.0-RELEASE - 11.1-p1 We can not get the Hyper-V hn interface to lock up like the customer can though. We can get the VMs to hang/stall regularly if the guests run ntpd - approx every 15 mins, but no real obvious pattern to it. Disabling ntpd fixes it. Paul. -- Paul Koch | Founder | CEO AKIPS Network Monitor | akips.com Brisbane, Australia ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
On Thu, 14 Sep 2017 09:54:56 +0800 Sepherosa Ziehauwrote: > If you have any updates on this, please let me know. There is still > time for 10.4. Still working on it. We are trying to replicate the FreeBSD 11.1 running in a Hyper-V VM setup in our test lab. We have ping/snmp/netflow network simulators that can create large amounts of real network traffic to see if it reliably triggers the problem. Paul. -- Paul Koch | Founder | CEO AKIPS Network Monitor | akips.com Brisbane, Australia ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
If you have any updates on this, please let me know. There is still time for 10.4. On Thu, Sep 7, 2017 at 11:04 PM, Paul Kochwrote: > On Thu, 7 Sep 2017 13:51:11 +0800 > Sepherosa Ziehau wrote: > >> Weird, your traffic pattern does not even belong to anything heavy. >> Sending is mainly UDP, which will never be able to saturate the TX >> buffer ring causing the RXBUF ACK sending failure. This is weird. > > It's a bit tricky. The poller is very fast. We ping every device every 15 > seconds, and collect every MIB object every 60 seconds. The poller "rate > limits" itself by dividing each minute into 100ms time slots and only sends a > specific amount of pings/snmp packets in each time slot. The problem is, it > blasts the request packets out really fast at the start of each time slot, > and then sits in a receive loop until the next time slot comes around. The > requests are not paced over the 100ms, therefore it will blast out a lot > of packets in a few milliseconds. > > We use to use a 1 second rate limiting time slot, and didn't interlace > ping/snmp requests, but we found certain interface types on Cisco 6509 > switches couldn't keep up with back-to-back pings and would lose them. > > >> Anyhow, make sure to test this patch: >> 8762017-Sep-07 02:19 hn_inc_txbr.diff > > Yep. Might take a bit of time to test though because we'll need to get the > customer to spin up a test VM on the same platform, and they are fairly > remote (Perth, Australia). We don't run any Microsoft servers/HyperV setups > in our lab. > > Paul. > -- > Paul Koch | Founder | CEO > AKIPS Network Monitor | akips.com > Brisbane, Australia -- Tomorrow Will Never Die ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
On Thu, 7 Sep 2017 13:51:11 +0800 Sepherosa Ziehauwrote: > Weird, your traffic pattern does not even belong to anything heavy. > Sending is mainly UDP, which will never be able to saturate the TX > buffer ring causing the RXBUF ACK sending failure. This is weird. It's a bit tricky. The poller is very fast. We ping every device every 15 seconds, and collect every MIB object every 60 seconds. The poller "rate limits" itself by dividing each minute into 100ms time slots and only sends a specific amount of pings/snmp packets in each time slot. The problem is, it blasts the request packets out really fast at the start of each time slot, and then sits in a receive loop until the next time slot comes around. The requests are not paced over the 100ms, therefore it will blast out a lot of packets in a few milliseconds. We use to use a 1 second rate limiting time slot, and didn't interlace ping/snmp requests, but we found certain interface types on Cisco 6509 switches couldn't keep up with back-to-back pings and would lose them. > Anyhow, make sure to test this patch: > 8762017-Sep-07 02:19 hn_inc_txbr.diff Yep. Might take a bit of time to test though because we'll need to get the customer to spin up a test VM on the same platform, and they are fairly remote (Perth, Australia). We don't run any Microsoft servers/HyperV setups in our lab. Paul. -- Paul Koch | Founder | CEO AKIPS Network Monitor | akips.com Brisbane, Australia ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
Weird, your traffic pattern does not even belong to anything heavy. Sending is mainly UDP, which will never be able to saturate the TX buffer ring causing the RXBUF ACK sending failure. This is weird. Anyhow, make sure to test this patch: 8762017-Sep-07 02:19 hn_inc_txbr.diff On Thu, Sep 7, 2017 at 1:07 PM, Paul Kochwrote: > On Thu, 7 Sep 2017 10:22:40 +0800 > Sepherosa Ziehau wrote: > >> Is it possible to tell me your workload? e.g. TX heavy or RX heavy. >> Enabled TSO or not. Details like how the send syscalls are issue will >> be interesting. And your Windows version, include the patch level, >> etc. >> >> Please try the following patch: >> https://people.freebsd.org/~sephe/hn_dec_txdesc.diff >> >> Thanks, >> sephe > > Hi Sephe, > > Here's a bit of an explanation of the environment... > > AKIPS Network Monitor workload: > - 22000 devices (routers/switches/APs/etc) > - 123000 interfaces (60 snmp polling) > - 131 netflow exporters > - ~1500 pings per second > - ~1000 snmp requests/responses per second (~1.9 million MIB object/min) > - ~250 netflow packets/sec (~4500 flows/sec incoming) > - ~130 syslog messages/sec (incoming) > - ~200 snmp traps/sec (incoming) > > The ping/snmp poller is a single monolithic process (no threads). > Separate processes for each of the syslog/trap/netflow collection. > > SNMP requests are sent using the sendto() system call over a non-blocking UDP > socket for both IPv4 and v6. We set the UDP socket receive buffer size to > 4 Mbytes. Nothing really complex with it. > > Pings are interlaced with snmp requests so we limit the bursty nature of > small back-to-back packets (eliminates issues with switch interfaces dropping > bursts of packets). Ping requests are sent using a raw icmp socket. We > don't read the responses from the icmp socket, instead we put the interface > into promiscuous mode and use the BPF info to measure the tx/rx RTT values. > > Syslog daemon just listens on a UDP socket with a 4 Mbyte receive buffer. > Same with the snmp trap daemon. > > > Here's some links to performance graphs of the VM: > https://www.akips.com/downloads/hyperv-fbsd11.1p1/system-graphs-last2h.pdf > https://www.akips.com/downloads/hyperv-fbsd11.1p1/system-graphs-last24h.pdf > https://www.akips.com/downloads/hyperv-fbsd11.1p1/system-graphs-last7d.pdf > > The OS was upgraded to 11.1p1 at 5pm on the 5th Sep. The hn0 interface hung > at 7:36pm. The interface hung three times before we reverted to 11.0p9. It > takes a few hours after rebooting the VM before the interface hangs. > > > Microsoft Host is running Windows 2012 R2. Waiting for patch level info from > the customer. > > I'll have to get the customer to spin up a new VM before trying your patch. > > > Here's some info (after a reboot of the VM) > > Guest VM dmesg: > > FreeBSD 11.1-RELEASE-p1 #0 r322350: Thu Aug 10 22:16:21 UTC 2017 > r...@shed31.akips.com:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM > 4.0.0) > VT(vga): text 80x25 > Hyper-V Version: 6.3.9600 [SP18] > > Features=0xe7f > PM Features=0x0 [C2] > Features3=0x7b2 > Timecounter "Hyper-V" frequency 1000 Hz quality 2000 > CPU: Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz (2300.00-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 > > Features=0x1f83fbff > Features2=0x80002001 > AMD Features=0x20100800 > AMD Features2=0x1 > Hypervisor: Origin = "Microsoft Hv" > real memory = 34359738368 (32768 MB) > avail memory = 33325903872 (31782 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 1 package(s) x 4 core(s) > random: unblocking device. > ioapic0: Changing APIC ID to 0 > ioapic0 irqs 0-23 on motherboard > SMP: AP CPU #1 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #2 Launched! > Timecounter "Hyper-V-TSC" frequency 1000 Hz quality 3000 > random: entropy device external interface > kbd1 at kbdmux0 > netmap: loaded module > module_register_init: MOD_LOAD (vesa, 0x80f5b220, 0) error 19 > nexus0 > vtvga0: on motherboard > cryptosoft0: on motherboard > acpi0: on motherboard > acpi0: Power Button (fixed) > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > attimer0: port 0x40-0x43 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > pcib0: port 0xcf8-0xcff
Re: 11.1 running on HyperV hn interface hangs
Ignore ths hn_dec_txdesc.diff, please try this done; should be more effective: https://people.freebsd.org/~sephe/hn_inc_txbr.diff On Thu, Sep 7, 2017 at 10:22 AM, Sepherosa Ziehauwrote: > Is it possible to tell me your workload? e.g. TX heavy or RX heavy. > Enabled TSO or not. Details like how the send syscalls are issue will > be interesting. And your Windows version, include the patch level, > etc. > > Please try the following patch: > https://people.freebsd.org/~sephe/hn_dec_txdesc.diff > > Thanks, > sephe > > > On Wed, Sep 6, 2017 at 11:23 PM, Julian Elischer wrote: >> On 6/9/17 7:02 pm, Pete French wrote: We recently moved our software from 11.0-p9 to 11.1-p1, but looks like there is a regression in 11.1-p1 running on HyperV (Windows/HyperV 2012 R2) where the virtual hn0 interface hangs with the following kernel messages: hn0: on vmbus0 hn0: Ethernet address: 00:15:5d:31:21:0f hn0: link state changed to UP ... hn0: RXBUF ack retry hn0: RXBUF ack failed last message repeated 571 times It requires a restart of the HyperV VM. This is a customer production server (remote customer ~4000km away) running fairly critical monitoring software, so we needed to roll it back to 11.0-p9. We only have two customers running our software in HyperV, vs lots in VMware and a handful on physical hardware. 11.0-p9 has been very stable. Has anyone seen this problem before with 11.1 ? >>> >>> >>> >>> I don't run anything on local hyper-v anymore, but I do run a ot of stuff >>> in Azure, and we havent seen anything like this. I track STABLE for things >>> though, updating after reading the commits and testing locally for a week or >>> so, so the version I am running currently is r320175, which was part of >>> 11.1-BETA2. I am going to upgrade to a more recent STABLE sometime this weke >>> or next though, will do that on a test amchine and let you now how it goes. >>> >>> I seem to recall that there were some large changes to the hn code in >>> August to add virtual function support. When does 11.1-p1 date from ? >> >> make sure you contact the FreeBSD/Microsoft guys. Very responsive.. don't >> know if they watch -stable.. >> I'll cc a couple.. >> >>> >>> -pete. >>> ___ >>> freebsd-stable@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" >>> >> > > > > -- > Tomorrow Will Never Die -- Tomorrow Will Never Die ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
On Thu, 7 Sep 2017 10:22:40 +0800 Sepherosa Ziehauwrote: > Is it possible to tell me your workload? e.g. TX heavy or RX heavy. > Enabled TSO or not. Details like how the send syscalls are issue will > be interesting. And your Windows version, include the patch level, > etc. > > Please try the following patch: > https://people.freebsd.org/~sephe/hn_dec_txdesc.diff > > Thanks, > sephe Hi Sephe, Here's a bit of an explanation of the environment... AKIPS Network Monitor workload: - 22000 devices (routers/switches/APs/etc) - 123000 interfaces (60 snmp polling) - 131 netflow exporters - ~1500 pings per second - ~1000 snmp requests/responses per second (~1.9 million MIB object/min) - ~250 netflow packets/sec (~4500 flows/sec incoming) - ~130 syslog messages/sec (incoming) - ~200 snmp traps/sec (incoming) The ping/snmp poller is a single monolithic process (no threads). Separate processes for each of the syslog/trap/netflow collection. SNMP requests are sent using the sendto() system call over a non-blocking UDP socket for both IPv4 and v6. We set the UDP socket receive buffer size to 4 Mbytes. Nothing really complex with it. Pings are interlaced with snmp requests so we limit the bursty nature of small back-to-back packets (eliminates issues with switch interfaces dropping bursts of packets). Ping requests are sent using a raw icmp socket. We don't read the responses from the icmp socket, instead we put the interface into promiscuous mode and use the BPF info to measure the tx/rx RTT values. Syslog daemon just listens on a UDP socket with a 4 Mbyte receive buffer. Same with the snmp trap daemon. Here's some links to performance graphs of the VM: https://www.akips.com/downloads/hyperv-fbsd11.1p1/system-graphs-last2h.pdf https://www.akips.com/downloads/hyperv-fbsd11.1p1/system-graphs-last24h.pdf https://www.akips.com/downloads/hyperv-fbsd11.1p1/system-graphs-last7d.pdf The OS was upgraded to 11.1p1 at 5pm on the 5th Sep. The hn0 interface hung at 7:36pm. The interface hung three times before we reverted to 11.0p9. It takes a few hours after rebooting the VM before the interface hangs. Microsoft Host is running Windows 2012 R2. Waiting for patch level info from the customer. I'll have to get the customer to spin up a new VM before trying your patch. Here's some info (after a reboot of the VM) Guest VM dmesg: FreeBSD 11.1-RELEASE-p1 #0 r322350: Thu Aug 10 22:16:21 UTC 2017 r...@shed31.akips.com:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) VT(vga): text 80x25 Hyper-V Version: 6.3.9600 [SP18] Features=0xe7f PM Features=0x0 [C2] Features3=0x7b2 Timecounter "Hyper-V" frequency 1000 Hz quality 2000 CPU: Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz (2300.00-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0x1f83fbff Features2=0x80002001 AMD Features=0x20100800 AMD Features2=0x1 Hypervisor: Origin = "Microsoft Hv" real memory = 34359738368 (32768 MB) avail memory = 33325903872 (31782 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) random: unblocking device. ioapic0: Changing APIC ID to 0 ioapic0 irqs 0-23 on motherboard SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! Timecounter "Hyper-V-TSC" frequency 1000 Hz quality 3000 random: entropy device external interface kbd1 at kbdmux0 netmap: loaded module module_register_init: MOD_LOAD (vesa, 0x80f5b220, 0) error 19 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 vmbus0: on pcib0 pci0: on pcib0 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 pci0: at device 7.3 (no driver attached) vgapci0: mem 0xf800-0xfbff irq 11 at device 8.0 on pci0 vgapci0: Boot video device atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer,
Re: 11.1 running on HyperV hn interface hangs
Is it possible to tell me your workload? e.g. TX heavy or RX heavy. Enabled TSO or not. Details like how the send syscalls are issue will be interesting. And your Windows version, include the patch level, etc. Please try the following patch: https://people.freebsd.org/~sephe/hn_dec_txdesc.diff Thanks, sephe On Wed, Sep 6, 2017 at 11:23 PM, Julian Elischerwrote: > On 6/9/17 7:02 pm, Pete French wrote: >>> >>> We recently moved our software from 11.0-p9 to 11.1-p1, but looks like >>> there >>> is a regression in 11.1-p1 running on HyperV (Windows/HyperV 2012 R2) >>> where >>> the virtual hn0 interface hangs with the following kernel messages: >>> >>> hn0: on vmbus0 >>> hn0: Ethernet address: 00:15:5d:31:21:0f >>> hn0: link state changed to UP >>> ... >>> hn0: RXBUF ack retry >>> hn0: RXBUF ack failed >>> last message repeated 571 times >>> >>> It requires a restart of the HyperV VM. >>> >>> This is a customer production server (remote customer ~4000km away) >>> running >>> fairly critical monitoring software, so we needed to roll it back to >>> 11.0-p9. >>> We only have two customers running our software in HyperV, vs lots in >>> VMware >>> and a handful on physical hardware. >>> >>> 11.0-p9 has been very stable. Has anyone seen this problem before with >>> 11.1 ? >> >> >> >> I don't run anything on local hyper-v anymore, but I do run a ot of stuff >> in Azure, and we havent seen anything like this. I track STABLE for things >> though, updating after reading the commits and testing locally for a week or >> so, so the version I am running currently is r320175, which was part of >> 11.1-BETA2. I am going to upgrade to a more recent STABLE sometime this weke >> or next though, will do that on a test amchine and let you now how it goes. >> >> I seem to recall that there were some large changes to the hn code in >> August to add virtual function support. When does 11.1-p1 date from ? > > make sure you contact the FreeBSD/Microsoft guys. Very responsive.. don't > know if they watch -stable.. > I'll cc a couple.. > >> >> -pete. >> ___ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" >> > -- Tomorrow Will Never Die ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
Paul Koch paul.koch at akips.com wrote on Wed Sep 6 09:33:26 UTC 2017 : > We recently moved our software from 11.0-p9 to 11.1-p1, but looks like there > is a regression in 11.1-p1 running on HyperV (Windows/HyperV 2012 R2) where > the virtual hn0 interface hangs with the following kernel messages: > > hn0: on vmbus0 > hn0: Ethernet address: 00:15:5d:31:21:0f > hn0: link state changed to UP > ... > hn0: RXBUF ack retry > hn0: RXBUF ack failed > last message repeated 571 times > > . . . > > Has anyone seen this problem before with 11.1 ? While it is/was a personal use/experiment I have used all the following under Windows 10 Pro's Hyper-V with networking via hn0 Ethernet as seen from the guest FreeBSD: releng/11.1 (no longer around to remind me of the most recent -r?? but various updates ) stable/11 (various updates, -r320807 currently) head(various updates, -r323147 currently) I had no problems with my use. (By no means a traffic match to your context but definitely used.) In all cases the Virtual Switch Manager was tied to the (builtin) "External network" that is listed as: Intel(R) I211 Gigabit Network Connection in the Virtual Switch Properties pop-up for External network. The machine is not a server. So not totally broken as far as I can tell. Something more specific to your context would seem to also be involved. Hyper-V has worked nicely for assigning 14 of the machine's 16 hardware threads to FreeBSD and doing buildworld buildkernel and poudriere based port builds. (Windows 10 Pro not being otherwise busy.) === Mark Millard markmi at dsl-only.net ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
On 6/9/17 7:02 pm, Pete French wrote: We recently moved our software from 11.0-p9 to 11.1-p1, but looks like there is a regression in 11.1-p1 running on HyperV (Windows/HyperV 2012 R2) where the virtual hn0 interface hangs with the following kernel messages: hn0: on vmbus0 hn0: Ethernet address: 00:15:5d:31:21:0f hn0: link state changed to UP ... hn0: RXBUF ack retry hn0: RXBUF ack failed last message repeated 571 times It requires a restart of the HyperV VM. This is a customer production server (remote customer ~4000km away) running fairly critical monitoring software, so we needed to roll it back to 11.0-p9. We only have two customers running our software in HyperV, vs lots in VMware and a handful on physical hardware. 11.0-p9 has been very stable. Has anyone seen this problem before with 11.1 ? I don't run anything on local hyper-v anymore, but I do run a ot of stuff in Azure, and we havent seen anything like this. I track STABLE for things though, updating after reading the commits and testing locally for a week or so, so the version I am running currently is r320175, which was part of 11.1-BETA2. I am going to upgrade to a more recent STABLE sometime this weke or next though, will do that on a test amchine and let you now how it goes. I seem to recall that there were some large changes to the hn code in August to add virtual function support. When does 11.1-p1 date from ? make sure you contact the FreeBSD/Microsoft guys. Very responsive.. don't know if they watch -stable.. I'll cc a couple.. -pete. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
On Wed, 6 Sep 2017 12:02:43 +0100 Pete Frenchwrote: > > We recently moved our software from 11.0-p9 to 11.1-p1, but looks like > > there is a regression in 11.1-p1 running on HyperV (Windows/HyperV 2012 > > R2) where the virtual hn0 interface hangs with the following kernel > > messages: > > > > hn0: on vmbus0 > > hn0: Ethernet address: 00:15:5d:31:21:0f > > hn0: link state changed to UP > > ... > > hn0: RXBUF ack retry > > hn0: RXBUF ack failed > > last message repeated 571 times > > > > It requires a restart of the HyperV VM. > > > > This is a customer production server (remote customer ~4000km away) > > running fairly critical monitoring software, so we needed to roll it back > > to 11.0-p9. We only have two customers running our software in HyperV, vs > > lots in VMware and a handful on physical hardware. > > > > 11.0-p9 has been very stable. Has anyone seen this problem before with > > 11.1 ? > > > I don't run anything on local hyper-v anymore, but I do run a ot of > stuff in Azure, and we havent seen anything like this. I track STABLE > for things though, updating after reading the commits and testing > locally for a week or so, so the version I am running currently is > r320175, which was part of 11.1-BETA2. I am going to upgrade to a more > recent STABLE sometime this weke or next though, will do that on a test > amchine and let you now how it goes. > > I seem to recall that there were some large changes to the hn code in > August to add virtual function support. When does 11.1-p1 date from ? Looks like 2017-08-10 Paul. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 11.1 running on HyperV hn interface hangs
We recently moved our software from 11.0-p9 to 11.1-p1, but looks like there is a regression in 11.1-p1 running on HyperV (Windows/HyperV 2012 R2) where the virtual hn0 interface hangs with the following kernel messages: hn0: on vmbus0 hn0: Ethernet address: 00:15:5d:31:21:0f hn0: link state changed to UP ... hn0: RXBUF ack retry hn0: RXBUF ack failed last message repeated 571 times It requires a restart of the HyperV VM. This is a customer production server (remote customer ~4000km away) running fairly critical monitoring software, so we needed to roll it back to 11.0-p9. We only have two customers running our software in HyperV, vs lots in VMware and a handful on physical hardware. 11.0-p9 has been very stable. Has anyone seen this problem before with 11.1 ? I don't run anything on local hyper-v anymore, but I do run a ot of stuff in Azure, and we havent seen anything like this. I track STABLE for things though, updating after reading the commits and testing locally for a week or so, so the version I am running currently is r320175, which was part of 11.1-BETA2. I am going to upgrade to a more recent STABLE sometime this weke or next though, will do that on a test amchine and let you now how it goes. I seem to recall that there were some large changes to the hn code in August to add virtual function support. When does 11.1-p1 date from ? -pete. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
11.1 running on HyperV hn interface hangs
No sure if -stable is the right mailing list for this one. We recently moved our software from 11.0-p9 to 11.1-p1, but looks like there is a regression in 11.1-p1 running on HyperV (Windows/HyperV 2012 R2) where the virtual hn0 interface hangs with the following kernel messages: hn0: on vmbus0 hn0: Ethernet address: 00:15:5d:31:21:0f hn0: link state changed to UP ... hn0: RXBUF ack retry hn0: RXBUF ack failed last message repeated 571 times It requires a restart of the HyperV VM. This is a customer production server (remote customer ~4000km away) running fairly critical monitoring software, so we needed to roll it back to 11.0-p9. We only have two customers running our software in HyperV, vs lots in VMware and a handful on physical hardware. 11.0-p9 has been very stable. Has anyone seen this problem before with 11.1 ? 11.1 is listed here https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/supported-freebsd-virtual-machines-on-hyper-v Paul. -- Paul Koch | Founder | CEO AKIPS Network Monitor | akips.com Brisbane, Australia ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"