Sorry to hear about your problems. First, sourceforge strips attachment so if you want to submit them you need to open a bug and attach the files there.
Second, if the hardware is Dell, you need to submit the issue to Dell and they will involve us if they need help. They want to troubleshoot problems with their hardware because they need to track the issues. If it is Dell hardware, don't open the bug here because we'll just have to tell you again to submit the issue to Dell. The third comment is that this looks like a possible known issue and with Dell hardware you need to use the Dell-approved firmware and drivers. They customize the hardware and firmware and you can't use the generic versions. Todd Fujinaka Software Application Engineer Data Center Group Intel Corporation todd.fujin...@intel.com -----Original Message----- From: Marc 'risson' Schmitt <ris...@cri.epita.fr> Sent: Tuesday, December 29, 2020 4:23 PM To: e1000-devel@lists.sourceforge.net Cc: c...@cri.epita.fr Subject: [E1000-devel] [i40e][bug] driver crashes machine under high network pressure Hi, During our benchmark of a Ceph cluster, we noticed one of our machines had a kernel panic and needed a reboot. After checking the kernel logs (attached), it turns out this error came from somewhere inside the i40e driver, which is used for our X722-DA2 network cards. After browsing Intel's forum, we found that someone had a similar problem, but in Windows with their cards being in a team. Ours were in a LACP bond, so we decided to stress test them when not in a bond, and were able to reproduce the problem. It also happens when stress testing only one of the interfaces, and in 1Gb/s mode (the previous tests were all done in 10Gb/s), although it takes longer. The amount of time after which the machine freezes/kernel panics is proportional to the network load. After further investigation, it seems that the number of softirqs is increasing by arbitrary steps (graph attached) to a point where the machine freezes. The fact that this bug also happens in 1Gb/s mode is leading us to believe that there is a memory leak in the i40e driver. During our tests, we observed that the slab memory is constantly increasing, without the machine doing anything else than network operations. The problem only happens when receiving traffic. Or at least we haven't been able to reproduce it when only sending traffic. After tracing the memory allocations and deallocations made by the kernel, we were able to confirm that the driver leaks memory. However, this leak doesn't happen when ntuples are off (disabled with `ethtool --features ens1f1 ntuple off`). We now plan to further analyze the memory operations made by the i40e driver and will report back if we find anything. In the meantime, we are opening this thread hoping that someone might already know of this issue and have a fix. Some useful information: We are running the latest version of the i40e downloaded from Intel's website because we had to upgrade the firmware of our network cards. This upgrade was necessary because the cards would otherwise randomly disconnect and the server had to be restarted or the cables un- and re-plugged for the card to work again (shutting the port off and back on on the switch did the trick too). We did not investigate this issue any further. The tests were ran as such: 4 iperf3 servers were listening on one node (hereafter called node-2), and two nodes (hereafter called node-1 and node-3) were each running 2 iperf3 clients. As such, each interface of node-2 was hit by two iperf3 clients, one from node-1, another from node-3. Note that we tried different combinations of this, with the same outputs, and as such we can conclude that the problem isn't due to one network card. The included kernel logs are from the test using only one interface. Look for the `[i40e]` pattern from the end of the file and you'll find the relevant stacktraces (there are many). The bug happens when using the out-of-tree module at version 2.13.10. It doesn't happen with the in-tree module at kernel versions 5.4.0, 5.8.2 and 5.10.2. Neither does it happen when using the out-of-tree module at 2.12.6. OS: Ubuntu 20.10 Kernel: 5.8.0-33-generic i40e version: 2.13.10 - BB96E598E7BFA4F229F7E53 X722 firmware: 5.15 0x8000275d 1.2829.0 iperf: 3.7 TSO and GRO are deactivated Server: Dell PowerEdge R6525 CPUs: 2x AMD EPYC 7352 24-Core Processor Regards, -- Marc 'risson' Schmitt CRI - EPITA _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
