On 01/22/2018 10:30 AM, Ben Greear wrote:
On 01/22/2018 10:16 AM, Eric Dumazet wrote:
On Mon, 2018-01-22 at 09:28 -0800, Ben Greear wrote:
My test case is to have 6 processes each create 5000 TCP IPv4 connections to
each other
on a system with 16GB RAM and send slow-speed data. This works fine on a 4.7
kernel, but
will not work at all on a 4.13. The 4.13 first complains about running out of
tcp memory,
but even after forcing those values higher, the max connections we can get is
around 15k.
Both kernels have my out-of-tree patches applied, so it is possible it is my
fault
at this point.
Any suggestions as to what this might be caused by, or if it is fixed in more
recent kernels?
I will start bisecting in the meantime...
Hi Ben
Unfortunately I have no idea.
Are you using loopback flows, or have I misunderstood you ?
How loopback connections can be slow-speed ?
I am sending to self, but over external network interfaces, by using
routing tables and rules and such.
On 4.13.16+, I see the Intel driver bouncing when I try to start 20k
connections. In this case, I have a pair of 10G ports doing 15k, and then
I try to start 5k on two of the 1G ports....
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
Down
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up
1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
Down
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up
1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
Down
Jan 22 10:15:41 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up
1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 22 10:15:43 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is
Down
Jan 22 10:15:45 lf1003-e3v2-13100124-f20x64 kernel: e1000e: eth3 NIC Link is Up
1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 22 10:15:51 lf1003-e3v2-13100124-f20x64 kernel: NETDEV WATCHDOG: eth3
(e1000e): transmit queue 0 timed out, trans_s...es: 1
Jan 22 10:15:51 lf1003-e3v2-13100124-f20x64 kernel: e1000e 0000:07:00.0 eth3:
Reset adapter unexpectedly
System reports 10+GB RAM free in this case, btw.
Actually, maybe the good kernel was even older than 4.7...I see same resets and
inability to do a full 20k
connections on 4.7 too. I double-checked with system-test and it seems 4.4
was a good kernel. I'll test
that next. Here is splat from 4.7:
[ 238.921679] ------------[ cut here ]------------
[ 238.921689] WARNING: CPU: 0 PID: 3 at
/home/greearb/git/linux-bisect/net/sched/sch_generic.c:272
dev_watchdog+0xd4/0x12f
[ 238.921690] NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out
[ 238.921691] Modules linked in: nf_conntrack_netlink nf_conntrack nfnetlink
nf_defrag_ipv4 cfg80211 macvlan pktgen bnep bluetooth fuse coretemp intel_rapl
ftdi_sio x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm iTCO_wdt
iTCO_vendor_support joydev ie31200_edac ipmi_devintf irqbypass serio_raw
ipmi_si edac_core
shpchp fjes video i2c_i801 tpm_tis lpc_ich ipmi_msghandler tpm nfsd auth_rpcgss
nfs_acl lockd grace sunrpc mgag200 i2c_algo_bit drm_kms_helper ttm drm i2c_core
e1000e ixgbe mdio hwmon dca ptp pps_core ipv6 [last unloaded: nf_conntrack]
[ 238.921720] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted 4.7.0 #62
[ 238.921721] Hardware name: Supermicro X9SCI/X9SCA/X9SCI/X9SCA, BIOS 2.0b
09/17/2012
[ 238.921723] 0000000000000000 ffff88041cdd7cd8 ffffffff81352a23
ffff88041cdd7d28
[ 238.921725] 0000000000000000 ffff88041cdd7d18 ffffffff810ea5dd
000001101cdd7d90
[ 238.921727] ffff880417a84000 0000000000000100 ffffffff8163ecff
ffff880417a84440
[ 238.921728] Call Trace:
[ 238.921733] [<ffffffff81352a23>] dump_stack+0x61/0x7d
[ 238.921736] [<ffffffff810ea5dd>] __warn+0xbd/0xd8
[ 238.921738] [<ffffffff8163ecff>] ? netif_tx_lock+0x81/0x81
[ 238.921740] [<ffffffff810ea63e>] warn_slowpath_fmt+0x46/0x4e
[ 238.921741] [<ffffffff8163ecf2>] ? netif_tx_lock+0x74/0x81
[ 238.921743] [<ffffffff8163edd3>] dev_watchdog+0xd4/0x12f
[ 238.921746] [<ffffffff8113cfbb>] call_timer_fn+0x65/0x11b
[ 238.921748] [<ffffffff8163ecff>] ? netif_tx_lock+0x81/0x81
[ 238.921749] [<ffffffff8113d73e>] run_timer_softirq+0x1ad/0x1d7
[ 238.921751] [<ffffffff810ee701>] __do_softirq+0xfb/0x25c
[ 238.921752] [<ffffffff810ee87b>] run_ksoftirqd+0x19/0x35
[ 238.921755] [<ffffffff81105ae8>] smpboot_thread_fn+0x169/0x1a9
[ 238.921756] [<ffffffff8110597f>] ? sort_range+0x1d/0x1d
[ 238.921759] [<ffffffff811031a1>] kthread+0xa0/0xa8
[ 238.921763] [<ffffffff816ce19f>] ret_from_fork+0x1f/0x40
[ 238.921764] [<ffffffff81103101>] ? init_completion+0x24/0x24
[ 238.921765] ---[ end trace 933912956c6ee5ff ]---
[ 238.961672] e1000e 0000:07:00.0 eth3: Reset adapter unexpectedly
So, on 4.4.8+, I see this and other splats related to e1000e. I guess that is
a separate
issue. I can easily start 40k connections however, 30k across the two 10G
ports,
and 10k more across a pair of mac-vlans on the 10G ports (since I was out of
address space to add a full 40k on the two physical ports).
Looks like the e1000e problem is a separate issue, so I'll just focus on the
10G NIC for now.
Thanks,
Ben
--
Ben Greear <gree...@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
