BBlack has submitted this change and it was merged.
Change subject: r::c::perf: un-mysterious somaxconn + syn_backlog
......................................................................
r::c::perf: un-mysterious somaxconn + syn_backlog
Both values are raised based on current stats and thinking, and
now better-documented. tlsproxy template also updated so that
nginx takes advantage of the raised somaxconn.
Change-Id: I174e5be9dbf2e42022ca322affca2525e70d8ec3
---
M modules/role/manifests/cache/perf.pp
M modules/tlsproxy/templates/localssl.erb
2 files changed, 18 insertions(+), 6 deletions(-)
Approvals:
BBlack: Verified; Looks good to me, approved
diff --git a/modules/role/manifests/cache/perf.pp
b/modules/role/manifests/cache/perf.pp
index feaf1fa..fdceffb 100644
--- a/modules/role/manifests/cache/perf.pp
+++ b/modules/role/manifests/cache/perf.pp
@@ -96,13 +96,25 @@
# will see drops in col 2 of /proc/net/softnet_stat
'net.core.netdev_max_backlog' => 60000,
- # Increase the queue size of new TCP connections
- 'net.core.somaxconn' => 4096, # 'mysterious'
- 'net.ipv4.tcp_max_syn_backlog' => 262144, # 'mysterious'
- 'net.ipv4.tcp_max_tw_buckets' => 360000, # 'mysterious'
+ # Our rate of incoming SYN on heaviest cp hosts peaks around
+ # 1-2K/sec. For somaxconn, the SYN numbers should be multiplied
+ # out for a few seconds of headroom (bursts, and userspace delays)
+ # and then perhaps doubled again to handle the influx of depooling
+ # large datacenters. Note somaxconn is just a parameter limit, the
+ # application still needs to set this explicitly (within the
+ # limit).
+ 'net.core.somaxconn' => 16384,
+
+ # Our active connection concurrency peaks in the ~100K-200K range
+ # per cp host (e.g. text esams as shown in ipvsadm). For
+ # max_syn_backlog, we probably want a small multiple of peak
+ # concurrency (maybe even just ~1x), as well as (again) dc failover
+ # and/or cp host depool headroom.
+ 'net.ipv4.tcp_max_syn_backlog' => 524288,
# Decrease FD/socket usage
'net.ipv4.tcp_tw_reuse' => 1,
+ 'net.ipv4.tcp_max_tw_buckets' => 360000, # 'mysterious'
'net.ipv4.tcp_fin_timeout' => 3, # 'mysterious'
'net.ipv4.tcp_max_orphans' => 262144, # 'mysterious'
'net.ipv4.tcp_synack_retries' => 2, # 'mysterious'
diff --git a/modules/tlsproxy/templates/localssl.erb
b/modules/tlsproxy/templates/localssl.erb
index 36bd037..d4b0635 100644
--- a/modules/tlsproxy/templates/localssl.erb
+++ b/modules/tlsproxy/templates/localssl.erb
@@ -9,8 +9,8 @@
# SSL proxying
server {
- listen [::]:443 <%= @default_server ? "default_server deferred
backlog=4096 reuseport ipv6only=on " : "" %>ssl http2;
- listen 443 <%= @default_server ? "default_server deferred backlog=4096
reuseport " : "" %>ssl http2;
+ listen [::]:443 <%= @default_server ? "default_server deferred
backlog=16384 reuseport ipv6only=on " : "" %>ssl http2;
+ listen 443 <%= @default_server ? "default_server deferred backlog=16384
reuseport " : "" %>ssl http2;
ssl on;
server_name <%= ([@server_name] + @server_aliases).join(" ") %>;
--
To view, visit https://gerrit.wikimedia.org/r/295540
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I174e5be9dbf2e42022ca322affca2525e70d8ec3
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: BBlack <bbl...@wikimedia.org>
Gerrit-Reviewer: BBlack <bbl...@wikimedia.org>
Gerrit-Reviewer: Ema <e...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
