Muehlenhoff has uploaded a new change for review. https://gerrit.wikimedia.org/r/240361
Change subject: Lower the conntrack tracking time for TIME_WAIT connections ...................................................................... Lower the conntrack tracking time for TIME_WAIT connections TCP connections in TIME_WAIT are maintained for sixty seconds by the Linux kernel. Note: There's many misleading Google hits indicating that this is configurable through the sysctl value net.ipv4.tcp_fin_timeout but after some digging that turned out to be bogus. It's a constant defined in the Linux lernel in include/net/tcp.h: However nf_conntrack tracks these for 120 seconds by default (configurable through the sysctl value nf_conntrack_tcp_timeout_time_wait) Reduce this to 65 (the maximum time used by the kernel plus a five seconds error margin). I tried this value manually on the initial job runner which has ferm enabled (mw1010) which had around 100k tracked connections of which nearly 58k were in TIME_WAIT. Reducing nf_conntrack_tcp_timeout_time_wait to 65 shrunk that down to about 70k. This patch also removes the custom setting for swift (it was initially reduced to 3 since the Swift performance docs recommend net.ipv4.tcp_fin_timeout=3 and it was assumed that this also effects the TIME_WAIT interval) for consistency. I've also tested this on ms-[fb]e1002 and it works fine. Bug: T105307 Change-Id: Ia3fb71591a9482daf022b314eb00d25397f5b492 --- M modules/base/manifests/firewall.pp M modules/swift/manifests/init.pp 2 files changed, 2 insertions(+), 5 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/61/240361/1 diff --git a/modules/base/manifests/firewall.pp b/modules/base/manifests/firewall.pp index bf0ae53..937aa71 100644 --- a/modules/base/manifests/firewall.pp +++ b/modules/base/manifests/firewall.pp @@ -19,7 +19,8 @@ # Increase the size of conntrack table size (default is 65536) sysctl::parameters { 'ferm_conntrack': values => { - 'net.netfilter.nf_conntrack_max' => 262144, + 'net.netfilter.nf_conntrack_max' => 262144, + 'net.netfilter.nf_conntrack_tcp_timeout_time_wait' => 65, }, } diff --git a/modules/swift/manifests/init.pp b/modules/swift/manifests/init.pp index cd49cbb..d34ace2 100644 --- a/modules/swift/manifests/init.pp +++ b/modules/swift/manifests/init.pp @@ -30,10 +30,6 @@ 'net.ipv4.tcp_max_orphans' => 262144, 'net.ipv4.tcp_synack_retries' => 2, 'net.ipv4.tcp_syn_retries' => 2, - - # even with NOTRACK enabled, conntrack will still keep track - # connections in TIME_WAIT, thus lower the respective timeout - 'net.netfilter.nf_conntrack_tcp_timeout_time_wait' => 3, }, } -- To view, visit https://gerrit.wikimedia.org/r/240361 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia3fb71591a9482daf022b314eb00d25397f5b492 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Muehlenhoff <mmuhlenh...@wikimedia.org> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits