> -----Original Message----- > From: [email protected] [mailto:lvs-users- > [email protected]] On Behalf Of Neal B > Sent: Friday, August 06, 2010 2:16 PM > To: [email protected] > Subject: [lvs-users] Kernel Tuning for LVS > > Over the past day I have been searching the net for the definitive answer for > the recommended tweaks to the kernel for ipvs. If you are doing this then > change this, but that does not exist. :) >
Hey, I've done a few things to tune the network for large-traffic here, and I'm going to share with you the values I set, what I set them to, and why. Please remember YMMV and don't modify something unless you completely understand what it does. In a nutshell, if you break stuff using these values, it's not my fault :). These values are on 2.6.27.46 -- I'm unsure as to what they do or translate to in newer or older kernels. SYSCTLS: These three values make the ARP cache larger. This is not needed in all environments, but generally only costs ram so it doesn't hurt to increase it: net.ipv4.neigh.default.gc_thresh3 = 8192 net.ipv4.neigh.default.gc_thresh2 = 4096 net.ipv4.neigh.default.gc_thresh1 = 1024 The next options are for TCP window scaling, I won't go into details because there are so many internet howtos on how to tweak your window scaling, but here are the places and what I usually set them to: net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65535 16777216 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_moderate_rcvbuf = 1 Next, I always disable proxy arp (RHEL, by default, sets the kernel to arp any IP on the box on any interface; usually this is undesirable and can break things for edge routers) -- make sure to set these values for net.ipv4.conf.[interface].arp_[ignore|announce] as well. net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.default.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.default.arp_announce = 2 A couple of other important sysctls imo: net.core.netdev_max_backlog = 2500 # Increases the receive queue in the kernel net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 86400 # lowers default timeout (default: 5d - I think it's a little silly) Non-SYSCTL tweaks: /sbin/ifconfig eth[#] txqueuelen 2000 # increases the size of the transmit queue Check to see what the max ring buffer size is for your NIC card (should be listed under Pre-set Maximums): ethtool -g eth[#] If any of the "Current Hardware Settings" values are less than the maximums, set them to the max: ethtool -G eth[#] rx [value] ethtool -G eth[#] tx [value] The biggest tweak I can advise you to do though is to *run a good network card*. I don't put a load balancer in production, regardless of load, unless it's running an e1000[e] driver NIC. These drivers have proven themselves the most stable of any I've run, and have an additional setting you can set to lower interrupts on a high-load system: (inside modprobe.conf) options e1000[e] InterruptThrottleRate=3000,3000,3000,3000 This tells the e1000 driver to *never* make more than 3000 interrupts per second. This is good if you want pure volume of traffic at the expense of a little more latency (very little more, really). I hope this has been helpful. Please let me know if you have any questions. -- Jason Faulkner Linux Engineer, Rackspace Email & Apps [email protected] _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - [email protected] Send requests to [email protected] or go to http://lists.graemef.net/mailman/listinfo/lvs-users
