Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
sys_ticrate 1 is bad. When there are enought resources in the box (tested with single hlds server, -15 priority and taskset'ed to single core, pingboost 3, game CS 1.6), everything will run faster than normal. Especially you can see it by HE flying speed, reload animation, running on the stairs. Every popular server hosting company is using sys_ticrate 1000. - en3my www.2Po.eu - Original Message - From: Kveri [EMAIL PROTECTED] To: Half-Life dedicated Linux server mailing list hlds_linux@list.valvesoftware.com Sent: Sunday, September 21, 2008 6:54 PM Subject: Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load yes, FPS jumping from 566 to 998, kernel is 2.6.24.3 hardened, 1000Hz timer, Core 2/never Xeon, sys_ticrate 1, TF2 war tickrate 66, public 33, pingboost 0 (not using pingboost, because kernel is boosted) rates at max. Marcel wrote / napísal(a): Kveri schrieb: second: 2x quad-core xeon E5335, 16gb ddr 667MHz, gentoo64 - hosting 1000FPS CS, TF2 servers, 15 CS1.6 1000FPS servers (~100ppls at max), 2TF2 32 slot publics (always full) and 4 TF2 match servers, ~70% utilization, no lags no problem. Do you get stable 1000FPS when this server is so heavy used (100+64 Slots)? If yes, what's your kernel and gs (pb,ticrate etc) config? cheers Marcel ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
I know, that was happening to me only when I was using -pingboost 3 param, without pingboost (I'm not using pingboost since I have 1000Hz kernel) it runs normal. Kveri en3my wrote / napísal(a): sys_ticrate 1 is bad. When there are enought resources in the box (tested with single hlds server, -15 priority and taskset'ed to single core, pingboost 3, game CS 1.6), everything will run faster than normal. Especially you can see it by HE flying speed, reload animation, running on the stairs. Every popular server hosting company is using sys_ticrate 1000. - en3my www.2Po.eu - Original Message - From: Kveri [EMAIL PROTECTED] To: Half-Life dedicated Linux server mailing list hlds_linux@list.valvesoftware.com Sent: Sunday, September 21, 2008 6:54 PM Subject: Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load yes, FPS jumping from 566 to 998, kernel is 2.6.24.3 hardened, 1000Hz timer, Core 2/never Xeon, sys_ticrate 1, TF2 war tickrate 66, public 33, pingboost 0 (not using pingboost, because kernel is boosted) rates at max. Marcel wrote / napísal(a): Kveri schrieb: second: 2x quad-core xeon E5335, 16gb ddr 667MHz, gentoo64 - hosting 1000FPS CS, TF2 servers, 15 CS1.6 1000FPS servers (~100ppls at max), 2TF2 32 slot publics (always full) and 4 TF2 match servers, ~70% utilization, no lags no problem. Do you get stable 1000FPS when this server is so heavy used (100+64 Slots)? If yes, what's your kernel and gs (pb,ticrate etc) config? cheers Marcel ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
Remember one machine has one kernel running (which itself is a program) and all your servers will be competing to get access to it. The sudden jump in user CPU is when each game is having to poll to get exclusive access to the kernel to perform the same sort of operation. Your kernel cannot keep up. There are certain lock/mutex operations where it polls rather than waits, as it doesn't expect to be held up. That polling would shoot up on user cpu time. This happens suddenly as it's either keeping up, or its not. When it can't keep up a polling queue suddenly forms. This is why faster cpus are better then many slower as the kernel can run faster and keep up with the requests from more game programs. You can prove this to yourself by running strace -c -p pid on one of the games when they are all lagging. Wait about 10 seconds, then ctrl-C to exit. This produces a summary of where all the time is being spent. Also run this on a game when everything is fine as a comparison. If the strace shows all the time is spent on kernel contention, you should look to run less servers on your box. It's the number of active servers in conjunction with the number of slots which is getting you in to that position. Each individual game server is trying to access the kernel. On UKCS we always go for the fastest latest generation processors, have a maximum of 4 cores per kernel, and never run more than one game per core. This avoids any contention issues. But we're a public game group, not a game rental group where I appreciate its not cost effective for you to go to that extreme. You could also look at your kernel config to ensure you've made it as near realtime as you can to optimise the situation. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Morgan Sent: 20 September 2008 23:08 To: hlds_linux@list.valvesoftware.com Subject: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load Hello ppl, I've a huge problem with performance of my servers. I'm hosting arround 30 gameservers per 1 server, mostly CS 1.6, few CS:S and CS:CZ. Allways when ~160-200 players play CPUs load jump from 30% to ~70-80% (suddenly all processes are using more power of CPU) and players get lags. No matter what kind of gameservers are working. On 1 server I'm put only CS 1.6, on another I mixed CS 1.6 with CS:S. Both has these same problem. I tried in 2 different Data Centers so it's not a network problem. Maybe my system was improperly setup. Configuration of servers: 2x Quad E5420, motherboard S5000PAL, 6x2GB FB-DIMM 667, 2x SAS with RAID 1 (i tried with 1 disk, no positive result). System: Slackware 12.1 and Centos 5.2 Kernel: 2.6.24.3, 2.6.25.4, 2.6.26.2 for Slack and 2.6.18-92.el5PAE for Centos (default) Kernel's Processor Settings which I tested: BIG SMP/PC Compatible, 300/1000Hz, low-latency/server, preempt big kernel on/off, Core 2/newer Xeon How it looks: Players: http://img81.imageshack.us/my.php?image=playersrt1.jpg CPU Load: http://img221.imageshack.us/my.php?image=cpucr5.jpg I don't know what can I do. Old Dual Xeons 3.0Ghz with HT works fine with ~100. I think that 2x Quad-Core Xeon should works fine with 300 playing players. Don't You think so? P.S. Sorry for my english ;-) Regards, J.Morgan ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
Try removing one processor. Btw I have 3 rigs, first: 2x dual-core xeon 5050, 8gb ddr 667MHz debian64 - sometimes lags, but I have mysql, apache and others stuff on this server, moving to dedicated server next month ;), second: 2x quad-core xeon E5335, 16gb ddr 667MHz, gentoo64 - hosting 1000FPS CS, TF2 servers, 15 CS1.6 1000FPS servers (~100ppls at max), 2TF2 32 slot publics (always full) and 4 TF2 match servers, ~70% utilization, no lags no problem. Third: 2x quad-core X5470 16gb ddr2 667MHz gentoo64 - hosting only 333FPS CS1.6 servers, now 63 servers at max ~450 ppls, util. 60% no lags no problems, but when I was using -pingboost 3 - lags, so if you're using -pingboost (1,2,3 no matter) at 1000Hz kernel, stop, server is already boosted just set sys_ticrate. Kveri John Morgan wrote / napísal(a): Hello ppl, I've a huge problem with performance of my servers. I'm hosting arround 30 gameservers per 1 server, mostly CS 1.6, few CS:S and CS:CZ. Allways when ~160-200 players play CPUs load jump from 30% to ~70-80% (suddenly all processes are using more power of CPU) and players get lags. No matter what kind of gameservers are working. On 1 server I'm put only CS 1.6, on another I mixed CS 1.6 with CS:S. Both has these same problem. I tried in 2 different Data Centers so it's not a network problem. Maybe my system was improperly setup. Configuration of servers: 2x Quad E5420, motherboard S5000PAL, 6x2GB FB-DIMM 667, 2x SAS with RAID 1 (i tried with 1 disk, no positive result). System: Slackware 12.1 and Centos 5.2 Kernel: 2.6.24.3, 2.6.25.4, 2.6.26.2 for Slack and 2.6.18-92.el5PAE for Centos (default) Kernel's Processor Settings which I tested: BIG SMP/PC Compatible, 300/1000Hz, low-latency/server, preempt big kernel on/off, Core 2/newer Xeon How it looks: Players: http://img81.imageshack.us/my.php?image=playersrt1.jpg CPU Load: http://img221.imageshack.us/my.php?image=cpucr5.jpg I don't know what can I do. Old Dual Xeons 3.0Ghz with HT works fine with ~100. I think that 2x Quad-Core Xeon should works fine with 300 playing players. Don't You think so? P.S. Sorry for my english ;-) Regards, J.Morgan ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
Kveri schrieb: second: 2x quad-core xeon E5335, 16gb ddr 667MHz, gentoo64 - hosting 1000FPS CS, TF2 servers, 15 CS1.6 1000FPS servers (~100ppls at max), 2TF2 32 slot publics (always full) and 4 TF2 match servers, ~70% utilization, no lags no problem. Do you get stable 1000FPS when this server is so heavy used (100+64 Slots)? If yes, what's your kernel and gs (pb,ticrate etc) config? cheers Marcel ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
yes, FPS jumping from 566 to 998, kernel is 2.6.24.3 hardened, 1000Hz timer, Core 2/never Xeon, sys_ticrate 1, TF2 war tickrate 66, public 33, pingboost 0 (not using pingboost, because kernel is boosted) rates at max. Marcel wrote / napísal(a): Kveri schrieb: second: 2x quad-core xeon E5335, 16gb ddr 667MHz, gentoo64 - hosting 1000FPS CS, TF2 servers, 15 CS1.6 1000FPS servers (~100ppls at max), 2TF2 32 slot publics (always full) and 4 TF2 match servers, ~70% utilization, no lags no problem. Do you get stable 1000FPS when this server is so heavy used (100+64 Slots)? If yes, what's your kernel and gs (pb,ticrate etc) config? cheers Marcel ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
My thought on this, if you are running 10+ servers with 500fps 100 tick then ya you are probably hitting the limits of the hardware in terms of context switching. I run a similar box and can't go more than about 30% cpu usage on it without problems with lag. John Morgan wrote: Hello ppl, I've a huge problem with performance of my servers. I'm hosting arround 30 gameservers per 1 server, mostly CS 1.6, few CS:S and CS:CZ. Allways when ~160-200 players play CPUs load jump from 30% to ~70-80% (suddenly all processes are using more power of CPU) and players get lags. No matter what kind of gameservers are working. On 1 server I'm put only CS 1.6, on another I mixed CS 1.6 with CS:S. Both has these same problem. I tried in 2 different Data Centers so it's not a network problem. Maybe my system was improperly setup. Configuration of servers: 2x Quad E5420, motherboard S5000PAL, 6x2GB FB-DIMM 667, 2x SAS with RAID 1 (i tried with 1 disk, no positive result). System: Slackware 12.1 and Centos 5.2 Kernel: 2.6.24.3, 2.6.25.4, 2.6.26.2 for Slack and 2.6.18-92.el5PAE for Centos (default) Kernel's Processor Settings which I tested: BIG SMP/PC Compatible, 300/1000Hz, low-latency/server, preempt big kernel on/off, Core 2/newer Xeon How it looks: Players: http://img81.imageshack.us/my.php?image=playersrt1.jpg CPU Load: http://img221.imageshack.us/my.php?image=cpucr5.jpg I don't know what can I do. Old Dual Xeons 3.0Ghz with HT works fine with ~100. I think that 2x Quad-Core Xeon should works fine with 300 playing players. Don't You think so? P.S. Sorry for my english ;-) Regards, J.Morgan ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
Hmm that something with game sourcecode :) I have the same configuration, but with only 1 CPU. I can hold about 180 players with 6 public CS 1.6 servers and 5 CS:S server (4 with TICK 100) Everything takes about 80% CPU but we don't feel any lagging on it. And tickrate in CS:S is stable and FPS in CS 1.6, crazy? But i tested it a month ago with 2 processors on the motherboard. We had about 30-40% CPU usage, we felt lags, tickrate (CS:S) wasn't stable and FPS (CS 1.6) So we remove one and problem disappeared (Of course i tested other models, because this one might be bad but problem was also on other processors) It is stupid but game quality is better on ONE full loaded CPU (~80%) than on two with smaller load (30%-40%). Crazy! :D So the second processor is lost of money. Maybe somebody find any solution? On Sun, Sep 21, 2008 at 1:54 AM, Midnight [EMAIL PROTECTED] wrote: My thought on this, if you are running 10+ servers with 500fps 100 tick then ya you are probably hitting the limits of the hardware in terms of context switching. I run a similar box and can't go more than about 30% cpu usage on it without problems with lag. John Morgan wrote: Hello ppl, I've a huge problem with performance of my servers. I'm hosting arround 30 gameservers per 1 server, mostly CS 1.6, few CS:S and CS:CZ. Allways when ~160-200 players play CPUs load jump from 30% to ~70-80% (suddenly all processes are using more power of CPU) and players get lags. No matter what kind of gameservers are working. On 1 server I'm put only CS 1.6, on another I mixed CS 1.6 with CS:S. Both has these same problem. I tried in 2 different Data Centers so it's not a network problem. Maybe my system was improperly setup. Configuration of servers: 2x Quad E5420, motherboard S5000PAL, 6x2GB FB-DIMM 667, 2x SAS with RAID 1 (i tried with 1 disk, no positive result). System: Slackware 12.1 and Centos 5.2 Kernel: 2.6.24.3, 2.6.25.4, 2.6.26.2 for Slack and 2.6.18-92.el5PAE for Centos (default) Kernel's Processor Settings which I tested: BIG SMP/PC Compatible, 300/1000Hz, low-latency/server, preempt big kernel on/off, Core 2/newer Xeon How it looks: Players: http://img81.imageshack.us/my.php?image=playersrt1.jpg CPU Load: http://img221.imageshack.us/my.php?image=cpucr5.jpg I don't know what can I do. Old Dual Xeons 3.0Ghz with HT works fine with ~100. I think that 2x Quad-Core Xeon should works fine with 300 playing players. Don't You think so? P.S. Sorry for my english ;-) Regards, J.Morgan ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
Pawel wrote: Hmm that something with game sourcecode :) I have the same configuration, but with only 1 CPU. I can hold about 180 players with 6 public CS 1.6 servers and 5 CS:S server (4 with TICK 100) Everything takes about 80% CPU but we don't feel any lagging on it. And tickrate in CS:S is stable and FPS in CS 1.6, crazy? But i tested it a month ago with 2 processors on the motherboard. We had about 30-40% CPU usage, we felt lags, tickrate (CS:S) wasn't stable and FPS (CS 1.6) So we remove one and problem disappeared (Of course i tested other models, because this one might be bad but problem was also on other processors) It is stupid but game quality is better on ONE full loaded CPU (~80%) than on two with smaller load (30%-40%). Crazy! :D So the second processor is lost of money. Maybe somebody find any solution? On Sun, Sep 21, 2008 at 1:54 AM, Midnight [EMAIL PROTECTED] wrote: My thought on this, if you are running 10+ servers with 500fps 100 tick then ya you are probably hitting the limits of the hardware in terms of context switching. I run a similar box and can't go more than about 30% cpu usage on it without problems with lag. John Morgan wrote: Hello ppl, I've a huge problem with performance of my servers. I'm hosting arround 30 gameservers per 1 server, mostly CS 1.6, few CS:S and CS:CZ. Allways when ~160-200 players play CPUs load jump from 30% to ~70-80% (suddenly all processes are using more power of CPU) and players get lags. No matter what kind of gameservers are working. On 1 server I'm put only CS 1.6, on another I mixed CS 1.6 with CS:S. Both has these same problem. I tried in 2 different Data Centers so it's not a network problem. Maybe my system was improperly setup. Configuration of servers: 2x Quad E5420, motherboard S5000PAL, 6x2GB FB-DIMM 667, 2x SAS with RAID 1 (i tried with 1 disk, no positive result). System: Slackware 12.1 and Centos 5.2 Kernel: 2.6.24.3, 2.6.25.4, 2.6.26.2 for Slack and 2.6.18-92.el5PAE for Centos (default) Kernel's Processor Settings which I tested: BIG SMP/PC Compatible, 300/1000Hz, low-latency/server, preempt big kernel on/off, Core 2/newer Xeon How it looks: Players: http://img81.imageshack.us/my.php?image=playersrt1.jpg CPU Load: http://img221.imageshack.us/my.php?image=cpucr5.jpg I don't know what can I do. Old Dual Xeons 3.0Ghz with HT works fine with ~100. I think that 2x Quad-Core Xeon should works fine with 300 playing players. Don't You think so? P.S. Sorry for my english ;-) Regards, J.Morgan ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux not it's not crazy. Running more than one cpu means the OS and programs have to keep track of which core they are on, maintain cache coherency, maintain the memory mappings..etc etc. If(when) the thread jumps to another cpu then the whole dataset has to get rarranged. I would start hard assigning servers to their own core and see if that helps out. ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
Re: [hlds_linux] Intel Quad-Core Xeon - problem with CPU load
I've done the CPU Core pinning thing, didn't help anything in my case. I don't think it is swapping cores that is the problem, it is just the ability to get CPU time in a stable and consistent rate with so many games swapping in/out of the CPU. I think less games with higher slot counts is the only way to maximize a 8-way Xeon setup. I have a theory that FB RAM also makes the situation worse by adding additional latency to the overall system performance. We see a DPC latency of 3000us vs 30-50us on a server board vs non server board. I have setup a new server just for running 500-1000FPS CS/CSS games. It is DDR3 non FB RAM, 1 CPU and overclocked to the max, basically an enthusiast board setup vs a server setup. So far in testing it seems to be able to run more high speed servers than a 2x Xeon server board. So we are looking to use that server to run more lower end game servers that are not cranked to the max on the tickrate. not it's not crazy. Running more than one cpu means the OS and programs have to keep track of which core they are on, maintain cache coherency, maintain the memory mappings..etc etc. If(when) the thread jumps to another cpu then the whole dataset has to get rarranged. I would start hard assigning servers to their own core and see if that helps out. ___ To unsubscribe, edit your list preferences, or view the list archives, please visit: http://list.valvesoftware.com/mailman/listinfo/hlds_linux
