Re: Question about threads
Im running 32bit. But I think that I have succeded creating more then 238 threads before on another system with the same setup. Anyway, 64bit might be the thing to have... If I want to have Debian, is it AMD64 version that I should go for? (OT) / E Poul-Henning Kamp skrev: In message [EMAIL PROTECTED], Erik Torlen writes: I still have the same problem :( The threads are created up to 238 where they are stopped, even if I set threads_max = 1000 and threads_pools = 2 (or 3). I also tested the tips and decreased the stack sixe to 512 and increased overflow_max to 1% . Any idea what could be wrong? Are you running on a 32bit or 64bit system ? On a 32bit system you may simply be running out of address-space... ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about threads
In message [EMAIL PROTECTED], Erik Torlen writes: Im running 32bit. But I think that I have succeded creating more then 238 threads before on another system with the same setup. Anyway, 64bit might be the thing to have... If I want to have Debian, is it AMD64 version that I should go for? (OT) Yes. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about threads
Erik Torlen skrev: Im running 32bit. But I think that I have succeded creating more then 238 threads before on another system with the same setup. Anyway, 64bit might be the thing to have... Maybe the init script should issue a warning that a 32bit arch is only usable as a test enviroment. Per. signature.asc Description: OpenPGP digital signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about threads
In message [EMAIL PROTECTED], Per Buer writes: Maybe the init script should issue a warning that a 32bit arch is only usable as a test enviroment. Well, Varnish is generally usable in 32bit, provided you have very small content, so I'm hessitant to rule it entirely out, but yes, we clearly need to push the 64bit angle. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Re: Question about threads
Thanks for the tips, I will test this and come back with the result. / E Original Message --- On Wed, 18 Jun 2008 13:24:52 -0700 Michael S. Fischer [EMAIL PROTECTED] wrote: On Wed, Jun 18, 2008 at 4:51 AM, Rafael Umann [EMAIL PROTECTED] wrote: If it is a 32bits system, probably the problem is that your stack size is 10Mb. So 238 * 10mb = ~2gb I decreased my stack size to 512Kb. Using 1gb storage files i can now open almost 1900 threads using all the 2gb that 32bits can alloc. So, my Varnish makes 2 hits/second serving clients! What is your request:connection ratio? Best regards, --Michael Unfortunately now i dont have servers doing 2 hits/second, and thats why i dont have stats for you. I have 6 servers runing this service now, each one doing 5500 hits/sec with 10% CPU usage, and this infrastructure scales to 2 hits/sec each server. For test purpose only i let just 2 servers with all the traffic, so i saw this limit i told you. In this 2 hits/sec im seeing two bottle necks: 1. The 32bits arch (cant open threads and the storage file is too small), so im moving into 64bits. 2. The cpu usage of the listener process with 2 hits/sec is getting to 100% in one CPU (we also made a patch to improve CPU usage, and it is avaliable in this last trunk version.. without this patch varnish was doing 8000 hits/sec). But anyway, i spoke to Poul, and he told me that the listener process will be changed soon (i really hope so, because i would love to use more than 30% of my cpu!). Anyway, varnish is doing a great job for me!!! Some stats taken now with 5500 hits/sec: # netstat -nap|grep :80 |grep SYN_REC |wc -l 166 # netstat -nap|grep :80 |grep ESTAB |wc -l 17139 # netstat -nap|grep :80 |grep FIN |wc -l 1454 # netstat -nap|grep :80 |grep TIME_W |wc -l 7766 # varnishstat Hitrate ratio: 10 13 13 Hitrate avg: 0.9990 0.9990 0.9990 36189916 476.66 250.71 Client connections accepted 404804204 5494.13 2804.30 Client requests received 391586079 5335.24 2712.74 Cache hits 10975643 157.8976.03 Cache hits for pass 1125709 1.00 7.80 Cache misses 12101360 158.8983.83 Backend connections success 0 0.00 0.00 Backend connections failures 0 0.00 0.00 Backend connections reuses 11139673 141.9077.17 Backend connections recycles 350-1.00 0.00 Backend connections unused 13447 .. N struct srcaddr 11353 .. N active struct srcaddr 30351 .. N struct sess_mem 17391 .. N struct sess 47069 .. N struct object 47080 .. N struct objecthead 97813 .. N struct smf 4185 .. N small free smf 5 .. N large free smf 0 .. N struct vbe_conn 556 .. N worker threads 556 0.00 0.00 N worker threads created 0 0.00 0.00 N worker threads not created 0 0.00 0.00 N worker threads limited 0 0.00 0.00 N queued work requests 556 0.00 0.00 N overflowed work requests 0 0.00 0.00 N dropped work requests 0 .. N expired objects 1079159 .. N LRU nuked objects 0 .. N LRU saved objects 0 .. N objects on deathrow 48 0.00 0.00 HTTP header overflows 0 0.00 0.00 Objects sent with sendfile 135051222 1812.72 935.58 Objects sent with write 36189893 494.65 250.71 Total Sessions 404810596 5491.13 2804.35 Total Requests 0 0.00 0.00 Total pipe 0 0.00 0.00 Total pass 12101371 158.8983.83 Total fetch 93389657948 1269372.98646962.32 Total header bytes 418107385660 5962840.69 2896463.38 Total body bytes 541249189.9437.50 Session Closed 0 0.00 0.00 Session Pipeline 1623343 0.0011.25 Session Read Ahead 398293975 5409.19 2759.20 Session herd 15780035398212996.82109317.12 SHM records 516154306 6307.55 3575.69 SHM writes 266181533.9818.44 SHM MTX contention 23279696 242.83 161.27 allocator requests 93623 .. outstanding allocations 910299136 .. bytes allocated []s, -- Rafael Umann [EMAIL PROTECTED] Suporte
Re: Question about threads
In message [EMAIL PROTECTED], [EMAIL PROTECTED] writes: 1. The 32bits arch (cant open threads and the storage file is too small), so im moving into 64bits. Yes, 32bit is generally not big enough to Varnish for non-trivial workloads. 2. The cpu usage of the listener process with 2 hits/sec is getting to 100% in one CPU. You may want to try to update to a -trunk version and play a bit with the session_linger parameter, every time you get a tick in the Session Linger counter, that's one request that took a shortcut. I have an idea for a more comprehensive shortcut for new sessions but that will have to wait until after 2.0. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about threads
how about your storage file size? []s, On Mon, 23 Jun 2008 21:20:59 +0200 Erik Torlen [EMAIL PROTECTED] wrote: I still have the same problem :( The threads are created up to 238 where they are stopped, even if I set threads_max = 1000 and threads_pools = 2 (or 3). I also tested the tips and decreased the stack sixe to 512 and increased overflow_max to 1% . Any idea what could be wrong? / E [EMAIL PROTECTED] skrev: Thanks for the tips, I will test this and come back with the result. / E E-mail verificado pelo Terra Anti-Spam. Para classificar como spam ou não spam, visite http://mail.terra.com.br/cgi-bin/reportspam.cgi?+_d=SCY1ODAyNDQ3I3Blcm0hdGVycmEmMSwxMjE0MjQ4ODYwLjkzNTUyMC4yNTIxOS5nYW5hbm9xdWUudGVycmEuY29tLDE3MTA= Verifique periodicamente a pasta Spam para garantir que apenas mensagens indesejadas sejam classificadas como Spam. Esta mensagem foi verificada pelo E-mail Protegido Terra. Atualizado em 23/06/2008 -- Rafael Umann [EMAIL PROTECTED] Suporte Engenharia 1 Terra Networks Brasil S/A Tel: 55 (51) 3284-4344 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about threads
In message [EMAIL PROTECTED], Erik Torlen writes: I still have the same problem :( The threads are created up to 238 where they are stopped, even if I set threads_max = 1000 and threads_pools = 2 (or 3). I also tested the tips and decreased the stack sixe to 512 and increased overflow_max to 1% . Any idea what could be wrong? Are you running on a 32bit or 64bit system ? On a 32bit system you may simply be running out of address-space... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about threads
On Wed, 18 Jun 2008 13:24:52 -0700 Michael S. Fischer [EMAIL PROTECTED] wrote: On Wed, Jun 18, 2008 at 4:51 AM, Rafael Umann [EMAIL PROTECTED] wrote: If it is a 32bits system, probably the problem is that your stack size is 10Mb. So 238 * 10mb = ~2gb I decreased my stack size to 512Kb. Using 1gb storage files i can now open almost 1900 threads using all the 2gb that 32bits can alloc. So, my Varnish makes 2 hits/second serving clients! What is your request:connection ratio? Best regards, --Michael Unfortunately now i dont have servers doing 2 hits/second, and thats why i dont have stats for you. I have 6 servers runing this service now, each one doing 5500 hits/sec with 10% CPU usage, and this infrastructure scales to 2 hits/sec each server. For test purpose only i let just 2 servers with all the traffic, so i saw this limit i told you. In this 2 hits/sec im seeing two bottle necks: 1. The 32bits arch (cant open threads and the storage file is too small), so im moving into 64bits. 2. The cpu usage of the listener process with 2 hits/sec is getting to 100% in one CPU (we also made a patch to improve CPU usage, and it is avaliable in this last trunk version.. without this patch varnish was doing 8000 hits/sec). But anyway, i spoke to Poul, and he told me that the listener process will be changed soon (i really hope so, because i would love to use more than 30% of my cpu!). Anyway, varnish is doing a great job for me!!! Some stats taken now with 5500 hits/sec: # netstat -nap|grep :80 |grep SYN_REC |wc -l 166 # netstat -nap|grep :80 |grep ESTAB |wc -l 17139 # netstat -nap|grep :80 |grep FIN |wc -l 1454 # netstat -nap|grep :80 |grep TIME_W |wc -l 7766 # varnishstat Hitrate ratio: 10 13 13 Hitrate avg: 0.9990 0.9990 0.9990 36189916 476.66 250.71 Client connections accepted 404804204 5494.13 2804.30 Client requests received 391586079 5335.24 2712.74 Cache hits 10975643 157.8976.03 Cache hits for pass 1125709 1.00 7.80 Cache misses 12101360 158.8983.83 Backend connections success 0 0.00 0.00 Backend connections failures 0 0.00 0.00 Backend connections reuses 11139673 141.9077.17 Backend connections recycles 350-1.00 0.00 Backend connections unused 13447 .. N struct srcaddr 11353 .. N active struct srcaddr 30351 .. N struct sess_mem 17391 .. N struct sess 47069 .. N struct object 47080 .. N struct objecthead 97813 .. N struct smf 4185 .. N small free smf 5 .. N large free smf 0 .. N struct vbe_conn 556 .. N worker threads 556 0.00 0.00 N worker threads created 0 0.00 0.00 N worker threads not created 0 0.00 0.00 N worker threads limited 0 0.00 0.00 N queued work requests 556 0.00 0.00 N overflowed work requests 0 0.00 0.00 N dropped work requests 0 .. N expired objects 1079159 .. N LRU nuked objects 0 .. N LRU saved objects 0 .. N objects on deathrow 48 0.00 0.00 HTTP header overflows 0 0.00 0.00 Objects sent with sendfile 135051222 1812.72 935.58 Objects sent with write 36189893 494.65 250.71 Total Sessions 404810596 5491.13 2804.35 Total Requests 0 0.00 0.00 Total pipe 0 0.00 0.00 Total pass 12101371 158.8983.83 Total fetch 93389657948 1269372.98646962.32 Total header bytes 418107385660 5962840.69 2896463.38 Total body bytes 541249189.9437.50 Session Closed 0 0.00 0.00 Session Pipeline 1623343 0.0011.25 Session Read Ahead 398293975 5409.19 2759.20 Session herd 15780035398212996.82109317.12 SHM records 516154306 6307.55 3575.69 SHM writes 266181533.9818.44 SHM MTX contention 23279696 242.83 161.27 allocator requests 93623 .. outstanding allocations 910299136 .. bytes allocated []s, -- Rafael Umann [EMAIL PROTECTED] Suporte Engenharia 1 Terra Networks Brasil S/A Tel: +55 (51) 3284-4344 ___
Re: Question about threads
On Thu, Jun 19, 2008 at 5:37 AM, Rafael Umann [EMAIL PROTECTED] wrote: What is your request:connection ratio? Unfortunately now i dont have servers doing 2 hits/second, and thats why i dont have stats for you. Actually, it's right there in your varnishstat output: 36189916 476.66 250.71 Client connections accepted 404804204 5494.13 2804.30 Client requests received Your request:connection ratio is 10:1! This is a very good situation to be in. varnish doesn't have to spawn nearly as many threads as it would if the ratio were lower, as is common at many other sites. I have 6 servers runing this service now, each one doing 5500 hits/sec with 10% CPU usage, and this infrastructure scales to 2 hits/sec each server. It's probably inaccurate to assume that things will scale linearly :) Best regards, --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about threads
On Thu, 19 Jun 2008 08:05:21 -0700 Michael S. Fischer [EMAIL PROTECTED] wrote: On Thu, Jun 19, 2008 at 5:37 AM, Rafael Umann [EMAIL PROTECTED] wrote: What is your request:connection ratio? Unfortunately now i dont have servers doing 2 hits/second, and thats why i dont have stats for you. Actually, it's right there in your varnishstat output: 36189916 476.66 250.71 Client connections accepted 404804204 5494.13 2804.30 Client requests received Your request:connection ratio is 10:1! This is a very good situation to be in. varnish doesn't have to spawn nearly as many threads as it would if the ratio were lower, as is common at many other sites. OK, what i meant is that i dont have the stats for the amount of hits/second that i told u. Anyway, with this amount of traffic, with this timeouts that i set, it will probablly be the same. I have 6 servers runing this service now, each one doing 5500 hits/sec with 10% CPU usage, and this infrastructure scales to 2 hits/sec each server. It's probably inaccurate to assume that things will scale linearly :) Its not with Varnish :) actually i was surprised with this, but it scales almost linearly to the 2 hits/sec (ok, ok, with a litle bit of response time decrease), but it dont go any further. Best regards, --Michael E-mail verificado pelo Terra Anti-Spam. Para classificar como spam ou não spam, visite http://mail.terra.com.br/cgi-bin/reportspam.cgi?+_d=SCY1ODAyNDQ3I3Blcm0hdGVycmEmMSwxMjEzODg3OTIzLjk1ODU0MC4xNjgyMC5nYW5hbm9xdWUudGVycmEuY29tLDU0Mzc= Verifique periodicamente a pasta Spam para garantir que apenas mensagens indesejadas sejam classificadas como Spam. Esta mensagem foi verificada pelo E-mail Protegido Terra. Atualizado em 19/06/2008 -- Rafael Umann [EMAIL PROTECTED] Suporte Engenharia 1 Terra Networks Brasil S/A Tel: 55 (51) 3284-4344 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about threads
On Wed, Jun 18, 2008 at 4:51 AM, Rafael Umann [EMAIL PROTECTED] wrote: If it is a 32bits system, probably the problem is that your stack size is 10Mb. So 238 * 10mb = ~2gb I decreased my stack size to 512Kb. Using 1gb storage files i can now open almost 1900 threads using all the 2gb that 32bits can alloc. So, my Varnish makes 2 hits/second serving clients! What is your request:connection ratio? Best regards, --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Question about threads
I recently made a loadtest against through varnish. First I received a very high response time and found out that varnish was maxing the maximum nr of threads. I updated thread_min = 5 and thread_max = 300 and recevied much better resp. times. Then I increased the nr of concurrent users and made another loadtest. The strange thing here was that I received high resp. times but the threads stopped at 238. The N worker threads not created increased rapidly. I increased the threads again and changed listen_depth to 2048. Here is all the numbers: 238 0.00 0.22 N worker threads created 1318 4.98 1.21 N worker threads not created 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory default_ttl120 [seconds] thread_pools 2 [pools] thread_pool_max400 [threads] thread_pool_min10 [threads] thread_pool_timeout120 [seconds] thread_pool_purge_delay1000 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_add_delay 10 [milliseconds] thread_pool_fail_delay 200 [milliseconds] overflow_max 100 [%] rush_exponent 3 [requests per request] sess_workspace 8192 [bytes] obj_workspace 8192 [bytes] sess_timeout 5 [seconds] pipe_timeout 60 [seconds] send_timeout 600 [seconds] auto_restart on [bool] fetch_chunksize128 [kilobytes] vcl_trace off [bool] listen_address 0.0.0.0:80 listen_depth 2048 [connections] srcaddr_hash 1049 [buckets] srcaddr_ttl30 [seconds] backend_http11 off [bool] client_http11 off [bool] cli_timeout5 [seconds] ping_interval 3 [seconds] lru_interval 2 [seconds] cc_command exec cc -fpic -shared -Wl,-x -o %o %s max_restarts 4 [restarts] max_esi_includes 5 [restarts] cache_vbe_connsoff [bool] connect_timeout400 [ms] cli_buffer 8192 [bytes] diag_bitmap0x0 [bitmap] Why do I get Create worker thread failed 12 Cannot allocate memory when I had 1900MB free RAM and 65GB free Disk on the server? Any ideas? If N worker threads not created is increasing, is that a bad sign? Thanks Duja ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about threads
Raising the number of threads will not significantly improve Varnish concurrency in most cases. I did a test a few months ago using 4 CPUs on RHEL 4.6 with very high request concurrency and a very low request-per-connection ratio (i.e., 1:1, no keepalives) and found that the magic number is about 16 threads/CPU. You should raise overflow_max to a very high value (1% worked just fine for us). Under optimal operating conditions you should not see the threads not created value increasing like this. Best regards, --Michael On Tue, Jun 17, 2008 at 3:37 AM, [EMAIL PROTECTED] wrote: I recently made a loadtest against through varnish. First I received a very high response time and found out that varnish was maxing the maximum nr of threads. I updated thread_min = 5 and thread_max = 300 and recevied much better resp. times. Then I increased the nr of concurrent users and made another loadtest. The strange thing here was that I received high resp. times but the threads stopped at 238. The N worker threads not created increased rapidly. I increased the threads again and changed listen_depth to 2048. Here is all the numbers: 238 0.00 0.22 N worker threads created 1318 4.98 1.21 N worker threads not created 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory 0 Debug- Create worker thread failed 12 Cannot allocate memory default_ttl120 [seconds] thread_pools 2 [pools] thread_pool_max400 [threads] thread_pool_min10 [threads] thread_pool_timeout120 [seconds] thread_pool_purge_delay1000 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_add_delay 10 [milliseconds] thread_pool_fail_delay 200 [milliseconds] overflow_max 100 [%] rush_exponent 3 [requests per request] sess_workspace 8192 [bytes] obj_workspace 8192 [bytes] sess_timeout 5 [seconds] pipe_timeout 60 [seconds] send_timeout 600 [seconds] auto_restart on [bool] fetch_chunksize128 [kilobytes] vcl_trace off [bool] listen_address 0.0.0.0:80 listen_depth 2048 [connections] srcaddr_hash 1049 [buckets] srcaddr_ttl30 [seconds] backend_http11 off [bool] client_http11 off [bool] cli_timeout5 [seconds] ping_interval 3 [seconds] lru_interval 2 [seconds] cc_command exec cc -fpic -shared -Wl,-x -o %o %s max_restarts 4 [restarts] max_esi_includes 5 [restarts] cache_vbe_connsoff [bool] connect_timeout400 [ms] cli_buffer 8192 [bytes] diag_bitmap0x0 [bitmap] Why do I get Create worker thread failed 12 Cannot allocate memory when I had 1900MB free RAM and 65GB free Disk on the server? Any ideas? If N worker threads not created is increasing, is that a bad sign? Thanks Duja ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
