[squid-users] Squid and Polygraph
Hi All, I need to run web polygraph for bench marking purposes. My scenario is 1 squid proxy (Freebsd 7.2 squid-2.7 stable 6) and a client. Which workload should I use. I am looking at WebAxe-4.pg and I noticed "addr-space" with a range of IP addresses. My client is 192.168.1.223 and my proxy is 192.168.1.156. I saw that I should not use simple.pg for bench marking purposes. Hope someone can give me some hints as to what to modify so I can run the test. Thanks Monah
[squid-users] Squid and polygraph
Hi guys, I would like to make benchmark in my squid configuration, but which workload is the best to test? I just test with polymix-1.pg with ntlm auth and polygraph version 3.1.5. When I know the squid is slow, how muck the max requests support? Thanks Marcos
Re: [squid-users] Squid and polygraph
2008/7/30 Marcos Dutra <[EMAIL PROTECTED]>: > Hi guys, > > I would like to make benchmark in my squid configuration, but which > workload is the best to test? > I just test with polymix-1.pg with ntlm auth and polygraph version 3.1.5. > When I know the squid is slow, how muck the max requests support? Well, Squid is going to be slower then some of the commercial proxies, but you should be able to obtain at least 500 req/sec on decent hardware + disks on polymix-4. http://www.cacheboy.net/benchmarks.html I say 'at least' because thats what I've benchmarked Squid-2 / Cacheboy at thus far. It looks like it'll go further but I don't have enough polygraph boxes yet to run the tests at a higher throughput. I don't see why I couldn't get to about 800 req/sec on my test hardware given enough polygraph boxes and using COSS instead of AUFS. At that point though I'll run out of steam on one of the Xeon cores. It should go much faster than 500 req/sec if you're not using disks. Again, I just don't have enough polygraph boxes yet to run the tests and polygraph still uses poll/select limiting its throughput on current hardware. :/ Adrian
Re: [squid-users] Squid and polygraph
Hi Adrian, I saw the cacheboy project, is very intersting but, the code of cacheboy has the same features of squid example NTLM auth? With squid-2.6.5 (RHEL 5) in my tests with polymix-1 arrive in 200req/sec (COSS + Aufs) but squid performance is very slow to open pages. I compiled squid-2.6.21 and I have the same performance above. My machine - Intel Xeon dual processor 3.6GHz with 8GB Ram and 2 disks 146GB in raid 1 on LVM. /var/sqool/squid - ext3 noatime nodiratime - 20GB cache_dir coss /var/spool/squid/coss 2048 block-size=512 max-stripe-waste=131072 max-size=131072 cache_dir aufs /var/spool/squid/aufs 2048 32 256 max-size=5000 cache_mem 2048 MB cache_swap_low 95% cache_swap_high 98% memory_pools on memory_pools_limit 5120 MB I have ntlm authentication in this machine. Thanks for advice Marcos 2008/7/29 Adrian Chadd <[EMAIL PROTECTED]>: > 2008/7/30 Marcos Dutra <[EMAIL PROTECTED]>: >> Hi guys, >> >> I would like to make benchmark in my squid configuration, but which >> workload is the best to test? >> I just test with polymix-1.pg with ntlm auth and polygraph version 3.1.5. >> When I know the squid is slow, how muck the max requests support? > > Well, Squid is going to be slower then some of the commercial proxies, > but you should be able to obtain at least 500 req/sec on decent > hardware + disks on polymix-4. > > http://www.cacheboy.net/benchmarks.html > > I say 'at least' because thats what I've benchmarked Squid-2 / > Cacheboy at thus far. It looks like it'll go further but I don't have > enough polygraph boxes yet to run the tests at a higher throughput. > > I don't see why I couldn't get to about 800 req/sec on my test > hardware given enough polygraph boxes and using COSS instead of AUFS. > At that point though I'll run out of steam on one of the Xeon cores. > > It should go much faster than 500 req/sec if you're not using disks. > Again, I just don't have enough polygraph boxes yet to run the tests > and polygraph still uses poll/select limiting its throughput on > current hardware. :/ > > > > Adrian >
Re: [squid-users] Squid and polygraph
(gah, why oh why won't gmail do what I said with my damned From:? Oh well..) 2008/7/30 Marcos Dutra <[EMAIL PROTECTED]>: > Hi Adrian, > > I saw the cacheboy project, is very intersting but, the code of > cacheboy has the same features of squid example NTLM auth? With cacheboy is "just" Squid-2.HEAD with a whole lot of code shuffling. Squid-2.HEAD should contain all of the 2.7 and 2.6 NTLM authentication stuff. > squid-2.6.5 (RHEL 5) in my tests with polymix-1 arrive in 200req/sec > (COSS + Aufs) but squid performance is very slow to open pages. I > compiled squid-2.6.21 and I have the same performance above. Polygraph can apparently speak NTLM authentication now. You may want to look into that. The lookup speed may be related to authentication. Have you tried disabling authentication for a specific desktop machine and try browsing? Adrian
Re: [squid-users] Squid and polygraph
2008/7/30 Adrian Chadd <[EMAIL PROTECTED]>: > (gah, why oh why won't gmail do what I said with my damned From:? Oh well..) Sorry :( > > > 2008/7/30 Marcos Dutra <[EMAIL PROTECTED]>: >> Hi Adrian, >> >> I saw the cacheboy project, is very intersting but, the code of >> cacheboy has the same features of squid example NTLM auth? With > > cacheboy is "just" Squid-2.HEAD with a whole lot of code shuffling. > Squid-2.HEAD should contain all of the 2.7 and 2.6 NTLM authentication > stuff. It's cool, I will test cacheboy. > >> squid-2.6.5 (RHEL 5) in my tests with polymix-1 arrive in 200req/sec >> (COSS + Aufs) but squid performance is very slow to open pages. I >> compiled squid-2.6.21 and I have the same performance above. > > Polygraph can apparently speak NTLM authentication now. You may want > to look into that. Yeah, I running polygraph with NTLM for test performance, but I would like more performance on this server. > > The lookup speed may be related to authentication. Have you tried > disabling authentication for a specific desktop machine and try > browsing? > > I don't it this yet, but the authentication I think is not a problem, I will try this. > > Adrian > One question, squid Vs cacheboy comparision which the better? Thanks Marcos
Re: [squid-users] Squid and polygraph
2008/7/30 Marcos Dutra <[EMAIL PROTECTED]>: > 2008/7/30 Adrian Chadd <[EMAIL PROTECTED]>: >> (gah, why oh why won't gmail do what I said with my damned From:? Oh well..) > > Sorry :( Hey, that isn't your fault. It just means I occasionally seem to misfile emails. Anyway.. >> 2008/7/30 Marcos Dutra <[EMAIL PROTECTED]>: >> cacheboy is "just" Squid-2.HEAD with a whole lot of code shuffling. >> Squid-2.HEAD should contain all of the 2.7 and 2.6 NTLM authentication >> stuff. > > It's cool, I will test cacheboy. Well, I'm all for cacheboy testing (as its mostly Squid-2.HEAD testing too!) but I'd be very surprised right now if there was a measurable performance difference between Squid-2.6/Squid-2.7 and Cacheboy in your environment. > Yeah, I running polygraph with NTLM for test performance, but I would > like more performance on this server. What metrics are you using for "more performance" ? >> The lookup speed may be related to authentication. Have you tried >> disabling authentication for a specific desktop machine and try >> browsing? > I don't it this yet, but the authentication I think is not a problem, > I will try this. Well, its worth ruling out authentication as a contributing factor. NTLM authentication using Samba isn't the fastest of things.. > One question, squid Vs cacheboy comparision which the better? At the present time? Mostly subjective. Cacheboy is "adrian's idea of where Squid-2 should've gone and should be going" ; the best way to compare Squid-(2, 3) and Cacheboy is to look at the developers, the development, the roadmap, and see which suits you better. There's no real performance or functionality reason to move over to Cacheboy but I won't say no to more users. I'd like the positive/negative feedback :) Adrian
Re: [squid-users] Squid and polygraph
2008/7/30 Adrian Chadd <[EMAIL PROTECTED]>: > 2008/7/30 Marcos Dutra <[EMAIL PROTECTED]>: >> 2008/7/30 Adrian Chadd <[EMAIL PROTECTED]>: >>> (gah, why oh why won't gmail do what I said with my damned From:? Oh well..) >> >> Sorry :( > > Hey, that isn't your fault. It just means I occasionally seem to > misfile emails. Anyway.. > >>> 2008/7/30 Marcos Dutra <[EMAIL PROTECTED]>: >>> cacheboy is "just" Squid-2.HEAD with a whole lot of code shuffling. >>> Squid-2.HEAD should contain all of the 2.7 and 2.6 NTLM authentication >>> stuff. >> >> It's cool, I will test cacheboy. > > Well, I'm all for cacheboy testing (as its mostly Squid-2.HEAD testing > too!) but I'd be very surprised right now if there was a measurable > performance difference between Squid-2.6/Squid-2.7 and Cacheboy in > your environment. Why do you working in two projects? > >> Yeah, I running polygraph with NTLM for test performance, but I would >> like more performance on this server. > > What metrics are you using for "more performance" ? I used basicaly req/sec and conections in port 3128. The time of open page in the browser is very important too. > >>> The lookup speed may be related to authentication. Have you tried >>> disabling authentication for a specific desktop machine and try >>> browsing? > >> I don't it this yet, but the authentication I think is not a problem, >> I will try this. > > Well, its worth ruling out authentication as a contributing factor. > NTLM authentication using Samba isn't the fastest of things.. Why in my tests with polygraph, when "netstat -an |grep 3306| wc" up to more 5000 connections and my squid don't open page or if open is much slow? When netstat up to 7000 connections, squidclient mgr:info show me around 13000 req/minute maximum number. After minutes this values down in 1 hour the medium is 8000 req/minute. > >> One question, squid Vs cacheboy comparision which the better? > > At the present time? Mostly subjective. Cacheboy is "adrian's idea of > where Squid-2 should've gone and should be going" ; the best way to > compare Squid-(2, 3) and Cacheboy is to look at the developers, the > development, the roadmap, and see which suits you better. > > There's no real performance or functionality reason to move over to > Cacheboy but I won't say no to more users. I'd like the > positive/negative feedback :) > > > > Adrian > Ps. I used your workload in my tests but failed :(. Well see this after lunch here! Thanks Marcos
Re: [squid-users] Squid and polygraph
2008/7/30 Marcos Dutra <[EMAIL PROTECTED]>: > Why do you working in two projects? Personal disagreements with what the current Squid "project direction" is, for whatever values of "is" it is at the present time. >>> Yeah, I running polygraph with NTLM for test performance, but I would >>> like more performance on this server. >> >> What metrics are you using for "more performance" ? > I used basicaly req/sec and conections in port 3128. The time of open > page in the browser is very important too. Ok, so what are the median service times in cachemgr:info ? >> Well, its worth ruling out authentication as a contributing factor. >> NTLM authentication using Samba isn't the fastest of things.. > > > Why in my tests with polygraph, when "netstat -an |grep 3306| wc" up > to more 5000 connections and my squid don't open page or if open is > much slow? Well, are you hitting 100% CPU when you reach this point? > When netstat up to 7000 connections, squidclient mgr:info show me > around 13000 req/minute maximum number. > After minutes this values down in 1 hour the medium is 8000 req/minute. Your best bet is to look at the slightly more instantaneous req/sec counts in mgr:5min. > Ps. I used your workload in my tests but failed :(. Well see this > after lunch here! My workload is just polymix-4! The trick is setting the thing up right. It took me a few goes! Adrian
Re: [squid-users] Squid and polygraph
I tested squid and cacheboy with polymix4 and I have a problem, my ntlm authentication has 200 connections but polymix used all connections and not open any site because this. How can I optimize this? Thanks Marcos
Re: [squid-users] Squid and polygraph
2008/8/1 Marcos Dutra <[EMAIL PROTECTED]>: > I tested squid and cacheboy with polymix4 and I have a problem, my > ntlm authentication has 200 connections but polymix used all > connections and not open any site because this. How can I optimize > this? Hm, whats the error? You may need to look at tuning your Samba configuration.. adrian
Re: [squid-users] Squid and polygraph
Hi Adrian, Well I ran polygraph with polymix 4 based in cacheboy statistics and added ntlm auth, I look squidclient mgr:ntlmauthenticator and the process are busy after minutes and the polygraph die when all 200 process are busy. When one process of auth is busy, don't open nothing! I tested open any page in firefox in this situation. My samba configuration is simple, bellow the conf. [global] workgroup = MARCOS netbios name = PROXY-TEST realm = AD.MARCOS server string = Marcos server test security = ADS encrypt passwords = Yes password server = TEST.AD.MARCOS log file = /var/log/samba/%m.log log level = 3 max log size = 0 socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192 preferred master = False local master = No domain master = False dns proxy = No wins server = TEST.AD.MARCOS winbind separator = + winbind enum users = Yes winbind enum groups = Yes winbind use default domain = Yes idmap uid = 1-2 idmap gid = 1-2 client schannel = no Thanks. Marcos 2008/8/1 Adrian Chadd <[EMAIL PROTECTED]>: > 2008/8/1 Marcos Dutra <[EMAIL PROTECTED]>: >> I tested squid and cacheboy with polymix4 and I have a problem, my >> ntlm authentication has 200 connections but polymix used all >> connections and not open any site because this. How can I optimize >> this? > > Hm, whats the error? > > You may need to look at tuning your Samba configuration.. > > > > adrian >
Re: [squid-users] Squid and polygraph
Right; and what happens when you disable authentication in Squid and polygraph? Does it cope fine? Samba/Winbind are known to not handle high authentication transaction rates. Well, 200/sec isn't "high" to me.. If it works fine without NTLM authentication but fails when you try using it, then I'd point fingers at Samba/Winbind. There's a hard-coded default of 200 concurrent "connections" to winbind in the winbind source; I thought they were going to improve that. Anyway, if its fine without NTLM auth but slow with it enabled I'd go ask the Samba team about it. In the meantime, there's a workaround - you can enable uhm, authenticate_ip_shortcircuit_ttl and authenticate_ip_shortcircuit_access. Adrian 2008/8/1 Marcos Dutra <[EMAIL PROTECTED]>: > Hi Adrian, > > Well I ran polygraph with polymix 4 based in cacheboy statistics and > added ntlm auth, I look squidclient mgr:ntlmauthenticator and the > process are busy after minutes and the polygraph die when all 200 > process are busy. > When one process of auth is busy, don't open nothing! I tested open > any page in firefox in this situation. > My samba configuration is simple, bellow the conf. > > [global] >workgroup = MARCOS >netbios name = PROXY-TEST >realm = AD.MARCOS >server string = Marcos server test >security = ADS >encrypt passwords = Yes >password server = TEST.AD.MARCOS >log file = /var/log/samba/%m.log >log level = 3 >max log size = 0 >socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192 >preferred master = False >local master = No >domain master = False >dns proxy = No >wins server = TEST.AD.MARCOS >winbind separator = + >winbind enum users = Yes >winbind enum groups = Yes >winbind use default domain = Yes >idmap uid = 1-2 >idmap gid = 1-2 >client schannel = no > > Thanks. > Marcos > > > 2008/8/1 Adrian Chadd <[EMAIL PROTECTED]>: >> 2008/8/1 Marcos Dutra <[EMAIL PROTECTED]>: >>> I tested squid and cacheboy with polymix4 and I have a problem, my >>> ntlm authentication has 200 connections but polymix used all >>> connections and not open any site because this. How can I optimize >>> this? >> >> Hm, whats the error? >> >> You may need to look at tuning your Samba configuration.. >> >> >> >> adrian >> > >
Re: [squid-users] Squid and polygraph
Hi Adrian, 2008/8/2 Adrian Chadd <[EMAIL PROTECTED]>: > Right; and what happens when you disable authentication in Squid and > polygraph? Does it cope fine? I disabled ntlm auth and acl with regex sites and squid work fine. With ntlm and acl I saw cpu time in mgr info go up 99%, then I this is a problem when I have much connections. Well I enabled in my conf only ntlm auth without acl regex, and enabled samba logs and I have this problem: [2008/08/04 15:32:39, 0] utils/ntlm_auth.c:get_winbind_netbios_name(172) could not obtain winbind netbios name! [2008/08/04 15:32:39, 0] utils/ntlm_auth.c:get_winbind_netbios_name(172) could not obtain winbind netbios name! [2008/08/04 15:32:39, 0] utils/ntlm_auth.c:get_winbind_netbios_name(172) could not obtain winbind netbios name! 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback data. Releasing helper '0x10922a8'. 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback data. Releasing helper '0x108e128'. 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback data. Releasing helper '0x10901e8'. 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback data. Releasing helper '0x1094368'. 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback data. Releasing helper '0x1096428'. > > Samba/Winbind are known to not handle high authentication transaction > rates. Well, 200/sec isn't "high" to me.. > > If it works fine without NTLM authentication but fails when you try > using it, then I'd point fingers at Samba/Winbind. There's a > hard-coded default of 200 concurrent "connections" to winbind in the > winbind source; I thought they were going to improve that. Anyway, if > its fine without NTLM auth but slow with it enabled I'd go ask the > Samba team about it. I will ask to samba list, thanks > > In the meantime, there's a workaround - you can enable uhm, > authenticate_ip_shortcircuit_ttl and > authenticate_ip_shortcircuit_access. > I used squid version 2.6.5 and I think this options above don't work, I used authenticate_ip_ttl to cache authentication. Thanks for all.
Re: [squid-users] Squid and polygraph
Try squid-2.7 for those options. Also, try no NTLM and with ACLs, see if you hit high CPU use there. adrian 2008/8/5 Marcos Dutra <[EMAIL PROTECTED]>: > Hi Adrian, > > 2008/8/2 Adrian Chadd <[EMAIL PROTECTED]>: >> Right; and what happens when you disable authentication in Squid and >> polygraph? Does it cope fine? > > I disabled ntlm auth and acl with regex sites and squid work fine. > With ntlm and acl I saw cpu time in mgr info go up 99%, then I this is > a problem when I have much connections. > > Well I enabled in my conf only ntlm auth without acl regex, and > enabled samba logs and I have this problem: > > [2008/08/04 15:32:39, 0] utils/ntlm_auth.c:get_winbind_netbios_name(172) > could not obtain winbind netbios name! > [2008/08/04 15:32:39, 0] utils/ntlm_auth.c:get_winbind_netbios_name(172) > could not obtain winbind netbios name! > [2008/08/04 15:32:39, 0] utils/ntlm_auth.c:get_winbind_netbios_name(172) > could not obtain winbind netbios name! > 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback > data. Releasing helper '0x10922a8'. > 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback > data. Releasing helper '0x108e128'. > 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback > data. Releasing helper '0x10901e8'. > 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback > data. Releasing helper '0x1094368'. > 2008/08/04 15:32:39| AuthenticateNTLMHandleReply: invalid callback > data. Releasing helper '0x1096428'. > >> >> Samba/Winbind are known to not handle high authentication transaction >> rates. Well, 200/sec isn't "high" to me.. >> >> If it works fine without NTLM authentication but fails when you try >> using it, then I'd point fingers at Samba/Winbind. There's a >> hard-coded default of 200 concurrent "connections" to winbind in the >> winbind source; I thought they were going to improve that. Anyway, if >> its fine without NTLM auth but slow with it enabled I'd go ask the >> Samba team about it. > > I will ask to samba list, thanks > > >> >> In the meantime, there's a workaround - you can enable uhm, >> authenticate_ip_shortcircuit_ttl and >> authenticate_ip_shortcircuit_access. >> > > I used squid version 2.6.5 and I think this options above don't work, > I used authenticate_ip_ttl to cache authentication. > > Thanks for all. >
