Re: [squid-users] squid 3.x expected max throughput
On 11/04/2013 12:23 a.m., Youssef Ghorbal wrote: I was aware of that page. As you said, it's often RPS so it's not relevant for me. It is more relevant than you seem to think. Squid processing overheads are tied tightly to the request parsing and ACL testing processes. These are relatively fixed overheads for each request regardless of the request size. IIRC the 50Mbps traffic was on average HTTP request traffic objects in a medium sized ISP - we do not have any good detaisl on what that average was though. With very little extra overheads the same proxy on same hardware might also reach 100Mbps. All that is required is the avg object size to go from, for example, 16KB to 32KB which are both within the bounds of average HTTP traffic sizes (11-63KB). Amos Youssef On Apr 10, 2013, at 2:02 PM, Kinkie gkin...@gmail.com wrote: You probably want to check http://wiki.squid-cache.org/KnowledgeBase/Benchmarks Unfortunately the benchmarks reported are often expressed as RPS and not bandwidth. On Wed, Apr 10, 2013 at 1:21 PM, Youssef Ghorbal d...@pasteur.fr wrote: On Apr 10, 2013, at 10:37 AM, Kinkie gkin...@gmail.com wrote: On Tue, Apr 9, 2013 at 11:35 PM, Youssef Ghorbal d...@pasteur.fr wrote: Hello, Is there any recent figures about max throughput to be expected from a squid 3.x install (on recent hardware) in the scenario of a single stream downloading a large file ( 1GB) (read not cacheable) I'm aware that's not a performance metric per se, but it's one of the scenarios we have to deal with. Few weeks ago, Amos talked about 50Mb/s (client + server) for a squid 3.1 Hi, 50mb/s seems a very conservative estimate to me; in that scenario Squid is essentially acting as a network pipe. Assuming this is a lab (and we can thus ignore bandwidth and latency on the internet link), the expectation is that this kind of scenario for squid will be CPU and network I/O bound, so in order to give any sensible answer we'd need to know what kind of network interface you would use (fast-ethernet? Giga-ethernet copper? Giga-ethernet fiber? Even faster?), what kind of CPU and what kind of system architecture (server-class? pc-class? virtual?) I'm agree, it's a LAB and we can ignore bandwidth and latency of Internet link. And it's exactly what you said it's a scenario where Squid is essentially acting as a network pipe. Here is a summary of the hardware : - 1Gb ethernet NIC (copper) : the same of client and server traffic. - 16GB RAM - 2 CPUs Quad Core (Xeon E5420 2.5Ghz per core) : I think that the many cores are not relevent since a single stream will eventually be handled by a single core. - FreeBSD 8.3 amd64 What I'm seeing right now is ~50Mb/s on 3.1 release (as Amos said earlier) which seems very conservative estimate to me too, and I was seeking infos on what can be expected in a perfect world. If it's the current best figures I can get, that's fine and I'll not be looking to optmise anything else. If it can do a lot better (let's say 10 times better) I'll try to investgate time to reach this (upgrade to the last release, tune the configuration, tune the system etc) Youssef
Re: [squid-users] squid 3.x expected max throughput
On Apr 11, 2013, at 12:13 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 11/04/2013 12:23 a.m., Youssef Ghorbal wrote: I was aware of that page. As you said, it's often RPS so it's not relevant for me. It is more relevant than you seem to think. Squid processing overheads are tied tightly to the request parsing and ACL testing processes. These are relatively fixed overheads for each request regardless of the request size. Thank you for the explanation. It confirms what I was expecting myself. What I meant, is that in my case I have ONE big object (the infamous 1G file) and 1 client and I want to know what to expect on the troughput point of view. As kinkie said, in my scenario, Squid is basically a network pipe. the overhead (ACL parsing etc is done one time) Maybe, the question here is how does Squid handle data buffers, or at which rate does it perform server reads on the one side and to client writes on the other. IIRC the 50Mbps traffic was on average HTTP request traffic objects in a medium sized ISP - we do not have any good detaisl on what that average was though. With very little extra overheads the same proxy on same hardware might also reach 100Mbps. All that is required is the avg object size to go from, for example, 16KB to 32KB which are both within the bounds of average HTTP traffic sizes (11-63KB). Yousef
Re: [squid-users] squid 3.x expected max throughput
On 11/04/2013 10:48 p.m., Youssef Ghorbal wrote: On Apr 11, 2013, at 12:13 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 11/04/2013 12:23 a.m., Youssef Ghorbal wrote: I was aware of that page. As you said, it's often RPS so it's not relevant for me. It is more relevant than you seem to think. Squid processing overheads are tied tightly to the request parsing and ACL testing processes. These are relatively fixed overheads for each request regardless of the request size. Thank you for the explanation. It confirms what I was expecting myself. What I meant, is that in my case I have ONE big object (the infamous 1G file) and 1 client and I want to know what to expect on the troughput point of view. As kinkie said, in my scenario, Squid is basically a network pipe. the overhead (ACL parsing etc is done one time) Maybe, the question here is how does Squid handle data buffers, or at which rate does it perform server reads on the one side and to client writes on the other. Well, in absence of any other loading it would be one read per packet just like you (or was it someone earlier) are seeing. With an internal event call passing the buffered data through Squid to write the same amount on the output. The select loop speed varies with a maximum loop time held around 10ms and decreasing if packets arrive faster. You will have to try to follow the code to understand the buffers unfortunately. I'm still learning thet part myself. For CONNECT requests its easy, the input buffer is the output buffer - one fread() causes fwrite() on the same buffer. For other requests is varies as Store systems are also involved which may (or may not) cause a file I/O to occur in the middle or a bufffer copy (I'm not sure here). Amos IIRC the 50Mbps traffic was on average HTTP request traffic objects in a medium sized ISP - we do not have any good detaisl on what that average was though. With very little extra overheads the same proxy on same hardware might also reach 100Mbps. All that is required is the avg object size to go from, for example, 16KB to 32KB which are both within the bounds of average HTTP traffic sizes (11-63KB). Yousef
Re: [squid-users] squid 3.x expected max throughput
On Tue, Apr 9, 2013 at 11:35 PM, Youssef Ghorbal d...@pasteur.fr wrote: Hello, Is there any recent figures about max throughput to be expected from a squid 3.x install (on recent hardware) in the scenario of a single stream downloading a large file ( 1GB) (read not cacheable) I'm aware that's not a performance metric per se, but it's one of the scenarios we have to deal with. Few weeks ago, Amos talked about 50Mb/s (client + server) for a squid 3.1 Hi, 50mb/s seems a very conservative estimate to me; in that scenario Squid is essentially acting as a network pipe. Assuming this is a lab (and we can thus ignore bandwidth and latency on the internet link), the expectation is that this kind of scenario for squid will be CPU and network I/O bound, so in order to give any sensible answer we'd need to know what kind of network interface you would use (fast-ethernet? Giga-ethernet copper? Giga-ethernet fiber? Even faster?), what kind of CPU and what kind of system architecture (server-class? pc-class? virtual?) -- /kinkie
Re: [squid-users] squid 3.x expected max throughput
On Apr 10, 2013, at 10:37 AM, Kinkie gkin...@gmail.com wrote: On Tue, Apr 9, 2013 at 11:35 PM, Youssef Ghorbal d...@pasteur.fr wrote: Hello, Is there any recent figures about max throughput to be expected from a squid 3.x install (on recent hardware) in the scenario of a single stream downloading a large file ( 1GB) (read not cacheable) I'm aware that's not a performance metric per se, but it's one of the scenarios we have to deal with. Few weeks ago, Amos talked about 50Mb/s (client + server) for a squid 3.1 Hi, 50mb/s seems a very conservative estimate to me; in that scenario Squid is essentially acting as a network pipe. Assuming this is a lab (and we can thus ignore bandwidth and latency on the internet link), the expectation is that this kind of scenario for squid will be CPU and network I/O bound, so in order to give any sensible answer we'd need to know what kind of network interface you would use (fast-ethernet? Giga-ethernet copper? Giga-ethernet fiber? Even faster?), what kind of CPU and what kind of system architecture (server-class? pc-class? virtual?) I'm agree, it's a LAB and we can ignore bandwidth and latency of Internet link. And it's exactly what you said it's a scenario where Squid is essentially acting as a network pipe. Here is a summary of the hardware : - 1Gb ethernet NIC (copper) : the same of client and server traffic. - 16GB RAM - 2 CPUs Quad Core (Xeon E5420 2.5Ghz per core) : I think that the many cores are not relevent since a single stream will eventually be handled by a single core. - FreeBSD 8.3 amd64 What I'm seeing right now is ~50Mb/s on 3.1 release (as Amos said earlier) which seems very conservative estimate to me too, and I was seeking infos on what can be expected in a perfect world. If it's the current best figures I can get, that's fine and I'll not be looking to optmise anything else. If it can do a lot better (let's say 10 times better) I'll try to investgate time to reach this (upgrade to the last release, tune the configuration, tune the system etc) Youssef
Re: [squid-users] squid 3.x expected max throughput
You probably want to check http://wiki.squid-cache.org/KnowledgeBase/Benchmarks Unfortunately the benchmarks reported are often expressed as RPS and not bandwidth. On Wed, Apr 10, 2013 at 1:21 PM, Youssef Ghorbal d...@pasteur.fr wrote: On Apr 10, 2013, at 10:37 AM, Kinkie gkin...@gmail.com wrote: On Tue, Apr 9, 2013 at 11:35 PM, Youssef Ghorbal d...@pasteur.fr wrote: Hello, Is there any recent figures about max throughput to be expected from a squid 3.x install (on recent hardware) in the scenario of a single stream downloading a large file ( 1GB) (read not cacheable) I'm aware that's not a performance metric per se, but it's one of the scenarios we have to deal with. Few weeks ago, Amos talked about 50Mb/s (client + server) for a squid 3.1 Hi, 50mb/s seems a very conservative estimate to me; in that scenario Squid is essentially acting as a network pipe. Assuming this is a lab (and we can thus ignore bandwidth and latency on the internet link), the expectation is that this kind of scenario for squid will be CPU and network I/O bound, so in order to give any sensible answer we'd need to know what kind of network interface you would use (fast-ethernet? Giga-ethernet copper? Giga-ethernet fiber? Even faster?), what kind of CPU and what kind of system architecture (server-class? pc-class? virtual?) I'm agree, it's a LAB and we can ignore bandwidth and latency of Internet link. And it's exactly what you said it's a scenario where Squid is essentially acting as a network pipe. Here is a summary of the hardware : - 1Gb ethernet NIC (copper) : the same of client and server traffic. - 16GB RAM - 2 CPUs Quad Core (Xeon E5420 2.5Ghz per core) : I think that the many cores are not relevent since a single stream will eventually be handled by a single core. - FreeBSD 8.3 amd64 What I'm seeing right now is ~50Mb/s on 3.1 release (as Amos said earlier) which seems very conservative estimate to me too, and I was seeking infos on what can be expected in a perfect world. If it's the current best figures I can get, that's fine and I'll not be looking to optmise anything else. If it can do a lot better (let's say 10 times better) I'll try to investgate time to reach this (upgrade to the last release, tune the configuration, tune the system etc) Youssef -- /kinkie
Re: [squid-users] squid 3.x expected max throughput
I was aware of that page. As you said, it's often RPS so it's not relevant for me. Youssef On Apr 10, 2013, at 2:02 PM, Kinkie gkin...@gmail.com wrote: You probably want to check http://wiki.squid-cache.org/KnowledgeBase/Benchmarks Unfortunately the benchmarks reported are often expressed as RPS and not bandwidth. On Wed, Apr 10, 2013 at 1:21 PM, Youssef Ghorbal d...@pasteur.fr wrote: On Apr 10, 2013, at 10:37 AM, Kinkie gkin...@gmail.com wrote: On Tue, Apr 9, 2013 at 11:35 PM, Youssef Ghorbal d...@pasteur.fr wrote: Hello, Is there any recent figures about max throughput to be expected from a squid 3.x install (on recent hardware) in the scenario of a single stream downloading a large file ( 1GB) (read not cacheable) I'm aware that's not a performance metric per se, but it's one of the scenarios we have to deal with. Few weeks ago, Amos talked about 50Mb/s (client + server) for a squid 3.1 Hi, 50mb/s seems a very conservative estimate to me; in that scenario Squid is essentially acting as a network pipe. Assuming this is a lab (and we can thus ignore bandwidth and latency on the internet link), the expectation is that this kind of scenario for squid will be CPU and network I/O bound, so in order to give any sensible answer we'd need to know what kind of network interface you would use (fast-ethernet? Giga-ethernet copper? Giga-ethernet fiber? Even faster?), what kind of CPU and what kind of system architecture (server-class? pc-class? virtual?) I'm agree, it's a LAB and we can ignore bandwidth and latency of Internet link. And it's exactly what you said it's a scenario where Squid is essentially acting as a network pipe. Here is a summary of the hardware : - 1Gb ethernet NIC (copper) : the same of client and server traffic. - 16GB RAM - 2 CPUs Quad Core (Xeon E5420 2.5Ghz per core) : I think that the many cores are not relevent since a single stream will eventually be handled by a single core. - FreeBSD 8.3 amd64 What I'm seeing right now is ~50Mb/s on 3.1 release (as Amos said earlier) which seems very conservative estimate to me too, and I was seeking infos on what can be expected in a perfect world. If it's the current best figures I can get, that's fine and I'll not be looking to optmise anything else. If it can do a lot better (let's say 10 times better) I'll try to investgate time to reach this (upgrade to the last release, tune the configuration, tune the system etc) Youssef
