Re: Linux filesystem speed comparison
On Mon, 2005/04/11 (MDT), <[EMAIL PROTECTED]> wrote: We currently get arount 70 reqs/sec using 25% CPU (5 minute average for both values) on this hardware. I'm confident that I'll get a pretty high number of requests/second through these proxies becase of the epoll patch. Polygraph using PolyMix-4 workload does 500 req/sec per client-server pair. With a decent pair of PCs and Gbit links, you can 1000 req/sec or more per pair. Thus, you should be able to stress your Squid proxies with just a couple of PCs. Alex.
RE: Linux filesystem speed comparison
On Tue, 12 Apr 2005, Steven Wilton wrote: My thoughts were that if the numbers for %CPU in system and user were similar, then a "more efficient" filesystem would arrange the data on disk in such a way that the disk spends less time performing the operations. Partly true, but the problem is that the %IOWAIT is quite voilatile and hard to make any reliable readings due to %USER and %SYS stealing time from %IOWAIT, making things very timing dependent, and even more so as the load increases. %IOWAIT works great for single-threaded blocking disk I/O applications (i.e. if using the ufs or coss cache_dir driver), but not so good in applications where I/O is done in some non-blocking fashion (i.e. aufs or diskd cache drivers). I will add graphs for the /proc/diskstats value that records the amount of time the disk is actually performing operations, and see how this compares across the different filesystems (I looked at the iostat source to see how it calculates the %util value). Looking forward to see your results. Regards Henrik
Re: Linux filesystem speed comparison
Steven Wilton wrote: Heheh..."only" is a relative term. Our old 500 Mhz boxes couldn't even work two 7200 RPM IDE disks (on different buses, of course) effectively, and three provided a barely measurable boost. I'd be surprised if you aren't able to easily max your CPU with ReiserFS on a polygraph run on these boxes...and it will probably happen at around 110 reqs/sec., assuming reasonable configuration of kernel and Squid. We currently get arount 70 reqs/sec using 25% CPU (5 minute average for both values) on this hardware. I'm confident that I'll get a pretty high number of requests/second through these proxies becase of the epoll patch. Ah, yes, epoll could very well be an interesting twist...and it might make the relative filesystem results come out very differently.
RE: Linux filesystem speed comparison
> Heheh..."only" is a relative term. Our old 500 Mhz boxes > couldn't even > work two 7200 RPM IDE disks (on different buses, of course) > effectively, > and three provided a barely measurable boost. I'd be > surprised if you > aren't able to easily max your CPU with ReiserFS on a > polygraph run on > these boxes...and it will probably happen at around 110 reqs/sec., > assuming reasonable configuration of kernel and Squid. > We currently get arount 70 reqs/sec using 25% CPU (5 minute average for both values) on this hardware. I'm confident that I'll get a pretty high number of requests/second through these proxies becase of the epoll patch. -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.308 / Virus Database: 266.9.6 - Release Date: 4/11/2005
Re: Linux filesystem speed comparison
Steven Wilton wrote: It depends on the balance of hardware, but I'd be extremely surprised if XFS performs better than either reiser or ext2/3 for Squid workloads on /any/ system. So I have to assume your methodology is slightly flawed. ;-) That's what I thought, but there has been a bit of XFS work in recent kernels, and after my initial observations I was wondering if this has improved the performance with squid's filesystem load. It's been at least a year since I tried XFS, so I reserve the right to be horribly wrong. But it was so far behind the other options when I last toyed with it that I completely wrote it off as wholly uninteresting for Squid workloads. ;-) While I have found that ext3 (when configured correctly) has improved performance for Squid quite a bit over ext2, it is still no match for ReiserFS on our hardware, which always has more than enough CPU for the disk bandwidth available. But, I can certainly imagine a hardware configuration that would lead to ext3 performing better than ReiserFS (especially since Duane has proven that it is possible by putting 6 10,000 RPM disks on a relatively wimpy CPU and testing the configuration extensively with polygraph). The machines are a bit old (P3-500), but they've only got 3x 9Gb SCSI cache disks, and they're not running anywhere near 100% load. Heheh..."only" is a relative term. Our old 500 Mhz boxes couldn't even work two 7200 RPM IDE disks (on different buses, of course) effectively, and three provided a barely measurable boost. I'd be surprised if you aren't able to easily max your CPU with ReiserFS on a polygraph run on these boxes...and it will probably happen at around 110 reqs/sec., assuming reasonable configuration of kernel and Squid. I'm always interested in conflicting reports, however. If you've got a case that makes XFS faster for Squid against polygraph, I'd love to see the results and configuration. I had a quick look at polygraph before, but I didn't get very far in testing it. I would like to produce some polygraph figures for the proxies, so I will see what I can do to make a test system. My only concern is that the proxies may be able to process requests faster than the polygraph hardware can serve them. From memory there are a lot of options available for polygraph, and I was not sure how to produce meaningful results. Any help would be appreciated. I've got no magic formula for making Polygraph go. I can say that it built easily on Linux and FreeBSD last time I tried on either platform, and the documentation that Alex wrote for preparing for the cacheoffs was very helpful in getting an environment that works for roughly replicating publicly available proxy performance numbers, including those that Duane, myself, and others have published for Squid. Oh, and as for your concern that the Polygraph boxes might not be able to work your Squid hard enough to stress it...don't worry. Polygraph has a lot less work to do than Squid, and Alex has done a nice job making it go plenty fast. No single Squid instance will be a match for even a single polygraph box of equal hardware. I've been able to use a laptop for /both/ sides of the polypair and successfully stress Squid on similar hardware to what you're testing (not recommended, since results from a single box polypair wouldn't be trustworthy or sanely comparable to a real pair of poly boxes--but in a pinch it will do). Just some thoughts. Performance has become progressively less important over the years as hardware has become so much faster. I only see a few cases a year where we even need more than one Squid box, for anything other than redundancy (though sometimes the one Squid box is obscenely powerful). So, my Squid performance testing has become an occasional hobby rather than a core interest...so new results for various configurations might surprise me just as much as anyone else.
RE: Linux filesystem speed comparison
> -Original Message- > From: Joe Cooper [mailto:[EMAIL PROTECTED] > Sent: Tuesday, April 12, 2005 5:41 AM > To: Steven Wilton > Cc: 'Squid Developers' > Subject: Re: Linux filesystem speed comparison > > The only test I know of that accurately predicts how a proxy will > perform when given real load is Polygraph. And depending on the > hardware configuration, either ext2/ext3 or reiserfs will easily > outperform xfs. In my experience, ReiserFS is a better performer > assuming CPU is not a bottleneck. But it is a much heavier > user of CPU, > and so some test results (like Duane's extensive benchmarks > from a year > or more ago) show ext2/3 performing measurably better than > ReiserFS. A > Polymix-4 test will fill the cache twice and then begin the > test...so it > takes into account the decline in performance that hits all > filesystems. > > It depends on the balance of hardware, but I'd be extremely > surprised if > XFS performs better than either reiser or ext2/3 for Squid > workloads on > /any/ system. So I have to assume your methodology is > slightly flawed. > ;-) That's what I thought, but there has been a bit of XFS work in recent kernels, and after my initial observations I was wondering if this has improved the performance with squid's filesystem load. > While I have found that ext3 (when configured correctly) has improved > performance for Squid quite a bit over ext2, it is still no match for > ReiserFS on our hardware, which always has more than enough > CPU for the > disk bandwidth available. But, I can certainly imagine a hardware > configuration that would lead to ext3 performing better than ReiserFS > (especially since Duane has proven that it is possible by putting 6 > 10,000 RPM disks on a relatively wimpy CPU and testing the > configuration > extensively with polygraph). The machines are a bit old (P3-500), but they've only got 3x 9Gb SCSI cache disks, and they're not running anywhere near 100% load. > I'm always interested in conflicting reports, however. If > you've got a > case that makes XFS faster for Squid against polygraph, I'd > love to see > the results and configuration. I had a quick look at polygraph before, but I didn't get very far in testing it. I would like to produce some polygraph figures for the proxies, so I will see what I can do to make a test system. My only concern is that the proxies may be able to process requests faster than the polygraph hardware can serve them. >From memory there are a lot of options available for polygraph, and I was not sure how to produce meaningful results. Any help would be appreciated. Regards Steven -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.308 / Virus Database: 266.9.6 - Release Date: 4/11/2005
RE: Linux filesystem speed comparison
> -Original Message- > From: Henrik Nordstrom [mailto:[EMAIL PROTECTED] > Sent: Monday, April 11, 2005 6:09 PM > To: Steven Wilton > Cc: 'Squid Developers' > Subject: Re: Linux filesystem speed comparison > > On Mon, 11 Apr 2005, Steven Wilton wrote: > > > We are running some large proxies in our Melbourne POP, and > we graph the CPU > > counters available in the 2.6 linux kernel to give us an > idea of what the > > CPU is doing. We noticed that the CPU was spending large > amounts of time > > (around 60%) in an I/O wait state, which is when the CPU is > idle, but there > > are pending disk i/o opeartions. > > Which as such isn't that harmful to Squid (aufs/diskd) as > Squid continues > processing requests while there is pending I/O request. But > on the other > hand when the disk %util level approaches 100 you reach the > limit of what > the drive can sustain. My thoughts were that if the numbers for %CPU in system and user were similar, then a "more efficient" filesystem would arrange the data on disk in such a way that the disk spends less time performing the operations. I will add graphs for the /proc/diskstats value that records the amount of time the disk is actually performing operations, and see how this compares across the different filesystems (I looked at the iostat source to see how it calculates the %util value). Regards Steven -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.308 / Virus Database: 266.9.6 - Release Date: 4/11/2005
Re: Linux filesystem speed comparison
Steven Wilton wrote: The interesting thing is that this test shows that in a 2.6.10 kernel, XFS is the clear winner for I/O wait, followed by ext3 writeback. I was not surprised to see reiser come off worse than ext3, as I have previously tried to use reiser on our proxies (on a 2.2 kernel), and noticed that initially the proxy was a lot quicker, but as the disk filled up, the cache performance dropped. I thought I'd post this to squid-dev for comments first, as I have read other posts that say that squid+reiser is the recommended combination, and was wondering if there are other tests that I should perform. The only test I know of that accurately predicts how a proxy will perform when given real load is Polygraph. And depending on the hardware configuration, either ext2/ext3 or reiserfs will easily outperform xfs. In my experience, ReiserFS is a better performer assuming CPU is not a bottleneck. But it is a much heavier user of CPU, and so some test results (like Duane's extensive benchmarks from a year or more ago) show ext2/3 performing measurably better than ReiserFS. A Polymix-4 test will fill the cache twice and then begin the test...so it takes into account the decline in performance that hits all filesystems. It depends on the balance of hardware, but I'd be extremely surprised if XFS performs better than either reiser or ext2/3 for Squid workloads on /any/ system. So I have to assume your methodology is slightly flawed. ;-) While I have found that ext3 (when configured correctly) has improved performance for Squid quite a bit over ext2, it is still no match for ReiserFS on our hardware, which always has more than enough CPU for the disk bandwidth available. But, I can certainly imagine a hardware configuration that would lead to ext3 performing better than ReiserFS (especially since Duane has proven that it is possible by putting 6 10,000 RPM disks on a relatively wimpy CPU and testing the configuration extensively with polygraph). I'm always interested in conflicting reports, however. If you've got a case that makes XFS faster for Squid against polygraph, I'd love to see the results and configuration.
Re: Linux filesystem speed comparison
On Mon, 11 Apr 2005, Steven Wilton wrote: We are running some large proxies in our Melbourne POP, and we graph the CPU counters available in the 2.6 linux kernel to give us an idea of what the CPU is doing. We noticed that the CPU was spending large amounts of time (around 60%) in an I/O wait state, which is when the CPU is idle, but there are pending disk i/o opeartions. Which as such isn't that harmful to Squid (aufs/diskd) as Squid continues processing requests while there is pending I/O request. But on the other hand when the disk %util level approaches 100 you reach the limit of what the drive can sustain. The more work Squid is having the less %iowait you should be seeing due to Squid (and other tasks) making use of that CPU time. For this reason it is better to measure the disk %util rather than the system %iowait. In the CPU utilization you should sum %iowait and %idle as idle CPU time. %user and %sys is interesting. Note: %util is reported by iostat -x. Regards Henrik
Linux filesystem speed comparison
We are running some large proxies in our Melbourne POP, and we graph the CPU counters available in the 2.6 linux kernel to give us an idea of what the CPU is doing. We noticed that the CPU was spending large amounts of time (around 60%) in an I/O wait state, which is when the CPU is idle, but there are pending disk i/o opeartions. Some other recent tests have shown that on linux the aufs disk type gives us the best performance, but I wanted to see if I could reduce the amount of I/O wait time on the proxy servers by changing the filesystem. In Perth we have 4 identical proxies (P3-500, 512Mb RAM, 3x9Gb cache disks, linux 2.6.10 kernel, squid s2_5-epoll tree), which we were running with the ext3 filesystem. I reformatted 3 of them with reiserfs, xfs and jfs to see what difference each of these filesystems would have on the I/O wait. The mount options for each are as follows: /dev/sdb1 on /var/spool/squid/disk1 type reiserfs (rw,noatime,notail) /dev/sdb1 on /var/spool/squid/disk1 type xfs (rw,noatime) /dev/sdb1 on /var/spool/squid/disk1 type ext3 (rw,noatime,data=writeback) /dev/sdb1 on /var/spool/squid/disk1 type jfs (rw,noatime) Below is a single set of results from the daily averages of the graphs we have. I have taken 10 samples of 5 mijnute averages over the past week, and they come up with similar figures (the 5 minute samples are pasted at the end of this e-mail): Filesystem UserSys IO Req/sec U/R S/R I/R Reiser 7.6 8.4 14.128 0.270.170.50 Xfs 8.4 5.3 4.4 27.30.310.190.16 Ext37.6 4.4 10.428.20.270.160.15 Jfs 7.3 4.1 15.826.60.270.150.59 The numbers are as follows: User- %CPU user Sys - %CPU system IO - %CPU IO wait Req/sec - Requests/sec for squid U/R - User/(Req/sec) S/R - Sys/(Req/sec) I/R - IO /(Req/sec) The interesting thing is that this test shows that in a 2.6.10 kernel, XFS is the clear winner for I/O wait, followed by ext3 writeback. I was not surprised to see reiser come off worse than ext3, as I have previously tried to use reiser on our proxies (on a 2.2 kernel), and noticed that initially the proxy was a lot quicker, but as the disk filled up, the cache performance dropped. I thought I'd post this to squid-dev for comments first, as I have read other posts that say that squid+reiser is the recommended combination, and was wondering if there are other tests that I should perform. Steven The UserSys IO Req/sec U/R S/R I/R 5/4 9:43am reiser 11.57.3 12.254 0.210.140.23 xfs 12.37.4 4.5 49.40.250.150.09 ext312.28.8 10 48 0.250.180.21 jfs 9 5.3 10.240.90.220.130.25 5/4 8:23pm reiser 11.88 13 46.10.260.170.28 xfs 12.98 6.2 56.50.230.140.11 ext313.48.8 14.359.70.220.150.24 jfs 12.47.8 12.856.20.220.140.23 6/4 7:23am reiser 4.3 2.2 5.1 21.20.200.100.24 xfs 5.2 2.6 1.2 24.30.210.110.05 ext34.1 2.3 5.1 13.90.290.170.37 jfs 5.9 2.7 4.7 17 0.350.160.28 6/4 10:47am reiser 10.97.6 12.348.10.230.160.26 xfs 11.57.5 5.6 50.70.230.150.11 ext311.17.4 14.151 0.220.150.28 jfs 10.26.2 13.242.10.240.150.31 6/4 12:02pm reiser 10.16.2 12.549.70.200.120.25 xfs 11.68.3 6.4 49 0.240.170.13 ext312.38.2 14.448.80.250.170.30 jfs 10.16.2 11.241.30.240.150.27 6/4 15:20pm reiser 10.26.8 12.547.80.210.140.26 xfs 13.99.9 7.6 58.90.240.170.13 ext311.97.9 13.546.90.250.170.29 jfs 13.46 13.441.90.320.140.32 7/4 07:54am reiser 7.8 4.7 10.734.80.220.140.31 xfs 8.4 5.6 4.7 26.90.310.210.17 ext37.5 5.2 10.229.40.260.180.35 jfs 6 3.7 9.3 24.80.240.150.38 7/4 1:44pm reiser 12 8.5 19.755.30.220.150.36 xfs 11.4
