Re: [squid-users] Opinions sought on best storage type for FreeBSD- suggestion for developers

2007-08-26 Thread Adrian Chadd
On Fri, Aug 24, 2007, Michel Santos wrote: this donate thing does not work :) It works fine for other projects; but it just hasn't caught on here with Squid. and then when comes an fearless like me and ask how much would that cost then the answer is -z ... -z ? Adrian, you have lot of

Re: [squid-users] Opinions sought on best storage type for FreeBSD- suggestion for developers

2007-08-24 Thread Michel Santos
Adrian Chadd disse na ultima mensagem: On Mon, Aug 20, 2007, Nicole wrote: [snip good points] It has been found that people are more likely to donate money for something specific than for a general cause. Maybe we're not doing it right; but people seem quite happy to suggest

Re: [squid-users] Opinions sought on best storage type for FreeBSD- suggestion for developers

2007-08-23 Thread Adrian Chadd
On Mon, Aug 20, 2007, Nicole wrote: [snip good points] It has been found that people are more likely to donate money for something specific than for a general cause. Maybe we're not doing it right; but people seem quite happy to suggest functionality but not be willing to donate to see it

Re: [squid-users] Opinions sought on best storage type for FreeBSD- suggestion for developers

2007-08-20 Thread Nicole
On 11-Aug-07 My Secret NSA Wiretap Overheard Adrian Chadd Saying : On Sat, Aug 11, 2007, Michel Santos wrote: I must admit I can't talk in there because I never could test it really but I do not convinve myself easy by reading papers. Good! Thats why you take the papers and try to

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-14 Thread Tek Bahadur Limbu
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Michel, I am coming back to the topic of Squid restarting itself when the load increases and the number of clients requests goes upto 100-200 per second. After digging around cache.log of my Squid boxes, I am seeing the following messages. And

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-14 Thread Tek Bahadur Limbu
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Mark, On Tue, 14 Aug 2007 10:35:03 +1000 Mark Nottingham [EMAIL PROTECTED] wrote: FreeBSD and aufs was discussed a while back, IIRC, and the upshot was that for FreeBSD 6, it's useful (threads on 4 is a no-no). The lingering doubt in my

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-14 Thread Henrik Nordstrom
On tis, 2007-08-14 at 12:29 +0545, Tek Bahadur Limbu wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Michel, I am coming back to the topic of Squid restarting itself when the load increases and the number of clients requests goes upto 100-200 per second. After digging around

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-14 Thread Michel Santos
Mark Nottingham disse na ultima mensagem: FreeBSD and aufs was discussed a while back, IIRC, and the upshot was that for FreeBSD 6, it's useful (threads on 4 is a no-no). The lingering doubt in my mind was this bug: http://www.freebsd.org/cgi/ query-pr.cgi?pr=103127, which appears to have

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-14 Thread Henrik Nordstrom
On tis, 2007-08-14 at 08:32 -0300, Michel Santos wrote: and it should work well, I had no problem at all with the aufs model itself beside queue-congestion alert msgs while swap.state rebuilding was in progress and sometimes under load. Some of these is perfectly normal and seen when there is

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-13 Thread Tek Bahadur Limbu
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Michel, On Sun, 12 Aug 2007 17:51:06 -0300 (BRT) Michel Santos [EMAIL PROTECTED] wrote: Tek Bahadur Limbu disse na ultima mensagem: Ok let me upgrade my memory before setting it to 2 GB or more. I will set it to 768 MB for now since I

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-13 Thread Michel Santos
Tek Bahadur Limbu disse na ultima mensagem: what size is your link? For each proxy, the link is burstable upto to 15 mbps. But they are grouped together in different groups. We have 6 groups. Each group has bandwidth ranging from 5 mbps to 20 mbps. However since our link comes via

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-13 Thread Michel Santos
Adrian Chadd disse na ultima mensagem: well that was my knowledge about chances but here are not so many options, or you are a hell of forseer or you create an algorithm, kind of inverting the usage of the actual or other cache policies applying them before caching the objects instead of

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-13 Thread Tek Bahadur Limbu
Hi Michel, Michel Santos wrote: Tek Bahadur Limbu disse na ultima mensagem: what size is your link? For each proxy, the link is burstable upto to 15 mbps. But they are grouped together in different groups. We have 6 groups. Each group has bandwidth ranging from 5 mbps to 20 mbps. However

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-13 Thread Mark Nottingham
FreeBSD and aufs was discussed a while back, IIRC, and the upshot was that for FreeBSD 6, it's useful (threads on 4 is a no-no). The lingering doubt in my mind was this bug: http://www.freebsd.org/cgi/ query-pr.cgi?pr=103127, which appears to have been patched in 6.1- RELEASE-p5. So, in a

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-13 Thread Henrik Nordstrom
On tis, 2007-08-14 at 10:35 +1000, Mark Nottingham wrote: So, in a nutshell, can it be safely said that aufs is stable and reasonably performant on FreeBSD = 6.2, as long as the described thread configuration is performed? Thats my understanding of the issue, but not being a FreeBSD user.

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-12 Thread Tek Bahadur Limbu
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sat, 11 Aug 2007 17:24:10 -0300 (BRT) Michel Santos [EMAIL PROTECTED] wrote: Tek Bahadur Limbu disse na ultima mensagem: Michel Santos wrote: Tek Bahadur Limbu disse na ultima mensagem: diskd indeed seems to fail under load especially

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-12 Thread Michel Santos
Tek Bahadur Limbu disse na ultima mensagem: how much mem the server has installed? Most of them have 1 GB memory well, I believe that is really too low for such a busy machine and you should think of 4-8 gigs (or more?) for such a server what is you kern.maxdsiz value? It's the default

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-12 Thread Tek Bahadur Limbu
Hi Michel, Michel Santos wrote: Tek Bahadur Limbu disse na ultima mensagem: how much mem the server has installed? Most of them have 1 GB memory well, I believe that is really too low for such a busy machine and you should think of 4-8 gigs (or more?) for such a server Well I think you

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-12 Thread Michel Santos
Tek Bahadur Limbu disse na ultima mensagem: Ok let me upgrade my memory before setting it to 2 GB or more. I will set it to 768 MB for now since I have only 1 GB of memory at the moment. I believe with stock maxdsiz your squid process can not use more than the 512MB limit ... so I do not

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-12 Thread Adrian Chadd
On Sat, Aug 11, 2007, Michel Santos wrote: * don't write everything cachable to disk! only write stuff that has a good chance of being read again; there is a good chance beeing hit by a car when sleeping in the middle of a highway as well there is a chance not beeing hit at all :) :)

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Tek Bahadur Limbu
Henrik Nordstrom wrote: On fre, 2007-08-10 at 06:50 -0300, Michel Santos wrote: please be a little bit more specific about comitting resources, what do you exactly mean? Gettin a developer working on fixing the bugs. what is what you agree to be broken beyond the shutdown issue? Bug #761

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Henrik Nordstrom
On lör, 2007-08-11 at 15:10 +0545, Tek Bahadur Limbu wrote: As far as I know and seen with my limited experience, diskd seems good for BSD boxes. But I guess I have to try other alternatives too. If I opt to use aufs, will the following compilations work? '--enable-async-io'

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Adrian Chadd
On Sat, Aug 11, 2007, Tek Bahadur Limbu wrote: Or simply, what is the best compilation parameters to use on a Linux/Unix machine if I want to use aufs? coss at first seemed a good choice but it's long rebuilding process is not suitable for production use. We know how to fix COSS. Time

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Michel Santos
Tek Bahadur Limbu disse na ultima mensagem: diskd indeed seems to fail under load especially when approaching 200/300 requests per second. are you sure this numbers are correct? where do you get them from? It causes Squid to crash and restart automatically. Though, the side effects are

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Michel Santos
Henrik Nordstrom disse na ultima mensagem: On lör, 2007-08-11 at 15:10 +0545, Tek Bahadur Limbu wrote: As far as I know and seen with my limited experience, diskd seems good for BSD boxes. But I guess I have to try other alternatives too. If I opt to use aufs, will the following

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Michel Santos
Adrian Chadd disse na ultima mensagem: On Sat, Aug 11, 2007, Tek Bahadur Limbu wrote: Or simply, what is the best compilation parameters to use on a Linux/Unix machine if I want to use aufs? coss at first seemed a good choice but it's long rebuilding process is not suitable for production

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Adrian Chadd
On Sat, Aug 11, 2007, Michel Santos wrote: nice words but IMO the fs usage is kind of fine-tuning because the difference between the actual competitors aufs and coss is not sooo big Yeah, but the difference between AUFS/COSS and whats actually possible and done in the commercial world - and

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Michel Santos
Adrian Chadd disse na ultima mensagem: On Sat, Aug 11, 2007, Michel Santos wrote: nice words but IMO the fs usage is kind of fine-tuning because the difference between the actual competitors aufs and coss is not sooo big Yeah, but the difference between AUFS/COSS and whats actually possible

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Adrian Chadd
On Sat, Aug 11, 2007, Michel Santos wrote: but look, easy to code but 6 month of concentrated work ar not so really the same things ... ;) Heh, I didn't mean that stuff is easy to code but more spending time improving squid and coding up faster storage/network/http/etc code is easier to

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Adrian Chadd
On Sat, Aug 11, 2007, Michel Santos wrote: I must admit I can't talk in there because I never could test it really but I do not convinve myself easy by reading papers. Good! Thats why you take the papers and try to duplicate/build from them to convince yourself. Google for UCFS web cache,

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Michel Santos
Adrian Chadd disse na ultima mensagem: On Sat, Aug 11, 2007, Michel Santos wrote: I must admit I can't talk in there because I never could test it really but I do not convinve myself easy by reading papers. Good! Thats why you take the papers and try to duplicate/build from them to

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Tek Bahadur Limbu
Michel Santos wrote: Tek Bahadur Limbu disse na ultima mensagem: diskd indeed seems to fail under load especially when approaching 200/300 requests per second. are you sure this numbers are correct? where do you get them from? Hi Michel, I am getting these numbers from one of my busy proxy

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Michel Santos
Tek Bahadur Limbu disse na ultima mensagem: Michel Santos wrote: Tek Bahadur Limbu disse na ultima mensagem: diskd indeed seems to fail under load especially when approaching 200/300 requests per second. are you sure this numbers are correct? where do you get them from? Hi Michel, I am

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-11 Thread Henrik Nordstrom
On lör, 2007-08-11 at 09:21 -0300, Michel Santos wrote: not sure, both thread implementations work well but kernel threads pretend to be faster. In order to sense it you need some real load on the machine and I am not sure if there is a difference at all on an UP machine for aufs you need

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-10 Thread Michel Santos
Henrik Nordstrom disse na ultima mensagem: On tor, 2007-08-09 at 10:18 -0700, Nicole wrote: As some have pointed out, it's a shame diskd is horked, since it seemed to be nice and fast. Well, it's been broken for several years now, an no one has been willing to commit any resources to get it

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-10 Thread Henrik Nordstrom
On tor, 2007-08-09 at 10:18 -0700, Nicole wrote: As some have pointed out, it's a shame diskd is horked, since it seemed to be nice and fast. Well, it's been broken for several years now, an no one has been willing to commit any resources to get it fixed. However, since I have not heard of

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-10 Thread Henrik Nordstrom
On fre, 2007-08-10 at 11:24 -0300, Alexandre Correa wrote: using aufs works fine.. my server receives about 4000 to 6000 req/min !! quite modest load. Not a very high load. file system of hard disk is reiserfs4 !! USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-10 Thread Alexandre Correa
using aufs works fine.. my server receives about 4000 to 6000 req/min !! file system of hard disk is reiserfs4 !! USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND squid32748 1.1 35.0 1493184 1446748 ? Sl Aug09 8:17 (squid) -D -s the server is very clean,

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-10 Thread Michel Santos
Alexandre Correa disse na ultima mensagem: after reading this email, i switched from aufs to diskd to see performance of them under high load .. with aufs, squid never used more than 10% of cpu and response time is very low (5ms to 150ms).. with diskd cpu usage goes to 50% +- and median

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-10 Thread Alexandre Correa
after reading this email, i switched from aufs to diskd to see performance of them under high load .. with aufs, squid never used more than 10% of cpu and response time is very low (5ms to 150ms).. with diskd cpu usage goes to 50% +- and median response time up to 900ms !! i´m running CentOS 5.0

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-10 Thread Michel Santos
Henrik Nordstrom disse na ultima mensagem: On fre, 2007-08-10 at 06:50 -0300, Michel Santos wrote: what is what you agree to be broken beyond the shutdown issue? Bug #761 unstable under high load when using diskd cache_dir diskd falls over under load due to internal design problems in how

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-10 Thread Michel Santos
Adrian Chadd disse na ultima mensagem: On Thu, Aug 09, 2007, Michel Santos wrote: the bug, I am curious what others have been using or prefer as their alternative to diskd and why? diskd for sure is the fastest specially on SMP machines but there are not so much people sharing my opinion

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-10 Thread Henrik Nordstrom
On fre, 2007-08-10 at 06:50 -0300, Michel Santos wrote: please be a little bit more specific about comitting resources, what do you exactly mean? Gettin a developer working on fixing the bugs. what is what you agree to be broken beyond the shutdown issue? Bug #761 unstable under high load

[squid-users] Opinions sought on best storage type for FreeBSD

2007-08-09 Thread Nicole
Hello I run a large number of FreeBSD based servers as cache accelerators for large scale image serving. (amd64 and most with dual core) Each server has (3) 147G disks and 36G of the boot disk. Altho I have some older servers that have 36G and (3) 72G disks. The older (smaller) servers

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-09 Thread Michel Santos
Nicole disse na ultima mensagem: Hello I run a large number of FreeBSD based servers as cache accelerators for large scale image serving. (amd64 and most with dual core) Each server has (3) 147G disks and 36G of the boot disk. Altho I have some older servers that have 36G and (3) 72G

Re: [squid-users] Opinions sought on best storage type for FreeBSD

2007-08-09 Thread Adrian Chadd
On Thu, Aug 09, 2007, Michel Santos wrote: the bug, I am curious what others have been using or prefer as their alternative to diskd and why? diskd for sure is the fastest specially on SMP machines but there are not so much people sharing my opinion ... Just supply real-world numbers