Re: multi-thread
Thus spake Jacques Frip' WERNERT ([EMAIL PROTECTED]): ok, on my Solaris, the qmail distribution is "forking" almost 10 to 20 processes per second. Solaris is shunned for its incredibly bad fork performance. Install Sparc-Linux or some BSD variant if that is a problem for you. So I'm trying to work on a threaded qmail-rspawn to avoid so many forks Bad idea. Very bad idea. Felix
Re: multi-thread
Hello, I'll take care of your advice Thanx Frip' - Original Message - From: "Felix von Leitner" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, February 08, 2001 12:15 PM Subject: Re: multi-thread Thus spake Jacques Frip' WERNERT ([EMAIL PROTECTED]): ok, on my Solaris, the qmail distribution is "forking" almost 10 to 20 processes per second. Solaris is shunned for its incredibly bad fork performance. Install Sparc-Linux or some BSD variant if that is a problem for you. So I'm trying to work on a threaded qmail-rspawn to avoid so many forks Bad idea. Very bad idea. Felix
Re: multi-thread
ok, on my Solaris, the qmail distribution is "forking" almost 10 to 20 processes per second. This cost a lot in system ressources and system calls Yes. Unfortunately, Solaris isn't Unix, and qmail was designed to run on Unix systems. Unix is rather good at forking, especially images as tiny as qmail; Solaris isn't. As Rob Pike once said, "perhaps if people had understood fork() better we wouldn't have threads". So I'm trying to work on a threaded qmail-rspawn to avoid so many forks Yikes. I'm going to put my manager's hat on for a moment. How much time do you intend to spend on developing and debugging this? How much does that time cost? How much would it cost to buy a fast PC, run a real Unix (I'd suggest OpenBSD, FreeBSD, or some version of Linux) on it, and make that your mail server? Tim.
Re: multi-thread
On Thu, Feb 08, 2001 at 12:06:05PM -, Tim Goodwin wrote: ok, on my Solaris, the qmail distribution is "forking" almost 10 to 20 processes per second. This cost a lot in system ressources and system calls Yes. Unfortunately, Solaris isn't Unix, and qmail was designed to run on Unix systems. Unix is rather good at forking, especially images as tiny as qmail; Solaris isn't. As Rob Pike once said, "perhaps if people had understood fork() better we wouldn't have threads". So I'm trying to work on a threaded qmail-rspawn to avoid so many forks Yikes. If all he's trying to achive is reduce forking on his Solaris box, I concur. However if we generalize the question, I don't know that I'd draw the same conclusion. If any area of qmail would benefit for threading, it might be the remote delivery mechanism - currently handled by Batman and Robin, er, sorry, qmail-rspawn and qmail-remote. First off, there is an amount of data they can share and cache, such as tcpok and recent DNS lookups. Second, remote delivery can have very high latency so any footprint saving is a big saving. Third, the state requirements are truly tiny. A socket and an fd is just about all that the thread needs. Fourth, there are few security issues. Neither qmail-rspawn nor qmail-remote need any special file system access. This is often a nasty complication for threaded implementations. Not so here. Fifth, the interface is simple and clean, plug in the threaded qmail-rspawn and no one is any the wiser. Sixth, the problem domain isn't that large: $ wc -l qmail-rspawn.c qmail-remote.c 103 qmail-rspawn.c 427 qmail-remote.c 530 total Having said that, in the scheme of things, qmail-remote borders on ridiculously tiny as it is. I recently wrote a queueless wrapper program that uses qmail-remote as the smtp engine (opt-in spam I call it). I rediscovered that a concurrency of 1,000 qmail-remotes consumes very little system resource on FreeBSD. I'm going to put my manager's hat on for a moment. How much time do you intend to spend on developing and debugging this? How much does that time cost? How much would it cost to buy a fast PC, run a real Unix (I'd suggest OpenBSD, FreeBSD, or some version of Linux) on it, and make that your mail server? As a solitary exercise solely designed to speed up one system, of course replacing the box may be a better solution. Regards.
Re: multi-thread
On Thu, Feb 08, 2001 at 04:51:39PM +, Mark Delany wrote: [snip] First off, there is an amount of data they can share and cache, such as tcpok and recent DNS lookups. That's what the dnscache running on your 127.0.0.1 is for. That's close enough. [snip] Fifth, the interface is simple and clean, plug in the threaded qmail-rspawn and no one is any the wiser. With nonblocking sockets and select(), one could write a single-threaded qmail-rspawn/remote. Only need to find a way to do the dns lookups in parallel. Greetz, Peter.
Re: multi-thread
Fifth, the interface is simple and clean, plug in the threaded qmail-rspawn and no one is any the wiser. With nonblocking sockets and select(), one could write a single-threaded qmail-rspawn/remote. Only need to find a way to do the dns lookups in parallel. Yes Virginia. There are at least three ways to skin a cat in Unix. And even the design of a good threaded implementation may not be the obvious one of one thread per socket. It might be, eg, a thread per function, or a thread per lockable area or a thread per external interface or... And yes Virginia. This is off topic now. Regards.
Re: multi-thread
Thus spake Mark Delany ([EMAIL PROTECTED]): If all he's trying to achive is reduce forking on his Solaris box, I concur. However if we generalize the question, I don't know that I'd draw the same conclusion. If any area of qmail would benefit for threading, it might be the remote delivery mechanism - currently handled by Batman and Robin, er, sorry, qmail-rspawn and qmail-remote. Nothing benefits from multithreading. It makes the code hard to understand, creates new problems (one thread dies, the whole app dies), kills resource limits, and is not even faster. There is no reason to use multithreading except if you are a marketing guy at Sun or Microsoft and your analysis says that it is cheaper to ram multithreading down people's throats than to fix the insanely huge process creation latency of your broken poor excuse of an operating system. Felix
Re: multi-thread
At 04:51 PM 08-02-2001 +, Mark Delany wrote: it). I rediscovered that a concurrency of 1,000 qmail-remotes consumes very little system resource on FreeBSD. What do we gain by multithreading? The design of qmail will theoretically result in more latency e.g. process 1 forking and exec'ing process 2 which execs process 3 and so on. Also there's less shareable memory that way. However the gain in simplicity, security and robustness is worth it, especially since the processes are very small making these nonissues in most cases and especially for email. Also most other email servers aren't even close to performing at that level in the first place :). I can see how it will be an issue for web servers though. Still if we start putting fat processes (a perl checkmail for instance) in the typical qmail process pipeline it could get ugly. Or do anything which needs a long startup time (make db connections). Cheerio, Link.
Re: multi-thread
On Wed, Feb 07, 2001 at 04:47:24PM +0100, Jacques Frip' WERNERT wrote: does anyone know if someone is working on a multithreaded release of qmail-smtpd and or qmail-rspawn? s/release/version/ As far as i know nobody does that right now. It might be fun, though. Regards, Uwe
Re: multi-thread
Hello :)) ok, on my Solaris, the qmail distribution is "forking" almost 10 to 20 processes per second. This cost a lot in system ressources and system calls So I'm trying to work on a threaded qmail-rspawn to avoid so many forks Frip' - Original Message - From: "Peter van Dijk" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, February 07, 2001 5:03 PM Subject: Re: multi-thread On Wed, Feb 07, 2001 at 04:47:24PM +0100, Jacques Frip' WERNERT wrote: Hello, does anyone know if someone is working on a multithreaded release of qmail-smtpd and or qmail-rspawn? What do you mean? Also, could you please start a new thread when stating a new question? Thank you. [Removed In-Reply-To to start new thread]. Greetz, Peter.
Re: multi-thread
On Wed, Feb 07, 2001 at 05:45:43PM +0100, Jacques Frip' WERNERT wrote: ok, on my Solaris, the qmail distribution is "forking" almost 10 to 20 processes per second. This cost a lot in system ressources and system calls Are you kidding? What kind of hardware are you using? On my Celeron PC, I can fork and exec 200 shared processes per second, and almost 300 staticly-linked processes per second. So I'm trying to work on a threaded qmail-rspawn to avoid so many forks I'd be willing to bet it doesn't buy you enough to make it worth the effort on most modern UNIX-type OSs. It's also a rather large task, as the existing code likely relies heavily on globals. -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/ PGP signature
Re: multi-thread
On Thu, Feb 08, 2001 at 01:07:42AM -0600, Bruce Guenter wrote: On Wed, Feb 07, 2001 at 05:45:43PM +0100, Jacques Frip' WERNERT wrote: ok, on my Solaris, the qmail distribution is "forking" almost 10 to 20 processes per second. This cost a lot in system ressources and system calls Are you kidding? What kind of hardware are you using? On my Celeron PC, I can fork and exec 200 shared processes per second, and almost 300 staticly-linked processes per second. Not all Unices are created equal... Linux is extremely good at forking (in fact so good that it's threads implementation is a special-case fork), whereas others are better at threads. If you look at where Apache is going (OS-specific optimizations to squeeze the best out of each platform), it would certainly make sense to do a threads version for some of the slower Unices... ...or simpler yet - stop using them... :-) -- Cheers Jason Haar Unix/Special Projects, Trimble NZ Phone: +64 3 9635 377 Fax: +64 3 9635 417
Re: multi-thread
Hello I'm using Solaris 7 E220/420 systems. I didn't say that my system can't fork. I just said that it wastes a lot with all this forks. I have a high volume server and I need to send over 1 million mails in few hours. So I'm trying to find the best way to achieve this. I've already built a qmail with 15 qmail-queues to have the best rate. Regards Frip' - Original Message - From: "Jason Haar" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, February 08, 2001 8:19 AM Subject: Re: multi-thread On Thu, Feb 08, 2001 at 01:07:42AM -0600, Bruce Guenter wrote: On Wed, Feb 07, 2001 at 05:45:43PM +0100, Jacques Frip' WERNERT wrote: ok, on my Solaris, the qmail distribution is "forking" almost 10 to 20 processes per second. This cost a lot in system ressources and system calls Are you kidding? What kind of hardware are you using? On my Celeron PC, I can fork and exec 200 shared processes per second, and almost 300 staticly-linked processes per second. Not all Unices are created equal... Linux is extremely good at forking (in fact so good that it's threads implementation is a special-case fork), whereas others are better at threads. If you look at where Apache is going (OS-specific optimizations to squeeze the best out of each platform), it would certainly make sense to do a threads version for some of the slower Unices... ...or simpler yet - stop using them... :-) -- Cheers Jason Haar Unix/Special Projects, Trimble NZ Phone: +64 3 9635 377 Fax: +64 3 9635 417
