Re: Trying to achieve maximum speed!
Russell Nelson was overheard saying: : It modifies various programs to use hashed todo and intd directories. : This allows you to inject mail faster than qmail-send can deal with : it. Otherwise, you end up with really big directories with more than : 1,000 files. Once that happens, the kernel spends more and more time : locked reading/writing those directories. Also, if you're injecting : 100,000 messages all at once, make your conf-split bigger -- more like : 231 than the default 23. I downloaded this patch because it seems to be the kind of thing I'm looking for. Does anyone know how to modify queue-fix to deal with this kind of queue directory? It looks like the intd and todo directories changed but I'm not sure in what manner. Thanks -- Matthew Harrell The perversity of the universe Bit Twiddlers, Inc. tends to a maximum. [EMAIL PROTECTED]
Re: Trying to achieve maximum speed!
Todd at NM Technet writes: > > My rule of thumb is that every directory should have <1000 files in it. > > is this filesystem-dependant? Yes. -- -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson Crynwr supports Open Source(tm) Software| PGPok | Government schools are so 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool!
Re: Trying to achieve maximum speed!
russ, all, a quick question below... On Wed, 14 Jul 1999, Russell Nelson wrote: > > Is there any ballpark threshhold where these changes become useful? > > My rule of thumb is that every directory should have <1000 files in it. is this filesystem-dependant? ie: does this assume the fs is using some O(n) algorithm for directories (some, like XFS on IRIX and newer fs's for linux like Reiserfs (i believe) use BTrees for directories giving them O(logn) behavior, which should mean massively larger numbers of files in a directory without a performance hit). maybe i'm misunderstanding the point of the change, though. todd [EMAIL PROTECTED]
Re: Trying to achieve maximum speed!
Keith Burdis writes: > On Tue 1999-07-13 (16:50), Russell Nelson wrote: > > [EMAIL PROTECTED] writes: > > > > > > Wow. Looks like I do :-) Can you explain what it does? > > > > It modifies various programs to use hashed todo and intd directories. > > This allows you to inject mail faster than qmail-send can deal with > > it. Otherwise, you end up with really big directories with more than > > 1,000 files. Once that happens, the kernel spends more and more time > > locked reading/writing those directories. Also, if you're injecting > > 100,000 messages all at once, make your conf-split bigger -- more like > > 231 than the default 23. > > When does it make sense to apply the big-todo patch and increase the size of > conf-split? conf-split needs to be bigger if you have more than 23K distinct messages (not addresses) in the queue. big-todo is needed if you're injecting messages faster than qmail-send can process them. > Is there any ballpark threshhold where these changes become useful? My rule of thumb is that every directory should have <1000 files in it. > Would it hurt performance in any way if I made these changes on a relatively > low volume system? Slightly, but not in any large manner. -- -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson Crynwr supports Open Source(tm) Software| PGPok | Government schools are so 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool!
Re: Trying to achieve maximum speed!
On Tue 1999-07-13 (16:50), Russell Nelson wrote: > [EMAIL PROTECTED] writes: > > > > Wow. Looks like I do :-) Can you explain what it does? > > It modifies various programs to use hashed todo and intd directories. > This allows you to inject mail faster than qmail-send can deal with > it. Otherwise, you end up with really big directories with more than > 1,000 files. Once that happens, the kernel spends more and more time > locked reading/writing those directories. Also, if you're injecting > 100,000 messages all at once, make your conf-split bigger -- more like > 231 than the default 23. When does it make sense to apply the big-todo patch and increase the size of conf-split? Is there any ballpark threshhold where these changes become useful? Would it hurt performance in any way if I made these changes on a relatively low volume system? Thanks. > -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson - Keith -- Keith Burdis - MSc (Com Sci) - Rhodes University, South Africa Email : [EMAIL PROTECTED] WWW : http://www.rucus.ru.ac.za/~keith/ IRC : Panthras JAPH "Any technology sufficiently advanced is indistinguishable from a perl script" Standard disclaimer. ---
Re: Trying to achieve maximum speed!
Ok, I've done all this, and it is most definitely faster, but I'm still getting fairly low numbers on concurrent remotes. Adveraging around 4 - 5 out of 75. I guess I need to understand why this becomes limited and why it doesn't try to fill its limit. We just started the process about 5 minutes ago and the queue is up to 4500 messages already and growing. It's going to hit about 100,000 before it's done, then it will take a very long time to empty out that queue. When I got in this morning, there were over 200,000 messages in the queue, and this is why I decided to wipe it out and start over. -jeremy > empty queue. Easiest way to do that is to install a second > installation of qmail in /var/qmail2. Make a symlink: > rm -rf /var/qmail2/control > ln -s /var/qmail/control /var/qmail2/control > > Associate /var/qmail2/bin/qmail-smtpd with port 25, and > /var/qmail2/bin/sendmail with /usr/lib/sendmail. In 7 days (aka > control/queuelifetime), the old queue will be empty, and you can > remove that installation of qmail. > > -- > -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson > Crynwr supports Open Source(tm) Software| PGPok | Government schools are so > 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur > Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool! > http://www.xxedgexx.com | [EMAIL PROTECTED] - Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
Markus Stumpf writes: > Russell Nelson wrote: > > locked reading/writing those directories. Also, if you're injecting > > 100,000 messages all at once, make your conf-split bigger -- more like > > 231 than the default 23. > > Can one just recompile qmail with a bigger conf-split and install it > with a existing queue or does one have to start over with an empty queue? empty queue. Easiest way to do that is to install a second installation of qmail in /var/qmail2. Make a symlink: rm -rf /var/qmail2/control ln -s /var/qmail/control /var/qmail2/control Associate /var/qmail2/bin/qmail-smtpd with port 25, and /var/qmail2/bin/sendmail with /usr/lib/sendmail. In 7 days (aka control/queuelifetime), the old queue will be empty, and you can remove that installation of qmail. -- -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson Crynwr supports Open Source(tm) Software| PGPok | Government schools are so 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool!
Re: Trying to achieve maximum speed!
Russell Nelson wrote: > locked reading/writing those directories. Also, if you're injecting > 100,000 messages all at once, make your conf-split bigger -- more like > 231 than the default 23. Can one just recompile qmail with a bigger conf-split and install it with a existing queue or does one have to start over with an empty queue? \Maex -- SpaceNet GmbH | http://www.Space.Net/ | Yeah, yo mama dresses Research & Development| mailto:[EMAIL PROTECTED] | you funny and you need Joseph-Dollinger-Bogen 14 | Tel: +49 (89) 32356-0| a mouse to delete files D-80807 Muenchen | Fax: +49 (89) 32356-299 |
Re: Trying to achieve maximum speed!
Russell Nelson wrote: > > [EMAIL PROTECTED] writes: > > > > Wow. Looks like I do :-) Can you explain what it does? > > It modifies various programs to use hashed todo and intd directories. > This allows you to inject mail faster than qmail-send can deal with > it. Otherwise, you end up with really big directories with more than > 1,000 files. Once that happens, the kernel spends more and more time > locked reading/writing those directories. Also, if you're injecting > 100,000 messages all at once, make your conf-split bigger -- more like > 231 than the default 23. I might say a big stupid thing but... why not use a not less stupid fat32 partition for a special big-todo queue directory then ? (forget it).
Re: Trying to achieve maximum speed!
Thank you Russell. I applied the patch and I already see things running much faster. I see my concurrentremote is filling up too! It's going up to about 20/75 instead of staying around 3 or 4. I will also do what you suggest on the split and see how that goes. Thanks again! -jeremy > [EMAIL PROTECTED] writes: > > > > Wow. Looks like I do :-) Can you explain what it does? > > It modifies various programs to use hashed todo and intd directories. > This allows you to inject mail faster than qmail-send can deal with > it. Otherwise, you end up with really big directories with more than > 1,000 files. Once that happens, the kernel spends more and more time > locked reading/writing those directories. Also, if you're injecting > 100,000 messages all at once, make your conf-split bigger -- more like > 231 than the default 23. > > -- > -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson > Crynwr supports Open Source(tm) Software| PGPok | Government schools are so > 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur > Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool! > http://www.xxedgexx.com | [EMAIL PROTECTED] - Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
[EMAIL PROTECTED] writes: > > Wow. Looks like I do :-) Can you explain what it does? It modifies various programs to use hashed todo and intd directories. This allows you to inject mail faster than qmail-send can deal with it. Otherwise, you end up with really big directories with more than 1,000 files. Once that happens, the kernel spends more and more time locked reading/writing those directories. Also, if you're injecting 100,000 messages all at once, make your conf-split bigger -- more like 231 than the default 23. -- -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson Crynwr supports Open Source(tm) Software| PGPok | Government schools are so 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool!
Re: Trying to achieve maximum speed!
It communicates with the localhost smtp server and basically relay's every message off the localhost. -jeremy > > So your saying that their cute little java program in no way uses an MTA, > and that it contacts the remote MTA of the recipients server directly. > > If that's the case, i dont see why you even have qmail installed on the > server? You are tuning for outbound performance, but according to you > their java server: > > "communicates directly with smtp" which means that no MTA will ever get > called to do outbound SMTP. > > > On Tue, 13 Jul 1999 [EMAIL PROTECTED] wrote: > > > > >SMTP is a port on your computer. Port 25. The program uses socket > >functions to connect to this port to do its work instead of shelling out > >and calling something like /usr/sbin/sendmail. Therefore this makes it > >difficult to have the program call qmail-inject or qmail-remote if it > >never calls external programs in the first place. > > > >-jeremy > > > >> > >> Maybee i'm confused. Can you please explain to me what: > >> > >> "communicates directly with smtp" means? > >> > >> > >> ___ _ __ _ > >> __ /___ ___ /__ John Gonzalez/Net.Tech > >> __ __ \ __ \ __/_ __ `__ \/ __ /_ ___/ MDC Computers/netMDC! > >> _ / / / `__/ /_ / / / / / / /_/ / / /__ (505)437-7600/fax-437-3052 > >> /_/ /_/\___/\__/ /_/ /_/ /_/\__,_/ \___/ http://www.netmdc.com > >> [-[system info]---] > >> 2:10pm up 158 days, 21:13, 2 users, load average: 0.00, 0.06, 0.10 > >> > > > > > >http://www.xxedgexx.com | [EMAIL PROTECTED] > >- > >Y2K. We're all gonna die. > > > > > > ___ _ __ _ > __ /___ ___ /__ John Gonzalez/Net.Tech > __ __ \ __ \ __/_ __ `__ \/ __ /_ ___/ MDC Computers/netMDC! > _ / / / `__/ /_ / / / / / / /_/ / / /__ (505)437-7600/fax-437-3052 > /_/ /_/\___/\__/ /_/ /_/ /_/\__,_/ \___/ http://www.netmdc.com > [-[system info]---] > 2:20pm up 158 days, 21:23, 2 users, load average: 0.11, 0.21, 0.18 > http://www.xxedgexx.com | [EMAIL PROTECTED] - Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
[EMAIL PROTECTED] writes: > > messages in queue: 93166 > messages in queue but not yet preprocessed: 81367 You need my big-todo patch. It's http://www.qmail.org/big-todo.103.patch . -- -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson Crynwr supports Open Source(tm) Software| PGPok | Government schools are so 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool!
Re: Trying to achieve maximum speed!
Wow. Looks like I do :-) Can you explain what it does? Thanks! -jeremy > [EMAIL PROTECTED] writes: > > > > messages in queue: 93166 > > messages in queue but not yet preprocessed: 81367 > > You need my big-todo patch. It's http://www.qmail.org/big-todo.103.patch . > > -- > -russ nelson <[EMAIL PROTECTED]> http://crynwr.com/~nelson > Crynwr supports Open Source(tm) Software| PGPok | Government schools are so > 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur > Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool! > http://www.xxedgexx.com | [EMAIL PROTECTED] - Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
Right. This is a nasty state to be in as qmail-send is not processing the inbound queue quickly enough. Are you in a position to let the "not yet preprocessed" number go down to zero then run the injection program and redo this sample? If the "not yet preprocessed" number climbs or gets into double or triple digits I worry. You are probably running out of disk spindle so a look at your I/O stats would be useful at this point. I'd be looking at a 10 second samples of iostat and vmstat (or their moral equivalent). Regards. At 04:22 PM Tuesday 7/13/99, [EMAIL PROTECTED] wrote: > >Here: > >messages in queue: 93166 >messages in queue but not yet preprocessed: 81367 >-- >messages in queue: 93215 >messages in queue but not yet preprocessed: 81414 >-- >messages in queue: 93268 >messages in queue but not yet preprocessed: 81469 >-- >messages in queue: 93327 >messages in queue but not yet preprocessed: 81507 >-- >messages in queue: 93365 >messages in queue but not yet preprocessed: 81564 >-- >messages in queue: 93428 >messages in queue but not yet preprocessed: 81618 >-- >messages in queue: 93480 >messages in queue but not yet preprocessed: 81672 >-- >messages in queue: 93540 >messages in queue but not yet preprocessed: 81720 >-- >messages in queue: 93590 >messages in queue but not yet preprocessed: 81770 >-- >messages in queue: 93637 >messages in queue but not yet preprocessed: 81819 >-- >messages in queue: 93693 >messages in queue but not yet preprocessed: 81876 >-- >messages in queue: 93714 >messages in queue but not yet preprocessed: 81880 >-- > >every 10 seconds. > >Thanks >-jeremy > >> Post 2-3 minutes of this to the list. >> >> >> Regards. >> > > >http://www.xxedgexx.com | [EMAIL PROTECTED] >- >Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
So your saying that their cute little java program in no way uses an MTA, and that it contacts the remote MTA of the recipients server directly. If that's the case, i dont see why you even have qmail installed on the server? You are tuning for outbound performance, but according to you their java server: "communicates directly with smtp" which means that no MTA will ever get called to do outbound SMTP. On Tue, 13 Jul 1999 [EMAIL PROTECTED] wrote: > >SMTP is a port on your computer. Port 25. The program uses socket >functions to connect to this port to do its work instead of shelling out >and calling something like /usr/sbin/sendmail. Therefore this makes it >difficult to have the program call qmail-inject or qmail-remote if it >never calls external programs in the first place. > >-jeremy > >> >> Maybee i'm confused. Can you please explain to me what: >> >> "communicates directly with smtp" means? >> >> >> ___ _ __ _ >> __ /___ ___ /__ John Gonzalez/Net.Tech >> __ __ \ __ \ __/_ __ `__ \/ __ /_ ___/ MDC Computers/netMDC! >> _ / / / `__/ /_ / / / / / / /_/ / / /__ (505)437-7600/fax-437-3052 >> /_/ /_/\___/\__/ /_/ /_/ /_/\__,_/ \___/ http://www.netmdc.com >> [-[system info]---] >> 2:10pm up 158 days, 21:13, 2 users, load average: 0.00, 0.06, 0.10 >> > > >http://www.xxedgexx.com | [EMAIL PROTECTED] >- >Y2K. We're all gonna die. > > ___ _ __ _ __ /___ ___ /__ John Gonzalez/Net.Tech __ __ \ __ \ __/_ __ `__ \/ __ /_ ___/ MDC Computers/netMDC! _ / / / `__/ /_ / / / / / / /_/ / / /__ (505)437-7600/fax-437-3052 /_/ /_/\___/\__/ /_/ /_/ /_/\__,_/ \___/ http://www.netmdc.com [-[system info]---] 2:20pm up 158 days, 21:23, 2 users, load average: 0.11, 0.21, 0.18
Re: Trying to achieve maximum speed!
SMTP is a port on your computer. Port 25. The program uses socket functions to connect to this port to do its work instead of shelling out and calling something like /usr/sbin/sendmail. Therefore this makes it difficult to have the program call qmail-inject or qmail-remote if it never calls external programs in the first place. -jeremy > > Maybee i'm confused. Can you please explain to me what: > > "communicates directly with smtp" means? > > > ___ _ __ _ > __ /___ ___ /__ John Gonzalez/Net.Tech > __ __ \ __ \ __/_ __ `__ \/ __ /_ ___/ MDC Computers/netMDC! > _ / / / `__/ /_ / / / / / / /_/ / / /__ (505)437-7600/fax-437-3052 > /_/ /_/\___/\__/ /_/ /_/ /_/\__,_/ \___/ http://www.netmdc.com > [-[system info]---] > 2:10pm up 158 days, 21:13, 2 users, load average: 0.00, 0.06, 0.10 > http://www.xxedgexx.com | [EMAIL PROTECTED] - Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
Here: messages in queue: 93166 messages in queue but not yet preprocessed: 81367 -- messages in queue: 93215 messages in queue but not yet preprocessed: 81414 -- messages in queue: 93268 messages in queue but not yet preprocessed: 81469 -- messages in queue: 93327 messages in queue but not yet preprocessed: 81507 -- messages in queue: 93365 messages in queue but not yet preprocessed: 81564 -- messages in queue: 93428 messages in queue but not yet preprocessed: 81618 -- messages in queue: 93480 messages in queue but not yet preprocessed: 81672 -- messages in queue: 93540 messages in queue but not yet preprocessed: 81720 -- messages in queue: 93590 messages in queue but not yet preprocessed: 81770 -- messages in queue: 93637 messages in queue but not yet preprocessed: 81819 -- messages in queue: 93693 messages in queue but not yet preprocessed: 81876 -- messages in queue: 93714 messages in queue but not yet preprocessed: 81880 -- every 10 seconds. Thanks -jeremy > Post 2-3 minutes of this to the list. > > > Regards. > http://www.xxedgexx.com | [EMAIL PROTECTED] - Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
On Tue, 13 Jul 1999 [EMAIL PROTECTED] wrote: > >Well, I'm not really whining. I realize there are things that I'm limited >by and unfortunately the people in charge don't really understand, but I'd >still like to try and optimize where I can. I'm willing to accept that >there's nothing I can do. I just want to be sure. > >Right now as it stands, calling external program is pointless because like >I said, their nice program communicates directly with smtp. They said >this is to have platform portability. I understand that I guess, but why >since it is running on a Unix machine. > Maybee i'm confused. Can you please explain to me what: "communicates directly with smtp" means? ___ _ __ _ __ /___ ___ /__ John Gonzalez/Net.Tech __ __ \ __ \ __/_ __ `__ \/ __ /_ ___/ MDC Computers/netMDC! _ / / / `__/ /_ / / / / / / /_/ / / /__ (505)437-7600/fax-437-3052 /_/ /_/\___/\__/ /_/ /_/ /_/\__,_/ \___/ http://www.netmdc.com [-[system info]---] 2:10pm up 158 days, 21:13, 2 users, load average: 0.00, 0.06, 0.10
Re: Trying to achieve maximum speed!
Well, I'm not really whining. I realize there are things that I'm limited by and unfortunately the people in charge don't really understand, but I'd still like to try and optimize where I can. I'm willing to accept that there's nothing I can do. I just want to be sure. Right now as it stands, calling external program is pointless because like I said, their nice program communicates directly with smtp. They said this is to have platform portability. I understand that I guess, but why since it is running on a Unix machine. Thanks -jeremy > On Tue, 13 Jul 1999 [EMAIL PROTECTED] wrote: > >I just sent out 300 messages using qmail-remote in about 4 seconds. > >*sigh*, qmail remote is fast, but you have to pass it information like you > >state below, such as the prefer MX host, which over course a simple perl > >wrapper could find, but well... > > > >I still remember qmail being very fast on other servers. Why isn't my > >concurrentremote filling up? I see in logs an average of 1-3/75 and > >that's it. > > > >-jeremy > > You'd be better off doing a couple other things with your time (rather > then whining here) > > 1.) Look at qmail-inject, and see if that can help you test the machine > for proper delivery statistics (might also look into getting qmail-analog) > > 2.) See if it's possible to use qmail-inject with their java system > >A. If it's not possible, then try convincing the higher ups that the > system needs to be modified by doing the following: > > 1.) Show them how much faster an untuned qmail > installation can be over their "custom" setup. > > 2.) Show them the benefits of the modularity of qmail, and > how easily it can be modified to do what they want. (there are people on > this list that will contract themselves to custom write anything you need, > i'm sure russ can help you out, he's a genius when it comes to qmail > (among other things)) > > > ___ _ __ _ > __ /___ ___ /__ John Gonzalez/Net.Tech > __ __ \ __ \ __/_ __ `__ \/ __ /_ ___/ MDC Computers/netMDC! > _ / / / `__/ /_ / / / / / / /_/ / / /__ (505)437-7600/fax-437-3052 > /_/ /_/\___/\__/ /_/ /_/ /_/\__,_/ \___/ http://www.netmdc.com > [-[system info]---] > 1:50pm up 158 days, 20:53, 2 users, load average: 1.09, 0.35, 0.17 > http://www.xxedgexx.com | [EMAIL PROTECTED] - Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
On Tue, 13 Jul 1999 [EMAIL PROTECTED] wrote: >I just sent out 300 messages using qmail-remote in about 4 seconds. >*sigh*, qmail remote is fast, but you have to pass it information like you >state below, such as the prefer MX host, which over course a simple perl >wrapper could find, but well... > >I still remember qmail being very fast on other servers. Why isn't my >concurrentremote filling up? I see in logs an average of 1-3/75 and >that's it. > >-jeremy You'd be better off doing a couple other things with your time (rather then whining here) 1.) Look at qmail-inject, and see if that can help you test the machine for proper delivery statistics (might also look into getting qmail-analog) 2.) See if it's possible to use qmail-inject with their java system A. If it's not possible, then try convincing the higher ups that the system needs to be modified by doing the following: 1.) Show them how much faster an untuned qmail installation can be over their "custom" setup. 2.) Show them the benefits of the modularity of qmail, and how easily it can be modified to do what they want. (there are people on this list that will contract themselves to custom write anything you need, i'm sure russ can help you out, he's a genius when it comes to qmail (among other things)) ___ _ __ _ __ /___ ___ /__ John Gonzalez/Net.Tech __ __ \ __ \ __/_ __ `__ \/ __ /_ ___/ MDC Computers/netMDC! _ / / / `__/ /_ / / / / / / /_/ / / /__ (505)437-7600/fax-437-3052 /_/ /_/\___/\__/ /_/ /_/ /_/\__,_/ \___/ http://www.netmdc.com [-[system info]---] 1:50pm up 158 days, 20:53, 2 users, load average: 1.09, 0.35, 0.17
Re: Trying to achieve maximum speed!
>I just got word that their program does not shell out to spawn mail >processes, but rather communicates directly with smtp, so unless they have >some control over this, it doesn't look like I'll be able to call anything >directly. > >Heh, the load average on this thing is averaging around 20 which is just >bad. > >I just sent out 300 messages using qmail-remote in about 4 seconds. >*sigh*, qmail remote is fast, but you have to pass it information like you >state below, such as the prefer MX host, which over course a simple perl >wrapper could find, but well... > >I still remember qmail being very fast on other servers. Why isn't my >concurrentremote filling up? I see in logs an average of 1-3/75 and >that's it. It may be that your Java program is simply injecting one mail at a time (via SMTP) and that is getting delivered before the Java prog is ready to inject the next. It may also be that you're hitting an inbound SMTP concurrency limit. How is your inbound SMTP connection managed? inetd, tcpserver, ? To find out more, run qmail-qstat every 10 seconds while the Java prog is running. Post 2-3 minutes of this to the list. Regards.
Re: Trying to achieve maximum speed!
The system is a Linux box, PII 450, 128meg o ram. It's doing nothing but this mail list, but the mail list is using Java, which makes the system crap as far as I'm concerned, but hey, that's out of my control, so I have to deal with it. I just got word that their program does not shell out to spawn mail processes, but rather communicates directly with smtp, so unless they have some control over this, it doesn't look like I'll be able to call anything directly. Heh, the load average on this thing is averaging around 20 which is just bad. I just sent out 300 messages using qmail-remote in about 4 seconds. *sigh*, qmail remote is fast, but you have to pass it information like you state below, such as the prefer MX host, which over course a simple perl wrapper could find, but well... I still remember qmail being very fast on other servers. Why isn't my concurrentremote filling up? I see in logs an average of 1-3/75 and that's it. -jeremy > At 02:47 PM Tuesday 7/13/99, [EMAIL PROTECTED] wrote: > > > >I'm trying to tune qmail to deliver outgoing mail as fast as possible. We > >have a mail list with about 100,000 subscribers. I'd use ezmlm, but > > Over what time period and what sort of hardware? > > >Qmail, just from tailing logs doesn't appear to be spawning more then > >about one delivery per second. I have concurrentremote boosted up to 75. > >What else do I need to do. It doesn't appear to be working to its > >capacity. I've seen qmail do unbelieve amounts of work on other systems. > > Right. You haven't provided any information about resource utilization on > your system. How busy are the disks? What else is happening? What does > qmail-qstat say? > > What sort of concurrency do you get after all the mails have been injected > into the queue. How long does the injection take? > > Btw. qmail-remote is not a direct replacement for any of the mail injection > programs (qmail-inject, sendmail, qmail-queue) and if you wish to use > qmail-remote directly you will need to provide all the queue and retry > mechanisms that qmail provides. > > > Regards. > http://www.xxedgexx.com | [EMAIL PROTECTED] - Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
At 03:09 PM Tuesday 7/13/99, [EMAIL PROTECTED] wrote: > >Hmm, so if they can possible alter their command to call qmail-remote, >instead of /usr/sbin/sendmail, this should increase speed? Nope and maybe. But only if your own queue and retry mechanism is more efficient than that provided by qmail-send. Oh, you don't have your own queue and retry mechanism? Then qmail-remote is irrelevant to you unless you are happy to discard mail that doesn't get thru first time. I sgguest that you stick with qmail-inject and investigage your resource usage. To give you a ballpark. Something like a P-II with 100MB of memory, a single SCSI disk that is well connected should be able to deliver 100K average sized messages in 3-4 hours. If your goal is in that ballpark or less, then it's a matter of homework and perhaps a better understanding of the best way to inject mail. Regards.
Re: Trying to achieve maximum speed!
At 02:47 PM Tuesday 7/13/99, [EMAIL PROTECTED] wrote: > >I'm trying to tune qmail to deliver outgoing mail as fast as possible. We >have a mail list with about 100,000 subscribers. I'd use ezmlm, but Over what time period and what sort of hardware? >Qmail, just from tailing logs doesn't appear to be spawning more then >about one delivery per second. I have concurrentremote boosted up to 75. >What else do I need to do. It doesn't appear to be working to its >capacity. I've seen qmail do unbelieve amounts of work on other systems. Right. You haven't provided any information about resource utilization on your system. How busy are the disks? What else is happening? What does qmail-qstat say? What sort of concurrency do you get after all the mails have been injected into the queue. How long does the injection take? Btw. qmail-remote is not a direct replacement for any of the mail injection programs (qmail-inject, sendmail, qmail-queue) and if you wish to use qmail-remote directly you will need to provide all the queue and retry mechanisms that qmail provides. Regards.
Re: Trying to achieve maximum speed!
Hmm, so if they can possible alter their command to call qmail-remote, instead of /usr/sbin/sendmail, this should increase speed? Thanks -jeremy > The main trick is to use qmail-remote directly. Only queue things when > delivery is not possible. > > Dirk > > On Tue, Jul 13, 1999 at 02:47:16PM -0400, [EMAIL PROTECTED] wrote: > > > > I'm trying to tune qmail to deliver outgoing mail as fast as possible. We > > have a mail list with about 100,000 subscribers. I'd use ezmlm, but > > unfortunately this is impossible due to prioritary stuff built into the > > mail list to generate reports, update a database, etc, etc. So the people > > before me implemented this Java mail delivery client thing using some > > common Java mail interface thing. I don't know much about it, but it's > > really not important. > > > > Qmail, just from tailing logs doesn't appear to be spawning more then > > about one delivery per second. I have concurrentremote boosted up to 75. > > What else do I need to do. It doesn't appear to be working to its > > capacity. I've seen qmail do unbelieve amounts of work on other systems. > > > > Any suggestion or pointers to something I may be missing would be great. > > > > Thanks! > > -jeremy > > > http://www.xxedgexx.com | [EMAIL PROTECTED] - Y2K. We're all gonna die.
Re: Trying to achieve maximum speed!
The main trick is to use qmail-remote directly. Only queue things when delivery is not possible. Dirk On Tue, Jul 13, 1999 at 02:47:16PM -0400, [EMAIL PROTECTED] wrote: > > I'm trying to tune qmail to deliver outgoing mail as fast as possible. We > have a mail list with about 100,000 subscribers. I'd use ezmlm, but > unfortunately this is impossible due to prioritary stuff built into the > mail list to generate reports, update a database, etc, etc. So the people > before me implemented this Java mail delivery client thing using some > common Java mail interface thing. I don't know much about it, but it's > really not important. > > Qmail, just from tailing logs doesn't appear to be spawning more then > about one delivery per second. I have concurrentremote boosted up to 75. > What else do I need to do. It doesn't appear to be working to its > capacity. I've seen qmail do unbelieve amounts of work on other systems. > > Any suggestion or pointers to something I may be missing would be great. > > Thanks! > -jeremy >
Trying to achieve maximum speed!
I'm trying to tune qmail to deliver outgoing mail as fast as possible. We have a mail list with about 100,000 subscribers. I'd use ezmlm, but unfortunately this is impossible due to prioritary stuff built into the mail list to generate reports, update a database, etc, etc. So the people before me implemented this Java mail delivery client thing using some common Java mail interface thing. I don't know much about it, but it's really not important. Qmail, just from tailing logs doesn't appear to be spawning more then about one delivery per second. I have concurrentremote boosted up to 75. What else do I need to do. It doesn't appear to be working to its capacity. I've seen qmail do unbelieve amounts of work on other systems. Any suggestion or pointers to something I may be missing would be great. Thanks! -jeremy
