Re: [Declude.JunkMail] OT: Issues with Windows 2003 FTP service
Sandy, I have a lot of interest in finding a workaround to the thread issue, and no doubt this has promise. I'm just saying that I'm reserving judgment until it is tested. I will probably test it myself later today since Peter didn't respond yet to your post. If you have some suggestions for what to set these registry values as, please don't hesitate to post them here or on the o.e.support group. Regarding the MS SMTP tarpitting, this was figured out long before issues with that client that turned out to be related to the interaction of IMail and MS SMTP. The MS SMTP tarpitting issue is much more widespread. Certainly it doesn't make sense to delay a 552 response when tarpitting, and either the delay itself, or the fact that this is combined with MS SMTP sending the response out of sequence, produces very understandable issues. When I disabled this tweak, almost all of the issues disappeared immediately. The only remaining issues are ones specifically related to the out of sequence response from MS SMTP, but they are much less frequent than what I was seeing. Matt Sanford Whiteman wrote: I am only passing on what I was told by Peter from VamSoft. There is a whole thread on this where Peter confirmed the affect on FTP (but not IIS) in VamSoft's o.e.support newsgroup, which I see you found after sending this message. Peter is very smart developer, but not a systems guy. You probably noticed our back-and-forth about ORF's 64-bit support. During our off-list continuation, I was struck equally by his lack of facility with certain common OS features (such as COM+) and yet the expertise with which he built ORF for scaleability and performance. I don't expect him to do in-depth tweaking outside of his app (and I'm sure he didn't touch these parameters here). I personally would be surprised to see a fixed (non-tweakable) thread limit, but I can't make a judgment without some form of verification of your suggestion considering the guidance that I was given (I'm not the programmer nor the expert). Tweak the settings and open up a lot of sessions to your server; you will see that the number of worker threads (as opposed to I/O ports) that can be created rises accordingly. Regarding the MS SMTP tarpitting, you have to trust me on that, or at least test it yourself before suggesting that I am wrong here. I'm sure you _are_ seeing aggravated problems with tarpitting enabled. Yet it's clear that you've observed problems with oversize messages _whether or not_ tarpitting is enabled, and you treated that fact insufficiently, it seemed to me. >From what you wrote, it sounds like running the sink that I wrote for the non-tarpitting server would solve both problems. As you may have noticed, it generates a DSN that does not include the original message, just the subject line, the message size, and the size limit. --Sandy Sanford Whiteman, Chief Technologist Broadleaf Systems, a division of Cypress Integrated Systems, Inc. e-mail: [EMAIL PROTECTED] SpamAssassin plugs into Declude! http://www.imprimia.com/products/software/freeutils/SPAMC32/download/release/ Defuse Dictionary Attacks: Turn Exchange or IMail mailboxes into IMail Aliases! http://www.imprimia.com/products/software/freeutils/exchange2aliases/download/release/ http://www.imprimia.com/products/software/freeutils/ldap2aliases/download/release/ --- [This E-mail was scanned for viruses by Declude EVA www.declude.com] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] OT: Issues with Windows 2003 FTP service
t pane, double-click the MaxDSNSize registry value
and enter the size limit (in bytes) that you want in the Value data
field. Messages that are larger than this value that generate an NDR do
not return attachments or full message properties.
Close the registry editor, and then restart the Simple Mail
Transfer Protocol (SMTP) service for the change to take effect.
For more information about the MaxDSNSize registry setting,
see the following Microsoft Knowledge Base articles:
308303, "XCON: Option to Strip Attachments for Messages That
Generate an NDR," (http://go.microsoft.com/fwlink/?linkid=3052kbid=308303)
323484, "XCON: Description of the Multipart/Report Internet Message
Format in Exchange 2000," (http://go.microsoft.com/fwlink/?linkid=3052kbid=323484)
2) One possible registry key that would scale up the number
of threads used by IIS (hey, it seems like a good entry):
The Microsoft Exchange Server
Best Practices Analyzer Tool reads the following registry entries to
determine whether the maximum number of Internet Information Services
(IIS) pool threads has been modified from the default value:
HKEY_LOCAL_MACHINE\
System\CurrentControlSet\Services\Inetinfo\Parameters\PoolThreadLimit
If the Exchange Server
Best Practices Analyzer finds PoolThreadLimit to be present and
configured with an alternate value, a non-default configuration message
is displayed.
The PoolThreadLimit
registry value specifies the maximum number of input/output (I/O)
worker threads that can be created in the Inetinfo.exe process, which
in turn controls the maximum number of simultaneous connections that
can be made to IIS. Each pool thread watches for a network request and
processes it, either by sending back a static file or by passing the
request to an ISAPI extension DLL (such as ASP.DLL) or to a Common
Gateway Interface (CGI) application. If the ISAPI extension processes a
request synchronously and it takes a long time to process requests, it
will tie up the worker thread, leaving IIS with fewer worker threads to
process other requests. For this reason, well-written ISAPI extensions,
such as ASP, implement their own thread pools, place requests in a
queue, and process them asynchronously with their own threads, so as
not to tie up IIS worker threads.
By default, IIS sets PoolThreadLimit
to:
2 * number of
megabytes of RAM present in the machine
If this value is
larger than 256, it will be clamped to 256. By default, this registry
value does not exist. If this value is present in the registry, it
overrides IIS's default calculation.
Generally, if you find
that the default limit of 256 threads is inadequate, you probably have
a poorly written ISAPI extension tying up IIS worker threads. 256 is a
lot of threads to have active simultaneously and will incur significant
overhead in synchronization and context switching.
The PoolThreadLimit
registry value is a hard limit that includes all IIS worker threads,
including the HTTP, FTP, NNTP, and SMTP services.
Andrew 8)
From:
[EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]] On Behalf Of Matt
Sent: Tuesday, January 10, 2006 4:45 PM
To: Declude.JunkMail@declude.com
Subject: Re: [Declude.JunkMail] OT: Issues with Windows 2003
FTP service
Sandy, you have such a delicate way with words...
I am only passing on what I was told by Peter from VamSoft. There is a
whole thread on this where Peter confirmed the affect on FTP (but not
IIS) in VamSoft's o.e.support newsgroup, which I see you found after
sending this message.
I personally would be surprised to see a fixed (non-tweakable) thread
limit, but I can't make a judgment without some form of verification of
your suggestion considering the guidance that I was given (I'm not the
programmer nor the expert). This server averages only about 2% CPU
utilization, so it would be a shame to have it top out so prematurely.
I did previously search for "CDO_OnArrival thread", but came up with
nothing at all that was useful. Looks like I should have searched for
"SMTPSVC threads" instead since I did come across an article mentioning
the registry parameters that you mentioned. I'll wait for Peter to
reply to your suggestion.
Regarding the MS SMTP tarpitting, you have to trust me on that, or at
least test it yourself before suggesting that I am wrong here. This
was a different issue than that other server had. The issue with that
server was that MS SMTP returns 552 errors in the middle of the DATA,
and while many servers support that, IMail doesn't. I have since
convinced people at Ipswitch that this needs to be changed even if it
isn't technically RFC compliant due to real-world conditions, and the
fact that Microsoft's implementation makes more sense and should be
easy enough to support. This hasn't apparently been worked into the
software yet because they were in the final stages of testing IMail
2006, and this isn't a
Re: [Declude.JunkMail] OT: Issues with Windows 2003 FTP service
Sandy, you have such a delicate way with words... I am only passing on what I was told by Peter from VamSoft. There is a whole thread on this where Peter confirmed the affect on FTP (but not IIS) in VamSoft's o.e.support newsgroup, which I see you found after sending this message. I personally would be surprised to see a fixed (non-tweakable) thread limit, but I can't make a judgment without some form of verification of your suggestion considering the guidance that I was given (I'm not the programmer nor the expert). This server averages only about 2% CPU utilization, so it would be a shame to have it top out so prematurely. I did previously search for "CDO_OnArrival thread", but came up with nothing at all that was useful. Looks like I should have searched for "SMTPSVC threads" instead since I did come across an article mentioning the registry parameters that you mentioned. I'll wait for Peter to reply to your suggestion. Regarding the MS SMTP tarpitting, you have to trust me on that, or at least test it yourself before suggesting that I am wrong here. This was a different issue than that other server had. The issue with that server was that MS SMTP returns 552 errors in the middle of the DATA, and while many servers support that, IMail doesn't. I have since convinced people at Ipswitch that this needs to be changed even if it isn't technically RFC compliant due to real-world conditions, and the fact that Microsoft's implementation makes more sense and should be easy enough to support. This hasn't apparently been worked into the software yet because they were in the final stages of testing IMail 2006, and this isn't a quick fix. I expect to see it happen soon though, hopefully right after they get the major issues worked out with the 2006 release. The MS SMTP tarpitting issue is one that I came across under other circumstances. When I had this turned on (not the ORF version, but the registry hack), I experienced the condition repeatedly from all sorts of servers where I was the recipient. After extensive research and testing, I found that it was this registry hack that was causing the majority of the servers to never recognize the 552 error code, and since all this did was delay the response, one has to assume that the delay of the 552 error was responsible. With this turned off (the default), I am still experiencing some issues with requeued incoming oversized E-mail, but they are much lower in number and related to the sending of 552 errors in the middle of the DATA command. Microsoft shouldn't be haphazardly delaying all 5xx errors, but instead should just be delaying the ones related to invalid recipients. This was a bad oversight on their part, and I am 100% positive about the cause. It may be almost unnoticeable to those that are running this in front of less E-mail addresses and don't actively monitor their bandwidth like I do. Due to my volume, this issue became so large that it was eating up many times my normal incoming bandwidth on a regular basis. Note that I tried to work around both MS SMTP 552 issues on incoming E-mail by setting MS SMTP to accept messages of unlimited size and then configuring IMail to signal the error, but MS SMTP hears the 552 error that IMail generates and then tries to bounce the entire oversized message through the configured Smart Host, which was the same IMail server that wouldn't accept the message in the first place. Why you would want to send a 50 MB bounce message is beyond me. I could kludge something together to bypass the problem, that's not the issue though. My point is that I have been through this stuff with a fine-toothed comb. Matt Sanford Whiteman wrote: The exact source of the problem was a limit in threads in MS SMTP, which is something like 20 per processor (not sure if hyperthreading counts in this case). . . This isn't an accurate claim. It sounds like you just haven't tried to tune MS SMTP appropriately for load/resources/growth. That's fine, since MS doesn't push you too hard on this, but please do some more lab work before deciding that MS SMTP is broken. Look into the following values: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\InetInfo\Parameters\PoolThreadLimit HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\SMTPSVC\Queuing\MaxPercentPoolThreads HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\SMTPSVC\Queuing\AdditionalPoolThreadsPerProc You also need to realize the differences between I/O completion port architecture, callbacks, and pure multithreading. It's possible to bottleneck anything, but that's an implementation problem with a given event sink, not a problem with the overall framework. . . . this limitation in the MS SMTP architecture makes it inappropriate for any full scale spam blocking solution such as Declude unless you have that application ride behind MS SMTP instead of being plugged into it. . . . Sorry, man, but
RE: [Declude.JunkMail] OT: Issues with Windows 2003 FTP service
System\CurrentControlSet\Services\Inetinfo\Parameters\PoolThreadLimit If the Exchange Server Best Practices Analyzer finds PoolThreadLimit to be present and configured with an alternate value, a non-default configuration message is displayed. The PoolThreadLimit registry value specifies the maximum number of input/output (I/O) worker threads that can be created in the Inetinfo.exe process, which in turn controls the maximum number of simultaneous connections that can be made to IIS. Each pool thread watches for a network request and processes it, either by sending back a static file or by passing the request to an ISAPI extension DLL (such as ASP.DLL) or to a Common Gateway Interface (CGI) application. If the ISAPI extension processes a request synchronously and it takes a long time to process requests, it will tie up the worker thread, leaving IIS with fewer worker threads to process other requests. For this reason, well-written ISAPI extensions, such as ASP, implement their own thread pools, place requests in a queue, and process them asynchronously with their own threads, so as not to tie up IIS worker threads. By default, IIS sets PoolThreadLimit to: 2 * number of megabytes of RAM present in the machine If this value is larger than 256, it will be clamped to 256. By default, this registry value does not exist. If this value is present in the registry, it overrides IIS's default calculation. Generally, if you find that the default limit of 256 threads is inadequate, you probably have a poorly written ISAPI extension tying up IIS worker threads. 256 is a lot of threads to have active simultaneously and will incur significant overhead in synchronization and context switching. The PoolThreadLimit registry value is a hard limit that includes all IIS worker threads, including the HTTP, FTP, NNTP, and SMTP services. Andrew 8) From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of MattSent: Tuesday, January 10, 2006 4:45 PMTo: Declude.JunkMail@declude.comSubject: Re: [Declude.JunkMail] OT: Issues with Windows 2003 FTP service Sandy, you have such a delicate way with words...I am only passing on what I was told by Peter from VamSoft. There is a whole thread on this where Peter confirmed the affect on FTP (but not IIS) in VamSoft's o.e.support newsgroup, which I see you found after sending this message.I personally would be surprised to see a fixed (non-tweakable) thread limit, but I can't make a judgment without some form of verification of your suggestion considering the guidance that I was given (I'm not the programmer nor the expert). This server averages only about 2% CPU utilization, so it would be a shame to have it top out so prematurely. I did previously search for "CDO_OnArrival thread", but came up with nothing at all that was useful. Looks like I should have searched for "SMTPSVC threads" instead since I did come across an article mentioning the registry parameters that you mentioned. I'll wait for Peter to reply to your suggestion.Regarding the MS SMTP tarpitting, you have to trust me on that, or at least test it yourself before suggesting that I am wrong here. This was a different issue than that other server had. The issue with that server was that MS SMTP returns 552 errors in the middle of the DATA, and while many servers support that, IMail doesn't. I have since convinced people at Ipswitch that this needs to be changed even if it isn't technically RFC compliant due to real-world conditions, and the fact that Microsoft's implementation makes more sense and should be easy enough to support. This hasn't apparently been worked into the software yet because they were in the final stages of testing IMail 2006, and this isn't a quick fix. I expect to see it happen soon though, hopefully right after they get the major issues worked out with the 2006 release.The MS SMTP tarpitting issue is one that I came across under other circumstances. When I had this turned on (not the ORF version, but the registry hack), I experienced the condition repeatedly from all sorts of servers where I was the recipient. After extensive research and testing, I found that it was this registry hack that was causing the majority of the servers to never recognize the 552 error code, and since all this did was delay the response, one has to assume that the delay of the 552 error was responsible. With this turned off (the default), I am still experiencing some issues with requeued incoming oversized E-mail, but they are much lower in number and related to the sending of 552 errors in the middle of the DATA command. Microsoft shouldn't be haphazardly delaying all 5xx errors, but instead should just be delaying the ones related to invalid recipients. This was a bad oversight on their part, and I am 100% positive about the cause. It may be almost
Re: [Declude.JunkMail] OT: Issues with Windows 2003 FTP service
Good morning all, I figured that I might save those that might respond some time...I found and fixed the issue. Turns out that the MS SMTP part of the metabase was still corrupt in some way...not sure exactly how...and this was causing FTP of all things to behave very, very slowly (while MS SMTP was operating normally). After a lot of playing around with things I figured out that it was the MS SMTP segment of the metabase that when enabled as it was originally would cause FTP to drag, and I also found that stopping the MS SMTP service would cause FTP to return to normal. Why??? Who really knows, but when my metabase was corrupted, it was a corruption in the MS SMTP portion of the file and somehow it is still bad (I'm thinking that my backup copy that I restored had the error that eventually caused the corruption). Thanks, Matt Matt wrote: I'm at wits end with this and I figured that I would put a feeler out here to see if anyone has a clue as to what the source of my issue might be. My MSFTPSVC on one server suddenly has slowed to a crawl, i.e. 15 to 60 seconds from issuing a command to receiving a response. This even happens with the FTP client on the same server going to 127.0.0.1. I have also tested by installing a third-party FTP server on the same box and that worked fine. There is nothing else that is remarkable going on with that server, and I am unsure as to what precipitated the issue, though one possibility is the last MS security rollout that caused my metabase to become corrupted following the reboot back on 12/22. I fixed that with a copy from a backup and all seemed normal. The corrupted metabase showed a block of random characters in the middle of the XML file, and it occurred in the SMTP segment. The current working metabase looks just fine, but I'm thinking that whatever caused the corruption might have also corrupted some other stuff that is affecting FTP. The release notes on those patches didn't suggest anything related to the FTP service or TCP/IP. I have tried many different things from uninstalling and reinstalling the FTP service, removing the last two MS patches (and reinstalling them), and a host of smaller tasks. I have run a rootkit detector and I have real-time virus protection on the server, but that was just to eliminate the very small possibility as the server is well firewalled, completely patched, has only one regular RD user (myself), unnecessary services are disabled, and I even stay away from often exploited software such as Perl and PHP. There is nothing else abnormal on the server that would suggest a bug or otherwise. Curiously this isn't affecting the Web server or SMTP services that are also part of IIS along with FTP. One clue to the problem is that when I reset my router, FTP works at full speed for maybe up to a minute. Although this makes no sense in the purest sense, the same thing happens when using a client on the same box FTPing to 127.0.0.1...the FTP will work at normal speed for a short while when FTPing to 127.0.0.1 immediately following a router reload. I am 99.9% positive that my network has nothing to do with causing the issue, but this one thing suggests that there is some interaction with TCP/IP and the FTP service that is contributing to the issue. This makes me think that it is a bug with the IIS rate limiting which requires QOS to be bound to the NIC, and maybe the router resets are resetting the QOS/rate limiting, allowing it to operate at full speed until it adjusts back to almost no throughput. I have rate limiting turned on for both Web and FTP, but this is only affecting FTP. I have tried turning off QOS and rebooting, but that had no affect on the issue, yet the way that rate limiting works, it seems to explain why a router reload causes things to work well for a few moments before degrading again. At this point my next try will probably be to uninstall and reinstall all of IIS, but I was hoping that maybe someone around here has seen this or a similar issue, or if there were any ideas about the possible interaction with QOS and rate limiting gone bad, and how to reinstall that part of Windows if possible. I would like to avoid rebuilding this box, but I won't keep it running in the present state with an unknown issue even though I could migrate to a third-party FTP server and avoid the issue. I would appreciate any glimmers of hope that anyone might have for me on this :) Thanks, Matt --- [This E-mail was scanned for viruses by Declude EVA www.declude.com] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude EVA www.declude.com] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an
RE: [Declude.JunkMail] OT: Issues with Windows 2003 FTP service
Thanks a lot for the follow up and answer to your own post. It may help us in the future. You are very kind. I am glad you were able to solve the problem. regards Luis Arango -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt Sent: Miércoles, 04 de Enero de 2006 08:37 a.m. To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] OT: Issues with Windows 2003 FTP service Good morning all, I figured that I might save those that might respond some time...I found and fixed the issue. Turns out that the MS SMTP part of the metabase was still corrupt in some way...not sure exactly how...and this was causing FTP of all things to behave very, very slowly (while MS SMTP was operating normally). After a lot of playing around with things I figured out that it was the MS SMTP segment of the metabase that when enabled as it was originally would cause FTP to drag, and I also found that stopping the MS SMTP service would cause FTP to return to normal. Why??? Who really knows, but when my metabase was corrupted, it was a corruption in the MS SMTP portion of the file and somehow it is still bad (I'm thinking that my backup copy that I restored had the error that eventually caused the corruption). Thanks, Matt Matt wrote: I'm at wits end with this and I figured that I would put a feeler out here to see if anyone has a clue as to what the source of my issue might be. My MSFTPSVC on one server suddenly has slowed to a crawl, i.e. 15 to 60 seconds from issuing a command to receiving a response. This even happens with the FTP client on the same server going to 127.0.0.1. I have also tested by installing a third-party FTP server on the same box and that worked fine. There is nothing else that is remarkable going on with that server, and I am unsure as to what precipitated the issue, though one possibility is the last MS security rollout that caused my metabase to become corrupted following the reboot back on 12/22. I fixed that with a copy from a backup and all seemed normal. The corrupted metabase showed a block of random characters in the middle of the XML file, and it occurred in the SMTP segment. The current working metabase looks just fine, but I'm thinking that whatever caused the corruption might have also corrupted some other stuff that is affecting FTP. The release notes on those patches didn't suggest anything related to the FTP service or TCP/IP. I have tried many different things from uninstalling and reinstalling the FTP service, removing the last two MS patches (and reinstalling them), and a host of smaller tasks. I have run a rootkit detector and I have real-time virus protection on the server, but that was just to eliminate the very small possibility as the server is well firewalled, completely patched, has only one regular RD user (myself), unnecessary services are disabled, and I even stay away from often exploited software such as Perl and PHP. There is nothing else abnormal on the server that would suggest a bug or otherwise. Curiously this isn't affecting the Web server or SMTP services that are also part of IIS along with FTP. One clue to the problem is that when I reset my router, FTP works at full speed for maybe up to a minute. Although this makes no sense in the purest sense, the same thing happens when using a client on the same box FTPing to 127.0.0.1...the FTP will work at normal speed for a short while when FTPing to 127.0.0.1 immediately following a router reload. I am 99.9% positive that my network has nothing to do with causing the issue, but this one thing suggests that there is some interaction with TCP/IP and the FTP service that is contributing to the issue. This makes me think that it is a bug with the IIS rate limiting which requires QOS to be bound to the NIC, and maybe the router resets are resetting the QOS/rate limiting, allowing it to operate at full speed until it adjusts back to almost no throughput. I have rate limiting turned on for both Web and FTP, but this is only affecting FTP. I have tried turning off QOS and rebooting, but that had no affect on the issue, yet the way that rate limiting works, it seems to explain why a router reload causes things to work well for a few moments before degrading again. At this point my next try will probably be to uninstall and reinstall all of IIS, but I was hoping that maybe someone around here has seen this or a similar issue, or if there were any ideas about the possible interaction with QOS and rate limiting gone bad, and how to reinstall that part of Windows if possible. I would like to avoid rebuilding this box, but I won't keep it running in the present state with an unknown issue even though I could
