Matt,
Excellent research. Very well put together. Thanks for the time and the links.
Keith
_____
From: Message Sniffer Community on behalf of Matt
Sent: Sun 6/10/2007 6:29 PM
To: Message Sniffer Community
Subject: [sniffer] Re: Error Messages since WeightGate
Here's a better page from someone at Microsoft all about the desktop heap.
This one suggests that you can change the limit from 48 MB to a value as much
as 450 MB. You will probably normally not need more than the total number of
processes that Declude can use times the amount of memory allocated per
session, so if you have 512 MB/session, and have 100 processes defined in
Declude, you would need about 50 MB, but adding something like weightgate to an
app that has latency could very well increase the needs even more.
http://blogs.msdn.com/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx
Matt
Matt wrote:
Keith,
When I looked at this several years ago, this is what I came up with:
Windows allows a total of 48 MB in the heap, and each service started process
uses the third setting in the chain, or 512 KB by default, and there is about
10 MB that gets used for other things. Based on what Scott Perry wrote
concerning this in a obscure page on the Declude site about Declude Queue,
there can only be a total of 77 service started processes before having issues,
and you can assume that there will be one for Declude up to my limit of 40, and
also often times another process whether it is a virus scanner or external
filter application in JunkMail.
Windows apparently starts to barf when the limit is reached, and applications
can go into a bad state, only partially launching and becoming corrupted. This
has a high association with load, but the true association seems to be the
number of processes, which typically correspond to load but not necessarily.
This is probably also what has caused McAfee to barf on occasion on my server
with similar errors. McAfee has a decent amount of latency compared to most
other things that Declude launches except of course for Eradispam due to the
timeout issues.
There are two camps on what to do with the mystery heap, aka desktop heap.
Some have indicated on the IMail and Declude lists in the past that setting it
to 2048 would resolve some issues with IMail's SMTP and also the 16 bit version
of F-Prot when run from Declude which is awfully slow and CPU intensive. That
change however would reduce the number of service started processes that were
possible by a factor of 4. Scott suggests that reducing it to maybe 256 would
help in high traffic servers, though this is a limit that you wouldn't want to
pass because it could cause instability.
FYI, the error messages will contribute to heap usage, so these must be
cleared, and when you have a bunch of these, it will limit what you can run,
and in fact make the problem worse.
If you are using Declude as a service, that certainly takes one process off the
top that used to count towards the heap, but it's likely what is running
concurrently that is causing the issue along with error dialogs. Weightgate
certainly adds to this issue, as well as other plugins and virus scanners. The
best solution for a high volume server that wants to do weight skipping would
be for either Sniffer or Declude to skip based on both a high and a low weight
within the config. I have been asking for over three years for this and have
even recently documented a solution for Declude that would be backwards
compatible with current configs should they opt to do this.
Here's a quote from the old Declude site authored by Scott:
Flaw #1 - Server crashing: Microsoft's Mystery Heap
Fortunately, not many people experience this problem. However, it is listed
first because it is more serious than the other flaw. This one can back up mail
for hours/days, and crash the server.
The problem here is that each process that is started by a service uses a
certain (unknown) amount of an undocumented type of memory that Windows
allocates. Without knowing how much of the mystery heap is used, or how much is
left, or how much is available when the system starts, it's impossible to know
when you will run out.
When you DO run out, Windows does a *terrible* job in handling it. Instead of
preventing the program from loading and recording an error to the event log,
Windows will keep the program half-loaded (the error almost always occurs while
loading .DLLs) and pop up an error message saying that it can't start the
program.
When this happens, unless you happen to be at the server, you won't have a
chance to close the box. So, another one will soon pop up as another SMTP
process is started. By the time you find out, there could be hundreds or
thousands of the pop-up boxes. Since Microsoft doesn't clear them
automatically, when the original 30 SMTP processes end, there still isn't
enough of this mystery heap left, because Microsoft is using it to display
these error messages. So until you click "OK" to all the hundreds of pop-up
boxes, or you reboot the computer, new mail will not be delivered. Eventually,
the server may crash completely.
Additional Help
Most versions of Windows NT/2000 will apparently by default allocate 512KB of
the "Mystery Heap" to each service-started process. There is also apparently a
total of 48MB of the "Mystery Heap" available. That means you can have a
maximum of about 77 service-started processes (48Megs minus (3Meg * 3 default
desktops) minus (1Meg system-wide) divided by 512). Changing it to 256KB should
approximately double the amount of service-started processes that can run
before the mystery heap is depleted. However, some people have reported better
results by raising the value to 2048KB -- that's one of the problems with
undocumented resources (there's no way to know for sure which value is better
or why).
We recommend going to
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676 and changing
the registry entry to use a value of "256" or "2048" (NOTE: Microsoft
recommends 512 in that article; if you use 512, make sure not to have IMail's
MaxQueProc registry entry set to more than 30).
Matt
Keith Johnson wrote:
Darrell,
Did you alter your heap size 3rd entry? If so, did you go to 1024 or other. I
found this article by crossing a Declude page, appears to be what I need to go
after.
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676
-Keith
_____
From: Message Sniffer Community on behalf of Darrell ([EMAIL
PROTECTED]<mailto:[EMAIL PROTECTED]>)
Sent: Sun 6/10/2007 2:31 PM
To: Message Sniffer Community
Subject: [sniffer] Re: Error Messages since WeightGate
After looking into it I am on board with what Pete said about the "heap"
issue. It makes sense to me that its the heap issue since were
launching weight gate -> SNF. Effectively doubling the amount of
processes being launched.
Darrell
-------------------------------------------
Check out http://www.invariantsystems.com for utilities for Declude,
Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring,
SURBL/URI integration, MRTG Integration, and Log Parsers.
Keith Johnson wrote:
Darrell,
You are right, a reboot will take care of it for a season, then it comes back
out of the blue. Very strange indeed.
Keith
_____
From: Message Sniffer Community on behalf of Darrell ([EMAIL
PROTECTED]<mailto:[EMAIL PROTECTED]>)
Sent: Sat 6/9/2007 9:36 PM
To: Message Sniffer Community
Subject: [sniffer] Re: Error Messages since WeightGate
Keith,
I was having the same problems last week. Just came out of the blue and
was across several of our servers as well. Same error verbatim. FWIW -
I also use weightgate. I rebooted the servers I was seeing this issue
on and the problem has not returned.
Very odd you mentioned that as I thought this was isolated to just me.
Darrell
-------------------------------------------
Check out http://www.invariantsystems.com for utilities for Declude,
Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring,
SURBL/URI integration, MRTG Integration, and Log Parsers.
Keith Johnson wrote:
It appears since installing WeightGate we have been receiving a lot of the
below Application PopUps indicating an error:
The application failed to initialize properly 0xc0000142. Click on OK to
terminate the application
The application entry is our Sniffer .exe. Today alone I saw over 300. I
thought it was an isolated issue. However, it is happening across all our
servers. We are running the latest Sniffer in Persistent mode. We never saw
these prior to WeightGate. Has anyone seen this before? Below is the actual
entry in Event Log.
-Keith
Event Type: Information
Event Source: Application Popup
Event Category: None
Event ID: 26
Date: 6/9/2007
Time: 12:12:35 AM
User: N/A
Computer: NAIMAIL2
Description:
Application popup: rrctp2ez.exe - Application Error : The application failed to
initialize properly (0xc0000142). Click on OK to terminate the application.
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[email protected]><mailto:[email protected]>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
--
-------------------------------------------
Check out http://www.invariantsystems.com for utilities for Declude, Imail,
mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI
integration, MRTG Integration, and Log Parsers.
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[email protected]><mailto:[email protected]>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[email protected]><mailto:[email protected]>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[email protected]><mailto:[email protected]>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[email protected]><mailto:[email protected]>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]><mailto:[EMAIL
PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]>
#############################################################
This message is sent to you because you are subscribed to
the mailing list <[email protected]>.
To unsubscribe, E-mail to: <[EMAIL PROTECTED]>
To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]>
To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]>
Send administrative queries to <[EMAIL PROTECTED]>
