Re: Pool consumption (was: Re: Virtualized server issue...)

2009-04-01 Thread Kurt Buff 
This issue is not related to the virtualizing of the machine. It seems
to have started when that happened, but it also is now happening when
we reverted to the physical machine, which had not been wiped.

P2V is irrelevant in *this* case, anyway.

On Wed, Apr 1, 2009 at 11:21, mse...@ont.com  wrote:
> You virtualized (P2v'd) your server. Sometimes it does not always work
> correctly or you can encounter problems. If the problem is with your VM you
> can do the following.
>
> 1. Delete problem VM. Rerun P2V if you have not already destroyed original
> server.
> 2. Check settings such as HAL to make sure it is correct. Also make sure
> people are not on server when performing P2V since open files can cause
> problems.
> 3. Set new VM with proper settings such as vCPU's, Memory, VMware Tools,
> etc.
>
> We have seen P2V's have problems. No harm in doing again. Just shut down
> original server after P2V.
>
> Original Message:
> -
> From: Brian Desmond br...@briandesmond.com
> Date: Wed, 1 Apr 2009 17:41:45 +
> To: ntsysadmin@lyris.sunbelt-software.com
> Subject: RE: Pool consumption (was: Re: Virtualized server issue...)
>
>
> Explain?
>
> Thanks,
> Brian Desmond
> br...@briandesmond.com
>
> c - 312.731.3132
>
> -Original Message-
> From: Mike Semon [mailto:mse...@ont.com]
> Sent: Wednesday, April 01, 2009 6:52 AM
> To: NT System Admin Issues
> Subject: RE: Pool consumption (was: Re: Virtualized server issue...)
>
> You virtualized or P2v'd your box. If not done correctly can create errors
> you are seeing.
>
> -Original Message-
> From: Brian Desmond [mailto:br...@briandesmond.com]
> Sent: Tuesday, March 31, 2009 4:57 PM
> To: NT System Admin Issues
> Subject: RE: Pool consumption (was: Re: Virtualized server issue...)
>
> What does that have to do with this?
>
> Thanks,
> Brian Desmond
> br...@briandesmond.com
>
> c - 312.731.3132
>
>
> -Original Message-
> From: mse...@ont.com [mailto:mse...@ont.com]
> Sent: Tuesday, March 31, 2009 4:56 PM
> To: NT System Admin Issues
> Subject: RE: Pool consumption (was: Re: Virtualized server issue...)
>
> Start with 1 vCPU then check your CPU and memory on your VM. Having
> multiple cpu's can actually hurt performance of a VM unless it truely needs
> multiple CPU's. We start with 1 vCPU on all of our VM's. Also check your
> HAL if you are trying to run a uniprocessor HAL on multiprocessor machine
>
> Mike
>
> Original Message:
> -
> From: Kurt Buff kurt.b...@gmail.com
> Date: Tue, 31 Mar 2009 11:45:59 -0700
> To: ntsysadmin@lyris.sunbelt-software.com
> Subject: Pool consumption (was: Re: Virtualized server issue...)
>
>
> Have added the reg entry, and am examining poolmon output.
>
> The server needed to be rebooted again today, so the reg entry should now be
> active. I note that in perfmon Pool Paged Allocs for this machine had
> slammed to the ceiling, although nothing else seemed to be
>
> Poolmon is proving problematic - I've examined the top two memory eaters,
> and am either getting too many or no hits.
>
> At 10:12 this morning, I took this reading, and a few minutes later had to
> reboot the box.
>
> Memory: 1047456K Avail:  220432K  PageFlts:    47   InRam Krnl: 2912K
> P:281220K
> Commit: 570560K Limit:3053000K Peak: 580612K            Pool N:35804K
> P:282140K
> System pool information
> Tag  Type     Allocs            Frees            Diff   Bytes       Per
> Alloc
>
> Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 (     0)
> 20
> MmSt Paged    755490 (   9)    753741 (   9)     1749 15544008 (     0)
> 8887
>
> Memory: 1047456K Avail:  229772K  PageFlts:    66   InRam Krnl: 2912K
> P:281000K
> Commit: 570396K Limit:3053000K Peak: 580612K            Pool N:35776K
> P:281920K
> System pool information
> Tag  Type     Allocs            Frees            Diff   Bytes       Per
> Alloc
>
> MmCm Nonp       1090 (   0)        15 (   0)     1075 7864864 (     0)
> 7316
> LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0)
> 2576384
>
> As the winner and new champeen, Se is at least an order of magnitude larger
> than its closes competitor. However, the Se tag is in lots of files, so I
> prefixed it in the findstr command with 'h' per the KB article. That
> narrowed it considerably, but what's left pretty much looks to be MSFT
> files, and there are still 21 files.
>
> MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
> srv.sys - I note that the Se tag is also found in the latter.
>
> Later today, perhaps after standard business

RE: Pool consumption (was: Re: Virtualized server issue...)

2009-04-01 Thread mse...@ont.com 
You virtualized (P2v'd) your server. Sometimes it does not always work
correctly or you can encounter problems. If the problem is with your VM you
can do the following.

1. Delete problem VM. Rerun P2V if you have not already destroyed original
server.
2. Check settings such as HAL to make sure it is correct. Also make sure
people are not on server when performing P2V since open files can cause
problems.
3. Set new VM with proper settings such as vCPU's, Memory, VMware Tools,
etc.

We have seen P2V's have problems. No harm in doing again. Just shut down
original server after P2V.

Original Message:
-
From: Brian Desmond br...@briandesmond.com
Date: Wed, 1 Apr 2009 17:41:45 +
To: ntsysadmin@lyris.sunbelt-software.com
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)


Explain?

Thanks,
Brian Desmond
br...@briandesmond.com

c - 312.731.3132

-Original Message-
From: Mike Semon [mailto:mse...@ont.com]
Sent: Wednesday, April 01, 2009 6:52 AM
To: NT System Admin Issues
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)

You virtualized or P2v'd your box. If not done correctly can create errors
you are seeing.

-Original Message-
From: Brian Desmond [mailto:br...@briandesmond.com]
Sent: Tuesday, March 31, 2009 4:57 PM
To: NT System Admin Issues
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)

What does that have to do with this?

Thanks,
Brian Desmond
br...@briandesmond.com

c - 312.731.3132


-Original Message-
From: mse...@ont.com [mailto:mse...@ont.com]
Sent: Tuesday, March 31, 2009 4:56 PM
To: NT System Admin Issues
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)

Start with 1 vCPU then check your CPU and memory on your VM. Having
multiple cpu's can actually hurt performance of a VM unless it truely needs
multiple CPU's. We start with 1 vCPU on all of our VM's. Also check your
HAL if you are trying to run a uniprocessor HAL on multiprocessor machine

Mike

Original Message:
-
From: Kurt Buff kurt.b...@gmail.com
Date: Tue, 31 Mar 2009 11:45:59 -0700
To: ntsysadmin@lyris.sunbelt-software.com
Subject: Pool consumption (was: Re: Virtualized server issue...)


Have added the reg entry, and am examining poolmon output.

The server needed to be rebooted again today, so the reg entry should now be
active. I note that in perfmon Pool Paged Allocs for this machine had
slammed to the ceiling, although nothing else seemed to be

Poolmon is proving problematic - I've examined the top two memory eaters,
and am either getting too many or no hits.

At 10:12 this morning, I took this reading, and a few minutes later had to
reboot the box.

Memory: 1047456K Avail:  220432K  PageFlts:47   InRam Krnl: 2912K
P:281220K
Commit: 570560K Limit:3053000K Peak: 580612KPool N:35804K
P:282140K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 ( 0)
20
MmSt Paged755490 (   9)753741 (   9) 1749 15544008 ( 0)
8887

Memory: 1047456K Avail:  229772K  PageFlts:66   InRam Krnl: 2912K
P:281000K
Commit: 570396K Limit:3053000K Peak: 580612KPool N:35776K
P:281920K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

MmCm Nonp   1090 (   0)15 (   0) 1075 7864864 ( 0)
7316
LSwi Nonp  1 (   0) 0 (   0)1 2576384 ( 0)
2576384

As the winner and new champeen, Se is at least an order of magnitude larger
than its closes competitor. However, the Se tag is in lots of files, so I
prefixed it in the findstr command with 'h' per the KB article. That
narrowed it considerably, but what's left pretty much looks to be MSFT
files, and there are still 21 files.

MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
srv.sys - I note that the Se tag is also found in the latter.

Later today, perhaps after standard business hours, I'll reenable Active
Protection for VIPRE on this machine, and copy a lot of files to it. If
that's the issue, it should start spiking pretty quickly.

On Mon, Mar 30, 2009 at 15:29, Brian Desmond  wrote:
> Kurt-
>
> Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll
registry value and reboot? This will allow you to generate a dump next time
this happens (the hang, specifically) by pressing the /right/ Ctrl key and
Scroll Lock twice.
>
> Also, Poolmon can help tremendously here too for logging.
>
> Thanks,
> Brian Desmond
> br...@briandesmond.com
>
> c - 312.731.3132
>
>
> -Original Message-
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Monday, March 30, 2009 5:00 PM
> To: NT System Admin Issues
> Subject: Virtualized server issue...
>
> All,
>
> Over the

RE: Pool consumption (was: Re: Virtualized server issue...)

2009-04-01 Thread Brian Desmond 
Explain?

Thanks,
Brian Desmond
br...@briandesmond.com

c - 312.731.3132

-Original Message-
From: Mike Semon [mailto:mse...@ont.com]
Sent: Wednesday, April 01, 2009 6:52 AM
To: NT System Admin Issues
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)

You virtualized or P2v'd your box. If not done correctly can create errors
you are seeing.

-Original Message-
From: Brian Desmond [mailto:br...@briandesmond.com]
Sent: Tuesday, March 31, 2009 4:57 PM
To: NT System Admin Issues
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)

What does that have to do with this?

Thanks,
Brian Desmond
br...@briandesmond.com

c - 312.731.3132


-Original Message-
From: mse...@ont.com [mailto:mse...@ont.com]
Sent: Tuesday, March 31, 2009 4:56 PM
To: NT System Admin Issues
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)

Start with 1 vCPU then check your CPU and memory on your VM. Having
multiple cpu's can actually hurt performance of a VM unless it truely needs
multiple CPU's. We start with 1 vCPU on all of our VM's. Also check your
HAL if you are trying to run a uniprocessor HAL on multiprocessor machine

Mike

Original Message:
-
From: Kurt Buff kurt.b...@gmail.com
Date: Tue, 31 Mar 2009 11:45:59 -0700
To: ntsysadmin@lyris.sunbelt-software.com
Subject: Pool consumption (was: Re: Virtualized server issue...)


Have added the reg entry, and am examining poolmon output.

The server needed to be rebooted again today, so the reg entry should now be
active. I note that in perfmon Pool Paged Allocs for this machine had
slammed to the ceiling, although nothing else seemed to be

Poolmon is proving problematic - I've examined the top two memory eaters,
and am either getting too many or no hits.

At 10:12 this morning, I took this reading, and a few minutes later had to
reboot the box.

Memory: 1047456K Avail:  220432K  PageFlts:47   InRam Krnl: 2912K
P:281220K
Commit: 570560K Limit:3053000K Peak: 580612KPool N:35804K
P:282140K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 ( 0)
20
MmSt Paged755490 (   9)753741 (   9) 1749 15544008 ( 0)
8887

Memory: 1047456K Avail:  229772K  PageFlts:66   InRam Krnl: 2912K
P:281000K
Commit: 570396K Limit:3053000K Peak: 580612KPool N:35776K
P:281920K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

MmCm Nonp   1090 (   0)15 (   0) 1075 7864864 ( 0)
7316
LSwi Nonp  1 (   0) 0 (   0)1 2576384 ( 0)
2576384

As the winner and new champeen, Se is at least an order of magnitude larger
than its closes competitor. However, the Se tag is in lots of files, so I
prefixed it in the findstr command with 'h' per the KB article. That
narrowed it considerably, but what's left pretty much looks to be MSFT
files, and there are still 21 files.

MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
srv.sys - I note that the Se tag is also found in the latter.

Later today, perhaps after standard business hours, I'll reenable Active
Protection for VIPRE on this machine, and copy a lot of files to it. If
that's the issue, it should start spiking pretty quickly.

On Mon, Mar 30, 2009 at 15:29, Brian Desmond  wrote:
> Kurt-
>
> Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll
registry value and reboot? This will allow you to generate a dump next time
this happens (the hang, specifically) by pressing the /right/ Ctrl key and
Scroll Lock twice.
>
> Also, Poolmon can help tremendously here too for logging.
>
> Thanks,
> Brian Desmond
> br...@briandesmond.com
>
> c - 312.731.3132
>
>
> -Original Message-
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Monday, March 30, 2009 5:00 PM
> To: NT System Admin Issues
> Subject: Virtualized server issue...
>
> All,
>
> Over the weekend we virtualized our file/print server, and it seemed
> to go well. Host is a Dell machine running ESX 3.5 update 2.
>
> The physical machine has an Intel HT processor and 1gbyte of RAM. I
> gave the VM 2 procs and 2gbytes of RAM, just for good measure.
>
> Both machines were talking to our LeftHand SAN, on a separate physical
> LAN, but today I had to reboot the VM, then a couple of hours later
> shut it down and revert to the physical machine after it stopped
> responding.
>
> The logs were indicating lack of server memory - specifically, these
> were being emitted to my syslog server:
>
> 2009-03-30 14:05:12 User.Notice home-01 Mar 30 14:05:12
> home-01 MSWinEventLog   1   System  13892   Mon Mar 30 14:05:08 2009
> 2020Srv Unknown UserN/A Err

RE: Pool consumption (was: Re: Virtualized server issue...)

2009-04-01 Thread Mike Semon 
You virtualized or P2v'd your box. If not done correctly can create errors
you are seeing.

-Original Message-
From: Brian Desmond [mailto:br...@briandesmond.com] 
Sent: Tuesday, March 31, 2009 4:57 PM
To: NT System Admin Issues
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)

What does that have to do with this?

Thanks,
Brian Desmond
br...@briandesmond.com

c - 312.731.3132


-Original Message-
From: mse...@ont.com [mailto:mse...@ont.com]
Sent: Tuesday, March 31, 2009 4:56 PM
To: NT System Admin Issues
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)

Start with 1 vCPU then check your CPU and memory on your VM. Having
multiple cpu's can actually hurt performance of a VM unless it truely needs
multiple CPU's. We start with 1 vCPU on all of our VM's. Also check your
HAL if you are trying to run a uniprocessor HAL on multiprocessor machine

Mike

Original Message:
-
From: Kurt Buff kurt.b...@gmail.com
Date: Tue, 31 Mar 2009 11:45:59 -0700
To: ntsysadmin@lyris.sunbelt-software.com
Subject: Pool consumption (was: Re: Virtualized server issue...)


Have added the reg entry, and am examining poolmon output.

The server needed to be rebooted again today, so the reg entry should now be
active. I note that in perfmon Pool Paged Allocs for this machine had
slammed to the ceiling, although nothing else seemed to be

Poolmon is proving problematic - I've examined the top two memory eaters,
and am either getting too many or no hits.

At 10:12 this morning, I took this reading, and a few minutes later had to
reboot the box.

Memory: 1047456K Avail:  220432K  PageFlts:47   InRam Krnl: 2912K
P:281220K
Commit: 570560K Limit:3053000K Peak: 580612KPool N:35804K
P:282140K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 ( 0)
20
MmSt Paged755490 (   9)753741 (   9) 1749 15544008 ( 0)
8887

Memory: 1047456K Avail:  229772K  PageFlts:66   InRam Krnl: 2912K
P:281000K
Commit: 570396K Limit:3053000K Peak: 580612KPool N:35776K
P:281920K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

MmCm Nonp   1090 (   0)15 (   0) 1075 7864864 ( 0)
7316
LSwi Nonp  1 (   0) 0 (   0)1 2576384 ( 0)
2576384

As the winner and new champeen, Se is at least an order of magnitude larger
than its closes competitor. However, the Se tag is in lots of files, so I
prefixed it in the findstr command with 'h' per the KB article. That
narrowed it considerably, but what's left pretty much looks to be MSFT
files, and there are still 21 files.

MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
srv.sys - I note that the Se tag is also found in the latter.

Later today, perhaps after standard business hours, I'll reenable Active
Protection for VIPRE on this machine, and copy a lot of files to it. If
that's the issue, it should start spiking pretty quickly.

On Mon, Mar 30, 2009 at 15:29, Brian Desmond  wrote:
> Kurt-
>
> Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll
registry value and reboot? This will allow you to generate a dump next time
this happens (the hang, specifically) by pressing the /right/ Ctrl key and
Scroll Lock twice.
>
> Also, Poolmon can help tremendously here too for logging.
>
> Thanks,
> Brian Desmond
> br...@briandesmond.com
>
> c - 312.731.3132
>
>
> -Original Message-
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Monday, March 30, 2009 5:00 PM
> To: NT System Admin Issues
> Subject: Virtualized server issue...
>
> All,
>
> Over the weekend we virtualized our file/print server, and it seemed
> to go well. Host is a Dell machine running ESX 3.5 update 2.
>
> The physical machine has an Intel HT processor and 1gbyte of RAM. I
> gave the VM 2 procs and 2gbytes of RAM, just for good measure.
>
> Both machines were talking to our LeftHand SAN, on a separate physical
> LAN, but today I had to reboot the VM, then a couple of hours later
> shut it down and revert to the physical machine after it stopped
> responding.
>
> The logs were indicating lack of server memory - specifically, these
> were being emitted to my syslog server:
>
> 2009-03-30 14:05:12 User.Notice home-01 Mar 30 14:05:12
> home-01 MSWinEventLog   1   System  13892   Mon Mar 30 14:05:08 2009
> 2020Srv Unknown UserN/A Error   HOME-01 None
> : 00 00 04 00 01 00 54 00   ...  0008: 00 00 00 00 e4 07 00 c0
>    0010: 00 00 00 00 9a 00 00 c0     0018: 00 00 00
> 00 00 00 00 00     0020: 00 00 00 00 00 00 00 00   
> 0028: ae 04 00 00 d0 02 70 00

Re: Pool consumption (was: Re: Virtualized server issue...)

2009-03-31 Thread Kurt Buff 
Yup, just now.

On Tue, Mar 31, 2009 at 14:57, Brian Desmond  wrote:
> Just replied offline to you directly - let me know if you get it ?
>
> Thanks,
> Brian Desmond
> br...@briandesmond.com
>
> c - 312.731.3132
>
>
> -Original Message-
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Tuesday, March 31, 2009 4:35 PM
> To: NT System Admin Issues
> Subject: Re: Pool consumption (was: Re: Virtualized server issue...)
>
> BTW - I think I've found the culprit, and will be testing this evening.
>
> I have a job that runs every night to check permissions, using
> fileacl.exe. Normally its report is a few kilobytes. Today's was over
> 10 megabytes - this happens occasionally when a certain department of
> ours needs to rearrange the directories allocated to it, and signals
> that thousands of files have been moved.
>
> I investigated, and found that this person was indeed performing this
> task around the time of the server having difficulties, both yesterday
> and today.
>
> This leads me to believe that it's indeed VIPRE and its Active
> Protection settings - I've disabled that for now, and will be testing
> it this evening by doing a mass copy from one directory to another on
> the file server, both with and without the Active Protection setting
> turned on. This should help pinpoint the issue..
>
> Kurt
>
> On Tue, Mar 31, 2009 at 13:58, Brian Desmond  wrote:
>> Kurt-
>>
>>
>>
>> Is pooltag.txt in that archive I sent you offline?
>>
>>
>>
>> If not, install the Debug Tools for Windows (free download) and go in the
>> triage folder  ┐  т Ь pooltag.txt has a dictionary so to speak of all the MS
>> tags
>>
>>
>>
>>   ┐ ┐  ┐  - nt!  ┐ ┐   ├  В  ├  ├  ┐  ├  ┐  - General security allocations
>>
>>
>>
>> Allocs is  ┐  т вt really terribly interesting, what is is the number of 
>> bytes
>> allocated which in this case is ~232MB. That is not normal. I have no idea
>> what this one is   ├  АЬ i  ┐  т вs not somethi  ├  ┐  ├  Двve seen before 
>> an  ├  ┐  ├  Двs not
>> normal. I would be inclined to tell you to call PSS and let them work on it.
>> I know how to get the stacks for the allocations with a live debug but
>> that  ├  Двs fairly involved as things go.
>>
>>
>>
>> MmSt is associated with memory tracking shared files (in a 
>> nutshel  ┐  ├  ┐  Ь
>> looks ok in your snapshot.
>>
>>
>>
>> Thanks,
>>
>> Brian Desmond
>>
>> br...@briandesmond.com
>>
>>
>>
>> c - 312.731.3132
>>
>>
>>
>> Active Directory, 4th Ed - http://www.briandesmond.com/ad4/
>>
>> Microsoft MVP - https://mvp.support.microsoft.com/profile/Brian
>>
>>
>>
>> From: Kurt Buff [mailto:kurt.b...@gmail.com]
>> Sent: Tuesday, March 31, 2009 1:46 PM
>> To: NT System Admin Issues
>> Subject: Pool consumption (was: Re: Virtualized server issue...)
>>
>>
>>
>> Have added the reg entry, and am examining poolmon output.
>>
>> The server needed to be rebooted again today, so the reg entry should now be
>> active. I note that in perfmon Pool Paged Allocs for this machine had
>> slammed to the ceiling, although nothing else seemed to be
>>
>> Poolmon is proving problematic - I've examined the top two memory eaters,
>> and am either getting too many or no hits.
>>
>> At 10:12 this morning, I took this reading, and a few minutes later had to
>> reboot the box.
>>
>> Memory: 1047456K Avail:  220432K  PageFlts:    47   InRam Krnl: 2912K
>> P:281220K
>> Commit: 570560K Limit:3053000K Peak: 580612K            Pool N:35804K
>> P:282140K
>> System pool information
>> Tag  Type     Allocs            Frees            Diff   Bytes       Per
>> Alloc
>>
>> Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 (     0)
>> 20
>> MmSt Paged    755490 (   9)    753741 (   9)     1749 15544008 (     0)
>> 8887
>>
>> Memory: 1047456K Avail:  229772K  PageFlts:    66   InRam Krnl: 2912K
>> P:281000K
>> Commit: 570396K Limit:3053000K Peak: 580612K            Pool N:35776K
>> P:281920K
>> System pool information
>> Tag  Type     Allocs            Frees            Diff   Bytes       Per
>> Alloc
>>
>> MmCm Nonp       1090 (   0)        15 (   0)     1075 7864864 (     0)
>> 7316
>> LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0)
>> 2576384
>>
>> As the winner and new champeen, Se is at least an order of magnit

Re: Pool consumption (was: Re: Virtualized server issue...)

2009-03-31 Thread Kurt Buff 
We're not running the VM at the moment - we've put it back to the
physical machine, which is still exhibiting the behavior.

On Tue, Mar 31, 2009 at 14:55, mse...@ont.com  wrote:
> Start with 1 vCPU then check your CPU and memory on your VM. Having
> multiple cpu's can actually hurt performance of a VM unless it truely needs
> multiple CPU's. We start with 1 vCPU on all of our VM's. Also check your
> HAL if you are trying to run a uniprocessor HAL on multiprocessor machine
>
> Mike
>
> Original Message:
> -
> From: Kurt Buff kurt.b...@gmail.com
> Date: Tue, 31 Mar 2009 11:45:59 -0700
> To: ntsysadmin@lyris.sunbelt-software.com
> Subject: Pool consumption (was: Re: Virtualized server issue...)
>
>
> Have added the reg entry, and am examining poolmon output.
>
> The server needed to be rebooted again today, so the reg entry should now be
> active. I note that in perfmon Pool Paged Allocs for this machine had
> slammed to the ceiling, although nothing else seemed to be
>
> Poolmon is proving problematic - I've examined the top two memory eaters,
> and am either getting too many or no hits.
>
> At 10:12 this morning, I took this reading, and a few minutes later had to
> reboot the box.
>
> Memory: 1047456K Avail:  220432K  PageFlts:    47   InRam Krnl: 2912K
> P:281220K
> Commit: 570560K Limit:3053000K Peak: 580612K            Pool N:35804K
> P:282140K
> System pool information
> Tag  Type     Allocs            Frees            Diff   Bytes       Per
> Alloc
>
> Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 (     0)
> 20
> MmSt Paged    755490 (   9)    753741 (   9)     1749 15544008 (     0)
> 8887
>
> Memory: 1047456K Avail:  229772K  PageFlts:    66   InRam Krnl: 2912K
> P:281000K
> Commit: 570396K Limit:3053000K Peak: 580612K            Pool N:35776K
> P:281920K
> System pool information
> Tag  Type     Allocs            Frees            Diff   Bytes       Per
> Alloc
>
> MmCm Nonp       1090 (   0)        15 (   0)     1075 7864864 (     0)
> 7316
> LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0)
> 2576384
>
> As the winner and new champeen, Se is at least an order of magnitude larger
> than its closes competitor. However, the Se tag is in lots of files, so I
> prefixed it in the findstr command with 'h' per the KB article. That
> narrowed it considerably, but what's left pretty much looks to be MSFT
> files, and there are still 21 files.
>
> MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
> srv.sys - I note that the Se tag is also found in the latter.
>
> Later today, perhaps after standard business hours, I'll reenable Active
> Protection for VIPRE on this machine, and copy a lot of files to it. If
> that's the issue, it should start spiking pretty quickly.
>
> On Mon, Mar 30, 2009 at 15:29, Brian Desmond  wrote:
>> Kurt-
>>
>> Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll
> registry value and reboot? This will allow you to generate a dump next time
> this happens (the hang, specifically) by pressing the /right/ Ctrl key and
> Scroll Lock twice.
>>
>> Also, Poolmon can help tremendously here too for logging.
>>
>> Thanks,
>> Brian Desmond
>> br...@briandesmond.com
>>
>> c - 312.731.3132
>>
>>
>> -Original Message-
>> From: Kurt Buff [mailto:kurt.b...@gmail.com]
>> Sent: Monday, March 30, 2009 5:00 PM
>> To: NT System Admin Issues
>> Subject: Virtualized server issue...
>>
>> All,
>>
>> Over the weekend we virtualized our file/print server, and it seemed
>> to go well. Host is a Dell machine running ESX 3.5 update 2.
>>
>> The physical machine has an Intel HT processor and 1gbyte of RAM. I
>> gave the VM 2 procs and 2gbytes of RAM, just for good measure.
>>
>> Both machines were talking to our LeftHand SAN, on a separate physical
>> LAN, but today I had to reboot the VM, then a couple of hours later
>> shut it down and revert to the physical machine after it stopped
>> responding.
>>
>> The logs were indicating lack of server memory - specifically, these
>> were being emitted to my syslog server:
>>
>> 2009-03-30 14:05:12     User.Notice     home-01     Mar 30 14:05:12
>> home-01 MSWinEventLog   1   System  13892   Mon Mar 30 14:05:08 2009
>>     2020    Srv     Unknown User    N/A Error   HOME-01     None
>> : 00 00 04 00 01 00 54 00   ...  0008: 00 00 00 00 e4 07 00 c0
>>    0010: 00 00 00 00 9a 00 00 c0     0018: 00 00 00
>> 00 00 00 00 00     0020: 00 00 00 00 00 00 00 00   
>> 0028: ae 04 00 00 d0 02 70 00   ...      The server was unable to
>> allocate from the system paged pool because the pool was empty.
>>
>> Then this, as I tried to log in to shut it down:
>>
>> 2009-03-30 14:09:39     User.Notice     zet-home-01     Mar 30
>> 14:09:39 zet-home-01 MSWinEventLog   1   Application     13935   Mon
>> Mar 30 14:09:39 2009        1512    Userenv SYSTEM  User
>> Error   ZET-HOME-01     None            Windows cannot unload your
>> registry file

RE: Pool consumption (was: Re: Virtualized server issue...)

2009-03-31 Thread Brian Desmond 
Just replied offline to you directly - let me know if you get it ?

Thanks,
Brian Desmond
br...@briandesmond.com

c - 312.731.3132


-Original Message-
From: Kurt Buff [mailto:kurt.b...@gmail.com]
Sent: Tuesday, March 31, 2009 4:35 PM
To: NT System Admin Issues
Subject: Re: Pool consumption (was: Re: Virtualized server issue...)

BTW - I think I've found the culprit, and will be testing this evening.

I have a job that runs every night to check permissions, using
fileacl.exe. Normally its report is a few kilobytes. Today's was over
10 megabytes - this happens occasionally when a certain department of
ours needs to rearrange the directories allocated to it, and signals
that thousands of files have been moved.

I investigated, and found that this person was indeed performing this
task around the time of the server having difficulties, both yesterday
and today.

This leads me to believe that it's indeed VIPRE and its Active
Protection settings - I've disabled that for now, and will be testing
it this evening by doing a mass copy from one directory to another on
the file server, both with and without the Active Protection setting
turned on. This should help pinpoint the issue..

Kurt

On Tue, Mar 31, 2009 at 13:58, Brian Desmond  wrote:
> Kurt-
>
>
>
> Is pooltag.txt in that archive I sent you offline?
>
>
>
> If not, install the Debug Tools for Windows (free download) and go in the
> triage folder pooltag.txt has a dictionary so to speak of all the MS
> tags
>
>
>
> � ��  - nt!� �  - General security allocations
>
>
>
> Allocs ist really terribly interesting, what is is the number of bytes
> allocated which in this case is ~232MB. That is not normal. I have no idea
> what this one is ��� is not somethi�ve seen before 
> an�s not
> normal. I would be inclined to tell you to call PSS and let them work on it.
> I know how to get the stacks for the allocations with a live debug but
> that���s fairly involved as things go.
>
>
>
> MmSt is associated with memory tracking shared files (in a nutshel
> looks ok in your snapshot.
>
>
>
> Thanks,
>
> Brian Desmond
>
> br...@briandesmond.com
>
>
>
> c - 312.731.3132
>
>
>
> Active Directory, 4th Ed - http://www.briandesmond.com/ad4/
>
> Microsoft MVP - https://mvp.support.microsoft.com/profile/Brian
>
>
>
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Tuesday, March 31, 2009 1:46 PM
> To: NT System Admin Issues
> Subject: Pool consumption (was: Re: Virtualized server issue...)
>
>
>
> Have added the reg entry, and am examining poolmon output.
>
> The server needed to be rebooted again today, so the reg entry should now be
> active. I note that in perfmon Pool Paged Allocs for this machine had
> slammed to the ceiling, although nothing else seemed to be
>
> Poolmon is proving problematic - I've examined the top two memory eaters,
> and am either getting too many or no hits.
>
> At 10:12 this morning, I took this reading, and a few minutes later had to
> reboot the box.
>
> Memory: 1047456K Avail:  220432K  PageFlts:47   InRam Krnl: 2912K
> P:281220K
> Commit: 570560K Limit:3053000K Peak: 580612KPool N:35804K
> P:282140K
> System pool information
> Tag  Type AllocsFreesDiff   Bytes   Per
> Alloc
>
> Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 ( 0)
> 20
> MmSt Paged755490 (   9)753741 (   9) 1749 15544008 ( 0)
> 8887
>
> Memory: 1047456K Avail:  229772K  PageFlts:66   InRam Krnl: 2912K
> P:281000K
> Commit: 570396K Limit:3053000K Peak: 580612KPool N:35776K
> P:281920K
> System pool information
> Tag  Type AllocsFreesDiff   Bytes   Per
> Alloc
>
> MmCm Nonp   1090 (   0)15 (   0) 1075 7864864 ( 0)
> 7316
> LSwi Nonp  1 (   0) 0 (   0)1 2576384 ( 0)
> 2576384
>
> As the winner and new champeen, Se is at least an order of magnitude larger
> than its closes competitor. However, the Se tag is in lots of files, so I
> prefixed it in the findstr command with 'h' per the KB article. That
> narrowed it considerably, but what's left pretty much looks to be MSFT
> files, and there are still 21 files.
>
> MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
> srv.sys - I note that the Se tag is also found in the latter.
>
> Later today, perhaps after standard business hours, I'll reenable Active
> Protection for VIPRE on this machine, and copy a lot of files to it. If
> that's the issue, it should start sp

RE: Pool consumption (was: Re: Virtualized server issue...)

2009-03-31 Thread Brian Desmond 
What does that have to do with this?

Thanks,
Brian Desmond
br...@briandesmond.com

c - 312.731.3132


-Original Message-
From: mse...@ont.com [mailto:mse...@ont.com]
Sent: Tuesday, March 31, 2009 4:56 PM
To: NT System Admin Issues
Subject: RE: Pool consumption (was: Re: Virtualized server issue...)

Start with 1 vCPU then check your CPU and memory on your VM. Having
multiple cpu's can actually hurt performance of a VM unless it truely needs
multiple CPU's. We start with 1 vCPU on all of our VM's. Also check your
HAL if you are trying to run a uniprocessor HAL on multiprocessor machine

Mike

Original Message:
-
From: Kurt Buff kurt.b...@gmail.com
Date: Tue, 31 Mar 2009 11:45:59 -0700
To: ntsysadmin@lyris.sunbelt-software.com
Subject: Pool consumption (was: Re: Virtualized server issue...)


Have added the reg entry, and am examining poolmon output.

The server needed to be rebooted again today, so the reg entry should now be
active. I note that in perfmon Pool Paged Allocs for this machine had
slammed to the ceiling, although nothing else seemed to be

Poolmon is proving problematic - I've examined the top two memory eaters,
and am either getting too many or no hits.

At 10:12 this morning, I took this reading, and a few minutes later had to
reboot the box.

Memory: 1047456K Avail:  220432K  PageFlts:47   InRam Krnl: 2912K
P:281220K
Commit: 570560K Limit:3053000K Peak: 580612KPool N:35804K
P:282140K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 ( 0)
20
MmSt Paged755490 (   9)753741 (   9) 1749 15544008 ( 0)
8887

Memory: 1047456K Avail:  229772K  PageFlts:66   InRam Krnl: 2912K
P:281000K
Commit: 570396K Limit:3053000K Peak: 580612KPool N:35776K
P:281920K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

MmCm Nonp   1090 (   0)15 (   0) 1075 7864864 ( 0)
7316
LSwi Nonp  1 (   0) 0 (   0)1 2576384 ( 0)
2576384

As the winner and new champeen, Se is at least an order of magnitude larger
than its closes competitor. However, the Se tag is in lots of files, so I
prefixed it in the findstr command with 'h' per the KB article. That
narrowed it considerably, but what's left pretty much looks to be MSFT
files, and there are still 21 files.

MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
srv.sys - I note that the Se tag is also found in the latter.

Later today, perhaps after standard business hours, I'll reenable Active
Protection for VIPRE on this machine, and copy a lot of files to it. If
that's the issue, it should start spiking pretty quickly.

On Mon, Mar 30, 2009 at 15:29, Brian Desmond  wrote:
> Kurt-
>
> Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll
registry value and reboot? This will allow you to generate a dump next time
this happens (the hang, specifically) by pressing the /right/ Ctrl key and
Scroll Lock twice.
>
> Also, Poolmon can help tremendously here too for logging.
>
> Thanks,
> Brian Desmond
> br...@briandesmond.com
>
> c - 312.731.3132
>
>
> -Original Message-
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Monday, March 30, 2009 5:00 PM
> To: NT System Admin Issues
> Subject: Virtualized server issue...
>
> All,
>
> Over the weekend we virtualized our file/print server, and it seemed
> to go well. Host is a Dell machine running ESX 3.5 update 2.
>
> The physical machine has an Intel HT processor and 1gbyte of RAM. I
> gave the VM 2 procs and 2gbytes of RAM, just for good measure.
>
> Both machines were talking to our LeftHand SAN, on a separate physical
> LAN, but today I had to reboot the VM, then a couple of hours later
> shut it down and revert to the physical machine after it stopped
> responding.
>
> The logs were indicating lack of server memory - specifically, these
> were being emitted to my syslog server:
>
> 2009-03-30 14:05:12 User.Notice home-01 Mar 30 14:05:12
> home-01 MSWinEventLog   1   System  13892   Mon Mar 30 14:05:08 2009
> 2020Srv Unknown UserN/A Error   HOME-01 None
> : 00 00 04 00 01 00 54 00   ...  0008: 00 00 00 00 e4 07 00 c0
>    0010: 00 00 00 00 9a 00 00 c0     0018: 00 00 00
> 00 00 00 00 00     0020: 00 00 00 00 00 00 00 00   
> 0028: ae 04 00 00 d0 02 70 00   ...  The server was unable to
> allocate from the system paged pool because the pool was empty.
>
> Then this, as I tried to log in to shut it down:
>
> 2009-03-30 14:09:39 User.Notice zet-home-01 Mar 30
> 14:09:39 zet-home-01 MSWinEventLog   1   Application 13935

RE: Pool consumption (was: Re: Virtualized server issue...)

2009-03-31 Thread Brian Desmond 
Interesting. Not sure where that A/V package would be in this, but, you should 
have them investigate.

Thanks,
Brian Desmond
br...@briandesmond.com

c - 312.731.3132


-Original Message-
From: Kurt Buff [mailto:kurt.b...@gmail.com]
Sent: Tuesday, March 31, 2009 4:35 PM
To: NT System Admin Issues
Subject: Re: Pool consumption (was: Re: Virtualized server issue...)

BTW - I think I've found the culprit, and will be testing this evening.

I have a job that runs every night to check permissions, using
fileacl.exe. Normally its report is a few kilobytes. Today's was over
10 megabytes - this happens occasionally when a certain department of
ours needs to rearrange the directories allocated to it, and signals
that thousands of files have been moved.

I investigated, and found that this person was indeed performing this
task around the time of the server having difficulties, both yesterday
and today.

This leads me to believe that it's indeed VIPRE and its Active
Protection settings - I've disabled that for now, and will be testing
it this evening by doing a mass copy from one directory to another on
the file server, both with and without the Active Protection setting
turned on. This should help pinpoint the issue..

Kurt

On Tue, Mar 31, 2009 at 13:58, Brian Desmond  wrote:
> Kurt-
>
>
>
> Is pooltag.txt in that archive I sent you offline?
>
>
>
> If not, install the Debug Tools for Windows (free download) and go in the
> triage folder ��� pooltag.txt has a dictionary so to speak of all the MS
> tags
>
>
>
> S�ÿ  - nt!s��  - General security allocations
>
>
>
> Allocs isn���t really terribly interesting, what is is the number of bytes
> allocated which in this case is ~232MB. That is not normal. I have no idea
> what this one i� it���s not somethinve seen before 
> ands not
> normal. I would be inclined to tell you to call PSS and let them work on it.
> I know how to get the stacks for the allocations with a live debug but
> th�s fairly involved as things go.
>
>
>
> MmSt is associated with memory tracking shared files (in a nutshell���
> looks ok in your snapshot.
>
>
>
> Thanks,
>
> Brian Desmond
>
> br...@briandesmond.com
>
>
>
> c - 312.731.3132
>
>
>
> Active Directory, 4th Ed - http://www.briandesmond.com/ad4/
>
> Microsoft MVP - https://mvp.support.microsoft.com/profile/Brian
>
>
>
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Tuesday, March 31, 2009 1:46 PM
> To: NT System Admin Issues
> Subject: Pool consumption (was: Re: Virtualized server issue...)
>
>
>
> Have added the reg entry, and am examining poolmon output.
>
> The server needed to be rebooted again today, so the reg entry should now be
> active. I note that in perfmon Pool Paged Allocs for this machine had
> slammed to the ceiling, although nothing else seemed to be
>
> Poolmon is proving problematic - I've examined the top two memory eaters,
> and am either getting too many or no hits.
>
> At 10:12 this morning, I took this reading, and a few minutes later had to
> reboot the box.
>
> Memory: 1047456K Avail:  220432K  PageFlts:47   InRam Krnl: 2912K
> P:281220K
> Commit: 570560K Limit:3053000K Peak: 580612KPool N:35804K
> P:282140K
> System pool information
> Tag  Type AllocsFreesDiff   Bytes   Per
> Alloc
>
> Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 ( 0)
> 20
> MmSt Paged755490 (   9)753741 (   9) 1749 15544008 ( 0)
> 8887
>
> Memory: 1047456K Avail:  229772K  PageFlts:66   InRam Krnl: 2912K
> P:281000K
> Commit: 570396K Limit:3053000K Peak: 580612KPool N:35776K
> P:281920K
> System pool information
> Tag  Type AllocsFreesDiff   Bytes   Per
> Alloc
>
> MmCm Nonp   1090 (   0)15 (   0) 1075 7864864 ( 0)
> 7316
> LSwi Nonp  1 (   0) 0 (   0)1 2576384 ( 0)
> 2576384
>
> As the winner and new champeen, Se is at least an order of magnitude larger
> than its closes competitor. However, the Se tag is in lots of files, so I
> prefixed it in the findstr command with 'h' per the KB article. That
> narrowed it considerably, but what's left pretty much looks to be MSFT
> files, and there are still 21 files.
>
> MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
> srv.sys - I note that the Se tag is also found in the latter.
>
> Later today, perhaps after standard business hours, I'll reenable Active
> Protection for VIPRE on this machine, and copy a lot of files to it. If
> that's

RE: Pool consumption (was: Re: Virtualized server issue...)

2009-03-31 Thread mse...@ont.com 
Start with 1 vCPU then check your CPU and memory on your VM. Having
multiple cpu's can actually hurt performance of a VM unless it truely needs
multiple CPU's. We start with 1 vCPU on all of our VM's. Also check your
HAL if you are trying to run a uniprocessor HAL on multiprocessor machine

Mike

Original Message:
-
From: Kurt Buff kurt.b...@gmail.com
Date: Tue, 31 Mar 2009 11:45:59 -0700
To: ntsysadmin@lyris.sunbelt-software.com
Subject: Pool consumption (was: Re: Virtualized server issue...)


Have added the reg entry, and am examining poolmon output.

The server needed to be rebooted again today, so the reg entry should now be
active. I note that in perfmon Pool Paged Allocs for this machine had
slammed to the ceiling, although nothing else seemed to be

Poolmon is proving problematic - I've examined the top two memory eaters,
and am either getting too many or no hits.

At 10:12 this morning, I took this reading, and a few minutes later had to
reboot the box.

Memory: 1047456K Avail:  220432K  PageFlts:47   InRam Krnl: 2912K
P:281220K
Commit: 570560K Limit:3053000K Peak: 580612KPool N:35804K
P:282140K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 ( 0)
20
MmSt Paged755490 (   9)753741 (   9) 1749 15544008 ( 0)
8887

Memory: 1047456K Avail:  229772K  PageFlts:66   InRam Krnl: 2912K
P:281000K
Commit: 570396K Limit:3053000K Peak: 580612KPool N:35776K
P:281920K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per
Alloc

MmCm Nonp   1090 (   0)15 (   0) 1075 7864864 ( 0)
7316
LSwi Nonp  1 (   0) 0 (   0)1 2576384 ( 0)
2576384

As the winner and new champeen, Se is at least an order of magnitude larger
than its closes competitor. However, the Se tag is in lots of files, so I
prefixed it in the findstr command with 'h' per the KB article. That
narrowed it considerably, but what's left pretty much looks to be MSFT
files, and there are still 21 files.

MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
srv.sys - I note that the Se tag is also found in the latter.

Later today, perhaps after standard business hours, I'll reenable Active
Protection for VIPRE on this machine, and copy a lot of files to it. If
that's the issue, it should start spiking pretty quickly.

On Mon, Mar 30, 2009 at 15:29, Brian Desmond  wrote:
> Kurt-
>
> Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll
registry value and reboot? This will allow you to generate a dump next time
this happens (the hang, specifically) by pressing the /right/ Ctrl key and
Scroll Lock twice.
>
> Also, Poolmon can help tremendously here too for logging.
>
> Thanks,
> Brian Desmond
> br...@briandesmond.com
>
> c - 312.731.3132
>
>
> -Original Message-
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Monday, March 30, 2009 5:00 PM
> To: NT System Admin Issues
> Subject: Virtualized server issue...
>
> All,
>
> Over the weekend we virtualized our file/print server, and it seemed
> to go well. Host is a Dell machine running ESX 3.5 update 2.
>
> The physical machine has an Intel HT processor and 1gbyte of RAM. I
> gave the VM 2 procs and 2gbytes of RAM, just for good measure.
>
> Both machines were talking to our LeftHand SAN, on a separate physical
> LAN, but today I had to reboot the VM, then a couple of hours later
> shut it down and revert to the physical machine after it stopped
> responding.
>
> The logs were indicating lack of server memory - specifically, these
> were being emitted to my syslog server:
>
> 2009-03-30 14:05:12 User.Notice home-01 Mar 30 14:05:12
> home-01 MSWinEventLog   1   System  13892   Mon Mar 30 14:05:08 2009
> 2020Srv Unknown UserN/A Error   HOME-01 None
> : 00 00 04 00 01 00 54 00   ...  0008: 00 00 00 00 e4 07 00 c0
>    0010: 00 00 00 00 9a 00 00 c0     0018: 00 00 00
> 00 00 00 00 00     0020: 00 00 00 00 00 00 00 00   
> 0028: ae 04 00 00 d0 02 70 00   ...  The server was unable to
> allocate from the system paged pool because the pool was empty.
>
> Then this, as I tried to log in to shut it down:
>
> 2009-03-30 14:09:39 User.Notice zet-home-01 Mar 30
> 14:09:39 zet-home-01 MSWinEventLog   1   Application 13935   Mon
> Mar 30 14:09:39 20091512Userenv SYSTEM  User
> Error   ZET-HOME-01 NoneWindows cannot unload your
> registry file. The memory used by the registry has not been freed.
> This is often caused by services running as a user account, try
> configuring the services to run in either the LocalService or
> NetworkService account. If this problem persists, contact your
> administrator.DETAIL - Insufficient system resources exist to
> complete the requested service.

Re: Pool consumption (was: Re: Virtualized server issue...)

2009-03-31 Thread Kurt Buff 
BTW - I think I've found the culprit, and will be testing this evening.

I have a job that runs every night to check permissions, using
fileacl.exe. Normally its report is a few kilobytes. Today's was over
10 megabytes - this happens occasionally when a certain department of
ours needs to rearrange the directories allocated to it, and signals
that thousands of files have been moved.

I investigated, and found that this person was indeed performing this
task around the time of the server having difficulties, both yesterday
and today.

This leads me to believe that it's indeed VIPRE and its Active
Protection settings - I've disabled that for now, and will be testing
it this evening by doing a mass copy from one directory to another on
the file server, both with and without the Active Protection setting
turned on. This should help pinpoint the issue..

Kurt

On Tue, Mar 31, 2009 at 13:58, Brian Desmond  wrote:
> Kurt-
>
>
>
> Is pooltag.txt in that archive I sent you offline?
>
>
>
> If not, install the Debug Tools for Windows (free download) and go in the
> triage folder ÿÿ“ pooltag.txt has a dictionary so to speak of all the MS
> tags
>
>
>
> Sÿÿ ÿ  - nt!sÿÿ ÿÿÂÿÿ ÿÿÂÿÿ  - General security allocations
>
>
>
> Allocs isnÿÿ™t really terribly interesting, what is is the number of bytes
> allocated which in this case is ~232MB. That is not normal. I have no idea
> what this one is ÿÿ“ itÿÿ™s not somethingÿÿâÿ™ve seen before and ÿÿâÿ™s not
> normal. I would be inclined to tell you to call PSS and let them work on it.
> I know how to get the stacks for the allocations with a live debug but
> thatÿÿ™s fairly involved as things go.
>
>
>
> MmSt is associated with memory tracking shared files (in a nutshellÿÿâÿ“
> looks ok in your snapshot.
>
>
>
> Thanks,
>
> Brian Desmond
>
> br...@briandesmond.com
>
>
>
> c - 312.731.3132
>
>
>
> Active Directory, 4th Ed - http://www.briandesmond.com/ad4/
>
> Microsoft MVP - https://mvp.support.microsoft.com/profile/Brian
>
>
>
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Tuesday, March 31, 2009 1:46 PM
> To: NT System Admin Issues
> Subject: Pool consumption (was: Re: Virtualized server issue...)
>
>
>
> Have added the reg entry, and am examining poolmon output.
>
> The server needed to be rebooted again today, so the reg entry should now be
> active. I note that in perfmon Pool Paged Allocs for this machine had
> slammed to the ceiling, although nothing else seemed to be
>
> Poolmon is proving problematic - I've examined the top two memory eaters,
> and am either getting too many or no hits.
>
> At 10:12 this morning, I took this reading, and a few minutes later had to
> reboot the box.
>
> Memory: 1047456K Avail:  220432K  PageFlts:    47   InRam Krnl: 2912K
> P:281220K
> Commit: 570560K Limit:3053000K Peak: 580612K            Pool N:35804K
> P:282140K
> System pool information
> Tag  Type     Allocs            Frees            Diff   Bytes       Per
> Alloc
>
> Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 (     0)
> 20
> MmSt Paged    755490 (   9)    753741 (   9)     1749 15544008 (     0)
> 8887
>
> Memory: 1047456K Avail:  229772K  PageFlts:    66   InRam Krnl: 2912K
> P:281000K
> Commit: 570396K Limit:3053000K Peak: 580612K            Pool N:35776K
> P:281920K
> System pool information
> Tag  Type     Allocs            Frees            Diff   Bytes       Per
> Alloc
>
> MmCm Nonp       1090 (   0)        15 (   0)     1075 7864864 (     0)
> 7316
> LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0)
> 2576384
>
> As the winner and new champeen, Se is at least an order of magnitude larger
> than its closes competitor. However, the Se tag is in lots of files, so I
> prefixed it in the findstr command with 'h' per the KB article. That
> narrowed it considerably, but what's left pretty much looks to be MSFT
> files, and there are still 21 files.
>
> MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in
> srv.sys - I note that the Se tag is also found in the latter.
>
> Later today, perhaps after standard business hours, I'll reenable Active
> Protection for VIPRE on this machine, and copy a lot of files to it. If
> that's the issue, it should start spiking pretty quickly.
>
> On Mon, Mar 30, 2009 at 15:29, Brian Desmond  wrote:
>> Kurt-
>>
>> Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll
>> registry value and reboot? This will allow you to generate a dump next time
>> this happens (the hang, specifically) by pressing the /right/ Ctrl key and
>> Scroll Lock twice.
>>
>> Also, Poolmon can help tremendously here too for logging.
>>
>> Thanks,
>> Brian Desmond
>> br...@briandesmond.com
>>
>> c - 312.731.3132
>>
>>
>> -Original Message-
>> From: Kurt Buff [mailto:kurt.b...@gmail.com]
>> Sent: Monday, March 30, 2009 5:00 PM
>> To: NT System Admin Issues
>> Subject: Virtualized server issue...
>>
>> All,
>>
>> Over the weekend we virtualized our file/print server, and it seemed
>> to

Re: Pool consumption (was: Re: Virtualized server issue...)

2009-03-31 Thread Kurt Buff 
Have received no emails from you offline, sorry to report.

Also, your message, as you can see below, is a bit hard to read, with
some funky characters spoiling things.

I will take a look at the Debug Tools for Windows and take a look.

On Tue, Mar 31, 2009 at 13:58, Brian Desmond  wrote:
>
> Kurt-
>
>
>
> Is pooltag.txt in that archive I sent you offline?
>
>
>
> If not, install the Debug Tools for Windows (free download) and go in the 
> triage folder ÿÿ“ pooltag.txt has a dictionary so to speak of all the MS tags
>
>
>
> Sÿÿ ÿ  - nt!sÿÿ ÿÿÂÿÿ ÿÿÂÿÿ  - General security allocations
>
>
>
> Allocs isnÿÿ™t really terribly interesting, what is is the number of bytes 
> allocated which in this case is ~232MB. That is not normal. I have no idea 
> what this one is ÿÿ“ itÿÿ™s not somethingÿÿâÿ™ve seen before and ÿÿâÿ™s not 
> normal. I would be inclined to tell you to call PSS and let them work on it. 
> I know how to get the stacks for the allocations with a live debug but 
> thatÿÿ™s fairly involved as things go.
>
>
>
> MmSt is associated with memory tracking shared files (in a nutshellÿÿâÿ“ 
> looks ok in your snapshot.
>
>
>
> Thanks,
>
> Brian Desmond
>
> br...@briandesmond.com
>
>
>
> c - 312.731.3132
>
>
>
> Active Directory, 4th Ed - http://www.briandesmond.com/ad4/
>
> Microsoft MVP - https://mvp.support.microsoft.com/profile/Brian
>
>
>
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Tuesday, March 31, 2009 1:46 PM
> To: NT System Admin Issues
> Subject: Pool consumption (was: Re: Virtualized server issue...)
>
>
>
> Have added the reg entry, and am examining poolmon output.
>
> The server needed to be rebooted again today, so the reg entry should now be 
> active. I note that in perfmon Pool Paged Allocs for this machine had slammed 
> to the ceiling, although nothing else seemed to be
>
> Poolmon is proving problematic - I've examined the top two memory eaters, and 
> am either getting too many or no hits.
>
> At 10:12 this morning, I took this reading, and a few minutes later had to 
> reboot the box.
>
> Memory: 1047456K Avail:  220432K  PageFlts:    47   InRam Krnl: 2912K 
> P:281220K
> Commit: 570560K Limit:3053000K Peak: 580612K            Pool N:35804K 
> P:282140K
> System pool information
> Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc
>
> Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 (     0)     
> 20
> MmSt Paged    755490 (   9)    753741 (   9)     1749 15544008 (     0)   8887
>
> Memory: 1047456K Avail:  229772K  PageFlts:    66   InRam Krnl: 2912K 
> P:281000K
> Commit: 570396K Limit:3053000K Peak: 580612K            Pool N:35776K 
> P:281920K
> System pool information
> Tag  Type     Allocs            Frees            Diff   Bytes       Per Alloc
>
> MmCm Nonp       1090 (   0)        15 (   0)     1075 7864864 (     0)   7316
> LSwi Nonp          1 (   0)         0 (   0)        1 2576384 (     0) 2576384
>
> As the winner and new champeen, Se is at least an order of magnitude larger 
> than its closes competitor. However, the Se tag is in lots of files, so I 
> prefixed it in the findstr command with 'h' per the KB article. That narrowed 
> it considerably, but what's left pretty much looks to be MSFT files, and 
> there are still 21 files.
>
> MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in srv.sys 
> - I note that the Se tag is also found in the latter.
>
> Later today, perhaps after standard business hours, I'll reenable Active 
> Protection for VIPRE on this machine, and copy a lot of files to it. If 
> that's the issue, it should start spiking pretty quickly.
>
> On Mon, Mar 30, 2009 at 15:29, Brian Desmond  wrote:
> > Kurt-
> >
> > Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll 
> > registry value and reboot? This will allow you to generate a dump next time 
> > this happens (the hang, specifically) by pressing the /right/ Ctrl key and 
> > Scroll Lock twice.
> >
> > Also, Poolmon can help tremendously here too for logging.
> >
> > Thanks,
> > Brian Desmond
> > br...@briandesmond.com
> >
> > c - 312.731.3132
> >
> >
> > -Original Message-
> > From: Kurt Buff [mailto:kurt.b...@gmail.com]
> > Sent: Monday, March 30, 2009 5:00 PM
> > To: NT System Admin Issues
> > Subject: Virtualized server issue...
> >
> > All,
> >
> > Over the weekend we virtualized our file/print server, and it seemed
> > to go well. Host is a Dell machine running ESX 3.5 update 2.
> >
> > The physical machine has an Intel HT processor and 1gbyte of RAM. I
> > gave the VM 2 procs and 2gbytes of RAM, just for good measure.
> >
> > Both machines were talking to our LeftHand SAN, on a separate physical
> > LAN, but today I had to reboot the VM, then a couple of hours later
> > shut it down and revert to the physical machine after it stopped
> > responding.
> >
> > The logs were indicating lack of server memory - specifically, these
> > were being emitted to my syslog server:
> >
> > 2009-03-30 14:

RE: Pool consumption (was: Re: Virtualized server issue...)

2009-03-31 Thread Brian Desmond 
Kurt-

Is pooltag.txt in that archive I sent you offline?

If not, install the Debug Tools for Windows (free download) and go in the 
triage folder pooltag.txt has a dictionary so to speak of all the MS tags

Se   - nt!se- General security allocations

Allocs isn���t really terribly interesting, what is is the number of bytes 
allocated which in this case is ~232MB. That is not normal. I have no idea what 
this one is is not something I���ve seen before and it���s not normal. 
I would be inclined to tell you to call PSS and let them work on it. I know how 
to get the stacks for the allocations with a live debug but th�s fairly 
involved as things go.

MmSt is associated with memory tracking shared files (in a nutshell) looks 
ok in your snapshot.

Thanks,
Brian Desmond
br...@briandesmond.com

c - 312.731.3132

Active Directory, 4th Ed - http://www.briandesmond.com/ad4/
Microsoft MVP - https://mvp.support.microsoft.com/profile/Brian

From: Kurt Buff [mailto:kurt.b...@gmail.com]
Sent: Tuesday, March 31, 2009 1:46 PM
To: NT System Admin Issues
Subject: Pool consumption (was: Re: Virtualized server issue...)

Have added the reg entry, and am examining poolmon output.

The server needed to be rebooted again today, so the reg entry should now be 
active. I note that in perfmon Pool Paged Allocs for this machine had slammed 
to the ceiling, although nothing else seemed to be

Poolmon is proving problematic - I've examined the top two memory eaters, and 
am either getting too many or no hits.

At 10:12 this morning, I took this reading, and a few minutes later had to 
reboot the box.

Memory: 1047456K Avail:  220432K  PageFlts:47   InRam Krnl: 2912K P:281220K
Commit: 570560K Limit:3053000K Peak: 580612KPool N:35804K P:282140K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per Alloc

Se   Paged  13914251 (   0)   1770996 (   0) 12143255 243309424 ( 0) 20
MmSt Paged755490 (   9)753741 (   9) 1749 15544008 ( 0)   8887

Memory: 1047456K Avail:  229772K  PageFlts:66   InRam Krnl: 2912K P:281000K
Commit: 570396K Limit:3053000K Peak: 580612KPool N:35776K P:281920K
System pool information
Tag  Type AllocsFreesDiff   Bytes   Per Alloc

MmCm Nonp   1090 (   0)15 (   0) 1075 7864864 ( 0)   7316
LSwi Nonp  1 (   0) 0 (   0)1 2576384 ( 0) 2576384

As the winner and new champeen, Se is at least an order of magnitude larger 
than its closes competitor. However, the Se tag is in lots of files, so I 
prefixed it in the findstr command with 'h' per the KB article. That narrowed 
it considerably, but what's left pretty much looks to be MSFT files, and there 
are still 21 files.

MsSt is simply not found anywhere, nor is MmCm,. though LSwi shows in srv.sys - 
I note that the Se tag is also found in the latter.

Later today, perhaps after standard business hours, I'll reenable Active 
Protection for VIPRE on this machine, and copy a lot of files to it. If that's 
the issue, it should start spiking pretty quickly.

On Mon, Mar 30, 2009 at 15:29, Brian Desmond 
mailto:br...@briandesmond.com>> wrote:
> Kurt-
>
> Can you add the http://support.microsoft.com/kb/244139 CrashOnCtrlScroll 
> registry value and reboot? This will allow you to generate a dump next time 
> this happens (the hang, specifically) by pressing the /right/ Ctrl key and 
> Scroll Lock twice.
>
> Also, Poolmon can help tremendously here too for logging.
>
> Thanks,
> Brian Desmond
> br...@briandesmond.com
>
> c - 312.731.3132
>
>
> -Original Message-
> From: Kurt Buff [mailto:kurt.b...@gmail.com]
> Sent: Monday, March 30, 2009 5:00 PM
> To: NT System Admin Issues
> Subject: Virtualized server issue...
>
> All,
>
> Over the weekend we virtualized our file/print server, and it seemed
> to go well. Host is a Dell machine running ESX 3.5 update 2.
>
> The physical machine has an Intel HT processor and 1gbyte of RAM. I
> gave the VM 2 procs and 2gbytes of RAM, just for good measure.
>
> Both machines were talking to our LeftHand SAN, on a separate physical
> LAN, but today I had to reboot the VM, then a couple of hours later
> shut it down and revert to the physical machine after it stopped
> responding.
>
> The logs were indicating lack of server memory - specifically, these
> were being emitted to my syslog server:
>
> 2009-03-30 14:05:12 User.Notice home-01 Mar 30 14:05:12
> home-01 MSWinEventLog   1   System  13892   Mon Mar 30 14:05:08 2009
> 2020Srv Unknown UserN/A Error   HOME-01 None
> : 00 00 04 00 01 00 54 00   ...  0008: 00 00 00 00 e4 07 00 c0
>    0010: 00 00 00 00 9a 00 00 c0     0018: 00 00 00
> 00 00 00 00 00     0020: 00 00 00 00 00 00 00 00   
> 0028: ae 04 00 00 d0 02 70 00   ...  The server was unable to
>