Half a dozen isn’t likely to be worth worrying about Thanks, Brian Desmond br...@briandesmond.com
w – 312.625.1438 | c – 312.731.3132 From: Kurt Buff [mailto:kurt.b...@gmail.com] Sent: Tuesday, February 14, 2012 1:55 PM To: NT System Admin Issues Subject: Re: Picking up file server tuning again RE: PST files. This might well be part of the issue: #net file | findstr /i pst 21310605 J:\Home\...\Archive PSTs\archive.pst USER1 0 21310606 J:\Home\USER1\Archive PSTs\xxxxx.PST USER1 0 21359101 J:\Home\...\xxxx Folders.pst USER2 2 21375086 J:\Home\...\Outlook\xxxxxxxx.pst USER3 1 21375089 J:\Home\...\Outlook\~xxxxxxxx.pst.tmp USER3 0 21375091 J:\Home\...\Outlook\Jokes.pst USER3 1 21375094 J:\Home\...\Outlook\~Jokes.pst.tmp USER3 0 21386255 J:\Home\USER4\Private\USER4.pst USER4 1 Kurt On Mon, Feb 13, 2012 at 19:25, Brian Desmond <br...@briandesmond.com<mailto:br...@briandesmond.com>> wrote: Well, the % Interrupts/DPC Time/Kernel Mode CPU time isn't necessarily going to be fixed by x64. It may very well mean you've got some crappy drivers in play. The disk stuff indicates the disk is not fast enough to keep up with demand. You can solve that with more spindles or faster spindles. Page Pool utilization will be resolved by x64 (or even x86 on 2008). That's indicative of crappy drivers, large tokens, and/or people doing things like using PSTs off file shares. Thanks, Brian Desmond br...@briandesmond.com<mailto:br...@briandesmond.com> w – 312.625.1438<tel:312.625.1438> | c – 312.731.3132<tel:312.731.3132> -----Original Message----- From: Michael B. Smith [mailto:mich...@smithcons.com<mailto:mich...@smithcons.com>] Sent: Monday, February 13, 2012 6:18 PM To: NT System Admin Issues Subject: RE: Picking up file server tuning again Well, the kernel mode, paged pool, and interrupt time are items that will be specifically reduced with an x64 OS. The I/O situation is indicative of disk queuing which is "hypervisor related". Dunno how you optimize that in VMware, there are a number of potentials in Hyper-V. Regards, Michael B. Smith Consultant and Exchange MVP http://TheEssentialExchange.com -----Original Message----- From: Kurt Buff [mailto:kurt.b...@gmail.com<mailto:kurt.b...@gmail.com>] Sent: Monday, February 13, 2012 5:33 PM To: NT System Admin Issues Subject: Re: Picking up file server tuning again It *is* a busy box, and migrating the iSCSI LUNs to a 64bit server is something I've definitely considered. I have a Dell R310 with 16gb RAM that I could use, but it's already got 9 active VMs, although they're not heavy hitters. AFAICT, probably the highest-use machines on the ESXi 4.1 box are the secondary DC (no FSMO roles, but does do DNS and WINS) and the issuing CA box. It's currently a VM on what I believe to be an underpowered ESX 3.5 box - I think it's possible that it's simply starved for resources on that ESX box. I'm sure there's something out there like perfmon for VMware that I can use to capture performance over time - I'd like to measure and analyze the performance of the ESX 3.5 box while the backups are happening against the file server. I'm also considering moving the Win2k3 file server VM to the ESX box and seeing if the situation improves. Kurt On Mon, Feb 13, 2012 at 12:08, Michael B. Smith <mich...@smithcons.com<mailto:mich...@smithcons.com>> wrote: > That's a busy box. I'd suggest moving to a 64-bit OS. > > Regards, > > Michael B. Smith > Consultant and Exchange MVP > http://TheEssentialExchange.com > > -----Original Message----- > From: Kurt Buff [mailto:kurt.b...@gmail.com<mailto:kurt.b...@gmail.com>] > Sent: Monday, February 13, 2012 3:00 PM > To: NT System Admin Issues > Subject: Re: Picking up file server tuning again > > Ran PAL against the log. > > Um, wow. It's a freaking christmas tree - red and yellow all over the > place in CPU and disk. > > Who should I be talking with to analyze this? > > A sample of the issues shown - all of which show up in more than one > time slice - some in every or almost every slice: > o- More than 50% Processor Utilization > o- More than 30% privileged (kernel) mode CPU usage > o- More than 2 packets are waiting in the output queue > o- Greater than 25ms physical disk READ response times > o- Greater than 25ms physical disk WRITE response times > o- More than 80% of Pool Paged Kernel Memory Used > o- More than 2 I/O's are waiting on the physical disk > o- 20 (Processor(_Total)\DPC Rate) > o- More than 30% Interrupt Time > o- Greater than 1000 page inputs per second (Memory\Pages Input/sec) > > Some things that showed no alerts: > o- Memory\Available MBytes > o- Memory\Free System Page Table Entrie > o- Memory\Pages/sec > o- Memory\System Cache Resident Bytes > o- Memory\Cache Bytes > o- Memory\% Committed Bytes In Use > o- Network Interface(*)\% Network Utilization > MS TCP Loopback interface > VMware Accelerated AMD PCNet Adapter > VMware Accelerated AMD PCNet Adapter#1 > o- Network Interface(*)\Packets Outbound Errors > MS TCP Loopback interface > VMware Accelerated AMD PCNet Adapter > VMware Accelerated AMD PCNet Adapter#1 > > > Kurt > > On Fri, Feb 10, 2012 at 16:04, Brian Desmond > <br...@briandesmond.com<mailto:br...@briandesmond.com>> wrote: >> Rather than trying to do this yourself, check out PAL - >> http://pal.codeplex.com/. It will setup all the right counters for you and >> crunch the data. >> >> Thanks, >> Brian Desmond >> br...@briandesmond.com<mailto:br...@briandesmond.com> >> >> w – 312.625.1438<tel:312.625.1438> | c – 312.731.3132<tel:312.731.3132> >> >> -----Original Message----- >> From: Kurt Buff [mailto:kurt.b...@gmail.com<mailto:kurt.b...@gmail.com>] >> Sent: Friday, February 10, 2012 4:43 PM >> To: NT System Admin Issues >> Subject: Picking up file server tuning again >> >> I'm getting back to monitoring my situation with the file server again, and >> just finished a perfmon session covering the 3rd through the 7th of this >> month. Simultaneously, I set up perfmon on the same workstation to monitor >> the backup server. >> >> If anyone cares to help, I'd be deeply appreciative. >> >> I set up perfmon on a Win7 VM on an ESXi 4.1 host to take measurements at 60 >> second intervals of a whole bunch of counters, many of them probably just >> noise. >> >> I'll describe the history of the configuration first, however: >> >> The file server is a Win2k3 R2 VM running on a ESX 3.5 host with 16g of RAM >> - it's one of 10 VMs, and is definitely the heaviest hitter in terms of disk >> I/O. About 2.5-3 months ago we noticed that the time to completion for the >> weekly full backups spiked dramatically. >> >> Prior to that time, the fulls would start around 7pm on a Friday, and finish >> by about 7pm on Sunday. >> >> Now they take until Thursday or Friday to complete. >> >> This coincided with some changes to the environment: I had to move >> the VM to a new host (it was a manual copy - we don't have vmotion >> licensed and configured for these hosts) and at about that time I >> also had to expand 2 of the 4 LUNS. Finally, the OS drive for the VM >> on the old host was on a LUN on our Lefthand unit - I had to migrate >> it to the local disk storage on the new home for the VM. The 4 data >> drives for this VM are attached via the MSFT iSCSI client running on >> the VM, not through VMWare's iSCSI client. So, at that point, all of >> the LUNS were on the Lefthand SAN, which is a 3-node cluster, and we >> use 2-way replication for all LUNS. The 2 LUNS that were expanded >> went to 2tb or slightly beyond. The Lefthand has two NSM 2060s and a >> P4300G2, with 6 and 8 disks each, respectively - a total of 20 disks >> >> Since that time, I've also added in our EMC VNXe 3100 with 6 disks in it in >> a RAID6 array. I mention this because this means that all of the file >> systems on the VNXe are clean and defragged. >> >> Currently, I've migrated 3 of the 4 data LUNs for the VM to the EMC. I made >> sure to align the partitions on the EMC to a megabyte boundary. >> >> So, to make this simpler to visualize, a little table: >> >> c: - local disk on ESX 3.5, 40gb, 23.6gb free >> j: - iSCSI LUN on Lefthand, 2.5tb, 900gb free >> k: - iSCSI LUN on VNXe, 1.98tb, 336gb free >> l: - iSCSI LUN on VNXe, 1tb, 79gb free >> m: - iSCSI LUN on VNXe 750gb, 425gb free >> >> I tried to capture separate disk queue stats for each LUN, but in spite of >> selecting and adding each drive letter separately in the perfmon interface, >> all I got was _Total. >> >> Selected stats are as follows: >> >> PhysicalDisk counters >> Current disk queue length - average 0.483, maximum 33.000 Average >> disk read queue length - 0.037, maximum 1.294 %disk time - average >> 34.068, maximum 153.877 Average disk write queue length - average >> 0.645, maximum 2.828 Average disk queue length - average 0.681, >> maximum 3.078 >> >> I have more data on PhysicalDisk, and data on other objects, including >> Memory, NetworkInterface, Paging File, Processor and Server Work Queues. >> >> If anyone has thoughts, I'd surely like to hear them. >> >> Thanks, >> >> Kurt >> >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >> >> --- >> To manage subscriptions click here: >> http://lyris.sunbelt-software.com/read/my_forums/ >> or send an email to >> listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> >> with the body: unsubscribe ntsysadmin >> >> >> ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ >> <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ >> >> --- >> To manage subscriptions click here: >> http://lyris.sunbelt-software.com/read/my_forums/ >> or send an email to >> listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> >> with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to > listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> > with the body: unsubscribe ntsysadmin > > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to > listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> > with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin
