Your situation helps me understand the internals of DMP that much more. Sorry my benefit comes at your disadvantage. Good luck with the mystery.
BTW, a tip-of-the-hat to you for using the word "ontoward". Nice to read a tech person using our language so well. On Fri, Jul 27, 2012 at 6:26 AM, <[email protected]> wrote: > Hi All, > Thanks for all the information and suggestions it's given me a lot > to > look at. Hopefully I've covered all your questions below... > > > I've checked and there's no evidence I can find that anyone has made any > changes against the arrays, either via the command line or vea - the latter > which we don't use. However, looking at the tuneable values and what > they're set to and what the defaults are there are two that are different > to the defaults... > > fuj411:/root# vxdmpadm gettune all > Tunable Current Value Default Value > ------------------------------- ------------- ------------- > dmp_failed_io_threshold 57600 57600 > dmp_retry_count 5 5 > dmp_pathswitch_blks_shift 11 11 > dmp_queue_depth 32 32 > dmp_cache_open off off > dmp_daemon_count 10 10 > dmp_scsi_timeout 30 30 > dmp_delayq_interval 15 15 > dmp_path_age 0 300 > dmp_stat_interval 1 1 > dmp_health_time 0 60 > dmp_probe_idle_lun on on > dmp_log_level 1 1 > dmp_retry_timeout 0 0 > fuj411:/root# > > I don't know who would've changed these or when, or at all. There's a lot > of disk seen down each path - physical HBA. We have an A and a B side to > the SAN with the disk being presented down both sides to separate HBA's on > the server. The errors and other messages are always to the same array > (IBM_SHARK1) which is the one that has the different value set, and is the > one that see's the throttling against it. > > > The version of VxVM is 4.1 MP2 so I know we are downlevel and doesn't > have the new default value of 512 that's come in with 5.1 MP3. > > > All paths are active and i/o is seen going down both paths to all disks. > I can see this at the server and the SAN port level. The storage team have > checked and the cannot find anything untoward. I've also had a look at the > HBA's as well (fcinfo hba-port -l <HBA_WWN>) > > > The recovery option on both arrays is the same and set to the > following... > > fuj411:/root# vxdmpadm getattr enclosure IBM_SHARK0 recoveryoption > ENCLR-NAME RECOVERY-OPTION DEFAULT[VAL] CURRENT[VAL] > > ============================================================================== > IBM_SHARK0 Throttle Timebound[10] Timebound > [10] > IBM_SHARK0 Error-Retry Fixed-Retry[5] Fixed-Retry > [5] > fuj411:/root# > fuj411:/root# vxdmpadm getattr enclosure IBM_SHARK1 recoveryoption > ENCLR-NAME RECOVERY-OPTION DEFAULT[VAL] CURRENT[VAL] > > ============================================================================== > IBM_SHARK1 Throttle Timebound[10] Timebound > [10] > IBM_SHARK1 Error-Retry Fixed-Retry[5] Fixed-Retry > [5] > fuj411:/root# > > My suspicion is that we have some kind of fault it's just identifying > where. I suspect this will involve getting the server up to the latest > patch levels in the o/s and within the version installed for starters and > getting the storage team to carry out a full end to end test of the > hardware. In the meantime thanks again for all your help with this, very > much appreciated. I think I will take Dmitry's advice and log a call with > Symantec and see if they can explain the difference and whether it's just > the way this array type operates. > > Thanks again. > > Cheers > Phil. > > > > > Dmitry Glushenok > <[email protected] > > To > William Havey <[email protected]> > 26/07/2012 21:09 [email protected] > cc > [email protected] > Subject > Re: [Veritas-vx] Question over DMP > partitionsize > > > > > > > > > > > Hello William, Phil, > > 26.07.2012, в 21:25, William Havey написал(а): > > Phil, > > I don't believe any DMP values are changed by vxconfigd > automatically. > > To be certain the partitionsize has not been manually changed, > Is VxVM command line logging enabled? run vxcmdlog -l to find out > If yes, look in /var/adm/vx/cmdlog for any vxdmpadm setattr commands. > > Phil using VCS, so cmdlog gets rewritten very fast. > If /etc/vx/dmppolicy.info contains nothing about partition size and other > VCS nodes have same partition size differences - it looks like ASL for > SHARK (or vxdmp itself) is making the changes. And only way to verify this > is contact symantec :) > > Do you use vea? If so, its commands are logged in > /var/adm/vx/veacmdlog. But, I don't believe these values can beset in > vea. > > I believe the "throttle" and "unthrottle" in the dmpevents log file > means holding back I/O from being addressed and sent out to storage > but I certainly can't say the two (partitionsize of 512 and log > entries for the SHARK1 with that size) are related or not. > > Path can be throttled if it holds SCSI command for more than specified > amount of time or if queue depth for the path reaches specified limit. It > can be checked with "vxdmpadm getattr enclosure <enc_name> recoveryoption" > and verified with iostat: asvc_t and actv/wait columns (not sure which > queue vxdmp counts, active or wait). > > > Are all paths being used for I/O just about the same? > vxdmpadm iostat start > vxdmpadm iostat reset > vxdmpadm iostat show enclosure=IBM_SHARK0 enclosure=IBM_SHARK1 > interval=5 > The first iteration of the utility shows cumulative statistics from > the time of mounting the file systems. So, ignore the first output. > If totals are about equal, then the value the partitionsize value may > be irrelevant. Perhaps I/O is truly random so that no I/O address is > within 512 sectors of the previous I/O. Even in a random environment > sometimes numerous successive I/O's can be within 512 sectors of each > other so that throttling those I/Os kicks-in. > > IHTH, > > Bill > > > On Thu, Jul 26, 2012 at 11:56 AM, <[email protected]> wrote: > Hi William, > Thanks for the reply. Not sure how to get this track cach > size? > What's confusing me most here is that the values are so different > between > the two arrays. They're identical models set-up up the same and > with the > same number of disks allocated to the servers - sorry, forgot to > mention > they're in an HA pair using VCS. The only difference is that > IBM_SHARK0 is > local to the server where the workload is currently running, and > IBM_SHARK1 > is in another building about 1.5KM's away. It's this disparity that > is > confusing me and making me wonder whether an issue we are seeing is > being > caused by this, or if it's an indication of an issue, though I've > been > informed by our storage people that there's nothing wrong with > either > array. > This is certainly not something we've changed so I don't > know if > VxVM/DMP is throttling things back because it's seeing an issue. I > have > seen messages in the /etc/vx/dmpevents.log file for disks in the > IBM_SHARK1 > array reporting 'Throttled Path' and then 'Un-throttled Path' and > I'm > trying to work out if the two are linked. > > Cheers > Phil. > > > > > William Havey > <[email protected] > m> > To > [email protected] > 26/07/2012 15:48 > cc > > > Subject > Re: [Veritas-vx] Question > over DMP > partitionsize > > > > > > > > > > > Partitionsize is in play when the iopolicy is Balanced, which is > the > default policy. You have Balanced. > > It is defined as "The partitionsize attribute: Each successive I/O > starting > within in this range (default is 2048 sectors) goes through the > same path > as the previous I/O" > > The man page has "Takes the track cache into consideration when > balancing > I/O across paths" > > What is the track cache size on each array? Seems that the > partitionsize > should be the same value as the track cache size. > > > > On Thu, Jul 26, 2012 at 7:09 AM, <[email protected]> wrote: > > Hi, > I'm trying to understand why one array has a wildly > different > partitionsize value to the other... > > fuj411:/root# vxdmpadm getattr arrayname IBM_SHARK partitionsize > ENCLR_NAME DEFAULT CURRENT > ============================================ > IBM_SHARK0 2048 2048 > IBM_SHARK1 256 512 > fuj411:/root# > > Both arrays are IBM ESS SHARK 2105's which are active/active and > operating > in a Balanced i/o policy... > > fuj411:/root# vxdmpadm getattr arrayname IBM_SHARK iopolicy > ENCLR_NAME DEFAULT CURRENT > ============================================ > IBM_SHARK0 Balanced Balanced > IBM_SHARK1 Balanced Balanced > fuj411:/root# > > The server is a Fujitsu PW850 running Solaris 10 with VxVM v4.1 > MP2. The > volume are mirrored between the two arrays. > > I'm trying to get a better understanding of what this means and > why it > would be so different between the two arrays. Any help greatly > appreciated. > > Cheers > Phil. > > -- > This message is private and confidential and may also be legally > privileged. If you have received this message in error, please > email it > back to the sender and immediately permanently delete it from > your > computer system. Please do not read, print, re-transmit, store > or act in > reliance on it or any attachments. > > British Airways may monitor email traffic data and also the > content of > emails, where permitted by law, for the purposes of security and > staff > training and in order to prevent or detect unauthorised use of > the > British Airways email system. > > Virus checking of emails (including attachments) is the > responsibility of > the recipient. > > British Airways Plc is a public limited company registered in > England and > Wales. Registered number: 1777777. Registered office: > Waterside, PO Box > 365, Harmondsworth, West Drayton, Middlesex, England, UB7 0GB. > > Additional terms and conditions are available on our website: > www.ba.com > > _______________________________________________ > Veritas-vx maillist - [email protected] > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx > > > -- > This message is private and confidential and may also be legally > privileged. If you have received this message in error, please > email it back to the sender and immediately permanently delete it > from your computer system. Please do not read, print, re-transmit, > store or act in reliance on it or any attachments. > > British Airways may monitor email traffic data and also the content > of emails, where permitted by law, for the purposes of security and > staff training and in order to prevent or detect unauthorised use > of the British Airways email system. > > Virus checking of emails (including attachments) is the > responsibility of the recipient. > > British Airways Plc is a public limited company registered in > England and Wales. Registered number: 1777777. Registered office: > Waterside, PO Box 365, Harmondsworth, West Drayton, Middlesex, > England, UB7 0GB. > > Additional terms and conditions are available on our website: > www.ba.com > > > > _______________________________________________ > Veritas-vx maillist - [email protected] > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx > > -- > Dmitry Glushenok > Jet Infosystems > > > > > ------------------------------------------------------------------------------------------------------------ > > Get the best from British Airways at ba.com > http://www.ba.com > -- > This message is private and confidential and may also be legally > privileged. If you have received this message in error, please email it > back to the sender and immediately permanently delete it from your computer > system. Please do not read, print, re-transmit, store or act in reliance > on it or any attachments. > > British Airways may monitor email traffic data and also the content of > emails, where permitted by law, for the purposes of security and staff > training and in order to prevent or detect unauthorised use of the British > Airways email system. > > Virus checking of emails (including attachments) is the responsibility of > the recipient. > > British Airways Plc is a public limited company registered in England and > Wales. Registered number: 1777777. Registered office: Waterside, PO Box > 365, Harmondsworth, West Drayton, Middlesex, England, UB7 0GB. > > Additional terms and conditions are available on our website: www.ba.com > >
_______________________________________________ Veritas-vx maillist - [email protected] http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
