On Wed, 26 Mar 2008 13:36:17 -0700, Skip Robinson <[EMAIL PROTECTED]> wrote:
>A comment on duplicate data. I haven't dug into the actual data produced by >IFASMFDL , but the doc says (in effect) that some duplicate records are >inevitable (my extrapolation) because the utility begins by dumping the >entire block that contains START time and continues through dumping the >entire block that contains END time. Thus some duplicate records are >unavoidable because on average the start and end time records are most >likely to fall somewhere within a block. > >However, we process data with MXG, which my SMF SME assures me has no >problem with duplicate data. > >. >. >JO.Skip Robinson >Southern California Edison Company >Electric Dragon Team Paddler >SHARE MVS Program Co-Manager >626-302-7535 Office >323-715-0595 Mobile >[EMAIL PROTECTED] > > > > Scott Barry > <[EMAIL PROTECTED] > COM> To > Sent by: IBM IBM-MAIN@bama.ua.edu > Mainframe cc > Discussion List > <[EMAIL PROTECTED] Subject > .edu> Re: SMF System Logger - limitations > of MANx > > 03/26/2008 11:02 > AM > > > Please respond to > IBM Mainframe > Discussion List > <[EMAIL PROTECTED] > .edu> > > > > > > >On Wed, 26 Mar 2008 09:38:10 -0700, Skip Robinson ><[EMAIL PROTECTED]> wrote: > >>We've had occasional episodes of lost data. >> > ><snip> > > >Two points for consideration: > >1) a possible technique for avoiding SMF data loss (such as SMF 101s or >116s -- you can decide an approach and types considered non-critical) is to > >setup an automation rule that fires when the SYSLOG message occurs, >stating "SMF is at 75% BUFFERS..." (message IEE986E). The SET SMF >command would then enable an alternate SMFPRMxx member deactivating >some subset of non-critical, large-volume contributor SMF record type(s), >such as SMF 101s. At some point, the condition is relieved and the normal >production SMFPRMxx member would be re-enabled in a similar fashion, or >after some defined time-period. > >2) the issue of duplicate SMF data occurring across IFASMFDL (SMF >Logstream enabled) dumps (see note #1 below) still does not get addressed, >without requiring a secondary data-filter pass (and/or sort with >noduplicates). Some would say that this duplicate-data condition is >unacceptable, given the extra data handling required. And it doesn't have >to >be linked to any accounting/chargeback scenario, either. > > >Scott Barry >SBBWorks, Inc. >_________________________ > >Note #1: most concerning is intraday dumping, for example, using CA MICS >Incremental Database Update feature -- although MICS will reject any data >already processed. > >Links: > >BUFSIZMAX, BUFUSEWARN, and NOBUFFS - Specifying SMF buffer options >(mind any broken URL): >http://publib.boulder.ibm.com/infocenter/zos/v1r9/topic/com.ibm.zos.r9.ieag2 0 > >0/buffopt.htm > >---------------------------------------------------------------------- >For IBM-MAIN subscribe / signoff / archive access instructions, >send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO >Search the archives at http://bama.ua.edu/archives/ibm-main.html For MXG data sources such as CICS, DB2, IMS (large-volume candidates and out-of-the-box MXG), there is no SORT NODUP operation performed during PDB build processing to remove adjacent duplicate records -- these PDB files are typically copied directly to their final destination, normally tape due to volume. Also, PDB build processing does not provide a check for "prior input data already processed - this record rejected". I do realize that SMF record types have header timestamps only granular to 1/100 of a second -- another challenge exacerbated by SMF System Logger deployment, because, again, dumping a MANx file represented a finite start and end point as compared to the SMF Syste Logger architecture providing a continuous data-pipe. Sincerely, Scott Barry SBBWorks, Inc. ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
