Re: IBM VSAM Statistics are often Bogus

[Ed Finnell]

 
In a message dated 7/14/2005 11:33:27 P.M. Central Standard Time,  
[EMAIL PROTECTED] writes:

Speaking  about M$ did you see where they are attempting to get into the 
storage  business? Talk about scary.



>>
wouldn't be too worried, after they claim the patents on all
known recording techniques, RAIDs, and SAN architecture we'll
be moved over to quantum computers by then.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Gould]

On Jul 14, 2005, at 9:36 AM, Ed Finnell wrote:



In a message dated 7/13/2005 8:35:44 P.M. Central Standard Time,
[EMAIL PROTECTED] writes:

Unfortunately, we are!






I though M$ had the patent on this delivery  model...


Speaking about M$ did you see where they are attempting to get into the 
storage business? Talk about scary.


Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Finnell]

 
In a message dated 7/13/2005 8:35:44 P.M. Central Standard Time,  
[EMAIL PROTECTED] writes:

Unfortunately, we are!



>>
I though M$ had the patent on this delivery  model...

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ted MacNEIL]

...
"Holy spaghetti Batman, who's testing these  things?"
...
Unfortunately, we are!
-teD

In God we Trust!
All others bring data!
  -- W. Edwards Deming

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Martin Kline]

> "Holy spaghetti Batman, who's testing these  things?"

Probably went to the lowest bidder.


CONFIDENTIALITY NOTICE:  This electronic transmission (including any
accompanying attachments) is intended solely for its authorized
recipient(s), and may contain confidential and/or legally privileged
information.  If you are not an intended recipient, or responsible for
delivering some or all of this transmission to an intended recipient, be
aware that any review, copying, printing, distribution, use or disclosure of
the contents of this message is strictly prohibited.  If you have received
this electronic message in error, please contact us immediately by
electronic mail at [EMAIL PROTECTED] or notify us
immediately by telephone at 1-800-345-2021 or 816-531-5575 and destroy the
original and all copies of this transmission (including any attachments).
Thank you.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Finnell]

 
In a message dated 7/13/2005 3:41:56 P.M. Central Standard Time,  
[EMAIL PROTECTED] writes:

the  conditions at the 1.4 and 1.5 levels (1G0 and 1H0) and reported on
validity  in the opposite way. Good is bad; bad is good. A philosophical and
ethical  conundrum if ever there was one. The 1.6 (1J0) level is  OK,
however.



"Holy spaghetti Batman, who's testing these  things?"

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Tim Hare]

SO, Skip,   are you saying 1.6 is bad ("good is bad, bad is good") or 1.6 
is good? But then, I just used "bad" when I meant "bad" but "bad is good" 
so did I mean "good"? 
I certainly meant well. 

Tim Hare
Senior Systems Programmer
Florida Department of Transportation
(850) 414-4209

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Skip Robinson]

Please forgive me if this is old news. OA11927 was PE'ed on 7/11 . This is
the guy that set out to 'restore' the bogus stats in LISTCAT so that
programs expecting numeric data would not blow up S0C7; the stats would be
marked invalid in some other field. Unfortunately, OA11927 somehow mixed up
the conditions at the 1.4 and 1.5 levels (1G0 and 1H0) and reported on
validity in the opposite way. Good is bad; bad is good. A philosophical and
ethical conundrum if ever there was one. The 1.6 (1J0) level is OK,
however.

.
.
.
JO.Skip Robinson
Southern California Edison Company
SHARE MVS Program Co-Manager
626-302-7535 Office
323-715-0595 Mobile
[EMAIL PROTECTED]

IBM Mainframe Discussion List  wrote on 07/02/2005
09:59:57:


>
> Anyhow, I only learned of OA11927 in this thread. Good decision. I
haven't
> seen the modified output but will install the fix ASAP. Sure hope I don't
> have to beat up on Serena again right after they finally fixed the
INVALID
> problem. ;-)
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: FW: IBM VSAM Statistics are often Bogus

[Shmuel Metz (Seymour J.)]

In <[EMAIL PROTECTED]>, on 07/08/2005
   at 10:37 PM, "Joel C. Ewing" <[EMAIL PROTECTED]> said:

>LISTCAT stats have been around for at least several decades, maybe
>since  the beginnings of OS/360. 

There was no AMS, LISTCAT or VSAM in OS/36 0. VSAM came in as an ICR
for OS/VS1 and OS/VS2 R1 (SVS).
 
-- 
 Shmuel (Seymour J.) Metz, SysProg and JOAT
 ISO position; see  
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Farley, Peter x23353]

Really, now, that's stretching what I said a bit far.  I only said
application programmers were responsible for care and feeding -- I did *not*
say that they actually touched the files themselves.  Authorization to
request that Operations run an On-Request reorg job is not a security hole,
especially if managerial approval falls between request and reorg.

The duty and responsibility to review and recommend are not the same as the
authority to corrupt.

Peter

-Original Message-
From: Shmuel Metz (Seymour J.) [mailto:[EMAIL PROTECTED]
Sent: Sunday, July 10, 2005 12:38 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: IBM VSAM Statistics are often Bogus


In <[EMAIL PROTECTED]>,
on 07/06/2005
   at 10:32 AM, "Farley, Peter x23353" <[EMAIL PROTECTED]> said:

>I never even envisioned automated tools looking at VSAM stats.  My
>ASSumption when reading Mark's posts was that he was referring to
>individual programmers looking at individual VSAM file stats for
>guidance.  My experience is obviously severely limited in this
>regard, as in my varied positions over the years the usual case was
>that all of the applications' datasets (including VSAM) were our (the
>application programmers') responsibility to feed and care for, and we
>never had thousands to contend with.

The application programmers had access to the production data sets?
There are security ramifications there.

_
This message and any attachments are intended only for the use of the addressee 
and
may contain information that is privileged and confidential. If the reader of 
the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: FW: IBM VSAM Statistics are often Bogus

[Ron and Jenny Hawkins]

Joel,

> 
> If those 20 minutes of stats show enough CI splits to justify a
> reorganize of the file, it doesn't matter that this only represents 20
> minutes of activity.  If the CI-split count is above your reorg
> threshold, it also doesn't matter if you lost 15 hours of CI-split
> counts because of a CICS crash, a reorg is still justified.  You may be
> late with a decision to reorganize, but better late than never.  With no
> stats at all, an automated process has no basis for choosing whether to
> reorganize or not, and arbitrarily choosing to reorganize could be a bad
> choice for a large, rarely-updated VSAM file - especially if doing an
> unnecessary  reorganize delays the availability of a critical online
> application.
> >
> 
I come from the Ron F. school of VSAM tuning... Highly likely that the high
split count in that last 20 minutes of batch has happened because the
dataset was reorged (by the restore). Refer to Ron's posts and books for
more info.

> LISTCAT stats have been around for at least several decades, maybe since
> the beginnings of OS/360.  Automated maintenance procedures and
> heuristic guidelines have been developed based on LISTCAT statistics and
> we have become dependent on them, because you build tools based on what
> is available, even if not perfect.  Without LISTCAT stats, we would have
> to expend resources to rethink and redesign a number of automated batch
> processes that have been working reasonable well up to now (z/OS 1.4).
> 

I've used LISTCAT to tune VSAM CI and freespace, but I put the listcat in a
step before and after the updates step. It is really much easier to use type
64 SMF records.

> ...
> 
> --
> Joel C. Ewing, Fort Smith, AR[EMAIL PROTECTED]
> 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Shmuel Metz (Seymour J.)]

In <[EMAIL PROTECTED]>,
on 07/06/2005
   at 10:32 AM, "Farley, Peter x23353" <[EMAIL PROTECTED]> said:

>I never even envisioned automated tools looking at VSAM stats.  My
>ASSumption when reading Mark's posts was that he was referring to
>individual programmers looking at individual VSAM file stats for
>guidance.  My experience is obviously severely limited in this
>regard, as in my varied positions over the years the usual case was
>that all of the applications' datasets (including VSAM) were our (the
>application programmers') responsibility to feed and care for, and we
>never had thousands to contend with.

The application programmers had access to the production data sets?
There are security ramifications there.
 
-- 
 Shmuel (Seymour J.) Metz, SysProg and JOAT
 ISO position; see  
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Shmuel Metz (Seymour J.)]

In <[EMAIL PROTECTED]>,
on 07/06/2005
   at 05:29 PM, Donald Pagdin <[EMAIL PROTECTED]> said:

>So it is assembler that can bite the unwary yet diligent programmer.

That's the theory, but there seem to be more problems with buffer
overruns in C programs than with other languages, including assembler.
 
-- 
 Shmuel (Seymour J.) Metz, SysProg and JOAT
 ISO position; see  
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Shmuel Metz (Seymour J.)]

In <[EMAIL PROTECTED]>,
on 07/06/2005
   at 12:00 AM, Ted MacNEIL <[EMAIL PROTECTED]> said:

>Open/Close processing is not cheap.
>Also, if you have (partial-)release on close, you can easily create a
>multi-(tiny)extent file in no time.

No. TCLOSE is not CLOSE. The data set remains open and there is no
release. 
 
-- 
 Shmuel (Seymour J.) Metz, SysProg and JOAT
 ISO position; see  
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Robert A. Rosenberg]

At 08:22 -0700 on 07/07/2005, Mark Thomen wrote about Re: IBM VSAM 
Statistics are often Bogus:



"Robert A. Rosenberg" <[EMAIL PROTECTED]> wrote in message
news:<[EMAIL PROTECTED]>...

 > Since CEMT P SHUT I causes a CONTROLLED issuance of an Abnormal
 > Termination, there is no reason why a request to flush the statistics

 BEFORE doing the Abnormal Termination can't be done.


Since once the ABEND is issued it's too late, it would be up to the
application (i.e. CICS) to "flush" the statistics by closing the files
before the ABEND is issued.


Which is what I was saying. CICS is running normally and the operator 
issues the  "CEMT P SHUT I" request/order. CICS accepts the request 
and can process it however it wants including flushing the stats 
BEFORE pulling the plug by doing the ABEND/Abnormal Termination. It 
is not as if the operator aborted CICS (Force Cancel/etc) instead of 
asking it to do the abort itself in an orderly manner.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Robert A. Rosenberg]

At 06:49 -0500 on 07/07/2005, Chase, John wrote about Re: IBM VSAM 
Statistics are often Bogus:



Aren't they the same IBM lab that gave us IGDZILLA?


If there was a IMG prefix, they could do IMGOJIRA ("I'm GOJIRA") - 
Gojira being the real name of the Japanese Monster called Godzilla in 
the US translations. In fact the Gojira/Godzilla issue was addressed 
in the US made Godzilla movie where the monster was originally called 
Gojira but was renamed by a TV Reporter mangling the name into 
Godzilla.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: FW: IBM VSAM Statistics are often Bogus

[Joel C. Ewing]

Ron and Jenny Hawkins wrote:

Tom, et al,

I was just thinking... CICS aside, when a job ABENDS while updating a KSDS
file, what is the most common thing that happens to allow the job to be
rerun? Delete and restore the KSDS, right? So what good are the accumulated
statistics at this point - valid or invalid? Well there worth zilcho,
because they were just deleted. Gone forever.


The ABENDing job steps can also use VSAM datasets READ only, in which 
case the stats will be inaccurate but no restore is needed.  Even in 
case of an updated file, it can depend on the application and nature of 
updates whether a restore is needed or not.


And the next day someone takes a LISTCAT of that file and makes some
decision based on those stats, assuming that it represents 24 hours of
activity, when it is really just the last 20 minutes of the batch run. Smart
move!


If those 20 minutes of stats show enough CI splits to justify a 
reorganize of the file, it doesn't matter that this only represents 20 
minutes of activity.  If the CI-split count is above your reorg 
threshold, it also doesn't matter if you lost 15 hours of CI-split 
counts because of a CICS crash, a reorg is still justified.  You may be 
late with a decision to reorganize, but better late than never.  With no 
stats at all, an automated process has no basis for choosing whether to 
reorganize or not, and arbitrarily choosing to reorganize could be a bad 
choice for a large, rarely-updated VSAM file - especially if doing an 
unnecessary  reorganize delays the availability of a critical online 
application.


I would rather we just got rid of all the stats from LISTCAT. We don't have
a listdb2, listdl1, listfastpath or listqsam, so why the religious fervour
around LISTC stats?



Ron



LISTCAT stats have been around for at least several decades, maybe since 
the beginnings of OS/360.  Automated maintenance procedures and 
heuristic guidelines have been developed based on LISTCAT statistics and 
we have become dependent on them, because you build tools based on what 
is available, even if not perfect.  Without LISTCAT stats, we would have 
to expend resources to rethink and redesign a number of automated batch 
processes that have been working reasonable well up to now (z/OS 1.4).


...

--
Joel C. Ewing, Fort Smith, AR[EMAIL PROTECTED]

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Gould]

On Jul 8, 2005, at 4:04 PM, Bruce Black wrote:


Job is updating the vsam cluster. Job abends.
Job is checkpoint restarted.
job completes (with closes to the dataset in question)

Are the stats now correct?



If by statistics, you mean those things listed under STATISTICS in a 
LISTCAT


STATISTICS
  REC-TOTAL--6 SPLITS-CI--0 
EXCPS-10REC-DELETED3 
SPLITS-CA--0 EXTENTS1
REC-INSERTED---7 FREESPACE-%CI--0 
SYSTEM-TIMESTAMP:   REC-UPDATED0 
FREESPACE-%CA--0  X'BC34491B1E5CB160'
REC-RETRIEVED-15 FREESPC---733184
the answer is probably no.  These numbers are updated at the time of a 
good CLOSE and the updates are lost on an ABEND CLOSE or system 
failure.  I don't believe that checkpoint saves the VSAM control 
blocks (AMDSB, I think) with this info.

--



Bruce,

Hmmm...I would think yes as IBM keeps said that they reason they can't 
clean up properly is that the vsam control blocks are in key 8 (user 
storage). Checkpoint restart should (at least I think it should) put 
the control blocks back as the job needs to continue.


IBM, anyone?

Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Bruce Black]

Job is updating the vsam cluster. Job abends.
Job is checkpoint restarted.
job completes (with closes to the dataset in question)

Are the stats now correct? 



If by statistics, you mean those things listed under STATISTICS in a LISTCAT

STATISTICS 

  REC-TOTAL--6 SPLITS-CI--0 
EXCPS-10  
  REC-DELETED3 SPLITS-CA--0 
EXTENTS1  
  REC-INSERTED---7 FREESPACE-%CI--0 
SYSTEM-TIMESTAMP: 
  REC-UPDATED0 FREESPACE-%CA--0  
X'BC34491B1E5CB160'  
  REC-RETRIEVED-15 FREESPC---733184

the answer is probably no.  These numbers are updated at the time of a 
good CLOSE and the updates are lost on an ABEND CLOSE or system 
failure.  I don't believe that checkpoint saves the VSAM control blocks 
(AMDSB, I think) with this info.   


--
Bruce A. Black
Senior Software Developer for FDR
Innovation Data Processing 973-890-7300
personal: [EMAIL PROTECTED]
sales info: [EMAIL PROTECTED]
tech support: [EMAIL PROTECTED]
web: www.innovationdp.fdr.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Martin Kline]

Ed asked:

>Which brings up a question (at least in my mind).

>Job is updating the vsam cluster. Job abends.
>Job is checkpoint restarted.
>job completes (with closes to the dataset in question)

>Are the stats now correct?

Since they are unreliable, it does not matter.


CONFIDENTIALITY NOTICE:  This electronic transmission (including any
accompanying attachments) is intended solely for its authorized
recipient(s), and may contain confidential and/or legally privileged
information.  If you are not an intended recipient, or responsible for
delivering some or all of this transmission to an intended recipient, be
aware that any review, copying, printing, distribution, use or disclosure of
the contents of this message is strictly prohibited.  If you have received
this electronic message in error, please contact us immediately by
electronic mail at [EMAIL PROTECTED] or notify us
immediately by telephone at 1-800-345-2021 or 816-531-5575 and destroy the
original and all copies of this transmission (including any attachments).
Thank you.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Gould]

On Jul 8, 2005, at 2:42 AM, R.S. wrote:


So, we come into philosofical considerations.
After VSAM restore (bacause of job failure) we have reorganized 
cluster. Assumed logical backup/restore.
BTW: We say 'cluster', but the considerations are mostly for KSDS or 
vRRDS. There are no splits in ESDS, LDS, or RRDS.

Ab ovo: What stats are for just-restored cluster ?
I guess that "original" - from the time of backup.
Second guess - zeroized, like for just defined cluster.

BOTH statistics are incorrect, when we want to decide about 
reorganization. As Ron F. pointed recently, the numbers are important 
when we know WHEN the splits occured, what is split distribution, I/O 
trends etc.
BTW: The last one is possible to collect - it is enough to write down 
the statistics before after every close. I mena some historical file. 
Just additional step using CSI or something.


--
Radoslaw Skorupka
Lodz, Poland


Which brings up a question (at least in my mind).

Job is updating the vsam cluster. Job abends.
Job is checkpoint restarted.
job completes (with closes to the dataset in question)

Are the stats now correct?

Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: FW: IBM VSAM Statistics are often Bogus

[R.S.]

So, we come into philosofical considerations.
After VSAM restore (bacause of job failure) we have reorganized cluster. 
Assumed logical backup/restore.
BTW: We say 'cluster', but the considerations are mostly for KSDS or 
vRRDS. There are no splits in ESDS, LDS, or RRDS.

Ab ovo: What stats are for just-restored cluster ?
I guess that "original" - from the time of backup.
Second guess - zeroized, like for just defined cluster.

BOTH statistics are incorrect, when we want to decide about 
reorganization. As Ron F. pointed recently, the numbers are important 
when we know WHEN the splits occured, what is split distribution, I/O 
trends etc.
BTW: The last one is possible to collect - it is enough to write down 
the statistics before after every close. I mena some historical file. 
Just additional step using CSI or something.


--
Radoslaw Skorupka
Lodz, Poland


Ron and Jenny Hawkins wrote:

Radoslaw,

As far as I recall it uses Export for Catalogs and REPRO for other VSAM.

Ron



Unless it was physical dump. It depends on backup technique and datamover.
BTW: is it REPRO or EXPORT ?



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: FW: IBM VSAM Statistics are often Bogus

[Ron and Jenny Hawkins]

Radoslaw,

As far as I recall it uses Export for Catalogs and REPRO for other VSAM.

Ron

> 
> Unless it was physical dump. It depends on backup technique and datamover.
> BTW: is it REPRO or EXPORT ?
> 
> --
> Radoslaw Skorupka
> Lodz, Poland
> 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Craddock, Chris]

Geez is this thread ever going to die? It has not edged 
towards Jaffe's conjecture yet, so it might keep limping
along. Darren could kill it at any second (if we're lucky)

>  I do want to thank Mark for his open and
> candid responses, and that he (at least) has tried to do 
> something about this long standing issue.

Mark's a stand-up dude and he's doing the best he can to
undo decades of neglect. We can all cut him some slack.
FWIW and IMO only, IBM could ditch the stats altogether and 
nobody would miss that function. There is just too much 
compelling evidence that their usage cases are even more 
wrong than the statistics themselves are. 

> I really don't have an issue with IBM either.  It has just 
> floored me to find out that they have uncharacteristically
> left a problem - and a very visual one at that - remain 
> without resolution for such an extended period of time.

My my my. If you only knew how many of such problems there
were... On second thought, lets not go there today.

:o)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Martin Kline]

> You wouldn't want to be doing a large number of TCLOSEs per run... on
>the order of 1's or maybe 10's but not 100's and certainly not 1000's.
>You'll be swatting a gnat with a nuclear device in short order with
>this approach.  (These statistics just aren't that important.)

I agree to doing a relative few TCLOSEs. 1 per 100,000 gets/puts is
probably sufficient.

Are the statistics ever important?

There's always an exception to the rule (except when there's not.)

Case in point - Batch job using 100 big VSAM files.  Code uses common
I/O module for all files. After executing for 16 hours, user declares
this is a problem. Problem has something to do with buffers and
numbers of sequential and random reads, inserts, deletes. The stats
for this problem are critical to fixing it. Unfortunately, they are
not recorded in the catalog, and are also missing from the type-64
records. Which file needs more buffers? Which files received updates
and now need to be restored before restarting the job? Which files
have the most activity? What kind of activity?

The current approach (without TCLOSE) will have to be to just
restore all files and rerun the job to let it complete. Only then
can someone examine the statistics.


CONFIDENTIALITY NOTICE:  This electronic transmission (including any
accompanying attachments) is intended solely for its authorized
recipient(s), and may contain confidential and/or legally privileged
information.  If you are not an intended recipient, or responsible for
delivering some or all of this transmission to an intended recipient, be
aware that any review, copying, printing, distribution, use or disclosure of
the contents of this message is strictly prohibited.  If you have received
this electronic message in error, please contact us immediately by
electronic mail at [EMAIL PROTECTED] or notify us
immediately by telephone at 1-800-345-2021 or 816-531-5575 and destroy the
original and all copies of this transmission (including any attachments).
Thank you.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Tom Schmidt]

On Thu, 7 Jul 2005 14:30:16 -0500, Martin Kline wrote:

>I believe John suggested using CLOSE TYPE=T to get a user equivalent of
>interval recording. This should update the catalog, and generate a type-64
>SMF record with bit 4 of SMF64RIN set.  My question is whether multiple
>such type-64 records have to be manually accumulated.  In other words, if I
>generate 20 such records for a single file, does each one include the
>statistics from the previous record in the "before OPEN" or in
>the "Change" section?


In the case of CLOSE TYPE=T (TCLOSE) I believe you should see multiple type
64 records with only one type 62 (VSAM Open) SMF record, and that type 62
will be (naturally) created before any of the type 64s.  If that's true you
would discard any but the last type 64 records' data.

(Someone might want to verify that first though.)

You wouldn't want to be doing a large number of TCLOSEs per run... on the
order of 1's or maybe 10's but not 100's and certainly not 1000's.  You'll
be swatting a gnat with a nuclear device in short order with this
approach.  (These statistics just aren't that important.)

--
Tom Schmidt
Madison, WI

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Martin Kline]

I believe John suggested using CLOSE TYPE=T to get a user equivalent of
interval recording. This should update the catalog, and generate a type-64
SMF record with bit 4 of SMF64RIN set.  My question is whether multiple
such
type-64 records have to be manually accumulated.  In other words, if I
generate 20 such records for a single file, does each one include the
statistics
from the previous record in the "before OPEN" or in the "Change" section?


CONFIDENTIALITY NOTICE:  This electronic transmission (including any
accompanying attachments) is intended solely for its authorized
recipient(s), and may contain confidential and/or legally privileged
information.  If you are not an intended recipient, or responsible for
delivering some or all of this transmission to an intended recipient, be
aware that any review, copying, printing, distribution, use or disclosure of
the contents of this message is strictly prohibited.  If you have received
this electronic message in error, please contact us immediately by
electronic mail at [EMAIL PROTECTED] or notify us
immediately by telephone at 1-800-345-2021 or 816-531-5575 and destroy the
original and all copies of this transmission (including any attachments).
Thank you.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Arsenault, Randy]

There is a product from Mainstar Software that we used many years ago
called VSAM Manager.

I checked their web site - www.mainstar.com and it is still listed with
the following description:

 
VSAM Manager analyzes logical record to CI relationships - checking for
potential efficiency problems, requested and achieved distributed
FREESPACE within CIs and CAs, and DASD space utilization of the data
set. It also provides information that identifies where record insertion
activity is occurring within the file - showing where CA splits have
been done - and how many are at each location. 


They had a sense of humour as well as every report carried a message at
the end something like:

THANK YOU FOR USING VSAM MANAGER. AS WE SAY IN THE BUSINESS, 'KEEP 'EM
SPLITTING'

-
Randy Arsenault
Enterprise Performance & Capacity Management
Hewlett-Packard (Canada) Co.
e-mail: [EMAIL PROTECTED]
--
For IBM-MAIN subscribe / signoff / archive access instructions, send
email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search
the archives at http://bama.ua.edu/archives/ibm-main.html

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Gould]

--SNIP---



If he's talking about VSAMAID from CA, the manual states:


Once installation has been completed, it is necessary to perform a
statistics collection run for the VSAM clusters in batch. Although this
process has been optimized to execute as quickly as possible, the
elapsed time depends on the number of clusters selected and how many
records each one contains. Each cluster is read in its entirety.





John,

Thanks that is interesting.

Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Dave Juraschek]

Tom said:
>I commend Mark for taking the issue in hand and trying to implement a long
>term solution, and for raising it to the level where it is discussed so
>that the appropriate awareness is reached, but Mark doesn't need my
>commendation for job satisfaction.
>
>Say what you want about IBM but its people are still among the best
>architects of software around.  This issue is convoluted at best.

Just, for the record, and knowing (now) that Mark has only been in this
arena a relatively short time, I do want to thank Mark for his open and
candid responses, and that he (at least) has tried to do something about
this long standing issue.

I really don't have an issue with IBM either.  It has just floored me to
find out that they have uncharacteristically left a problem - and a very
visual one at that - remain without resolution for such an extended period
of time.

Thanks.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[McKown, John]

> -Original Message-
> From: IBM Mainframe Discussion List 
> [mailto:[EMAIL PROTECTED] On Behalf Of Ed Gould
> Sent: Thursday, July 07, 2005 11:36 AM
> To: IBM-MAIN@BAMA.UA.EDU
> Subject: Re: IBM VSAM Statistics are often Bogus
> 
> 



> Bruce,
> 
> Not sure it was either... But getting back to the issue on incorrect 
> stats... does it use the stats in the catalog or does it 
> "examine" the 
> vsam dataset to make the recommendation?
> 
> Ed

If he's talking about VSAMAID from CA, the manual states:


Once installation has been completed, it is necessary to perform a
statistics collection run for the VSAM clusters in batch. Although this
process has been optimized to execute as quickly as possible, the
elapsed time depends on the number of clusters selected and how many
records each one contains. Each cluster is read in its entirety.


--
John McKown
Senior Systems Programmer
UICI Insurance Center
Information Technology

This message (including any attachments) contains confidential
information intended for a specific individual and purpose, and its'
content is protected by law.  If you are not the intended recipient, you
should delete this message and are hereby notified that any disclosure,
copying, or distribution of this transmission, or taking any action
based on it, is strictly prohibited.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Gould]

On Jul 7, 2005, at 8:39 AM, Bruce Black wrote:

Somewhere in the back of my paged out (permanently?) memory is there 
used to be (or there is one currently) a utility that went through 
the catalog and looked for vsam datasets and listed any  that 
according to its recommendation ones that needed
 re-orging. My mind just can't remember the name. It may have built 
idcams control cards as well.


Can anyone come up with the name of the product?


Ed, I don't know if this is what you were thinking of, but our product 
FDRREORG does exactly that for VSAM, IAM, and PDS datasets.  You 
provide the criteria and we select and reorg the datasets.  Doesn't 
use IDCAMS.





Bruce,

Not sure it was either... But getting back to the issue on incorrect 
stats... does it use the stats in the catalog or does it "examine" the 
vsam dataset to make the recommendation?


Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Mark Thomen]

"Robert A. Rosenberg" <[EMAIL PROTECTED]> wrote in message
news:<[EMAIL PROTECTED]>...
> At 07:36 -0500 on 07/06/2005, Chase, John wrote about Re: IBM VSAM
> Statistics are often Bogus:
>
> >The fact remains that CEMT P SHUT I causes an ABNORMAL termination of
CICS,
> >as would the MVS CANCEL or FORCE command.  The fact also remains that on
a
> >commanded ABNORMAL termination of (anything), VSAM cannot update the
dataset
> >statistics because the issuer of the command has INTENTIONALLY prevented
it
> >from doing so.
>
> Since CEMT P SHUT I causes a CONTROLLED issuance of an Abnormal
> Termination, there is no reason why a request to flush the statistics
> BEFORE doing the Abnormal Termination can't be done.

Since once the ABEND is issued it's too late, it would be up to the
application (i.e. CICS) to "flush" the statistics by closing the files
before the ABEND is issued.

However most people use immediate shutdown because they don't WANT to wait
for the files to get closed normally.  Their choice.


Thanks,
Mark Thomen
Catalog/IDCAMS/VSAM Development

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ray Mullins]

Don't overtax yourself looking for it.  :-)

It's the company where Herr Schiradin works.  But, for a translation - Alte
- old - Leipziger - of the city of Leipzig.  (It's an insurance company,
located near Frankfurt.)

Later,
Ray

-- 
M. Ray Mullins 
Roseville, CA, USA 
http://www.catherdersoftware.com/
http://www.mrmullins.big-bear-city.ca.us/ 
http://www.the-bus-stops-here.org/ 

 

> -Original Message-
> From: IBM Mainframe Discussion List 
> [mailto:[EMAIL PROTECTED] On Behalf Of Ted MacNEIL
> Sent: Tuesday 05 July 2005 17:00
> To: IBM-MAIN@BAMA.UA.EDU
> Subject: Re: IBM VSAM Statistics are often Bogus
> 
> ...
> As for Alte-Leipziger I can say "it's not issue for us".
> ...
> 
> I couldn't find the meaning of that compound word.
> 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Compton, John]

...& Macro-4 used to do one called VSAMTUNE, didn't they? (tho' it might've
been only for use on VSE...)

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf
Of McKown, John
Sent: 07 July 2005 14:18
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: IBM VSAM Statistics are often Bogus

> -Original Message-
> From: IBM Mainframe Discussion List 
> [mailto:[EMAIL PROTECTED] On Behalf Of Ed Gould
> Sent: Wednesday, July 06, 2005 9:47 PM
> To: IBM-MAIN@BAMA.UA.EDU
> Subject: Re: IBM VSAM Statistics are often Bogus
> 
> 



> 
> Peter,
> 
> Somewhere in the back of my paged out (permanently?) memory is there 
> used to be (or there is one currently) a utility that went 
> through the 
> catalog and looked for vsam datasets and listed any  that 
> according to 
> its recommendation ones that needed
>   re-orging. My mind just can't remember the name. It may have built 
> idcams control cards as well.
> 
> Can anyone come up with the name of the product?
> 
> Ed

I don't know if it is what you're speaking of, but we have a CA product
called VSAMAID which does that.


--
John McKown
Senior Systems Programmer
UICI Insurance Center
Information Technology

This message (including any attachments) contains confidential
information intended for a specific individual and purpose, and its'
content is protected by law.  If you are not the intended recipient, you
should delete this message and are hereby notified that any disclosure,
copying, or distribution of this transmission, or taking any action
based on it, is strictly prohibited.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Bruce Black]

Somewhere in the back of my paged out (permanently?) memory is there 
used to be (or there is one currently) a utility that went through the 
catalog and looked for vsam datasets and listed any  that according to 
its recommendation ones that needed
 re-orging. My mind just can't remember the name. It may have built 
idcams control cards as well.


Can anyone come up with the name of the product? 


Ed, I don't know if this is what you were thinking of, but our product 
FDRREORG does exactly that for VSAM, IAM, and PDS datasets.  You provide 
the criteria and we select and reorg the datasets.  Doesn't use IDCAMS.


--
Bruce A. Black
Senior Software Developer for FDR
Innovation Data Processing 973-890-7300
personal: [EMAIL PROTECTED]
sales info: [EMAIL PROTECTED]
tech support: [EMAIL PROTECTED]
web: www.innovationdp.fdr.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Bruce Black]

The programme prefix for dfpSMS is IGD.

Years ago, either the main module or an alias was IGDZILLA.

“I, God-Zilla”.


Still true.  Back when SMS was new, the lmod size was about 1M, making
it the largest module in LPA.  Today the size is about 2.3M, still the
largest module in LPA by a goodly factor.  We once had a presentation
slide with a title of SMS and a picture of you know who.

I don't know for sure but I suspect the name was a developer's in-joke
that was probably intended to be changed before it got released.

--
Bruce A. Black
Senior Software Developer for FDR
Innovation Data Processing 973-890-7300
personal: [EMAIL PROTECTED]
sales info: [EMAIL PROTECTED]
tech support: [EMAIL PROTECTED]
web: www.innovationdp.fdr.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[McKown, John]

> -Original Message-
> From: IBM Mainframe Discussion List 
> [mailto:[EMAIL PROTECTED] On Behalf Of Ed Gould
> Sent: Wednesday, July 06, 2005 9:47 PM
> To: IBM-MAIN@BAMA.UA.EDU
> Subject: Re: IBM VSAM Statistics are often Bogus
> 
> 



> 
> Peter,
> 
> Somewhere in the back of my paged out (permanently?) memory is there 
> used to be (or there is one currently) a utility that went 
> through the 
> catalog and looked for vsam datasets and listed any  that 
> according to 
> its recommendation ones that needed
>   re-orging. My mind just can't remember the name. It may have built 
> idcams control cards as well.
> 
> Can anyone come up with the name of the product?
> 
> Ed

I don't know if it is what you're speaking of, but we have a CA product
called VSAMAID which does that.


--
John McKown
Senior Systems Programmer
UICI Insurance Center
Information Technology

This message (including any attachments) contains confidential
information intended for a specific individual and purpose, and its'
content is protected by law.  If you are not the intended recipient, you
should delete this message and are hereby notified that any disclosure,
copying, or distribution of this transmission, or taking any action
based on it, is strictly prohibited.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Chase, John]

> -Original Message-
> From: IBM Mainframe Discussion List On Behalf Of Ron and Jenny Hawkins
> 
> Tom, et al,
> 
> I was just thinking... CICS aside, when a job ABENDS while 
> updating a KSDS file, what is the most common thing that 
> happens to allow the job to be rerun? Delete and restore the 
> KSDS, right? So what good are the accumulated statistics at 
> this point - valid or invalid? Well there worth zilcho, 
> because they were just deleted. Gone forever.
> 
> And the next day someone takes a LISTCAT of that file and 
> makes some decision based on those stats, assuming that it 
> represents 24 hours of activity, when it is really just the 
> last 20 minutes of the batch run. Smart move!
> 
> I would rather we just got rid of all the stats from LISTCAT. 
> We don't have a listdb2, listdl1, listfastpath or listqsam, 
> so why the religious fervour around LISTC stats?

"Hysterical" reasons.  :-)

With today's DASD being almost completely virtualized, it would seem that
the only relevant stats are space used (i.e., number of records,
HARBA/HURBA, and for KSDSes number of index levels and records).  For most
application-oriented intents and purposes, the other stats are little more
than "tourist info".

-jc-

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: FW: IBM VSAM Statistics are often Bogus

[R.S.]

Ron and Jenny Hawkins wrote:


Radoslaw,

Now there's an interesting point. If I recall correctly a Logical Dataset
Restore uses IDCAMS REPRO under DFDSS, so what you get is a reorged KSDS
with stats that say it isn't.


Unless it was physical dump. It depends on backup technique and datamover.
BTW: is it REPRO or EXPORT ?

--
Radoslaw Skorupka
Lodz, Poland

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Chase, John]

> -Original Message-
> From: IBM Mainframe Discussion List On Behalf Of Ted MacNEIL
> 
> ...
> 1) One can always do one's own interval accounting by opening 
> and closing the file periodically.
> ...
> 
> I hope you are joking!

That would be somewhat like sending separate 18-wheelers to deliver
individual pencils.

-jc-

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Chase, John]

> -Original Message-
> From: IBM Mainframe Discussion List On Behalf Of Edward E. Jaffe
> 
> Mark Thomen wrote:
> 
> >That's what module IFG0TC0A ("IFG-gotcha") does.
> 
> I've always called it "I-F-Gotcha!" or just "I've Gotcha!"

And VSE VSAM gives us IKQVDU (IKQ Voodoo).

-jc-

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Chase, John]

> -Original Message-
> From: IBM Mainframe Discussion List On Behalf Of Bruce Black
> 
> >That's what module IFG0TC0A ("IFG-gotcha") does.
> >
> Who said IBM doesn't have a sense of humor?   Thanks,Mark, 
> that gave me the giggles.

Aren't they the same IBM lab that gave us IGDZILLA?

And don't forget the Dilbert fans at JES2 ($DOGBERT, et al).

-jc-

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: FW: IBM VSAM Statistics are often Bogus

[Ron and Jenny Hawkins]

Radoslaw,

Now there's an interesting point. If I recall correctly a Logical Dataset
Restore uses IDCAMS REPRO under DFDSS, so what you get is a reorged KSDS
with stats that say it isn't.

Ron

> -Original Message-
> From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
> Behalf Of R.S.
> Sent: Thursday, 7 July 2005 6:40 PM
> To: IBM-MAIN@BAMA.UA.EDU
> Subject: Re: FW: IBM VSAM Statistics are often Bogus
> 
> Ron,
> Please, correct me if I'm wrong.
> Job fails, so cluster must be deleted and restored from backup. Am I
> right ?
> So, IMHO restore also restores VSAM stats, doesn't it ?
> I mean restore, not REPRO or IMPORT.
> 
> --
> Radoslaw Skorupka
> Lodz, Poland
> 
> 
> 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: FW: IBM VSAM Statistics are often Bogus

[R.S.]

Ron,
Please, correct me if I'm wrong.
Job fails, so cluster must be deleted and restored from backup. Am I 
right ?

So, IMHO restore also restores VSAM stats, doesn't it ?
I mean restore, not REPRO or IMPORT.

--
Radoslaw Skorupka
Lodz, Poland



Ron and Jenny Hawkins wrote:


Tom, et al,

I was just thinking... CICS aside, when a job ABENDS while updating a KSDS
file, what is the most common thing that happens to allow the job to be
rerun? Delete and restore the KSDS, right? So what good are the accumulated
statistics at this point - valid or invalid? Well there worth zilcho,
because they were just deleted. Gone forever.

And the next day someone takes a LISTCAT of that file and makes some
decision based on those stats, assuming that it represents 24 hours of
activity, when it is really just the last 20 minutes of the batch run. Smart
move!

I would rather we just got rid of all the stats from LISTCAT. We don't have
a listdb2, listdl1, listfastpath or listqsam, so why the religious fervour
around LISTC stats?

Ron


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Robert A. Rosenberg]

At 07:36 -0500 on 07/06/2005, Chase, John wrote about Re: IBM VSAM 
Statistics are often Bogus:



The fact remains that CEMT P SHUT I causes an ABNORMAL termination of CICS,
as would the MVS CANCEL or FORCE command.  The fact also remains that on a
commanded ABNORMAL termination of (anything), VSAM cannot update the dataset
statistics because the issuer of the command has INTENTIONALLY prevented it
from doing so.


Since CEMT P SHUT I causes a CONTROLLED issuance of an Abnormal 
Termination, there is no reason why a request to flush the statistics 
BEFORE doing the Abnormal Termination can't be done.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Gould]

On Jul 6, 2005, at 9:48 PM, Ed Finnell wrote:


--SNIP-



I'm decent, but was able to just cut and paste! Then I went and
did a review of the Oracle Grid. Hmmm, wonder how in the world they
stay up forever with no abends and no catalogs???  Hmmm.




Well, I am not a ORACLE fan but at least their catalog stats are 
probably correct.


Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM statistics are often bogus

[Edward E. Jaffe]

Richards.Bob wrote:


... For now, I off to search the Internet for the meaning of "boustrophedon". 

 



Back and forth like an old dot matrix printer.

--
-
| Edward E. Jaffe||
| Mgr, Research & Development| [EMAIL PROTECTED]|
| Phoenix Software International | Tel: (310) 338-0400 x318   |
| 5200 W Century Blvd, Suite 800 | Fax: (310) 338-0801|
| Los Angeles, CA 90045  | http://www.phoenixsoftware.com |
- 


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM statistics are often bogus

[Richards.Bob]

Not quite!

It is a writing style compare to an ox plowing a field where 
  .noitcerid etisoppo eht ni seog dna snrut ylerem eh

  

Bob 

 -Original Message-
From:   IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED]  On Behalf Of 
Ed Finnell
Sent:   Wednesday, July 06, 2005 10:57 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject:Re: IBM VSAM statistics are often bogus

 
In a message dated 7/6/2005 9:54:54 P.M. Central Standard Time,  
[EMAIL PROTECTED] writes:

boustrophedon


'Hot air balloon'

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html 
  
  
  
LEGAL DISCLAIMER 
The information transmitted is intended solely for the individual or entity to 
which it is addressed and may contain confidential and/or privileged material. 
Any review, retransmission, dissemination or other use of or taking action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you have received this email in error please 
contact the sender and delete the material from any computer. 
  
Seeing Beyond Money is a service mark of SunTrust Banks, Inc. 
[ST:XCL] 
 
 
 
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM statistics are often bogus

[Ed Finnell]

 
In a message dated 7/6/2005 9:54:54 P.M. Central Standard Time,  
[EMAIL PROTECTED] writes:

boustrophedon


'Hot air balloon'

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM statistics are often bogus

[Richards.Bob]

John,

I'll take your word on the automation coding time. For now, I off to search the 
Internet for the meaning of "boustrophedon". 

Bob 

 -Original Message-
From:   IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED]  On Behalf Of 
john gilmore
Sent:   Wednesday, July 06, 2005 10:47 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject:    Re: IBM VSAM statistics are often bogus

>  The  penance for both of us is to write "Invalid data is invalid data" 
>ten  thousand  times. Under ISPF EDIT, it should take us roughly ten 
>seconds. 

A more exiguous penalty is clearly called for.

My preference would be to require you format it in boustrophedon 
parametrically in full n-character lines, n <= 131, which it would take you 
rather longer to automate.

John Gilmore
Ashland, MA 01721
USA 
  
  
  
LEGAL DISCLAIMER 
The information transmitted is intended solely for the individual or entity to 
which it is addressed and may contain confidential and/or privileged material. 
Any review, retransmission, dissemination or other use of or taking action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you have received this email in error please 
contact the sender and delete the material from any computer. 
  
Seeing Beyond Money is a service mark of SunTrust Banks, Inc. 
[ST:XCL] 
 
 
 
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Finnell]

 
In a message dated 7/6/2005 9:43:31 P.M. Central Standard Time,  
[EMAIL PROTECTED] writes:

You must  be a typist. It took me 7 of the 10 to type the  words!



>>
I'm decent, but was able to just cut and paste! Then I went and
did a review of the Oracle Grid. Hmmm, wonder how in the world they
stay up forever with no abends and no catalogs???  Hmmm.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM statistics are often bogus

[john gilmore]

 The  penance for both of us is to write "Invalid data is invalid data" 
ten  thousand  times. Under ISPF EDIT, it should take us roughly ten 
seconds. 


A more exiguous penalty is clearly called for.

My preference would be to require you format it in boustrophedon 
parametrically in full n-character lines, n <= 131, which it would take you 
rather longer to automate.


John Gilmore
Ashland, MA 01721
USA

_
Don’t just search. Find. Check out the new MSN Search! 
http://search.msn.click-url.com/go/onm00200636ave/direct/01/


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Gould]

On Jul 6, 2005, at 9:32 AM, Farley, Peter x23353 wrote:


I never even envisioned automated tools looking at VSAM stats.  My
ASSumption when reading Mark's posts was that he was referring to 
individual

programmers looking at individual VSAM file stats for guidance.  My
experience is obviously severely limited in this regard, as in my 
varied
positions over the years the usual case was that all of the 
applications'

datasets (including VSAM) were our (the application programmers')
responsibility to feed and care for, and we never had thousands to 
contend

with.

As for my ASSumption on criticality, the normal performance variations 
(once

a good initial design and shakeout was done) for an individual (set of)
application file(s) were never that critical unless severe volume 
increases

occurred.  A parochial point of view, I must freely admit.

I stand humbly corrected.  My vision is obviously way too narrow.

Peter
-SNIP_



Peter,

Somewhere in the back of my paged out (permanently?) memory is there 
used to be (or there is one currently) a utility that went through the 
catalog and looked for vsam datasets and listed any  that according to 
its recommendation ones that needed
 re-orging. My mind just can't remember the name. It may have built 
idcams control cards as well.


Can anyone come up with the name of the product?

Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Richards.Bob]

Ed, 

You must be a typist. It took me 7 of the 10 to type the words!

Bob 

 -Original Message-
From:   IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED]  On Behalf Of 
Ed Finnell
Sent:   Wednesday, July 06, 2005 10:36 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject:Re: IBM VSAM Statistics are often Bogus

 
In a message dated 7/6/2005 6:52:42 P.M. Central Standard Time,  
[EMAIL PROTECTED] writes:

The  penance for both of us is to write "Invalid data is invalid data" ten 
thousand  times. Under ISPF EDIT, it should take us roughly ten seconds.   



R10 Invalid data is invalid data
rr
rr
 
3.7 secs. 
  
  
  
LEGAL DISCLAIMER 
The information transmitted is intended solely for the individual or entity to 
which it is addressed and may contain confidential and/or privileged material. 
Any review, retransmission, dissemination or other use of or taking action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you have received this email in error please 
contact the sender and delete the material from any computer. 
  
Seeing Beyond Money is a service mark of SunTrust Banks, Inc. 
[ST:XCL] 
 
 
 
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Finnell]

 
In a message dated 7/6/2005 6:52:42 P.M. Central Standard Time,  
[EMAIL PROTECTED] writes:

The  penance for both of us is to write "Invalid data is invalid data" ten 
thousand  times. Under ISPF EDIT, it should take us roughly ten seconds.   



R10 Invalid data is invalid data
rr
rr
 
3.7 secs.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

FW: IBM VSAM Statistics are often Bogus

[Ron and Jenny Hawkins]

Tom, et al,

I was just thinking... CICS aside, when a job ABENDS while updating a KSDS
file, what is the most common thing that happens to allow the job to be
rerun? Delete and restore the KSDS, right? So what good are the accumulated
statistics at this point - valid or invalid? Well there worth zilcho,
because they were just deleted. Gone forever.

And the next day someone takes a LISTCAT of that file and makes some
decision based on those stats, assuming that it represents 24 hours of
activity, when it is really just the last 20 minutes of the batch run. Smart
move!

I would rather we just got rid of all the stats from LISTCAT. We don't have
a listdb2, listdl1, listfastpath or listqsam, so why the religious fervour
around LISTC stats?

Ron

> -Original Message-
> From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
> Behalf Of Tom Savor
> Sent: Thursday, 7 July 2005 1:28 AM
> To: IBM-MAIN@BAMA.UA.EDU
> Subject: Re: IBM VSAM Statistics are often Bogus
> 
> Ok, I'll be the whipping boy.
> 
> Why can't the Operating System be "Enhanced" to intercept the Abend and
> close the files.
> Close any input file and/or VSAM file.  There is already alot of
> processing
> that occurs after
> the abend by building the dump thru Abend-Aid and other debuggers.  We
> already have CICS
> with DTB feature that will backout changes and adds back to the last
> SYNCPOINT.  DB2 can
> backout changes and adds back to the last COMMIT.  So why can't Z/OS be
> changed to make sure that
> VSAM files are closed during Abend process ??  I'm probably wrong, but
> this
> "sounds" much
> easier to do, than what CICS and DB2 does.
> 
> 
> Tom Savor
> Certegy Card Services
> 11720 Amber Park Drive, Suite 500
> Alpharetta, GA  30004
> 
> Phone: 678-867-8431
> cell:  404-660-6898
> E-Mail: [EMAIL PROTECTED]
> /\/\_00_/\/\

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Gould]

On Jul 5, 2005, at 7:00 PM, Ted MacNEIL wrote:


...
1) One can always do one's own interval accounting by opening and 
closing

the file periodically.
...

I hope you are joking!



Ted,

Along similar lines we had a programmer close and open a vsam dataset 
after each search. I agree with you.


Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Richards.Bob]

Peter,

Well, as long as we are bearing our sins, I must humbly admit that I have coded 
REXX to process the entire catalog structure based on LISTCAT output (then 
changed to CSI output) and then taken actions. The shame of it all. I have Ron 
F. to thank for showing me the error of my ways of trusting stats on open 
and/or corrupted files and leading me back to the path of right-stat-ness. 


My problem is not having a narrow vision. Mine is having visions that are often 
too grand! :)

The penance for both of us is to write "Invalid data is invalid data" ten 
thousand times. Under ISPF EDIT, it should take us roughly ten seconds.  

Bob 

 -Original Message-
From:   IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED]  On Behalf Of 
Farley, Peter x23353
Sent:   Wednesday, July 06, 2005 10:32 AM
To: IBM-MAIN@BAMA.UA.EDU
Subject:        Re: IBM VSAM Statistics are often Bogus

I never even envisioned automated tools looking at VSAM stats.  My
ASSumption when reading Mark's posts was that he was referring to individual
programmers looking at individual VSAM file stats for guidance.  My
experience is obviously severely limited in this regard, as in my varied
positions over the years the usual case was that all of the applications'
datasets (including VSAM) were our (the application programmers')
responsibility to feed and care for, and we never had thousands to contend
with.

As for my ASSumption on criticality, the normal performance variations (once
a good initial design and shakeout was done) for an individual (set of)
application file(s) were never that critical unless severe volume increases
occurred.  A parochial point of view, I must freely admit.

I stand humbly corrected.  My vision is obviously way too narrow. 
  
  
  
LEGAL DISCLAIMER 
The information transmitted is intended solely for the individual or entity to 
which it is addressed and may contain confidential and/or privileged material. 
Any review, retransmission, dissemination or other use of or taking action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you have received this email in error please 
contact the sender and delete the material from any computer. 
  
Seeing Beyond Money is a service mark of SunTrust Banks, Inc. 
[ST:XCL] 
 
 
 
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Bill Fairchild]

 
In a message dated 7/6/2005 4:24:41 P.M. Central Daylight Time,  
[EMAIL PROTECTED] writes:

>A  Devil's Advocate might be tempted to say just because you don't ABEND   
>does not mean that
>a user did not overlay or incorrectly modify  VSAM control blocks that are 
>in user key.  How can
>you  ever trust the stats?

and there is of course a sense in which his  argument is unanswerable; but it 
is finally irrelevant because too fertile  (and sophomoric): It can be used 
to argue that no result of any program's  execution can be trusted.



My point was that it does not seem to me to be any more likely that the  
stats have been hosed if all we know is that an ABEND has occurred.  If you  
want 
to determine the likelihood that the stats have been hosed, you should  
examine the stats for reasonableness, whether ABENDing or NORMENDing.
 
Bill Fairchild

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ted MacNEIL]

...
As for Alte-Leipziger I can say "it's not issue for us".
...

I couldn't find the meaning of that compound word.

-teD

In God we Trust!
All others bring data!
  --Deming

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Schiradin,Roland HG-Dir itb-db/dc]

Will this thread never end? Mark Thomen explained now several times the why and 
the current state of this issue. Please raise requirements if you relay on 
those statistics and explain the business case
and also how did you handle this the last nn years in the past. I believe a lot 
of customers 
will not vote for such a requirement because of the possible performance issue. 

As for Alte-Leipziger I can say "it's not issue for us".

Roland

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ted MacNEIL]

...
Why? I've been known to do CLOSE TYPE=T on sequential datasets after
processing "n" records. Of course, IIRC, it was a specialized
application. Lost is the mists of pre-history, now.
...

Open/Close processing is not cheap.
Also, if you have (partial-)release on close, you can easily create a 
multi-(tiny)extent file in no time.


-teD

In God we Trust!
All others bring data!
  --Deming

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ted MacNEIL]

...
Who said IBM doesn't have a sense of humor?   Thanks,Mark, that gave me 
the giggles.
...

I verified this one years ago.
I don't know if it's still true.

The programme prefix for dfpSMS is IGD.

Years ago, either the main module or an alias was IGDZILLA.

“I, God-Zilla”.

(8-{]}


-teD

In God we Trust!
All others bring data!
  --Deming

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Donald Pagdin]

>A Devil's Advocate might be tempted to say just because you don't ABEND  does
>not mean that a user did not overlay or incorrectly modify VSAM control
>blocks that are in user key.  How can you ever trust the stats?
>Bill Fairchild

Note:
CICS, recovering an application abend (via DTB) does not have to Close a VSAM 
file, since the in-flight work is backed out, and the CIs that are "frozen" 
(enqueued) are released at sync-point rollback. This lets other transactions 
continue against that VSAM file. Closing during/after DTB is irrelevant. DB2 
and IMS are careful users of VSAM also.

Another thread on this list is discussing "destructive" moves, etc.  Why 
mention it?
The point is this. In higher level languages one has to be particularly lax as 
a programmer to cause VSAM control block overlays and destructive (when 
unintended) moves.

So it is assembler that can bite the unwary yet diligent programmer. But if you 
are coding assembler in the first place, it is because you need some function 
that should be denied to Joe programmer. This language imposes a stiffer 
testing methodology and (dare I say) greater knowledge about how things work on 
the developer.

TRUST is inherent in a (debuged) language and (debuged) OS, since the hardware 
will do exactly what it is told... no more, no less.
What user? We're talking programmer.
What programmer? Using his native language?
Perhaps there should be a "TRUST" quotient assigned to each program based on 
the author's skill and care and test time, divided by the language complexity 
and factored by its size and host of the program (CICS, batch, Websphere, etc).

If you want VSAM to make the stats trustworthy, write trustworthy programs. At 
least acknowledge the probability of a failure and plan for it.

(I'll not debate the imagined solution of user-written ESTAEx and the problems 
of having an FFR, VSAM under SRB, VSAM in a Plex under RRS in the CCF, 
distributed LUWs, etc)



--

This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy this e-mail. Any unauthorized 
copying, disclosure or distribution of the material in this e-mail is strictly 
forbidden.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[john gilmore]

Bill Fairchild writes:

A Devil's Advocate might be tempted to say just because you don't ABEND  
does not mean that
a user did not overlay or incorrectly modify VSAM control blocks that are 
in user key.  How can

you ever trust the stats?


and there is of course a sense in which his argument is unanswerable; but it 
is finally irrelevant because too fertile (and sophomoric): It can be used 
to argue that no result of any program's execution can be trusted.


I have already made it clear that the tone of many of the posts in this 
thread is offensive.  Much of it is also silly.  He who uses a report format 
as a programming interface may think himself clever, but he has no basis for 
complaint when t changes.


John Gilmore
Ashland, MA 01721
USA

_
Express yourself instantly with MSN Messenger! Download today - it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[McKown, John]

> -Original Message-
> From: IBM Mainframe Discussion List 
> [mailto:[EMAIL PROTECTED] On Behalf Of Ted MacNEIL
> Sent: Tuesday, July 05, 2005 7:00 PM
> To: IBM-MAIN@BAMA.UA.EDU
> Subject: Re: IBM VSAM Statistics are often Bogus
> 
> 
> ...
> 1) One can always do one's own interval accounting by opening 
> and closing 
> the file periodically.
> ...
> 
> I hope you are joking!
> 
> 
> -teD
> 

Why? I've been known to do CLOSE TYPE=T on sequential datasets after
processing "n" records. Of course, IIRC, it was a specialized
application. Lost is the mists of pre-history, now.


--
John McKown
Senior Systems Programmer
UICI Insurance Center
Information Technology

This message (including any attachments) contains confidential
information intended for a specific individual and purpose, and its'
content is protected by law.  If you are not the intended recipient, you
should delete this message and are hereby notified that any disclosure,
copying, or distribution of this transmission, or taking any action
based on it, is strictly prohibited.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ted MacNEIL]

...
1) One can always do one's own interval accounting by opening and closing 
the file periodically.
...

I hope you are joking!


-teD

In God we Trust!
All others bring data!
  --Deming

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Edward E. Jaffe]

Mark Thomen wrote:


That's what module IFG0TC0A ("IFG-gotcha") does.



I've always called it "I-F-Gotcha!" or just "I've Gotcha!"

--
-
| Edward E. Jaffe||
| Mgr, Research & Development| [EMAIL PROTECTED]|
| Phoenix Software International | Tel: (310) 338-0400 x318   |
| 5200 W Century Blvd, Suite 800 | Fax: (310) 338-0801|
| Los Angeles, CA 90045  | http://www.phoenixsoftware.com |
-

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Bruce Black]

That's what module IFG0TC0A ("IFG-gotcha") does.

Who said IBM doesn't have a sense of humor?   Thanks,Mark, that gave me 
the giggles.


--
Bruce A. Black
Senior Software Developer for FDR
Innovation Data Processing 973-890-7300
personal: [EMAIL PROTECTED]
sales info: [EMAIL PROTECTED]
tech support: [EMAIL PROTECTED]
web: www.innovationdp.fdr.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Bill Fairchild]

 
In a message dated 7/6/2005 2:51:54 P.M. Central Daylight Time,  
[EMAIL PROTECTED] writes:

Most of  the VSAM control blocks are in user key.  If these have been
overlaid  or incorrectly modified, writing this data to the catalog may
BREAK THE  DATA SET.  The effects of breaking the data set are much worse
than  "throwing away" the data and verifying the data set on the next open.
No  one wants an online application to have to recover a major data set and
be  down for hours because it was broken closing it during an ABEND  because
data was overlaid.



A Devil's Advocate might be tempted to say just because you don't ABEND  does 
not mean that a user did not overlay or incorrectly modify VSAM control  
blocks that are in user key.  How can you ever trust the stats?
 
Bill Fairchild

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Mark Thomen]

<[EMAIL PROTECTED]> wrote in message
news:...
> Ok, I'll be the whipping boy.
>
> Why can't the Operating System be "Enhanced" to intercept the Abend and
> close the files.

That's what module IFG0TC0A ("IFG-gotcha") does.It is an exit invoked
by task termination to process open files.  However, there are problems
closing files as I mentioned before.

Most of the VSAM control blocks are in user key.  If these have been
overlaid or incorrectly modified, writing this data to the catalog may
BREAK THE DATA SET.  The effects of breaking the data set are much worse
than "throwing away" the data and verifying the data set on the next open.
No one wants an online application to have to recover a major data set and
be down for hours because it was broken closing it during an ABEND because
data was overlaid.

> Close any input file and/or VSAM file.  There is already alot of
processing
> that occurs after
> the abend by building the dump thru Abend-Aid and other debuggers.  We
> already have CICS
> with DTB feature that will backout changes and adds back to the last
> SYNCPOINT.  DB2 can
> backout changes and adds back to the last COMMIT.  So why can't Z/OS be
> changed to make sure that
> VSAM files are closed during Abend process ??

Because the only code at the moment is during task termination.  As I
mentioned before, the access methods do NOT have recovery routines because
of the overhead and performance implications.  Not that we're not trying to
solve that problem, but it's going to take quite a while to do it.
Movement of the control blocks into protected storage is a goal also, but
given the number of users/vendors/applications that try to touch these
blocks (and sometimes change them) means there are a lot of issues with
doing this.  Not that we're not going to do it, but existing references by
programs will have to be changed.

So when CICS and/or DB2 recovery routines get entered, they can do backout,
etc.  However control has been yanked away from the access method until
task termination processing, WAY after the application recovery routines
have been deleted.

Thanks,
Mark Thomen
Catalog/IDCAMS/VSAM Development
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Bill Fairchild]

 
In a message dated 7/6/2005 12:56:40 P.M. Central Daylight Time,  
[EMAIL PROTECTED] writes:

It  doesn't take a weatherman to know
>which the wind is  blowing.

...which WAY the wind is blowing.  (Unless you were  trying to imply
no 'way' in your original  sentence?)



If we are going to quibble over the lyrics, let's go to the Principles of  
Operation itself:
"You  don’t need a weather man
To  know which way the wind blows" [Bob Dylan:  1965;  “Subterranean Homesick 
Blues”]
 
Bill Fairchild

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Tom Schmidt]

On Wed, 6 Jul 2005 13:23:13 -0400, Kirk Talman wrote:

>7/6/2005 11:32 AM Tom Schmidt wrote
>>There is a more fundamental issue here: VSAM keeps its statistics records
>>in KEY 8 storage in the address space's private area, ...  Those
>>statistics records can, and often are in the case of a storage overlay,
>>subject to corruption and/or destruction because of that inherent lack of
>>protection.
>
>>VSAM (and other access methods perhaps) needs to have a protected area for
>>such apparently customer-important data as statistics.  ...
>
>...
>
>>The problem with using interval accounting to try to anticipate the ABENDs
>...
>
>1) One can always do one's own interval accounting by opening and closing
>the file periodically.

One can, true enough, but at a rather high cost in performance.  Still, if
you are picky about the stats this is one way to roll your own.

>2) In a prior life about a decade ago I remember that the control block
>containing at least some of the statistics was in CSA, where it was much
>less likely to be overlaid.  With a CSA overlay the devil jumps much
>faster and higher and sooner and is easily detected.

CSA overlays are still a little too common, although far less so than in
my 'youth'.  As for "easily detected" - ha!!

>It may be that that control block was only in CSA when shroptns=3, which
>we used -- an option sane people don't use.  The control block was small
>and could be easily be put in CSA always with minimal cost in these days
>of ubiquitous VSCR.

Not really such a minimal cost if one considers the tens of thousands of
VSAM files open at any arbitrary point in time in larger shops.

>3) Don't "eyecatchers" act as effective canaries to know when an overlay
>has occurred?  I know we used them extensively in the prior life to know
>when we were wounded but not yet informed thereof.

No.  Overlays can come in very small packages -- a single bit overlay is
extraordinarily difficult to detect, especially with an eyecatcher canary.
But a single bit overlay can cause big issues if, say, the bit is nearer
the left than the right in any given byte (or word).

>4) There has been much discussion of the desirability of "bad" statistics.
>
>If I am using "bad" statistics for billing, I can use reasonability checks
>to determine if the basis of the bill is bogus and eat the cost rather
>than offend the client.

Or in the case of a large basis bill to flag the bill for manual post
processing perhaps?

>If I am decisioning with possibly "bad" statistics, I can adjust my
>algorithm to be tolerant.  When I was trained as a physicist, accomodating
>error was a standard part of any measurement process.  The conclusion can
>often be made using "bad" data, even if an automaton is doing the work.
>The first paper in Physical Review Letters (~1964 Penzias & Wilson) on the
>"big bang theory" showed a graph with very large error on the
>measurements.  But in that case merely the order of magnitude of the data
>points allowed a conclusion to be drawn.  Or as one could hear said in the
>60's (when the first dinosaur ...) It doesn't take a weatherman to know
>which the wind is blowing.

...which WAY the wind is blowing.  (Unless you were trying to imply
no 'way' in your original sentence?)


--
Tom Schmidt
Madison, WI

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Tom Savor]

Ok, I'll be the whipping boy.

Why can't the Operating System be "Enhanced" to intercept the Abend and
close the files.
Close any input file and/or VSAM file.  There is already alot of processing
that occurs after
the abend by building the dump thru Abend-Aid and other debuggers.  We
already have CICS
with DTB feature that will backout changes and adds back to the last
SYNCPOINT.  DB2 can
backout changes and adds back to the last COMMIT.  So why can't Z/OS be
changed to make sure that
VSAM files are closed during Abend process ??  I'm probably wrong, but this
"sounds" much
easier to do, than what CICS and DB2 does.


Tom Savor
Certegy Card Services
11720 Amber Park Drive, Suite 500
Alpharetta, GA  30004

Phone: 678-867-8431
cell:  404-660-6898
E-Mail: [EMAIL PROTECTED]
/\/\_00_/\/\







--
This message contains information from Certegy, Inc which may be confidential 
and privileged.  If you are not an intended recipient, please refrain from any 
disclosure, copying, distribution or use of this information and note that such 
actions are prohibited.  If you have received this transmission in error, 
please notify by e:mail [EMAIL PROTECTED]
==

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Kirk Talman]

7/6/2005 11:32 AM Tom Schmidt <[EMAIL PROTECTED]> wrote
There is a more fundamental issue here: VSAM keeps its statistics records
in KEY 8 storage in the address space's private area, ...  Those 
statistics
records can, and often are in the case of a storage overlay, subject to
corruption and/or destruction because of that inherent lack of protection.

VSAM (and other access methods perhaps) needs to have a protected area for
such apparently customer-important data as statistics.  ...

...

The problem with using interval accounting to try to anticipate the ABENDs 
...

...

1) One can always do one's own interval accounting by opening and closing 
the file periodically.

2) In a prior life about a decade ago I remember that the control block 
containing at least some of the statistics was in CSA, where it was much 
less likely to be overlaid.  With a CSA overlay the devil jumps much 
faster and higher and sooner and is easily detected.

It may be that that control block was only in CSA when shroptns=3, which 
we used -- an option sane people don't use.  The control block was small 
and could be easily be put in CSA always with minimal cost in these days 
of ubiquitous VSCR.

3) Don't "eyecatchers" act as effective canaries to know when an overlay 
has occurred?  I know we used them extensively in the prior life to know 
when we were wounded but not yet informed thereof.

4) There has been much discussion of the desirability of "bad" statistics.

If I am using "bad" statistics for billing, I can use reasonability checks 
to determine if the basis of the bill is bogus and eat the cost rather 
than offend the client.

If I am decisioning with possibly "bad" statistics, I can adjust my 
algorithm to be tolerant.  When I was trained as a physicist, accomodating 
error was a standard part of any measurement process.  The conclusion can 
often be made using "bad" data, even if an automaton is doing the work. 
The first paper in Physical Review Letters (~1964 Penzias & Wilson) on the 
"big bang theory" showed a graph with very large error on the 
measurements.  But in that case merely the order of magnitude of the data 
points allowed a conclusion to be drawn.  Or as one could hear said in the 
60's (when the first dinosaur ...) It doesn't take a weatherman to know 
which the wind is blowing.

pup strolling down memory lane

p.s. I think Mr Thomen is to be commended for his persistent patience.


-
The information contained in this communication (including any attachments
hereto) is confidential and is intended solely for the personal and
confidential use of the individual or entity to whom it is addressed.  The
information may also constitute a legally privileged confidential
communication.  If the reader of this message is not the intended recipient
or an agent responsible for delivering it to the intended recipient, you
are hereby notified that you have received this communication in error and
that any review, dissemination, copying, or unauthorized use of this
information, or the taking of any action in reliance on the contents of
this information is strictly prohibited.  If you have received this
communication in error, please notify us immediately by e-mail, and delete
the original message.  Thank you

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Thomas Conley]

- Original Message - 
From: "Mark Thomen" <[EMAIL PROTECTED]>

Newsgroups: bit.listserv.ibm-main
Sent: Wednesday, July 06, 2005 11:26 AM
Subject: Re: IBM VSAM Statistics are often Bogus




Had I worked in VSAM for 30+ years I'd have made this change a long time
ago and this issue would never have been so contentious.  But I've only
been working directly in VSAM for a couple of years now so I apologize for
not publicizing this sooner.  And as several users have pointed out,
invalid data is invalid data - and how can you make business decisions on
invalid data?  If your checking statement said you had $3,127.47 but next
to it in parentheses it said "(but this amount is invalid)", would you go
out and try to spend the money?  No, I think you'd be kinda cautious so 
you

didn't get arrested for fraud.  I wonder how businesses can make decisions
on the same invalid data.

We do have plans to correct the problem, but it's dependent on resources,
and time.



Mark,

The only beef I had with this whole thing is that IBM did not protect the 
customer investment with the first fix.  I know you hate the compromise fix, 
but it was the right thing to do.  Ensuring that customer code runs 
uninterrupted wherever possible should be one of the highest development 
priorities.  The new fix allows application code to run unchanged (although 
REXX will probably still fail because the '*' is abutted to the end of the 
number), while giving the new information that the data is potentially 
invalid.


Regards,
Tom Conley 


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[McKown, John]

You want "good" stats to base things such as REORGs on? Go to DB2. And
pay for it. 

Only in FOSS will you get something for "free" (either Libre or Gratis).
And, in FOSS, if you don't like it, you can "fix" it yourself. If you
don't have the talent to "fix" it yourself, then either: (1) nicely ask
the maintainer to "fix" it; (2) pay somebody to "fix" it; (3) learn how
to "fix" it yourself; (4) learn to live with it.

With z/OS, you don't have option (3). And options (1) and (2) are
similar in that IBM must do any "fixing". Option (4) is always
available. I tried to refrain from mentioning that option (4) is "the
Windows way".


--
John McKown
Senior Systems Programmer
UICI Insurance Center
Information Technology

This message (including any attachments) contains confidential
information intended for a specific individual and purpose, and its'
content is protected by law.  If you are not the intended recipient, you
should delete this message and are hereby notified that any disclosure,
copying, or distribution of this transmission, or taking any action
based on it, is strictly prohibited.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Finnell]

 
In a message dated 7/6/2005 10:26:57 A.M. Central Standard Time,  
[EMAIL PROTECTED] writes:

invalid  data?  If your checking statement said you had $3,127.47 but next
to  it in parentheses it said "(but this amount is invalid)", would you go
out  and try to spend the money?  No, I think you'd be kinda cautious so  you
didn't get arrested for fraud.  I wonder how businesses can make  decisions
on the same invalid data.



>>
We had a competitor who had a customer withdraw his mistakenly
debited $3M. His claim was that he had prayed for riches and
his prayers were answered. The glee was that the competitor had
to explain under oath that it was human error totally and that
their recon system needed a little work. No jail time for the
fervent, but the competitor got scheduled visits from the Fed.
 
Guess another consideration is that the stats have been bogus  for
so long many, many application delete and redefine nightly adding
cycles to an already tight window. No way for 24/7. Runs  customers
off in droves. Can you say tanked revenue?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Tom Schmidt]

On Wed, 6 Jul 2005 06:57:38 EDT, Bill Fairchild wrote:
>
>In a message dated 7/6/2005 5:22:42 A.M. Central Daylight Time,
>Bob Richards writes:
>
>> In the case of an ABEND or CICS "close immediate" situation, what
>information would you like Mark to write to the file? And where is this
>>information supposed to come from?
>
>I think you are assuming there is only one way to "fix" this  problem.
>E.g., in the case of SMF data the solution is called interval
>accounting.  Every X minutes SMF records are written to reflect the
>activity during the last X minutes.  The VSAM statistics in question
>could  be updated on intervals.  You will miss all the activity that
>occurs during  the last partial interval just before a system crash.
>Application programs  or even access methods (e.g., at OPEN time) could be
>enhanced to add ABEND  recovery routines that invoke a new
>service which would add the last partial  interval's stats into the proper
>buckets.  This assumes the stats are kept in storage that was not trashed
>as part of the error leading to the ABEND.   If the operator FORCEs the
>job or the system crashes, there is no hope.   But for most abnormal ends
>there are ways to "fix" the stats.  The  information comes from the same
>place where it comes from now in the case of  normal CLOSE.  The minimum
>we would need is a new service which the user  would invoke to update
>stats.
>
>Bill Fairchild

There is a more fundamental issue here: VSAM keeps its statistics records
in KEY 8 storage in the address space's private area, since it normally
runs as an extension to the application program code.  Those statistics
records can, and often are in the case of a storage overlay, subject to
corruption and/or destruction because of that inherent lack of protection.

VSAM (and other access methods perhaps) needs to have a protected area for
such apparently customer-important data as statistics.  Anything less is a
band-aid.

VSAM could move the statistics to a data space, either owned by the
application address space or perhaps owned by the system (somewhere).  But
it would probably still end up being a KEY 8 data space since it is still
an extension of the application program code... so it would be 'security by
obscurity' and not iron-clad.  That should still offer a marked improvement
over the private area.  But it would also introduce performance concerns in
the middle of an obvious performance path.  Sticky detail.

The problem with using interval accounting to try to anticipate the ABENDs
is that storage overlays do not always (or even often) inflict the fatal
damage on the first strike.  The statistics records could, possibly, be the
first storage to be overlaid (or worse, partially overlaid) and the result
would be difficult to see... certainly difficult for poorly written LISTCAT
automation to see.  Again, this introduces a (smaller? maybe not)
performance concern by pushing out excess SMF records in the middle of the
I/O (performance) path.

A problem with having the access method inflict its own recovery during
ABENDs would be the new conflict(s) that would result, for example which
percolates to which and how could the access method get control "first" ...
should the access method REALLY get control first?  LE would probably
vote 'no' and other products (or applications) might vote similarly... then
what?  Even if VSAM's shiny new recovery intercept got control at the ABEND
who is to say that its statistics data wasn't already overlaid?  Too late.

As for why it took IBM so long to address the issue... I personally don't
blame IBM for taking so long, since the statistics were always (as far as I
know) documented to be an indicator and there was (as is obvious by this
thread) a huge misunderstanding in the customer (and vendor) community over
the scope and impact of the problem, if any.  There was documentation
warning against use of LISTCAT as a programming interface that goes back to
(at least) 1981 ... caveat emptor.

I commend Mark for taking the issue in hand and trying to implement a long
term solution, and for raising it to the level where it is discussed so
that the appropriate awareness is reached, but Mark doesn't need my
commendation for job satisfaction.

Say what you want about IBM but its people are still among the best
architects of software around.  This issue is convoluted at best.

--
Tom Schmidt
Madison, WI

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Mark Thomen]

"Dave Juraschek" <[EMAIL PROTECTED]> wrote in message
news:<[EMAIL PROTECTED]>...
> I guess what really peeves me is not that the stats are invalid -
although
> that is certainly disturbing to say the least.
> That is clear (to me now).
>
> What really is unconscionable is Mark's admission that IBM has known this
> for 30+ years and has not cared or bothered to fix it yet.  Mark's
response
> (whether he likes to admit is or not) and IBM's response seems to be
"shame
> on you for using our bogus [for John] numbers", and never "shame on us
for
> knowing about this for so long and just not giving a rip".  A lousy
> disclaimer in one paragraph of one manual is hardly notification to end-
> users who might need this kind of information that what IBM is providing
> them is ... um, again ... bogus.  Not having fixed this in 30+ years
> (according to Mark) is both arrogant and sloppy on IBM's part.

I am truly sorry you are offended.  However, if you understood the
evolution of this operating system better you would recognize that none of
the access methods since the beginning have had recovery routines.  Back in
the days when processors were slow and memory was limited, customers
screamed about performance.  Every instruction counted.  Adding recovery
routines to access methods for EVERY SINGLE I/O request would have killed
the system.  And blocks that generally need to be updated during a request
obviously had to be in user key, since there was no simple way back in the
old days to update these blocks if they were in protected storage.

VSAM is a particularly complex access method, much more so than the others.
Obviously we expect that people start learning VSAM by reading the manuals,
learning the macros, understanding the processes, etc.  There are many
warnings and notices in the manuals that need to be read and understood,
but you're not the only one that doesn't read them.  We regularly run into
these scenarios and we have to explain the limitations of the components to
customers.  It's unfortunate, but that's life.  Everything is not perfect.

How do we fix the problem?  Well, there are several ways, but the major
problem hinges on resources.  Resources - you know, those things that give
you new function in every release?  And the emphasis by most customers is
"gimme something new and improved in each release", not "can you go fix
these statistics"?  So guess where most of the resources go?

Statistics for VSAM files should be a NIT; let me say that again - a NIT.
Most people seriously misuse the statistics.  They look at split counts and
take down major applications to reorg files because they think that "helps"
performance.  There's enough information out there to prove that this is
not true.  Ask Ron Ferguson who preaches "don't reorg based on split
counts", or check out the VSAM Demystified Redbook.  Or do your own
studies.  Which is better - downtime for a major application, or reorging a
file that doesn't need it?  As a result of this change to IDCAMS I've had
discussions with several customers who were SHOCKED to discover they didn't
need to reorg.  They quietly said "thank you" and changed their processes.

Had I worked in VSAM for 30+ years I'd have made this change a long time
ago and this issue would never have been so contentious.  But I've only
been working directly in VSAM for a couple of years now so I apologize for
not publicizing this sooner.  And as several users have pointed out,
invalid data is invalid data - and how can you make business decisions on
invalid data?  If your checking statement said you had $3,127.47 but next
to it in parentheses it said "(but this amount is invalid)", would you go
out and try to spend the money?  No, I think you'd be kinda cautious so you
didn't get arrested for fraud.  I wonder how businesses can make decisions
on the same invalid data.

We do have plans to correct the problem, but it's dependent on resources,
and time.

Thanks,
Mark Thomen
Catalog/IDCAMS/VSAM Development
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Mark Thomen]

"Dave Juraschek" <[EMAIL PROTECTED]> wrote in message
news:<[EMAIL PROTECTED]>...
> > ... CICS "close immediate" situation
>
> This really is a red herring.
>
> The CICS CEMT PERFORM SHUT IMMEDIATE is completely IBM code.  This is an
> example of where this is absolutely not a user/application/customer
problem.
>
> If it is violating some IBM VSAM file rule, it's still an IBM problem and
> one would think that the IBM VSAM folks would have asked the IBM CICS
folks
> to play by their restrictive rules years and years and years ago -
> especially, if this is a primary cause of the statistics corruption as
Mark
> claims.

CICS distributes a program that can be tailored by the installation to
determine whether an application is waiting for an I/O to complete, or just
sitting idle.  CICS does not attempt to terminate a transaction that is
performing I/O and uses a wait timer (I believe also specified by the
installation) on how long to wait before deciding to terminate the
transaction on an immediate shutdown.

The solution is there for the customer - they just have to make use of it.
Almost nobody has.  It's called "options"...

Thanks,
Mark Thomen
Catalog/IDCAMS/VSAM Development
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Farley, Peter x23353]

See my earlier reply to Mr. Richards for the ASSumptions behind that
statement.

Peter

-Original Message-
From: Ted MacNEIL [mailto:[EMAIL PROTECTED]
Sent: Tuesday, July 05, 2005 6:14 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: IBM VSAM Statistics are often Bogus

...
Just because they are not written when something BAD happens is NOT a reason
to see them as invalid or useless.
...

I don't get it!
Invalid data is invalid data!

We make enough bad decisions with valid data.

How can you say that with a straight face?

_
This message and any attachments are intended only for the use of the addressee 
and
may contain information that is privileged and confidential. If the reader of 
the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Farley, Peter x23353]

I never even envisioned automated tools looking at VSAM stats.  My
ASSumption when reading Mark's posts was that he was referring to individual
programmers looking at individual VSAM file stats for guidance.  My
experience is obviously severely limited in this regard, as in my varied
positions over the years the usual case was that all of the applications'
datasets (including VSAM) were our (the application programmers')
responsibility to feed and care for, and we never had thousands to contend
with.

As for my ASSumption on criticality, the normal performance variations (once
a good initial design and shakeout was done) for an individual (set of)
application file(s) were never that critical unless severe volume increases
occurred.  A parochial point of view, I must freely admit.

I stand humbly corrected.  My vision is obviously way too narrow.

Peter

-Original Message-
From: Richards.Bob [mailto:[EMAIL PROTECTED]
Sent: Tuesday, July 05, 2005 6:32 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: IBM VSAM Statistics are often Bogus



Your paragraph:

 "Just because they are not written when something BAD happens is NOT a
reason to see them as invalid or useless.  Like anything else in this
business (or life, for that matter), you need to know what you are talking
about when you use statistics to justify a decision."

If these statistics were being inspected one-by-one, I might concede that
last half of the second sentence. In my experience though, decisions of this
sort are being made in an automated fashion without consideration for
"knowing what you are talking about". They are being made under *the
assumption that the statistics are accurate*, a point that has clearly been
demonstrated as wrong!

Urgent or critical? Without empirical evidence to the contrary, how can *you
assume* that it is neither/nor?  

I raise your two cents and make it my $.04 cents. 

Bob 

_
This message and any attachments are intended only for the use of the addressee 
and
may contain information that is privileged and confidential. If the reader of 
the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Chase, John]

> -Original Message-
> From: IBM Mainframe Discussion List On Behalf Of Dave Juraschek
> 
> > ... CICS "close immediate" situation
> 
> This really is a red herring.

More than a few shops use CEMT P SHUT I as their "normal" means of
terminating CICS, despite the warnings in the CICS documentation of the
adverse consequences IBM GUARANTEES will occur.  "Haste makes waste."

> The CICS CEMT PERFORM SHUT IMMEDIATE is completely IBM code.

So are the MVS CANCEL and FORCE commands, among others.
  
> This is an example of where this is absolutely not a 
> user/application/customer problem.
> 
> If it is violating some IBM VSAM file rule, it's still an IBM 
> problem and one would think that the IBM VSAM folks would 
> have asked the IBM CICS folks to play by their restrictive 
> rules years and years and years ago - especially, if this is 
> a primary cause of the statistics corruption as Mark claims.

One should likewise expect the same considerations from the folks who wrote
the CANCEL and FORCE commands.

The fact remains that CEMT P SHUT I causes an ABNORMAL termination of CICS,
as would the MVS CANCEL or FORCE command.  The fact also remains that on a
commanded ABNORMAL termination of (anything), VSAM cannot update the dataset
statistics because the issuer of the command has INTENTIONALLY prevented it
from doing so.

-jc-

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Dave Juraschek]

> ... CICS "close immediate" situation

This really is a red herring.

The CICS CEMT PERFORM SHUT IMMEDIATE is completely IBM code.  This is an
example of where this is absolutely not a user/application/customer problem.

If it is violating some IBM VSAM file rule, it's still an IBM problem and
one would think that the IBM VSAM folks would have asked the IBM CICS folks
to play by their restrictive rules years and years and years ago -
especially, if this is a primary cause of the statistics corruption as Mark
claims.

-Dave

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Richards.Bob]

Bill,

My reply assumed nothing. I merely posed some questions. Here are two more:

Where are the development $$$ for this new service to fix an issue that has 
been known for 30 years? 
Do you really want to increase system overhead by adding *code bloat* for 
normal close processing just so you can get statistics of files closed 
improperly?

I would prefer we fix the problems not the statistics. Stop the "close 
immediate" or accept the consequences. For abends, shoot the programmer 
responsible!  Better yet, let him code his own error handling routines if 
the stats are that important to him. 

Bob 

 -Original Message-
From:   IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED]  On Behalf Of 
Bill Fairchild
Sent:   Wednesday, July 06, 2005 6:58 AM
To: IBM-MAIN@BAMA.UA.EDU
Subject:        Re: IBM VSAM Statistics are often Bogus

 
In a message dated 7/6/2005 5:22:42 A.M. Central Daylight Time,  
[EMAIL PROTECTED] writes:
 
> In the case of an ABEND or CICS "close immediate" situation, what  
information would you like Mark to write to the file? And where is this  
information 
supposed to come from?

 
I think you are assuming there is only one way to "fix" this  problem.  E.g., 
in the case of SMF data the solution is called interval  accounting.  Every X 
minutes SMF records are written to reflect the  activity during the last X 
minutes.  The VSAM statistics in question could  be updated on intervals.  You 
will miss all the activity that occurs during  the last partial interval just 
before a system crash.  Application programs  or even access methods (e.g., at 
OPEN time) could be enhanced to add ABEND  recovery routines that invoke a new 
service which would add the last partial  interval's stats into the proper 
buckets.  This assumes the stats are kept  in storage that was not trashed as 
part of the error leading to the ABEND.   If the operator FORCEs the job or the 
system crashes, there is no hope.   But for most abnormal ends there are ways 
to "fix" the stats.  The  information comes from the same place where it comes 
from now in the case of  normal CLOSE.  The minimum we would need is a new 
service which the user  would invoke to update stats.
 
Bill Fairchild


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html 
  
  
  
LEGAL DISCLAIMER 
The information transmitted is intended solely for the individual or entity to 
which it is addressed and may contain confidential and/or privileged material. 
Any review, retransmission, dissemination or other use of or taking action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you have received this email in error please 
contact the sender and delete the material from any computer. 
  
Seeing Beyond Money is a service mark of SunTrust Banks, Inc. 
[ST:XCL] 
 
 
 
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Bill Fairchild]

 
In a message dated 7/6/2005 5:22:42 A.M. Central Daylight Time,  
[EMAIL PROTECTED] writes:
 
> In the case of an ABEND or CICS "close immediate" situation, what  
information would you like Mark to write to the file? And where is this  
information 
supposed to come from?

 
I think you are assuming there is only one way to "fix" this  problem.  E.g., 
in the case of SMF data the solution is called interval  accounting.  Every X 
minutes SMF records are written to reflect the  activity during the last X 
minutes.  The VSAM statistics in question could  be updated on intervals.  You 
will miss all the activity that occurs during  the last partial interval just 
before a system crash.  Application programs  or even access methods (e.g., at 
OPEN time) could be enhanced to add ABEND  recovery routines that invoke a new 
service which would add the last partial  interval's stats into the proper 
buckets.  This assumes the stats are kept  in storage that was not trashed as 
part of the error leading to the ABEND.   If the operator FORCEs the job or the 
system crashes, there is no hope.   But for most abnormal ends there are ways 
to "fix" the stats.  The  information comes from the same place where it comes 
from now in the case of  normal CLOSE.  The minimum we would need is a new 
service which the user  would invoke to update stats.
 
Bill Fairchild


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Bill Fairchild]

 
In a message dated 7/5/2005 10:04:27 P.M. Central Daylight Time,  
[EMAIL PROTECTED] writes:
 
>BTW.

>My set of initials was:

>BTDT. DLI.  DWTDIA!

>Been there, done that.
>Didn't like it.
>Don't  want to do it again!





Thanks for explaining your abbreviations, initials, acridnyms.
 
IPTSWOC. [1]
 
ICTVF. [2]
 
RDHTGWIM. [3]
 
Bill Fairchild
 
[1] I Prefer To Spell Words Out Completely.
 
[2] I Can Type Very Fast.
 
[3] Readers Don't Have To Guess WTF I Meant.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Richards.Bob]

Dave,

Your reply is wrong on so many levels that I don't know where to start. It is 
obvious to me that the topic of this thread *is not clear to you.* 

In the case of an ABEND or CICS "close immediate" situation, what information 
would you like Mark to write to the file? And where is this information 
supposed to come from?

Remember, we are only talking about VSAM statistics for datasets that were not 
closed *normally*. For normal closures, the statistics are just fine. 

To use your words, "the lousy disclaimer" has been known to me since I first 
learned about VSAM. Ron Ferguson has traveled around the world for a few 
decades now telling all who will listen these two major points:

1) Statistics garnered from listcats of open VSAM datasets are invalid
2) Statistics garnered from listcats of VSAM datasets that were not closed 
properly are also invalid

Your final comments about SMF and RMF are pure nonsense. Once again, to quote 
Ted, invalid data is invalid data!   

Bob 

 -Original Message-
From:   IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED]  On Behalf Of 
Dave Juraschek
Sent:   Wednesday, July 06, 2005 4:32 AM
To: IBM-MAIN@BAMA.UA.EDU
Subject:    Re: IBM VSAM Statistics are often Bogus

I guess what really peeves me is not that the stats are invalid - although
that is certainly disturbing to say the least.
That is clear (to me now).

What really is unconscionable is Mark's admission that IBM has known this
for 30+ years and has not cared or bothered to fix it yet.  Mark's response
(whether he likes to admit is or not) and IBM's response seems to be "shame
on you for using our bogus [for John] numbers", and never "shame on us for
knowing about this for so long and just not giving a rip".  A lousy
disclaimer in one paragraph of one manual is hardly notification to end-
users who might need this kind of information that what IBM is providing
them is ... um, again ... bogus.  Not having fixed this in 30+ years
(according to Mark) is both arrogant and sloppy on IBM's part.

I wonder where the disclaimer paragraph is that I missed about SMF stats,
RMF stats, and ever other statistic IBM provides disclaiming it's validity
too.

-Dave

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html 
  
  
  
LEGAL DISCLAIMER 
The information transmitted is intended solely for the individual or entity to 
which it is addressed and may contain confidential and/or privileged material. 
Any review, retransmission, dissemination or other use of or taking action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you have received this email in error please 
contact the sender and delete the material from any computer. 
  
Seeing Beyond Money is a service mark of SunTrust Banks, Inc. 
[ST:XCL] 
 
 
 
 
 
 
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Dave Juraschek]

I guess what really peeves me is not that the stats are invalid - although
that is certainly disturbing to say the least.
That is clear (to me now).

What really is unconscionable is Mark's admission that IBM has known this
for 30+ years and has not cared or bothered to fix it yet.  Mark's response
(whether he likes to admit is or not) and IBM's response seems to be "shame
on you for using our bogus [for John] numbers", and never "shame on us for
knowing about this for so long and just not giving a rip".  A lousy
disclaimer in one paragraph of one manual is hardly notification to end-
users who might need this kind of information that what IBM is providing
them is ... um, again ... bogus.  Not having fixed this in 30+ years
(according to Mark) is both arrogant and sloppy on IBM's part.

I wonder where the disclaimer paragraph is that I missed about SMF stats,
RMF stats, and ever other statistic IBM provides disclaiming it's validity
too.

-Dave

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ed Gould]

On Jul 4, 2005, at 7:00 PM, Ted MacNEIL wrote:


-SNIP--


In God we Trust!
All others bring data!
  --Deming




And don't trust VSAM stats either:-)

Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ted MacNEIL]

...
Straight face? Heck, how about a clean conscience?
...

I have been burned too many times making decisions/recommendations
based on valid data.

Invalid data? ... don't think so!

BTW.

My set of initials was:

BTDT. DLI. DWTDIA!

Been there, done that.
Didn't like it.
Don't want to do it again!


-teD

In God we Trust!
All others bring data!
  --Deming

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Richards.Bob]

Ted,

I don't get it either. That line of thought carried to an extreme is...well, 
let's just say I shudder at the thought that if those lines of reasoning were 
being extended to actuarial tables, government decisions on labor statistics, 
etc, what the outcome might be.

Straight face? Heck, how about a clean conscience? 

Bob 

 -Original Message-
From:   IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED]  On Behalf Of 
Ted MacNEIL
Sent:   Monday, July 04, 2005 8:00 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject:        Re: IBM VSAM Statistics are often Bogus

...
Just because they are not written when something BAD happens is NOT a reason
to see them as invalid or useless.
...

I don't get it!
Invalid data is invalid data!

We make enough bad decisions with valid data.

How can you say that with a straight face?

-teD 
  
  
  
LEGAL DISCLAIMER 
The information transmitted is intended solely for the individual or entity to 
which it is addressed and may contain confidential and/or privileged material. 
Any review, retransmission, dissemination or other use of or taking action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you have received this email in error please 
contact the sender and delete the material from any computer. 
  
Seeing Beyond Money is a service mark of SunTrust Banks, Inc. 
[ST:XCL] 
 
 
 
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Richards.Bob]

PMFJI also, but I totally disagree with what you said, Peter.

All of what is being said *does not* assume anything. Mark should know; he 
coded it for a specific reason so that others, attempting to use the stats, 
would know they were inaccurate and that *they not assume* any information 
there as being relevant to reality.

Making any decision knowing that the information is inaccurate is, at best, 
foolish. At worst, possibly harmful to response time, wall clock time, CPU 
time, necessary/unnecessary REORG time, space calculations, performance 
indications, etc. The list is not endless, but you get my point. And if this 
foolishness is extended to thousands of VSAM datasets through automated 
processes, the costs of those *inaccurate decisions* could be very significant 
indeed.  

Mentioning the ones with stats that are fine in the context of this thread is 
irrelevant. Analyze those to your heart's content.

Your paragraph:

 "Just because they are not written when something BAD happens is NOT a reason 
to see them as invalid or useless.  Like anything else in this business (or 
life, for that matter), you need to know what you are talking about when you 
use statistics to justify a decision."

If these statistics were being inspected one-by-one, I might concede that last 
half of the second sentence. In my experience though, decisions of this sort 
are being made in an automated fashion without consideration for "knowing what 
you are talking about". They are being made under *the assumption that the 
statistics are accurate*, a point that has clearly been demonstrated as wrong!

Urgent or critical? Without empirical evidence to the contrary, how can *you 
assume* that it is neither/nor?  

I raise your two cents and make it my $.04 cents. 

Bob 

 -Original Message-
From:   IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED]  On Behalf Of 
Farley, Peter x23353
Sent:   Tuesday, July 05, 2005 5:52 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject:    Re: IBM VSAM Statistics are often Bogus

PMFJI here, but I cannot hold my (virtual) tongue any longer.

ALL of what is being argued about here ASSuMEs that somthing BAD has
happened.  Either an ABEND or a CICS IMMediate shutdown or some other
ABNORMAL event.

ISTM that for the vast majority of programmers who might use those VSAM
statistics, there are LONG stretches of time where their VSAM files are JUST
FINE, with no abends or other "bad" events to disturb the statistics.

In those cases, the stats are just what they say they are, and can
reasonably be used to make decisions (like when to reorganize or review
performance, etc.).

Just because they are not written when something BAD happens is NOT a reason
to see them as invalid or useless.  Like anything else in this business (or
life, for that matter), you need to know what you are talking about when you
use statistics to justify a decision.

Mark, I think your view might be warped by only getting involved from the
IBM side when "something BAD" has already happened to the hapless user.

Lighten up guys.  It's not that urgent or critical, really.  It's only
stats, after all.

Just my USD$0.02 worth.

Peter 
  
  
  
LEGAL DISCLAIMER 
The information transmitted is intended solely for the individual or entity to 
which it is addressed and may contain confidential and/or privileged material. 
Any review, retransmission, dissemination or other use of or taking action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you have received this email in error please 
contact the sender and delete the material from any computer. 
  
Seeing Beyond Money is a service mark of SunTrust Banks, Inc. 
[ST:XCL] 
 
 
 
 
 
 
 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ted MacNEIL]

...
Just because they are not written when something BAD happens is NOT a reason
to see them as invalid or useless.
...

I don't get it!
Invalid data is invalid data!

We make enough bad decisions with valid data.

How can you say that with a straight face?

-teD

In God we Trust!
All others bring data!
  --Deming

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Farley, Peter x23353]

PMFJI here, but I cannot hold my (virtual) tongue any longer.

ALL of what is being argued about here ASSuMEs that somthing BAD has
happened.  Either an ABEND or a CICS IMMediate shutdown or some other
ABNORMAL event.

ISTM that for the vast majority of programmers who might use those VSAM
statistics, there are LONG stretches of time where their VSAM files are JUST
FINE, with no abends or other "bad" events to disturb the statistics.

In those cases, the stats are just what they say they are, and can
reasonably be used to make decisions (like when to reorganize or review
performance, etc.).

Just because they are not written when something BAD happens is NOT a reason
to see them as invalid or useless.  Like anything else in this business (or
life, for that matter), you need to know what you are talking about when you
use statistics to justify a decision.

Mark, I think your view might be warped by only getting involved from the
IBM side when "something BAD" has already happened to the hapless user.

Lighten up guys.  It's not that urgent or critical, really.  It's only
stats, after all.

Just my USD$0.02 worth.

Peter

-Original Message-
From: Mark Thomen [mailto:[EMAIL PROTECTED]
Sent: Tuesday, July 05, 2005 5:41 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: IBM VSAM Statistics are often Bogus


"Joel C. Ewing" <[EMAIL PROTECTED]> wrote in message
news:<[EMAIL PROTECTED]>...
> I strongly disagree on the desirability of suppressing VSAM statistics
> in these cases.  It would be much better to display them while
> indicating that they are incomplete.
>
> The delta in the LISTCAT VSAM statistics correctly reflects the activity
> to the VSAM data set between two points in time, as long as you compare
>   between two points where the VSAM file is closed and also know there
> are no intervening OPENs without a valid CLOSE.  This is extremely
> useful as an activity measurement tool, even if the absolute values are
> missing counts from earlier accesses.

Most people don't do this.  They look at the stats and make decisions.
They don't look at the "delta" before and after using the data set.

As I said earlier, the biggest place where we see the problem are people
that do immediate shutdowns of CICS.  You load a file with 100K records on
Sunday night, and by Saturday night you've done 150K adds, 379K updates,
and 75K deletions.  Then you do an immediate shutdown and guess what you
get?  Stats that show 100K records in the file, no updates, no deletes, no
splits, no nothing.

>
> I would hate to see one program ABEND make it impossible to get any
> statistics from all VSAM datasets used by the program until those
> datasets have been reorganized by unload/reload, as in some cases it may
> not be practical to reorganize the files for weeks or months.

Sorry, but the stats don't get updated on an ABEND.

>
> This also suggests that calling the statistics "INVALID" may be too
> strong a pejorative, as all the counts reflect actual activity to the
> file -  it's just that they do not represent ALL the activity to the
file.

I'm afraid I don't understand how this doesn't make the results "invalid".
If your gas gauge says 1/4 tank as your engine coughs and halts, wouldn't
you consider your gas gauge registering "invalid"?

_
This message and any attachments are intended only for the use of the addressee 
and
may contain information that is privileged and confidential. If the reader of 
the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ted MacNEIL]

...
> This also suggests that calling the statistics "INVALID" may be too
> strong a pejorative, as all the counts reflect actual activity to the
> file -  it's just that they do not represent ALL the activity to the
file.

I'm afraid I don't understand how this doesn't make the results "invalid".
If your gas gauge says 1/4 tank as your engine coughs and halts, wouldn't
you consider your gas gauge registering "invalid"?
...

Hear! Hear!

Incomplete. Invalid. Meaningless.
Semantics.

The bottom line: the stats cannot help you!

Everything else is noise.


-teD

In God we Trust!
All others bring data!
  --Deming

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Mark Thomen]

"Joel C. Ewing" <[EMAIL PROTECTED]> wrote in message
news:<[EMAIL PROTECTED]>...
> I strongly disagree on the desirability of suppressing VSAM statistics
> in these cases.  It would be much better to display them while
> indicating that they are incomplete.
>
> The delta in the LISTCAT VSAM statistics correctly reflects the activity
> to the VSAM data set between two points in time, as long as you compare
>   between two points where the VSAM file is closed and also know there
> are no intervening OPENs without a valid CLOSE.  This is extremely
> useful as an activity measurement tool, even if the absolute values are
> missing counts from earlier accesses.

Most people don't do this.  They look at the stats and make decisions.
They don't look at the "delta" before and after using the data set.

As I said earlier, the biggest place where we see the problem are people
that do immediate shutdowns of CICS.  You load a file with 100K records on
Sunday night, and by Saturday night you've done 150K adds, 379K updates,
and 75K deletions.  Then you do an immediate shutdown and guess what you
get?  Stats that show 100K records in the file, no updates, no deletes, no
splits, no nothing.

>
> I would hate to see one program ABEND make it impossible to get any
> statistics from all VSAM datasets used by the program until those
> datasets have been reorganized by unload/reload, as in some cases it may
> not be practical to reorganize the files for weeks or months.

Sorry, but the stats don't get updated on an ABEND.

>
> This also suggests that calling the statistics "INVALID" may be too
> strong a pejorative, as all the counts reflect actual activity to the
> file -  it's just that they do not represent ALL the activity to the
file.

I'm afraid I don't understand how this doesn't make the results "invalid".
If your gas gauge says 1/4 tank as your engine coughs and halts, wouldn't
you consider your gas gauge registering "invalid"?

Thanks,
Mark Thomen
Catalog/IDCAMS/VSAM Development
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: IBM VSAM Statistics are often Bogus

[Ron and Jenny Hawkins]

Joel,

Why do you wait to the next day - when I want SMF data in a hurry I just
read the SYS1.MANx directly. Waiting until the next day to read 1 million
plus SMF records is a procedure problem and not a technical issue.

As far as the accuracy of VSAM statistics go, I concede that using SMF is no
more accurate than using LISTCAT. The same job that abends without updating
the statistics will also fail produce a type 64 records. I just happen to
think it is simpler to read a type 64 record than it is to insert a LISTCAT
before and after every job step that access a VSAM file. YMMV.

There are two basic problems with VSAM statistics. One is that you may be
missing frequency counts for inserts, splits, deletions, EXCP and whatever
for anything that has a VSAM file open when the power fails, the system
crashes, or the address itself fails for whatever reason. The counts for
those jobs are gone forever. I really don't see how a verify could possibly
correct that. How important these counters are to you really depends on what
you are doing with them, but where I have worked it is common practice to
start out any LISTC based analysis with a freshly reorged file, and discard
anything that has been touched by a failure.

The second problem is that there are counts such as number of records that
can become invalid. For example, if I have a KSDS with 1000 records, and I
have inserted 1500 more records when my CICS region abends, then my LISTCAT
will still show 1000 records. If subsequent processing then deletes 1001
records then my LISTCAT will be negative 1 record, while the actual count is
1499. If Verify is to rectify this statistic then it is going to have to
read the whole KSDS from front to back and count the records. And the logic
to actually figure out how the number of CI and CA splits that have occurred
in prior Opens really escapes me.

Calculating the difference for various statistics from a LISTCAT before and
after a job step is a perfectly valid and simple way to measure VSAM
activity for that open. Comparing before and after statistics in this way is
unaffected by the accuracy of the statistics, because it is the difference
between the two that you are measuring. But if you want to take a LISTCAT of
a VSAM dataset that was defined 18 months ago and use those statistics for
some sort meaningful tuning or capacity planning - well I for one fail to
see how it would be useful whether the statistics were in error or not.

Ron

> -Original Message-
> From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
> Behalf Of Joel C. Ewing
> Sent: Saturday, 2 July 2005 3:50 PM
> To: IBM-MAIN@BAMA.UA.EDU
> Subject: Re: IBM VSAM Statistics are often Bogus
> 
> Ask yourself which is easier and faster in a case where one has reason
> to suspect a VSAM usage problem:
> (1) running LISTCAT, running program, running LISTCAT;
> or
> (2) running program, waiting until following day when offloaded SMF
> records become generally available, running or requesting a run of a
> batch program to read through 1M+ SMF records to possibly locate
> relevant statistics?
> 
> The "simplest" approach is frequently "better".
> 
> The fallacy of John's casino analogy is that the underreporting of VSAM
> statistics is not caused by unpredictable and unknown actions of
> individuals, but by specific understood system events which are mostly
> observable (ABEND or cancellation of address spaces) or associated with
> the running of specific programs.  You may not know exactly what was
> lost, but generally can determine when the losses occurred.  There are
> many useful cases where one can be 100% sure that you have an interval
> in which no statistics have been lost, and can therefore accurately
> assess the number of additional EXCP's, CI/CA splits, records
> fetched/added/deleted, etc. resulting from applications run in that
> interval.
> 
> As with many statistics related to MVS, one has to understand what they
> mean and their limitations in order to use them intelligently.  I don't
> accept that it makes sense to eliminate LISTCAT VSAM statistics just
> because some users don't understand the limitations.  Flagging the
> statistics as incomplete or questionable ought to be sufficient to warn
> neophyte users that there may be something about the values they don't
> understand - hopefully prompting them to research and understand the
> limitations.
> 
> 
> 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html