Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Seymour J Metz]

What is a physical block? Even without taking into accounnt that contemporary 
DASD subsystems are cached, on old CKD DASD you could read any of count, key 
and data without reading the others.


-- 
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3
עַם יִשְׂרָאֵל חַי




From: IBM Mainframe Assembler List  on behalf 
of Bob Raicer 
Sent: Monday, November 13, 2023 5:59 PM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

There have been a few posts in this thread that more or less stated
that only complete physical blocks could be read into storage from a
device.

This is not strictly true.

There is the obvious case of truncation, where the number of bytes
the device has been requested to transfer is less than the actual
physical size of the block.

For CCW types of I/O operations (as opposed to TCW types) there is
also the possibility of exploiting the "Skip" feature of the
channel.

Bit 35 (format-0 CCWs) or bit 11 (format-1 CCWs) of the CCW is
referred to as the Skip Flag Bit.  When this bit is one, the channel
suppresses the transfer of data into storage.  This flag bit is
valid ONLY for read, read-backward, sense ID, or sense operations.
Note that the storage device still engages with the data transfer to
the channel and all of the data error checking functions (such as
ECC) still occur.  It is the channel that suppresses the storage of
data into memory.  When combined with Data Chaining this can be used
to store subsets of a data block into potentially discontiguous
areas of memory.

I do not believe that any of the access methods exploit this,
however.

OPEN/CLOSE/EOV at one time used this feature when processing
Standard User Label tapes.  The interesting portion of the channel
program was a Read with the Skip Bit on, command chained to a
Transfer In Channel back to the Read.  When no errors occurred, the
channel program ended with Unit Exception status when the Tape Mark
at the end of the label set was encountered.

Bob

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Paul Gilmartin]

On 11/13/23 15:59:45, Bob Raicer wrote:

There have been a few posts in this thread that more or less stated
that only complete physical blocks could be read into storage from a
device.

This is not strictly true.> 

We don't know the OP's objective: performance, security, or other.


--
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Bob Raicer]

There have been a few posts in this thread that more or less stated
that only complete physical blocks could be read into storage from a
device.

This is not strictly true.

There is the obvious case of truncation, where the number of bytes
the device has been requested to transfer is less than the actual
physical size of the block.

For CCW types of I/O operations (as opposed to TCW types) there is
also the possibility of exploiting the "Skip" feature of the
channel.

Bit 35 (format-0 CCWs) or bit 11 (format-1 CCWs) of the CCW is
referred to as the Skip Flag Bit.  When this bit is one, the channel
suppresses the transfer of data into storage.  This flag bit is
valid ONLY for read, read-backward, sense ID, or sense operations.
Note that the storage device still engages with the data transfer to
the channel and all of the data error checking functions (such as
ECC) still occur.  It is the channel that suppresses the storage of
data into memory.  When combined with Data Chaining this can be used
to store subsets of a data block into potentially discontiguous
areas of memory.

I do not believe that any of the access methods exploit this,
however.

OPEN/CLOSE/EOV at one time used this feature when processing
Standard User Label tapes.  The interesting portion of the channel
program was a Read with the Skip Bit on, command chained to a
Transfer In Channel back to the Read.  When no errors occurred, the
channel program ended with Unit Exception status when the Tape Mark
at the end of the label set was encountered.

Bob

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Ed Jaffe]

On 11/13/2023 7:23 AM, Schmitt, Michael wrote:

As long no one ever uses DISP=MOD

...or checkpoints. Checkpoints can cause short blocks to be written.


What about CLOSE TYPE=T? Would that flush the current block?

--
Phoenix Software International
Edward E. Jaffe
831 Parkview Drive North
El Segundo, CA 90245
https://www.phoenixsoftware.com/



This e-mail message, including any attachments, appended messages and the
information contained therein, is for the sole use of the intended
recipient(s). If you are not an intended recipient or have otherwise
received this email message in error, any use, dissemination, distribution,
review, storage or copying of this e-mail message and the information
contained therein is strictly prohibited. If you are not an intended
recipient, please contact the sender by reply e-mail and destroy all copies
of this email message and do not otherwise utilize or retain this email
message or any or all of the information contained therein. Although this
email message and any attachments or appended messages are believed to be
free of any virus or other defect that might affect any computer system into
which it is received and opened, it is the responsibility of the recipient
to ensure that it is virus free and no responsibility is accepted by the
sender for any loss or damage arising in any way from its opening or use.

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Seymour J Metz]

At one time there was a component called Actual Block Processor that was used 
both by VSAM and paging, and the paging data sets were CI format.  I have no 
idea how much of that, if any, remains in z/OS, but essentially my question was 
whether pages were now stored in 4 KiB physical records.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3
עַם יִשְׂרָאֵל חַי




From: IBM Mainframe Assembler List  on behalf 
of Bob Raicer 
Sent: Monday, November 13, 2023 2:14 PM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

A PDSE is essentially a form of VSAM Linear, where the physical data
set is a collection of 4K-byte blocks.  It does not have the notion
of CA's and CI's like other VSAM data sets.  The notion of BLKSIZE
is synthesized and mapped onto the internal PDSE structures; the
BLKSIZE value is not the physical unit of transfer size as is the
typical case for a traditional Physical Sequential data set.

Here is a URL for a SHARE presentation that provides a nice
explanation of how a PDSE is put together:

https://share.confex.com/share/124/webprogram/Handout/Session16956/PDSE%20Nuts%20and%20Bolts.pdf

Bob

On 2023-11-13 7:48 AM, Seymour J Metz wrote:
>
> Does anybody know whether CI size = block size for PDSE and zFS (linear) data 
> sets? VSAM used to use multiple blocks for a CI.
>
>
>

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Bob Raicer]

A PDSE is essentially a form of VSAM Linear, where the physical data
set is a collection of 4K-byte blocks.  It does not have the notion
of CA's and CI's like other VSAM data sets.  The notion of BLKSIZE
is synthesized and mapped onto the internal PDSE structures; the
BLKSIZE value is not the physical unit of transfer size as is the
typical case for a traditional Physical Sequential data set.

Here is a URL for a SHARE presentation that provides a nice
explanation of how a PDSE is put together:

https://share.confex.com/share/124/webprogram/Handout/Session16956/PDSE%20Nuts%20and%20Bolts.pdf

Bob

On 2023-11-13 7:48 AM, Seymour J Metz wrote:


Does anybody know whether CI size = block size for PDSE and zFS (linear) data 
sets? VSAM used to use multiple blocks for a CI.

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Paul Gilmartin]

On 11/13/23 11:54:07, Schmitt, Michael wrote:

That I don't know. I've only dealt with GSAM checkpoints (which may not support 
RECFM=VBS). And one other: checkpoints in z/OS FTP.

My *guess* is that since a checkpoint has to flush the buffers to disk before 
you fill up the block, even VBS would have a short block.



I stand corrected.  I intended to inquire about RECFM=FBS.


-Original Message-
From: IBM Mainframe Assembler List  On Behalf 
Of Paul Gilmartin
Sent: Monday, November 13, 2023 12:50 PM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

On 11/13/23 08:23:44, Schmitt, Michael wrote:

As long no one ever uses DISP=MOD


...or checkpoints. Checkpoints can cause short blocks to be written.

(My knowledge is with IMS/DB GSAM checkpoints.)


How does checkpoint interact with RECFM=VBS?

--
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Schmitt, Michael]

That I don't know. I've only dealt with GSAM checkpoints (which may not support 
RECFM=VBS). And one other: checkpoints in z/OS FTP.

My *guess* is that since a checkpoint has to flush the buffers to disk before 
you fill up the block, even VBS would have a short block.

-Original Message-
From: IBM Mainframe Assembler List  On Behalf 
Of Paul Gilmartin
Sent: Monday, November 13, 2023 12:50 PM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

On 11/13/23 08:23:44, Schmitt, Michael wrote:
>>> As long no one ever uses DISP=MOD
>
> ...or checkpoints. Checkpoints can cause short blocks to be written.
>
> (My knowledge is with IMS/DB GSAM checkpoints.)

How does checkpoint interact with RECFM=VBS?

--
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Steve Smith]

Same as above.

On Mon, Nov 13, 2023 at 1:50 PM Paul Gilmartin <
0014e0e4a59b-dmarc-requ...@listserv.uga.edu> wrote:

> On 11/13/23 08:23:44, Schmitt, Michael wrote:
> >>> As long no one ever uses DISP=MOD
> >
> > ...or checkpoints. Checkpoints can cause short blocks to be written.
> >
> > (My knowledge is with IMS/DB GSAM checkpoints.)
>
> How does checkpoint interact with RECFM=VBS?
>
> --
> gil
>

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Steve Smith]

While the original question isn't really about assembler, it is at least
about programming.  This tangent has nothing to do with either... if you
want uninformed speculation on random topics, that's what IBM-MAIN is for.

sas

On Mon, Nov 13, 2023 at 9:48 AM Seymour J Metz  wrote:

> I believe that the OP wanted a solution for BSAM. If VSAM is an option
> then it's a nobrainer.
>
> Does anybody know whether CI size = block size for PDSE and zFS (linear)
> data sets? VSAM used to use multiple blocks for a CI.
>
>

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Paul Gilmartin]

On 11/13/23 08:23:44, Schmitt, Michael wrote:

As long no one ever uses DISP=MOD


...or checkpoints. Checkpoints can cause short blocks to be written.

(My knowledge is with IMS/DB GSAM checkpoints.)


How does checkpoint interact with RECFM=VBS?

--
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Schmitt, Michael]

>> As long no one ever uses DISP=MOD

...or checkpoints. Checkpoints can cause short blocks to be written.

(My knowledge is with IMS/DB GSAM checkpoints.)

-Original Message-
From: IBM Mainframe Assembler List  On Behalf 
Of Jon Perryman
Sent: Sunday, November 12, 2023 5:18 PM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

Everyone is making this far more complicated than it needs to be. Positioning a 
file is about you designing a method that can calculate a position on the disk 
that meets your requirements. z/OS, Unix, CMS, z/VSE and MS-Windows have file / 
dataset positioning because it greatly reduces I/O. There are many designs to 
available to calculate position.

Specifically asked was z/OS RECFM=FB. As long no one ever uses DISP=MOD, then 
the calculation is simple because number of records per block and blocks per 
track is consistent. You simply divide record number by records per block 
giving the block number used in the point and remainder is the number of 
records you must read after the point. Not quite correct but you get the idea.

If you need DISP=MOD functionality, then you must create the functionality. All 
programs needing DISP=MOD must open the file for update, position to the last 
block and read until EOF.

The simpler solution is VSAM RRDS or ESDS which allows you to position to a 
specific record number without the worry of DISP=MOD problems.

As I said before, there are many solutions and your requirements will determine 
which solution best fits your needs.

Suggestions have been made for a USS file. The advantage is that positioning is 
at the byte level instead of block. The calculation is greatly simplified to 
records times record size and DISP=MOD is greatly simplified because Unix does 
an append to the end of the file. The drawback is that the read record size 
must be handled by the programmer.

Most of the suggestions are viable solutions. You could even maintain an index 
file but then RRDS already does this for you. The important thing is a 
consistent method for POINT is determining the block and record for positioning.

On Sun, 12 Nov 2023 12:17:12 -0700, Paul Gilmartin  wrote:
>what does the EXECIO linenum option do?
>
>EXECIO arose on CMS and antedates REXX.  CMS files are indexed by
>line number.  I'd expect EXECIO on CMS to do True Skip-Sequential
>Processing.  Less likely on z/OS, even for RECFM=FBS.

As for CMS files, It's very doubtful they are indexed because there is no real 
benefit when the majority of files are processed sequentially. Records in a CMS 
file are probably stored in what z/OS people would think of as RECFM=VBS where 
blocksize is probably track size.

I doubt that CMS files have line number associations. CMS edit has number 
on/off but it has a different use. More likely is that CMS keeps a counter with 
the record number.

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Seymour J Metz]

I believe that the OP wanted a solution for BSAM. If VSAM is an option then 
it's a nobrainer.

Does anybody know whether CI size = block size for PDSE and zFS (linear) data 
sets? VSAM used to use multiple blocks for a CI.



-- 
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3
עַם יִשְׂרָאֵל חַי




From: IBM Mainframe Assembler List  on behalf 
of Jon Perryman 
Sent: Sunday, November 12, 2023 8:27 PM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

On Sun, 12 Nov 2023 17:14:49 -0700, Paul Gilmartin  wrote:

>RECFM=FBS prevents that.

That is certainly a partial solution but the correct solution is VSAM RRDS or 
ESDS which eliminates the special handling.

>That's mostly how the Divide instruction works.

Division is not the hard part. Implementation is what's being discussed which 
involves division..

>... Fill the block with additional records and rewrite in place.

What ever floats your boat. If the OP wants to fully implement that logic, it's 
available.

>Why, in this 21st Century hasn't that function been added to the
>access methods!?  UNIX does it, as you say below.

What makes you falsely claim that function doesn't exist. There is no 
difference between z/OS and Unix files in this respect. Your programs must deal 
with those extra records by ignoring them. Both must retain a pointer to the 
first available instead of using EOF.

>I think that can be done by the Multiply instruction, provided
>by most modern computer hardware.

Wow, the computers you use can't multiply? I stated "times" which is another 
word for multiply.

> But fixed-length records are
>alien to UNIX -- I think they're a relic of UR-device practice.

RECFM=FB & VB are not relics. It's a vital concept used by databases. A max 
length is required by VARCHAR for a reason. They know it's more efficient to 
add a number instead of scanning for X'00'. Databases encourage you to supply 
max length for everything and truly unknown lengths are avoided.  Just because 
a basic concept is not understood by Unix programmers doesn't mean it doesn't 
have great value.

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Jonathan Scott]

Someone mentioned the CMS file system in a recent post.

The CMS file system uses a hierarchical block map system for
each file, which gives very efficient direct access by record
number (both for fixed-length and varying-length records)
avoiding the need to read any previous file data blocks
(although it may be necessary to read higher-level map blocks,
which are likely to be cached anyway).

Jonathan Scott, HLASM
IBM Hursley, UK

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Jon Perryman]

On Sun, 12 Nov 2023 17:14:49 -0700, Paul Gilmartin  wrote:

>RECFM=FBS prevents that.

That is certainly a partial solution but the correct solution is VSAM RRDS or 
ESDS which eliminates the special handling.

>That's mostly how the Divide instruction works.

Division is not the hard part. Implementation is what's being discussed which 
involves division..

>... Fill the block with additional records and rewrite in place.

What ever floats your boat. If the OP wants to fully implement that logic, it's 
available.

>Why, in this 21st Century hasn't that function been added to the
>access methods!?  UNIX does it, as you say below.

What makes you falsely claim that function doesn't exist. There is no 
difference between z/OS and Unix files in this respect. Your programs must deal 
with those extra records by ignoring them. Both must retain a pointer to the 
first available instead of using EOF. 

>I think that can be done by the Multiply instruction, provided
>by most modern computer hardware.  

Wow, the computers you use can't multiply? I stated "times" which is another 
word for multiply.

> But fixed-length records are
>alien to UNIX -- I think they're a relic of UR-device practice.

RECFM=FB & VB are not relics. It's a vital concept used by databases. A max 
length is required by VARCHAR for a reason. They know it's more efficient to 
add a number instead of scanning for X'00'. Databases encourage you to supply 
max length for everything and truly unknown lengths are avoided.  Just because 
a basic concept is not understood by Unix programmers doesn't mean it doesn't 
have great value.

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Paul Gilmartin]

On 11/12/23 16:18:04, Jon Perryman wrote:


Specifically asked was z/OS RECFM=FB. As long no one ever uses DISP=MOD, then 
the calculation is simple because number of records per block and blocks per 
track is consistent.



RECFM=FBS prevents that.

 You simply divide record number by records per block giving the block number 
used in the point and remainder is the number of records you must read after 
the point. Not quite correct but you get the idea.




That's mostly how the Divide instruction works.


If you need DISP=MOD functionality, then you must create the functionality. All 
programs needing DISP=MOD must open the file for update, position to the last 
block and read until EOF.



... Fill the block with additional records and rewrite in place.
Why, in this 21st Century hasn't that function been added to the
access methods!?  UNIX does it, as you say below.



Suggestions have been made for a USS file. The advantage is that positioning is 
at the byte level instead of block. The calculation is greatly simplified to 
records times record size and DISP=MOD is greatly simplified because Unix does 
an append to the end of the file. The drawback is that the read record size 
must be handled by the programmer.



I think that can be done by the Multiply instruction, provided
by most modern computer hardware.  But fixed-length records are
alien to UNIX -- I think they're a relic of UR-device practice.

--
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Jon Perryman]

Everyone is making this far more complicated than it needs to be. Positioning a 
file is about you designing a method that can calculate a position on the disk 
that meets your requirements. z/OS, Unix, CMS, z/VSE and MS-Windows have file / 
dataset positioning because it greatly reduces I/O. There are many designs to 
available to calculate position. 

Specifically asked was z/OS RECFM=FB. As long no one ever uses DISP=MOD, then 
the calculation is simple because number of records per block and blocks per 
track is consistent. You simply divide record number by records per block 
giving the block number used in the point and remainder is the number of 
records you must read after the point. Not quite correct but you get the idea. 

If you need DISP=MOD functionality, then you must create the functionality. All 
programs needing DISP=MOD must open the file for update, position to the last 
block and read until EOF.

The simpler solution is VSAM RRDS or ESDS which allows you to position to a 
specific record number without the worry of DISP=MOD problems.

As I said before, there are many solutions and your requirements will determine 
which solution best fits your needs. 

Suggestions have been made for a USS file. The advantage is that positioning is 
at the byte level instead of block. The calculation is greatly simplified to 
records times record size and DISP=MOD is greatly simplified because Unix does 
an append to the end of the file. The drawback is that the read record size 
must be handled by the programmer.

Most of the suggestions are viable solutions. You could even maintain an index 
file but then RRDS already does this for you. The important thing is a 
consistent method for POINT is determining the block and record for positioning.

On Sun, 12 Nov 2023 12:17:12 -0700, Paul Gilmartin  wrote:
>what does the EXECIO linenum option do?
>
>EXECIO arose on CMS and antedates REXX.  CMS files are indexed by
>line number.  I'd expect EXECIO on CMS to do True Skip-Sequential
>Processing.  Less likely on z/OS, even for RECFM=FBS.

As for CMS files, It's very doubtful they are indexed because there is no real 
benefit when the majority of files are processed sequentially. Records in a CMS 
file are probably stored in what z/OS people would think of as RECFM=VBS where 
blocksize is probably track size. 

I doubt that CMS files have line number associations. CMS edit has number 
on/off but it has a different use. More likely is that CMS keeps a counter with 
the record number. 

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Paul Gilmartin]

On 11/11/23 19:36:36, Seymour J Metz wrote:

It is possible to do skip sequential reading of physical records. Itt is not 
possible to skip logical records.

There are two cases where it is possible; RECFM=FBS and you know the physical 
record number; you know the TTR of  the physical record.



what does the EXECIO linenum option do?


EXECIO arose on CMS and antedates REXX.  CMS files are indexed by
line number.  I'd expect EXECIO on CMS to do True Skip-Sequential
Processing.  Less likely on z/OS, even for RECFM=FBS.

--
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Seymour J Metz]

I'm not sure what ISPF does as far as indexing sequence numbers.

For other applications, the devil is in the detail. There are various cases 
where the initialization to determine or store TTRs is noise compared to read 
time. I had one application where the data changed twice a year. Storing TTRs 
in the records twice a year is a negligible amount of time. BDAM or ISAM would 
have been overkill.


-- 
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3
עַם יִשְׂרָאֵל חַי




From: IBM Mainframe Assembler List  on behalf 
of Paul Gilmartin <0014e0e4a59b-dmarc-requ...@listserv.uga.edu>
Sent: Saturday, November 11, 2023 9:36 PM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

On Sun, 12 Nov 2023 02:24:50 +, Seymour J Metz wrote:

>It doesn't have to be FBS if you know the location of the block.
>
Doesn't ISPF BROWSE do  something of that sort, creating an
index for rapid positioning to a requested record?  But it must
(except for FBS?) read once to build the index.


On Sun, 12 Nov 2023 02:21:26 +, Seymour J Metz wrote:

>Are you a betting man?
>
what's your proposition?

--
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Paul Gilmartin]

On Sun, 12 Nov 2023 02:24:50 +, Seymour J Metz wrote:

>It doesn't have to be FBS if you know the location of the block.
>
Doesn't ISPF BROWSE do  something of that sort, creating an
index for rapid positioning to a requested record?  But it must
(except for FBS?) read once to build the index.


On Sun, 12 Nov 2023 02:21:26 +, Seymour J Metz wrote:

>Are you a betting man? 
>
what's your proposition?

-- 
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Seymour J Metz]

It is possible to do skip sequential reading of physical records. Itt is not 
possible to skip logical records.

There are two cases where it is possible; RECFM=FBS and you know the physical 
record number; you know the TTR of  the physical record.

It's actually possible to do it in QSAM but this margin is too small to hold 
the proof. Don't do it at home.

--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3
עַם יִשְׂרָאֵל חַי




From: IBM Mainframe Assembler List  on behalf 
of David S. 
Sent: Saturday, November 11, 2023 8:59 AM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

To help resolve a question posted to a LinkedIn group I manage:
http://www.linkedin.com/feed/update/urn:li:groupPost:910927-7128598004344786944
... I'd like to find out if there's any way to achieve *true*
Skip-Sequential processing with a Fixed Block Sequential File with a fairly
short record length (i.e. DCB=(DSORG=PS,RECFM=FB,LRECL=80)?
For example: Begin sequential processing at record number 100, *without*
having to read the first 99 records.
Note: We already know certain VSAM formats can do this, but the file in
question is a DSORG=PS *Sequential* file, *not* VSAM. This is a rock-solid
requirement and cannot be changed. We also already know how certain
utilities such as SORT and REXX can *mimic* skip-sequential functionality
by *discarding* unwanted records until the specified record number is
reached. This is a likewise rock-solid requirement. Sequential processing
*must* begin at specified starting point and there can be *no* reading of
any records prior to that point.
My gut feeling is it *cannot* be done - at least not with RECFM=FB.  It
*might* be possible with RECFM=F, but efficiency would then be so
compromised it would  probably outweigh any advantage from *true*
skip-sequential processing.

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Seymour J Metz]

It doesn't have to be FBS if you know the location of the block.


-- 
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3
עַם יִשְׂרָאֵל חַי




From: IBM Mainframe Assembler List  on behalf 
of Binyamin Dissen 
Sent: Saturday, November 11, 2023 12:30 PM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

On Sat, 11 Nov 2023 10:07:14 -0500 Steve Smith  wrote:

:>Sure it's possible... BSAM POINT should work fine in general, and BDAM is
:>even more powerful.  However, the previous mention about short blocks is
:>important; that will foul up block calculations, so you'd probably want to
:>make it FBS, and make sure there are no short blocks.  Presuming your
:>application knows what record it wants to start with, you'll have to
:>calculate the TTR and block offset from that.  And of course, by using
:>BxAM, you have to do your own de-blocking, and I/O overlap.

Yes, it would have to be FBS. The block address can be calculated and a BDAM
direct read could retrieve the block.

However this is the wrong organization for the file if that is your need. No
matter how hard you try, a hammer is a poor tool to put in a screw.

--
Binyamin Dissen 
http://secure-web.cisco.com/1CMWALOVNRtco6uxkLaJ4W4d5JF61Tll8r5RfxJTQEZbMc0O-lpe5e2A9DOpYv_4hXIlCPyote0JZQD3L9oZN1dUI02GMKQfcMamv_R7boNUpJcPEbzJCuxHWwQj9VAntNbivcZmnBEFIdzDCcp3GfRdeHQs4VKQSUy3UsJgz0zW9gdmFFDkdPCZqn-ntK8RfwbqundRpx5zebeWyLnrh67VcdU9y660GP4yTEljwk-alYcIjV3v03Eq7M35DRpt-goBbGfYmo4XPV-GXq3jrJJfxBA8trlWOope9h1kSN7LexAcYipOwfEXsfMCHlVHs37vJfEm28-8Ykhne0naHKVcKNVZv_dlzUf3pkRCxSsAIVwsy-B_VXOsz_JfMhvC3Z2PwK9trVQ0vFaqrcoF79TnshPF20V0YMWfPH89tAJU/http%3A%2F%2Fwww.dissensoftware.com

Director, Dissen Software, Bar & Grill - Israel

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Seymour J Metz]

The word record is ambiguous. With BSAM you can point to and read a single 
physical record, but it may contain multiple logical records.

--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3
עַם יִשְׂרָאֵל חַי




From: IBM Mainframe Assembler List  on behalf 
of dave.g4...@gmail.com 
Sent: Saturday, November 11, 2023 6:48 PM
To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
Subject: Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

> -Original Message-
> From: IBM Mainframe Assembler List 
> On Behalf Of David S.
> Sent: Saturday, November 11, 2023 1:59 PM
> To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
> Subject: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?
>
> To help resolve a question posted to a LinkedIn group I manage:
> http://www.linkedin.com/feed/update/urn:li:groupPost:910927-
> 7128598004344786944
> ... I'd like to find out if there's any way to achieve *true* Skip-Sequential
> processing with a Fixed Block Sequential File with a fairly short record 
> length (i.e.
> DCB=(DSORG=PS,RECFM=FB,LRECL=80)?

You don't say what the block size is. You don't say what the access methos is 
QSAM/BSAM

> For example: Begin sequential processing at record number 100, *without*
> having to read the first 99 records.

Probably not. The data transfer is driven/implemented by a "channel program" 
and from what I remember a channel program can only read "blocks" from a disk, 
not parts of a block. So if the block size is 8000 bytes each read reads 100 80 
byte records. The POINT  MACRO does allow positioning.

However this is a more complex question. Whilst with an "OS" type operating 
system allows an arbitrary physical record size to be defined for a dataset, on 
any current disk type, this is achieved by mapping this onto fixed size blocks 
which are what is actually written to the disk. So reading an 8k block will 
generally involve reading multiple blocks from the disk media

> Note: We already know certain VSAM formats can do this, but the file in
> question is a DSORG=PS *Sequential* file, *not* VSAM.

Well again VSAM only reads BLOCKS so if we are talking large block sizes, you 
can only position to the block containing the record you want.

> This is a rock-solid
> requirement and cannot be changed. We also already know how certain utilities
> such as SORT and REXX can *mimic* skip-sequential functionality by
> *discarding* unwanted records until the specified record number is reached.
> This is a likewise rock-solid requirement. Sequential processing
> *must* begin at specified starting point and there can be *no* reading of any
> records prior to that point.

> My gut feeling is it *cannot* be done - at least not with RECFM=FB.  It
> *might* be possible with RECFM=F, but efficiency would then be so
> compromised it would  probably outweigh any advantage from *true* skip-
> sequential processing.

Like I say POINT can do this but you are then limited to BSAM...
... YOU DIDN'T SAY QSAM ONLY NOT VSAM

Dave

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[dave . g4ugm]

> -Original Message-
> From: IBM Mainframe Assembler List 
> On Behalf Of David S.
> Sent: Saturday, November 11, 2023 1:59 PM
> To: ASSEMBLER-LIST@LISTSERV.UGA.EDU
> Subject: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?
> 
> To help resolve a question posted to a LinkedIn group I manage:
> www.linkedin.com/feed/update/urn:li:groupPost:910927-
> 7128598004344786944
> ... I'd like to find out if there's any way to achieve *true* Skip-Sequential
> processing with a Fixed Block Sequential File with a fairly short record 
> length (i.e.
> DCB=(DSORG=PS,RECFM=FB,LRECL=80)?

You don't say what the block size is. You don't say what the access methos is 
QSAM/BSAM 

> For example: Begin sequential processing at record number 100, *without*
> having to read the first 99 records.

Probably not. The data transfer is driven/implemented by a "channel program" 
and from what I remember a channel program can only read "blocks" from a disk, 
not parts of a block. So if the block size is 8000 bytes each read reads 100 80 
byte records. The POINT  MACRO does allow positioning.

However this is a more complex question. Whilst with an "OS" type operating 
system allows an arbitrary physical record size to be defined for a dataset, on 
any current disk type, this is achieved by mapping this onto fixed size blocks 
which are what is actually written to the disk. So reading an 8k block will 
generally involve reading multiple blocks from the disk media 

> Note: We already know certain VSAM formats can do this, but the file in
> question is a DSORG=PS *Sequential* file, *not* VSAM. 

Well again VSAM only reads BLOCKS so if we are talking large block sizes, you 
can only position to the block containing the record you want.

> This is a rock-solid
> requirement and cannot be changed. We also already know how certain utilities
> such as SORT and REXX can *mimic* skip-sequential functionality by
> *discarding* unwanted records until the specified record number is reached.
> This is a likewise rock-solid requirement. Sequential processing
> *must* begin at specified starting point and there can be *no* reading of any
> records prior to that point.

> My gut feeling is it *cannot* be done - at least not with RECFM=FB.  It
> *might* be possible with RECFM=F, but efficiency would then be so
> compromised it would  probably outweigh any advantage from *true* skip-
> sequential processing.

Like I say POINT can do this but you are then limited to BSAM...
... YOU DIDN'T SAY QSAM ONLY NOT VSAM

Dave

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Steve Smith]

I suspect that a dive into the Unix side is not what he wants to do.  But
it's an interesting idea.

My suggestion was primarily for BSAM with POINT.  That seems to meet the
OP's needs.  The BDAM mention was an aside, and would be overkill.

Although I might say BDAM deserves more respect than it gets from IBM.
Nevertheless, I don't think it would be smart to start developing anything
based on it.  There are easier and better modern alternatives, such as
RRDS, ESDS, LDS, Unix files... I spent a few years working with BDAM
programs & files that would have naturally fit LDS.  But, LDS hadn't been
invented yet, so we made do.

sas


On Sat, Nov 11, 2023 at 5:19 PM Paul Gilmartin <
0014e0e4a59b-dmarc-requ...@listserv.uga.edu> wrote:

> On 11/11/23 06:59:07, David S. wrote:
> > To help resolve a question posted to a LinkedIn group I manage:
> > www.linkedin.com/feed/update/urn:li:groupPost:910927-7128598004344786944
> > ... I'd like to find out if there's any way to achieve *true*
> > Skip-Sequential processing with a Fixed Block Sequential File with a
> fairly
> > short record length (i.e. DCB=(DSORG=PS,RECFM=FB,LRECL=80)?
> > For example: Begin sequential processing at record number 100, *without*
> > having to read the first 99 records.
> >
> >
> This feels like a job for DSFS.
>  >
>
> UNIX readily solves the problem with seek() and DSFS is supposed
> to mimic a UNIX file with the content of a Classic data set.
>
> Where's the User's Guide for DSFS?
>
> Is the skip count fixed, or is it dynamic, varying up or down
> with successive executions of your program?
>
> --
> gil
>

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Paul Gilmartin]

On 11/11/23 06:59:07, David S. wrote:

To help resolve a question posted to a LinkedIn group I manage:
www.linkedin.com/feed/update/urn:li:groupPost:910927-7128598004344786944
... I'd like to find out if there's any way to achieve *true*
Skip-Sequential processing with a Fixed Block Sequential File with a fairly
short record length (i.e. DCB=(DSORG=PS,RECFM=FB,LRECL=80)?
For example: Begin sequential processing at record number 100, *without*
having to read the first 99 records.



This feels like a job for DSFS.


UNIX readily solves the problem with seek() and DSFS is supposed
to mimic a UNIX file with the content of a Classic data set.

Where's the User's Guide for DSFS?

Is the skip count fixed, or is it dynamic, varying up or down
with successive executions of your program?

--
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Bob Raicer]

I agree with Paul regarding the absurdity he mentioned.

Here's another scheme you might want to consider.

Let's call the set of logical records currently at the beginning of
the data set the Restricted Access Set.  Presumably, the remaining
records comprise the Unrestricted Access Set.

You could create two files:
File One contains the Restricted Access Set records.
File Two contains the Unrestricted Access Set records.

All file I/O can be done with QSAM without needing to perform any
type of record location calculations and the corresponding
positioning operations.  You can make the Block Size anything that
makes you happy; let Data Management do all of the blocking and
unblocking and deal with all of the device idiosyncrasies.  The two
files could even reside on very different types of devices and
possibly different file systems.

Both sets could be read as a logical single entity by concatenating
the two files.  The Unrestricted Set could be read (and written if
so desired) by simply OPENing that file.  The same thing goes for
the Restricted Set.  The concatenation could be done in JCL and/or
Dynamic Allocation.

If you do not intend to Extend either set, the two files could be
members of a PDS, and you can access the members using QSAM.

If you DO intend to extend either set (where the containing file is
a PDS), then DISP=MOD (or OPEN EXTEND) against a PDS member is
disallowed by Data Management.  In this case you would need to
create a "new" member by copying the "old" member data to the "new"
member, and then (without having CLOSEd the "new" member) add the
new logical records.  After CLOSEing the "new" member you would need
to delete the "old" member and rename the "new" member, giving it
the "old" member name.  In this kind of scenario a PDSE would be
better since it never needs to be compressed.

Clearly, the two separate files approach is the simplest.

Bob

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Paul Gilmartin]

On Sat, 11 Nov 2023 08:59:07 -0500, David S. wrote:

   ...

. Sequential processing

*must* begin at specified starting point and there can be *no* reading of
any records prior to that point.


That sounds absurd.  So if a block contains 10 records, you want to
be able to read the last 5 with "*no* reading of" the first 5!?
DASD don't work that way.


My gut feeling is it *cannot* be done - at least not with RECFM=FB.  It
*might* be possible with RECFM=F, but efficiency would then be so
compromised it would  probably outweigh any advantage from *true*
skip-sequential processing.


How about UNIX files with RECFM=FB,FILEDATA=BINARY and using
seek()?  (But that still reads entire blocks into a buffer.)

--
gil

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Bob Raicer]

More than a decade ago when I was working as a developer on the
implementation and support of Extended Address Volumes, the plan
of record was to "functionally stabilize" BDAM.  At that time there
was zero work planned or being done on BDAM for EAV support.  To be
honest, I haven't paid any attention to BDAM since, so I cannot say
for certain what BDAM improvements may have occurred.  That said, I
would avoid using BDAM unless you have absolutely no other choice.

As others have advised, you can likely achieve your goal using BSAM,
subject to the constraints and caveats they've stated.

I agree with the sentiment stated by Binyamin Dissen:

"However this is the wrong organization for the file if that is your
need.  No matter how hard you try, a hammer is a poor tool to put in
a screw."

Bob

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Binyamin Dissen]

On Sat, 11 Nov 2023 10:07:14 -0500 Steve Smith  wrote:

:>Sure it's possible... BSAM POINT should work fine in general, and BDAM is
:>even more powerful.  However, the previous mention about short blocks is
:>important; that will foul up block calculations, so you'd probably want to
:>make it FBS, and make sure there are no short blocks.  Presuming your
:>application knows what record it wants to start with, you'll have to
:>calculate the TTR and block offset from that.  And of course, by using
:>BxAM, you have to do your own de-blocking, and I/O overlap.

Yes, it would have to be FBS. The block address can be calculated and a BDAM
direct read could retrieve the block.

However this is the wrong organization for the file if that is your need. No
matter how hard you try, a hammer is a poor tool to put in a screw.

--
Binyamin Dissen 
http://www.dissensoftware.com

Director, Dissen Software, Bar & Grill - Israel

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Mike Shaw]

We do this with DSORG=PS and RECFM=F now.

 We load the file with QSAM and keep track of RBNs as we go. At EOF we
rewrite index blocks to predetermined locations in the file with EXCP, then
at READ time we use EXCP to read blocks and convert RBNs to CCHHRs based on
the device geometry as we go.

Mike Shaw
MVS/QuickRef Support Group
Chicago-Soft, Ltd

On Sat, Nov 11, 2023, 8:58 AM David S.  wrote:

> To help resolve a question posted to a LinkedIn group I manage:
> www.linkedin.com/feed/update/urn:li:groupPost:910927-7128598004344786944
> ... I'd like to find out if there's any way to achieve *true*
> Skip-Sequential processing with a Fixed Block Sequential File with a fairly
> short record length (i.e. DCB=(DSORG=PS,RECFM=FB,LRECL=80)?
> For example: Begin sequential processing at record number 100, *without*
> having to read the first 99 records.
> Note: We already know certain VSAM formats can do this, but the file in
> question is a DSORG=PS *Sequential* file, *not* VSAM. This is a rock-solid
> requirement and cannot be changed. We also already know how certain
> utilities such as SORT and REXX can *mimic* skip-sequential functionality
> by *discarding* unwanted records until the specified record number is
> reached. This is a likewise rock-solid requirement. Sequential processing
> *must* begin at specified starting point and there can be *no* reading of
> any records prior to that point.
> My gut feeling is it *cannot* be done - at least not with RECFM=FB.  It
> *might* be possible with RECFM=F, but efficiency would then be so
> compromised it would  probably outweigh any advantage from *true*
> skip-sequential processing.
>

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Steve Smith]

Sure it's possible... BSAM POINT should work fine in general, and BDAM is
even more powerful.  However, the previous mention about short blocks is
important; that will foul up block calculations, so you'd probably want to
make it FBS, and make sure there are no short blocks.  Presuming your
application knows what record it wants to start with, you'll have to
calculate the TTR and block offset from that.  And of course, by using
BxAM, you have to do your own de-blocking, and I/O overlap.

sas

Re: Is True Skip-Sequential Processing Possible with RECFM=FB,DSORG=PS?

[Martin Trübner]

David,


it can be done. It doe not matter if you have FB or F- all you have to 
know is the geometry of the device it is stored on (or the device it is 
does mimic).



If have created a program to do just that (on VSE platform). All it does 
is it sets the seeks address for the next record to read to whatever the 
calculation says it is for a certain record.



saved a hell of a lot of processing/elapsed time. Shall I go and look 
for it  (it is somewhere on the PC here).



Best


Martin


PS: I have red the thread on lnkedt - and I am starting to look just now.

Am 11.11.23 um 14:59 schrieb David S.:


To help resolve a question posted to a LinkedIn group I manage:
www.linkedin.com/feed/update/urn:li:groupPost:910927-7128598004344786944
... I'd like to find out if there's any way to achieve *true*
Skip-Sequential processing with a Fixed Block Sequential File with a fairly
short record length (i.e. DCB=(DSORG=PS,RECFM=FB,LRECL=80)?
For example: Begin sequential processing at record number 100, *without*
having to read the first 99 records.
Note: We already know certain VSAM formats can do this, but the file in
question is a DSORG=PS *Sequential* file, *not* VSAM. This is a rock-solid
requirement and cannot be changed. We also already know how certain
utilities such as SORT and REXX can *mimic* skip-sequential functionality
by *discarding* unwanted records until the specified record number is
reached. This is a likewise rock-solid requirement. Sequential processing
*must* begin at specified starting point and there can be *no* reading of
any records prior to that point.
My gut feeling is it *cannot* be done - at least not with RECFM=FB.  It
*might* be possible with RECFM=F, but efficiency would then be so
compromised it would  probably outweigh any advantage from *true*
skip-sequential processing.