Re: [CMS-PIPELINES] Replace single record in file
On Monday, 09/14/2020 at 10:07 GMT, "Stanislawski, Shawn (National VM Capability)" wrote: > For RECFM V when new record is longer and it would be preferable to avoid > truncation, then there does not appear to be a "pretty" solution short of > re-platforming (e.g. "go use SFS", etc). I actually meant "not CMS" unless you want to create your own ISAM- or VSAM-like infrastructure. By that I mean a file that has data records and a file that has index records. You must occasionally "reorg" the data file to compress out deleted records (you can re-order if you like). > Attempted to use VSAM under CMS, but the CMS VSAM handling code appears to be > significantly older than the z/OS counterparts. > Other than just using SFS, not sure if re-platforming to either BFS or CMS > OpenEditions might solve. VM/VSAM (5686-081) was withdrawn from marketing in 2005 and from service in 2007, so it's only an option if you have kept your VM/VSAM license active all these years. Some people have kept NetView on VM going in their shop, and it requires VSAM. VM/VSAM was actually VSE/VSAM. The programming interface was OS/VSAM since that's what the compilers used. An API "shim" in CMS converted between OS/VSAM and VSE/VSAM APIs. Alan Altmark Senior Managing z/VM and Linux Consultant IBM Systems Lab Services IBM Z Delivery Practice ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott
Re: [CMS-PIPELINES] Replace single record in file
>> Would you mind to mark the original as "deleted" and append the >> longer record to the RECFM V file maybe? But I guess you were going >> index by record number rather than a key. Indeed, indexing by record. So when the file is sitting on a CMS minidisk, it sounds like we have decent solutions for index-by-record when dealing with both RECFM F and RECFM V provided the new record is shorter. For RECFM F when new record is longer shouldn't be a problem unless longer than file's LRECL, and there are known methods for dealing with that situation already (e.g. split it, truncate it, etc). For RECFM V when new record is longer and it would be preferable to avoid truncation, then there does not appear to be a "pretty" solution short of re-platforming (e.g. "go use SFS", etc). Attempted to use VSAM under CMS, but the CMS VSAM handling code appears to be significantly older than the z/OS counterparts. Other than just using SFS, not sure if re-platforming to either BFS or CMS OpenEditions might solve. --Shawn S. -Original Message- From: CMSTSO Pipelines Discussion List On Behalf Of Alan Altmark Sent: Saturday, September 12, 2020 12:07 PM To: CMS-PIPELINES@VM.MARIST.EDU Subject: Re: [CMS-PIPELINES] Replace single record in file The CMS EDF file system is a very lightweight file system that has survived 40+ years with this restriction. (EDF was introduced in SEPP or BSEPP, 40+ I think.) Back then VSAM was available to solve more complex data management issues. Now, you re-platform. Regards, Alan Altmark IBM > On Sep 12, 2020, at 12:27 PM, Paul Gilmartin wrote: > > On 2020-09-12, at 06:18:25, Rob van der Heij wrote: >> >>> On Sat, 12 Sep 2020 at 01:18, Stanislawski, Shawn (National VM Capability) < >>> shaw...@dxc.com> wrote: >>> >>> Would be preferable to avoid truncation of a longer new record when >>> dealing with recfm=VARIABLE. >>> But at this point, it sounds as if there is not an elegant resource-light >>> method for so doing? >>> >> >> Would you mind to mark the original as "deleted" and append the >> longer record to the RECFM V file maybe? But I guess you were going >> index by record number rather than a key. >> > I understand that MDFS indexes blocks (not records) in a tree. > I was disappointed that does not support "elegant resource-light" > insertion, deletion, and replacement of records. (Yes, that would > require splitting blocks when records are inserted, marking parts of > blocks as unused, and balancing the tree for performance.) > > SFS is merely secret. > > -- gil >
Re: [CMS-PIPELINES] Replace single record in file
On Saturday, 09/12/2020 at 05:25 GMT, "John P. Hartmann" wrote: > EDF came as an add-on to VM/370 release 6 in 1979. Along with FBA > support if memory serves. I'm pretty sure that add-on was BSEPP or SEPP ([Basic] System Extensions Program Product, I think), which was subsequently rolled in with VM/370 to create VM/SP. I used to have a SEPP/BSEPP Redbook and I'm pretty sure it was there that the EDF file system was explained. But I may be misremembering. FWIW, that information was retained in the CMS Diagnosis Reference, last published with VM/ESA V2, as far as I can tell. (I'm trying to get the information resurrected before it's lost forever.) > Alan is not doing justice to the designers. Being copy-on-write, it was > the first CMS file system that had no leakage of disk blocks on crashes > or power outages. CDF had the choice of trashed files or block leakage > and clearly chose the latter, but as a consequence, you might have 10% > wasted blocks on a minidisk after a while. Ray Mansell recently noted over in IBMVM his contributions to EDF in the 80s. He's probably a good source of historical information. And I should point out that MVS sequential datasets (on which the CMS file system semantics are modeled) have similar restrictions. You cannot change the record length of an existing record in such a dataset (blocked or not). If I recall, ISAM provided a way to put records in logical sequence without dependency on physical sequence, allowing records to change length since a replacement record could be physically added anywhere there was space. This was subsequently replaced by VSAM RRDS. Alan Altmark Senior Managing z/VM and Linux Consultant IBM Systems Lab Services IBM Z Delivery Practice ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott
Re: [CMS-PIPELINES] Replace single record in file
On 2020-09-12, at 06:18:25, Rob van der Heij wrote: > > On Sat, 12 Sep 2020 at 01:18, Stanislawski, Shawn (National VM Capability) < > shaw...@dxc.com> wrote: > >> Would be preferable to avoid truncation of a longer new record when >> dealing with recfm=VARIABLE. >> But at this point, it sounds as if there is not an elegant resource-light >> method for so doing? >> > > Would you mind to mark the original as "deleted" and append the longer > record to the RECFM V file maybe? But I guess you were going index by > record number rather than a key. > I understand that MDFS indexes blocks (not records) in a tree. I was disappointed that does not support "elegant resource-light" insertion, deletion, and replacement of records. (Yes, that would require splitting blocks when records are inserted, marking parts of blocks as unused, and balancing the tree for performance.) SFS is merely secret. -- gil
Re: [CMS-PIPELINES] Replace single record in file
EDF came as an add-on to VM/370 release 6 in 1979. Along with FBA support if memory serves. Alan is not doing justice to the designers. Being copy-on-write, it was the first CMS file system that had no leakage of disk blocks on crashes or power outages. CDF had the choice of trashed files or block leakage and clearly chose the latter, but as a consequence, you might have 10% wasted blocks on a minidisk after a while. On 9/12/20 19:14, Kris Buelens wrote: Wasn't EDF new with VM/SP? (I remember performance problems with CDF for large RECFM V files in my first IBM years) Kris Buelens, --- VM/VSE consultant, Belgium --- --- Op za 12 sep. 2020 om 19:07 schreef Alan Altmark : The CMS EDF file system is a very lightweight file system that has survived 40+ years with this restriction. (EDF was introduced in SEPP or BSEPP, I think.) Back then VSAM was available to solve more complex data management issues. Now, you re-platform. Regards, Alan Altmark IBM On Sep 12, 2020, at 12:27 PM, Paul Gilmartin wrote: On 2020-09-12, at 06:18:25, Rob van der Heij wrote: On Sat, 12 Sep 2020 at 01:18, Stanislawski, Shawn (National VM Capability) < shaw...@dxc.com> wrote: Would be preferable to avoid truncation of a longer new record when dealing with recfm=VARIABLE. But at this point, it sounds as if there is not an elegant resource-light method for so doing? Would you mind to mark the original as "deleted" and append the longer record to the RECFM V file maybe? But I guess you were going index by record number rather than a key. I understand that MDFS indexes blocks (not records) in a tree. I was disappointed that does not support "elegant resource-light" insertion, deletion, and replacement of records. (Yes, that would require splitting blocks when records are inserted, marking parts of blocks as unused, and balancing the tree for performance.) SFS is merely secret. -- gil
Re: [CMS-PIPELINES] Replace single record in file
Wasn't EDF new with VM/SP? (I remember performance problems with CDF for large RECFM V files in my first IBM years) Kris Buelens, --- VM/VSE consultant, Belgium --- --- Op za 12 sep. 2020 om 19:07 schreef Alan Altmark : > The CMS EDF file system is a very lightweight file system that has survived > 40+ years with this restriction. (EDF was introduced in SEPP or BSEPP, I > think.) > > Back then VSAM was available to solve more complex data management issues. > Now, you re-platform. > > Regards, > Alan Altmark > IBM > > > On Sep 12, 2020, at 12:27 PM, Paul Gilmartin > wrote: > > > > On 2020-09-12, at 06:18:25, Rob van der Heij wrote: > >> > >>> On Sat, 12 Sep 2020 at 01:18, Stanislawski, Shawn (National VM > Capability) < > >>> shaw...@dxc.com> wrote: > >>> > >>> Would be preferable to avoid truncation of a longer new record when > >>> dealing with recfm=VARIABLE. > >>> But at this point, it sounds as if there is not an elegant > resource-light > >>> method for so doing? > >>> > >> > >> Would you mind to mark the original as "deleted" and append the longer > >> record to the RECFM V file maybe? But I guess you were going index by > >> record number rather than a key. > >> > > I understand that MDFS indexes blocks (not records) in a tree. > > I was disappointed that does not support "elegant resource-light" > > insertion, deletion, and replacement of records. (Yes, that > > would require splitting blocks when records are inserted, marking > > parts of blocks as unused, and balancing the tree for performance.) > > > > SFS is merely secret. > > > > -- gil > > >
Re: [CMS-PIPELINES] Replace single record in file
The CMS EDF file system is a very lightweight file system that has survived 40+ years with this restriction. (EDF was introduced in SEPP or BSEPP, I think.) Back then VSAM was available to solve more complex data management issues. Now, you re-platform. Regards, Alan Altmark IBM > On Sep 12, 2020, at 12:27 PM, Paul Gilmartin wrote: > > On 2020-09-12, at 06:18:25, Rob van der Heij wrote: >> >>> On Sat, 12 Sep 2020 at 01:18, Stanislawski, Shawn (National VM Capability) < >>> shaw...@dxc.com> wrote: >>> >>> Would be preferable to avoid truncation of a longer new record when >>> dealing with recfm=VARIABLE. >>> But at this point, it sounds as if there is not an elegant resource-light >>> method for so doing? >>> >> >> Would you mind to mark the original as "deleted" and append the longer >> record to the RECFM V file maybe? But I guess you were going index by >> record number rather than a key. >> > I understand that MDFS indexes blocks (not records) in a tree. > I was disappointed that does not support "elegant resource-light" > insertion, deletion, and replacement of records. (Yes, that > would require splitting blocks when records are inserted, marking > parts of blocks as unused, and balancing the tree for performance.) > > SFS is merely secret. > > -- gil >
Re: [CMS-PIPELINES] Replace single record in file
On Sat, 12 Sep 2020 at 01:18, Stanislawski, Shawn (National VM Capability) < shaw...@dxc.com> wrote: > Would be preferable to avoid truncation of a longer new record when > dealing with recfm=VARIABLE. > But at this point, it sounds as if there is not an elegant resource-light > method for so doing? > Would you mind to mark the original as "deleted" and append the longer record to the RECFM V file maybe? But I guess you were going index by record number rather than a key. Rob
Re: [CMS-PIPELINES] Replace single record in file
Would be preferable to avoid truncation of a longer new record when dealing with recfm=VARIABLE. But at this point, it sounds as if there is not an elegant resource-light method for so doing? --Shawn S. -Original Message- From: CMSTSO Pipelines Discussion List On Behalf Of Rob van der Heij Sent: Friday, September 11, 2020 2:51 AM To: CMS-PIPELINES@VM.MARIST.EDU Subject: Re: [CMS-PIPELINES] Replace single record in file On Fri, 11 Sep 2020 at 01:00, Glenn Knickerbocker wrote: > If you want to pad or truncate as needed, probably the simplest way is > to convert V2C and then back C2V. Assuming the input is numbered for > DISKUPDATE: > Good to have another pair of eyes on it. I didn't interpret the question as that Shawn was willing to truncate or pad so it would fit. That's indeed a nice plumbing challenge for a Friday afternoon. Rob -Original Message- From: CMSTSO Pipelines Discussion List On Behalf Of Rob van der Heij Sent: Thursday, September 10, 2020 3:31 PM To: CMS-PIPELINES@VM.MARIST.EDU Subject: Re: [CMS-PIPELINES] Replace single record in file Nope. As you found, that's due to CMS restrictions for RECFM V files. Depending on the use case, it might be possible to have a separate file to list the replacements, and use the record from the replacements when present, and otherwise go to the original one. Rob
Re: [CMS-PIPELINES] Replace single record in file
On 9/11/2020 11:10 AM, John P. Hartmann wrote: > The interaction with STRIP also needed a loving hand. Ouch, now I'm thinking way too hard about STRIP C2V and lengths around 16K. ¬R
Re: [CMS-PIPELINES] Replace single record in file
On 9/11/20 14:07, John P. Hartmann wrote: There is no test case of the placement option with v2c. Turns out there was a single one and it had the wrong result. The interaction with STRIP also needed a loving hand.
Re: [CMS-PIPELINES] Replace single record in file
On 9/11/20 01:00, Glenn Knickerbocker wrote: Uh-oh. I must never have tried THIS before: pipe literal 123| specs 1-* v2c 1.3 right | cons 1 That's just wrong. ¬R This is a bug that no-one has complained about before. The correct output is "123". That is, the placement has incorrectly been ignored when the v2c conversion is truncated. Documentation is quite clear about conversion being performed before output placement. The code should be fixed. I speculate the reason is that the output field is a catenation of an internal literal of the byte count and data from the input record. This two-input output arises only with v2c conversion. Processing is correct when the output field is padded: pipe literal 123| specs 1-* v2c 1.6 right |fmtdmp|cons Record 1 43F1 F2F3 * 123 * There is no test case of the placement option with v2c. Indeed with any of the conversions, he says after a quick peek at SPC TESTIN. I agree that message 393 is the result of paranoia. It should not be issued; one or two bytes should be output.
Re: [CMS-PIPELINES] Replace single record in file
On Fri, 11 Sep 2020 at 01:00, Glenn Knickerbocker wrote: > pipe literal 123| specs 1-* v2c 1.3 right | cons > > 1 > > That's just wrong. Yes, the x0001 that goes in is not the "length of the input field" but rather what still fits in the output field after we reserved two bytes for the length. There's been too much water through the pipes to change that or to consider it an error. The best I could do is change the book to explain what happens when the output field is too short. Fortunately you have the "addrdw cms" now that probably does what you meant to do. While I like "parcel" for the reverse operation, it's harder to make that work on multiple records. I'll think about the wording of the error message (like "length and field" rather than "field length"). Considering what the code does now, v2c into an output field of length 2 is not very useful (it would produce x). If you just wanted to know the length of the input field, there are a few different idioms. Sir Rob the Plumber
Re: [CMS-PIPELINES] Replace single record in file
On Fri, 11 Sep 2020 at 01:00, Glenn Knickerbocker wrote: > If you want to pad or truncate as needed, probably the simplest way is > to convert V2C and then back C2V. Assuming the input is numbered for > DISKUPDATE: > Good to have another pair of eyes on it. I didn't interpret the question as that Shawn was willing to truncate or pad so it would fit. That's indeed a nice plumbing challenge for a Friday afternoon. Rob
Re: [CMS-PIPELINES] Replace single record in file
On 9/10/2020 3:09 PM, Stanislawski, Shawn (National VM Capability) wrote: > Anyone have a great idea on how to use PIPE to replace single record > at a specified record number in a file without having to read / > write out the entire file, with the following caveats? > > * the file could be either recfm FIXED or VARIABLE > * the new record could potentially be either shorter or longer than its > original If you want to pad or truncate as needed, probably the simplest way is to convert V2C and then back C2V. Assuming the input is numbered for DISKUPDATE: (end /) ... | in: fanout | chop 10 | copy | diskrandom fn ft fm | xlate 00-ff 40 | out: specs 1-* v2c 11 select 1 1.10 1 11-* 13 | specs 1.10 1 11-* c2v 11 | diskupdate fn ft fm / in: | out: I was always a little disappointed that COMBINE wouldn't truncate and the f2V conversions wouldn't pad. And speaking of V format, I don't think I ever got around to complaining about this error way back: > pipe literal 123| specs 1-* v2c 1.2 | cons > Output field too short to contain field length. > ... Issued from stage 2 of pipeline 1. > ... Running "specs 1-* v2c 1.2". > Processing item number 1: 1-* v2c 1.2. The message is wrong, since 2 is just long enough to contain the field length, but even if I asked for only one character I don't get why this is an error. V2B and V2X don't care if there's room for the output; why should V2C? Uh-oh. I must never have tried THIS before: > pipe literal 123| specs 1-* v2c 1.3 right | cons > 1 That's just wrong. ¬R
Re: [CMS-PIPELINES] Replace single record in file
On Thu, 10 Sep 2020 at 21:09, Stanislawski, Shawn (National VM Capability) < shaw...@dxc.com> wrote: > Anyone have a great idea on how to use PIPE to replace single record at a > specified record number in a file without having to read / write out the > entire file, with the following caveats? > Nope. As you found, that's due to CMS restrictions for RECFM V files. Depending on the use case, it might be possible to have a separate file to list the replacements, and use the record from the replacements when present, and otherwise go to the original one. Rob
