Re: [IBM External] Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)
On Wed, 29 Sep 2021 06:54:09 +, Martin Packer wrote: > >Between steps can't be pipes, can be VIO. Between jobs can be pipes, can't >be VIO. > Pipes could work between steps if the PIPE SUBSYS has an implied ELASTIC stage. But that's just reinventing VIO. PIPE SUBSYS must do some buffering to handle BSAM RECFM=VB. Or does it require that RECFM and BLKSIZE or the producer and consumer be identical? >That second sentence depends on the ability to schedule two jobs (possibly >originally steps of the same job) alongside each other. > Alas, the Initiator won't do the latter for you. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: [IBM External] Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)
Between steps can't be pipes, can be VIO. Between jobs can be pipes, can't be VIO. That second sentence depends on the ability to schedule two jobs (possibly originally steps of the same job) alongside each other. Fun stuff but / and somewhat complex - which is what inspired me to start writing what would become SG24-2557 Parallel Sysplex Batch Performance in late 1990. :-) I did a lot of presenting on Pipes to individual customers and conferences in the 1990s. It would be fun to do it again... :-) Cheers, Martin Sent from my iPad > On 29 Sep 2021, at 05:23, Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu> wrote: > > On Mon, 27 Sep 2021 11:39:17 -0500, Hobart Spitz wrote: >> >> Intra-JOB pipe - This would be similar to VIO; it doesn't exist otherwise. >> I think it would be a great feature. >> > Between steps? One might just as well use VIO. > > -- gil > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAINUnless > stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)
On Mon, 27 Sep 2021 11:39:17 -0500, Hobart Spitz wrote: > >Intra-JOB pipe - This would be similar to VIO; it doesn't exist otherwise. >I think it would be a great feature. > Between steps? One might just as well use VIO. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)
Thanks for the clarity. So BatchPipes are the plumbing to implement streaming almost like the JCL equivalent of the Apache Kafka Streams API? On 28/09/2021 4:00 pm, Martin Packer wrote: I think you have to remember that BatchPipes/MVS' origin story is connecting existing batch job steps. I wouldn't want customers to have to re-write e.g. in java. One advantage is that it's easier to rework a pipeline - whether as a fitting or between batch jobs - than rework some java code. As someone who first proselytised Pipes in 1992 and first wrote about it in a Redbook in 1997 (and wrote about it again in 2011 and 2013) you can consider me a fan. Note: I don't have an enormous amount of influence on those that make decisions about either IBM flavour of pipes. I do recognise it's not as simple as "set the code free". There is the cost of bringing it to market and supporting it. The latter in particular would be much more costly than it's ever been before - if it were built in. So, I hope we can make both Pipelines and BatchPipes/MVS available as part of z/OS. I'd love to be talking about them again and supporting customers using them. But I'm just a field guy. Cheers, Martin Martin Packer WW z/OS Performance, Capacity and Architecture, IBM Technology Sales +44-7802-245-584 email: martin_pac...@uk.ibm.com Twitter / Facebook IDs: MartinPacker Blog: https://mainframeperformancetopics.com Mainframe, Performance, Topics Podcast Series (With Marna Walle): https://anchor.fm/marna-walle Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA From: "David Crayford" To: IBM-MAIN@LISTSERV.UA.EDU Date: 28/09/2021 08:17 Subject:[EXTERNAL] Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...) Sent by:"IBM Mainframe Discussion List" This first question I would ask is does IBM actually "own" BatchPipes or did the flog it off to an ISV like most of their other software? If it's the latter then they will be in no position to make it freely available. Secondly, rather than pine for something that isn't available why not just switch technologies and use a language that supports functional programming features that are similar to pipes. Even Java has supported functional programming since Java 8 came with streams in 2014. https://stackify.com/streams-guide-java-8/ On 28/09/2021 12:39 am, Hobart Spitz wrote: Gil wrote: On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote: I'm going to pivot here. I'm putting my support behind putting BatchPipes in the z/OS base (rather than just Pipes). If you agree, please write/support such a requirement and/or educate your management to get interested. BatchPipes includes BatchPipesWorks, a not so current, but still highly useful, version of TSO Pipelines. Is an RFE for an update required? Conway's Law. Great question. I guess it would depend on whether the were a lot of current BatchPipes customers needed the missing builtin stages and/o fix(es) or whether it was more important to get BatchPipes in the z/OS base. It might also depend on how much additional work would be required to get a more current version of Pipes into BatchPipes, and whether the staff, skills, and funding were available. IMHO, the update would delay the benefits of BatchPipes in the base (especially global warming mitigation), and not be important to current BatchPipes customers. In my estimation, BatchPipesWorks has 90% of the needed stages and 99% of the fix(es) as would be in an up to date version of TSO Pipelines. I find the IBM decision making process obscure, so these may not be the considerations that affect the final decision. AFAIK, there is only one person now working on Pipes, so I must be missing something in applying Conway's law. Does BatchPipes support connecting two Classic modules with an intervening small Pipeline filter? How? Is a coordinated third job needed? Let me clarify some terms, answer you questions in the process, and clarify my previous post: Inter-JOB pipe - This is a ppe that let's two JOB pass records from one to the next thru memory. The BatchPipes subsystem(s) is/are required. AFAIK: There are no obvious limitations to the topology. You can have multiple JOB connected in a single "stream", split and/or join streams, or even have loop(s). In this respect, it is similar to the PIPE command, except that the record flow with split and rejoined streams may not be predictable between JOBs. Half Pipe Fitting or Intra-STEP pipe - This is a series of Pipes stages that operate between a program in a step and the storage medium. Intra-JOB pipe - This would be similar to VIO; it doesn't exist otherwise. I think it would be a great feature. Hence I incorrectly used the term to refer to Half Pipe Fittings in a generic sense. Apologies for any confusion. AFAIK, and if I understood your question, connecting two classic programs in the same step with a series of
Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)
I think you have to remember that BatchPipes/MVS' origin story is connecting existing batch job steps. I wouldn't want customers to have to re-write e.g. in java. One advantage is that it's easier to rework a pipeline - whether as a fitting or between batch jobs - than rework some java code. As someone who first proselytised Pipes in 1992 and first wrote about it in a Redbook in 1997 (and wrote about it again in 2011 and 2013) you can consider me a fan. Note: I don't have an enormous amount of influence on those that make decisions about either IBM flavour of pipes. I do recognise it's not as simple as "set the code free". There is the cost of bringing it to market and supporting it. The latter in particular would be much more costly than it's ever been before - if it were built in. So, I hope we can make both Pipelines and BatchPipes/MVS available as part of z/OS. I'd love to be talking about them again and supporting customers using them. But I'm just a field guy. Cheers, Martin Martin Packer WW z/OS Performance, Capacity and Architecture, IBM Technology Sales +44-7802-245-584 email: martin_pac...@uk.ibm.com Twitter / Facebook IDs: MartinPacker Blog: https://mainframeperformancetopics.com Mainframe, Performance, Topics Podcast Series (With Marna Walle): https://anchor.fm/marna-walle Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA From: "David Crayford" To: IBM-MAIN@LISTSERV.UA.EDU Date: 28/09/2021 08:17 Subject:[EXTERNAL] Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...) Sent by:"IBM Mainframe Discussion List" This first question I would ask is does IBM actually "own" BatchPipes or did the flog it off to an ISV like most of their other software? If it's the latter then they will be in no position to make it freely available. Secondly, rather than pine for something that isn't available why not just switch technologies and use a language that supports functional programming features that are similar to pipes. Even Java has supported functional programming since Java 8 came with streams in 2014. https://stackify.com/streams-guide-java-8/ On 28/09/2021 12:39 am, Hobart Spitz wrote: > Gil wrote: >> On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote: >>> I'm going to pivot here. I'm putting my support behind putting BatchPipes >>> in the z/OS base (rather than just Pipes). If you agree, please >>> write/support such a requirement and/or educate your management to get >>> interested. BatchPipes includes BatchPipesWorks, a not so current, but >>> still highly useful, version of TSO Pipelines. >>> >> Is an RFE for an update required? Conway's Law. > Great question. I guess it would depend on whether the were a lot of > current BatchPipes customers needed the missing builtin stages and/o > fix(es) or whether it was more important to get BatchPipes in the z/OS > base. It might also depend on how much additional work would be required > to get a more current version of Pipes into BatchPipes, and whether the > staff, skills, and funding were available. IMHO, the update would delay > the benefits of BatchPipes in the base (especially global warming > mitigation), and not be important to current BatchPipes customers. In my > estimation, BatchPipesWorks has 90% of the needed stages and 99% of the > fix(es) as would be in an up to date version of TSO Pipelines. I find the > IBM decision making process obscure, so these may not be the considerations > that affect the final decision. > > AFAIK, there is only one person now working on Pipes, so I must be missing > something in applying Conway's law. > >> Does BatchPipes support connecting two Classic modules with an intervening >> small Pipeline filter? How? Is a coordinated third job needed? > Let me clarify some terms, answer you questions in the process, and clarify > my previous post: > > > Inter-JOB pipe - This is a ppe that let's two JOB pass records from one to > the next thru memory. The BatchPipes subsystem(s) is/are required. > AFAIK: There are no obvious limitations to the topology. You can have > multiple JOB connected in a single "stream", split and/or join streams, or > even have loop(s). In this respect, it is similar to the PIPE command, > except that the record flow with split and rejoined streams may not be > predictable between JOBs. > > Half Pipe Fitting or Intra-STEP pipe - This is a series of Pipes stages > that operate between a program in a step and the storage medium. > > Intra-JOB pipe - This would be similar to VIO; it doesn't exist otherwise. > I think it would be a great feature. Hence I incorrectly used the term to > refer to Half Pipe Fittings in a generic sense. Apologies for any > confusion. > > > AFAIK, and if I understood your question, connecting two classic programs > in the same step with a series of Pipes stages may not be possible now, at > least without resorting to Assembler.
Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)
This first question I would ask is does IBM actually "own" BatchPipes or did the flog it off to an ISV like most of their other software? If it's the latter then they will be in no position to make it freely available. Secondly, rather than pine for something that isn't available why not just switch technologies and use a language that supports functional programming features that are similar to pipes. Even Java has supported functional programming since Java 8 came with streams in 2014. https://stackify.com/streams-guide-java-8/ On 28/09/2021 12:39 am, Hobart Spitz wrote: Gil wrote: On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote: I'm going to pivot here. I'm putting my support behind putting BatchPipes in the z/OS base (rather than just Pipes). If you agree, please write/support such a requirement and/or educate your management to get interested. BatchPipes includes BatchPipesWorks, a not so current, but still highly useful, version of TSO Pipelines. Is an RFE for an update required? Conway's Law. Great question. I guess it would depend on whether the were a lot of current BatchPipes customers needed the missing builtin stages and/o fix(es) or whether it was more important to get BatchPipes in the z/OS base. It might also depend on how much additional work would be required to get a more current version of Pipes into BatchPipes, and whether the staff, skills, and funding were available. IMHO, the update would delay the benefits of BatchPipes in the base (especially global warming mitigation), and not be important to current BatchPipes customers. In my estimation, BatchPipesWorks has 90% of the needed stages and 99% of the fix(es) as would be in an up to date version of TSO Pipelines. I find the IBM decision making process obscure, so these may not be the considerations that affect the final decision. AFAIK, there is only one person now working on Pipes, so I must be missing something in applying Conway's law. Does BatchPipes support connecting two Classic modules with an intervening small Pipeline filter? How? Is a coordinated third job needed? Let me clarify some terms, answer you questions in the process, and clarify my previous post: Inter-JOB pipe - This is a ppe that let's two JOB pass records from one to the next thru memory. The BatchPipes subsystem(s) is/are required. AFAIK: There are no obvious limitations to the topology. You can have multiple JOB connected in a single "stream", split and/or join streams, or even have loop(s). In this respect, it is similar to the PIPE command, except that the record flow with split and rejoined streams may not be predictable between JOBs. Half Pipe Fitting or Intra-STEP pipe - This is a series of Pipes stages that operate between a program in a step and the storage medium. Intra-JOB pipe - This would be similar to VIO; it doesn't exist otherwise. I think it would be a great feature. Hence I incorrectly used the term to refer to Half Pipe Fittings in a generic sense. Apologies for any confusion. AFAIK, and if I understood your question, connecting two classic programs in the same step with a series of Pipes stages may not be possible now, at least without resorting to Assembler. I think the main reason is the potential for DD name conflict, both in the TIOT (DD name table) and in JCL. Most COBOL programs require SYSOUT, so the output could be intermixed. Much less tolerable, I suspect, would be a common DD (e.g. OUTPUT) conflict. Putting a record that doesn't belong between two records that must be consecutive would probably be a disaster. That said, it may be possible for a REXX filter to start two COBOL (e.g.) programs as subtasks (address attach ... , etc.) and feed and receive their records via an Assembler(?) Pipe Fitting. You would have to resolve any DD name conflicts. I don't think anyone has contemplated the capability for creating multiple TIOTs in a single step, let alone how to reference them in JCL. I think such things have been done under CMS using Assembler. Also: pipe - a generic connection, in memory, between two processes or tasks. Pipes - the TSO/CMS Pipelines software, singular. Pipe - a specific instance of Pipes. PIPE - the command that runs Pipes. OREXXMan Would you rather pass data in move mode (*nix piping) or locate mode (Pipes) or via disk (JCL)? Why do you think you rarely see *nix commands with more than a dozen filters, while Pipelines specifications are commonly over 100s of stages, and 1000s of stages are not uncommon. IBM has been looking for an HLL for program products; REXX is that language. On Mon, Sep 27, 2021 at 9:10 AM Paul Gilmartin < 000433f07816-dmarc-requ...@listserv.ua.edu> wrote: On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote: I'm going to pivot here. I'm putting my support behind putting BatchPipes in the z/OS base (rather than just Pipes). If you agree, please write/support such a requirement and/or educate your management to
Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)
Gil wrote: >On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote: >>I'm going to pivot here. I'm putting my support behind putting BatchPipes >>in the z/OS base (rather than just Pipes). If you agree, please >>write/support such a requirement and/or educate your management to get >>interested. BatchPipes includes BatchPipesWorks, a not so current, but >>still highly useful, version of TSO Pipelines. >> >Is an RFE for an update required? Conway's Law. Great question. I guess it would depend on whether the were a lot of current BatchPipes customers needed the missing builtin stages and/o fix(es) or whether it was more important to get BatchPipes in the z/OS base. It might also depend on how much additional work would be required to get a more current version of Pipes into BatchPipes, and whether the staff, skills, and funding were available. IMHO, the update would delay the benefits of BatchPipes in the base (especially global warming mitigation), and not be important to current BatchPipes customers. In my estimation, BatchPipesWorks has 90% of the needed stages and 99% of the fix(es) as would be in an up to date version of TSO Pipelines. I find the IBM decision making process obscure, so these may not be the considerations that affect the final decision. AFAIK, there is only one person now working on Pipes, so I must be missing something in applying Conway's law. > Does BatchPipes support connecting two Classic modules with an intervening > small Pipeline filter? How? Is a coordinated third job needed? Let me clarify some terms, answer you questions in the process, and clarify my previous post: Inter-JOB pipe - This is a ppe that let's two JOB pass records from one to the next thru memory. The BatchPipes subsystem(s) is/are required. AFAIK: There are no obvious limitations to the topology. You can have multiple JOB connected in a single "stream", split and/or join streams, or even have loop(s). In this respect, it is similar to the PIPE command, except that the record flow with split and rejoined streams may not be predictable between JOBs. Half Pipe Fitting or Intra-STEP pipe - This is a series of Pipes stages that operate between a program in a step and the storage medium. Intra-JOB pipe - This would be similar to VIO; it doesn't exist otherwise. I think it would be a great feature. Hence I incorrectly used the term to refer to Half Pipe Fittings in a generic sense. Apologies for any confusion. AFAIK, and if I understood your question, connecting two classic programs in the same step with a series of Pipes stages may not be possible now, at least without resorting to Assembler. I think the main reason is the potential for DD name conflict, both in the TIOT (DD name table) and in JCL. Most COBOL programs require SYSOUT, so the output could be intermixed. Much less tolerable, I suspect, would be a common DD (e.g. OUTPUT) conflict. Putting a record that doesn't belong between two records that must be consecutive would probably be a disaster. That said, it may be possible for a REXX filter to start two COBOL (e.g.) programs as subtasks (address attach ... , etc.) and feed and receive their records via an Assembler(?) Pipe Fitting. You would have to resolve any DD name conflicts. I don't think anyone has contemplated the capability for creating multiple TIOTs in a single step, let alone how to reference them in JCL. I think such things have been done under CMS using Assembler. Also: pipe - a generic connection, in memory, between two processes or tasks. Pipes - the TSO/CMS Pipelines software, singular. Pipe - a specific instance of Pipes. PIPE - the command that runs Pipes. OREXXMan Would you rather pass data in move mode (*nix piping) or locate mode (Pipes) or via disk (JCL)? Why do you think you rarely see *nix commands with more than a dozen filters, while Pipelines specifications are commonly over 100s of stages, and 1000s of stages are not uncommon. IBM has been looking for an HLL for program products; REXX is that language. On Mon, Sep 27, 2021 at 9:10 AM Paul Gilmartin < 000433f07816-dmarc-requ...@listserv.ua.edu> wrote: > On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote: > > >I'm going to pivot here. I'm putting my support behind putting BatchPipes > >in the z/OS base (rather than just Pipes). If you agree, please > >write/support such a requirement and/or educate your management to get > >interested. BatchPipes includes BatchPipesWorks, a not so current, but > >still highly useful, version of TSO Pipelines. > > > Is an RFE for an update rrequired? Conway's Law. > > Does BatchPipes support connecting two Classic modules with an intervening > small Pipeline filter? How? Is a coordinated third job needed? > > >The reasons are: [Snip! See the archive.] > > -- gil > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to
Re: The Business Case for BatchPipes in the z/OS Base (was: ... Pipes ...)
On Mon, 27 Sep 2021 07:26:51 -0500, Hobart Spitz wrote: >I'm going to pivot here. I'm putting my support behind putting BatchPipes >in the z/OS base (rather than just Pipes). If you agree, please >write/support such a requirement and/or educate your management to get >interested. BatchPipes includes BatchPipesWorks, a not so current, but >still highly useful, version of TSO Pipelines. > Is an RFE for an update rrequired? Conway's Law. Does BatchPipes support connecting two Classic modules with an intervening small Pipeline filter? How? Is a coordinated third job needed? >The reasons are: [Snip! See the archive.] -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
