Re: IBM python documentation? [was: RE: IBM SMF Export with Python]
On 4/10/22 01:56, Farley, Peter x23353 wrote: I have read that tagging behavior page and it makes sense when applied to files stored in the Unix file system. MVS datasets could just as easily just be considered always tagged as cp1047 (with an option to change the CCSID to 037 or one of the European EBCIDIC code pages) and then you just go on from there. Connecting to the MVS file system "under the covers" is the runtime implementor's job, not the application developer's job. As to providing a package to specifically access MVS datasets, my first question is why IBM didn't provide one in the first place, or better yet build it into the python I/O subsystem from the start? Why does the community need to provide such a basic functionality? "zoautil_py" is one such implementation, but it is extraordinarily "heavyweight" and expensive to use, and is not an acceptable solution in a world where every second counts and costs real money. I am quite sure an MVS I/O package will happen just like it did for golang and Node.js. Yet again, open source code documented in Github and the respective package manager repositories. You may not like that but it's how these things work. https://github.com/ibmruntimes/go-recordio The lack of easy-to-use and efficient MVS file access may well discourage more widespread use of python on z/OS for everyday computing tasks. Maybe not as much as you think. The tooling coming online will require Python so it's mandatory. The vast majority on this forum would rather use REXX anyway! And yes, I can accept that my expectations are not realistic. It is just a shame that an easy-to-use and efficient access to MVS datasets isn't and won't be there unless the "community" provides it. IMHO it should have been part-and-parcel of the original language implementation, but I know what my opinions and expectations are worth. The community involves IBM and the team that ported Python, Golang, Node.js etc. The team from IBM Toronto have setup and open source community that are actively ported FOSS to z/OS. Interestingly, this includes the zsh which I'm delighted about. Peter -Original Message- From: IBM Mainframe Discussion List On Behalf Of David Crayford Sent: Monday, October 3, 2022 1:48 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: IBM python documentation? [was: RE: IBM SMF Export with Python] On 3/10/22 07:18, Farley, Peter x23353 wrote: Where, for instance, in the documentation that you linked to is there information on whether the python "open()" function for files will or will not support direct access to MVS datasets without transferring them down to the Unix file system? I.E., are either of these calls to the open() function valid or not: Myfile = open("//'MVS.DATA.SET.NAME", "rt", ccsid="1047") Myfile = open("//:DDNAME", "rt", ccsid="1047") # This of course assumes that a call to "bpxunix" (or by other means) has been used to establish a DD allocation in the AS where the python program is executing This will *never* happen. If you read the doc WRT tagging behaviors you will understand why https://urldefense.com/v3/__https://www.ibm.com/docs/en/python-zos/3.9?topic=translation-tagging-behaviors__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!O2czdpcmZt_bBNIuRr4UXedStf34QpBfTBkZ-wQNwNFosWZR2zrDSP1nat8ZFfmZnRA9s7TVhDN6jaUYFy-CaGg$ . Rocket's Python port supported reading MVS data sets but it was not a documented feature and was used with great care. Patching the Python core I/O libraries to support MVS data sets is not a great idea. It's much better to write a specific library that *only* support MVS data sets and not z/OS UNIX files. A good reference implementation is the JZOS ZFile class. If you open an RFE IBM may do this. Or maybe the community can contribute? Either way it's going to be implemented as a specific package. Porting tools to support MVS data sets in trivial. They use open() because it's a low-level API and read/write support sockets, FIFO's, pipes etc. It's not just a case of replacing open with fopen(). Python implements a fdopen() https://urldefense.com/v3/__https://docs.python.org/3/library/os.html*os.fdopen__;Iw!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!O2czdpcmZt_bBNIuRr4UXedStf34QpBfTBkZ-wQNwNFosWZR2zrDSP1nat8ZFfmZnRA9s7TVhDN6jaUY6MzWLWc$ method for wrapping file descriptors. -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately
Re: IBM python documentation? [was: RE: IBM SMF Export with Python]
I have read that tagging behavior page and it makes sense when applied to files stored in the Unix file system. MVS datasets could just as easily just be considered always tagged as cp1047 (with an option to change the CCSID to 037 or one of the European EBCIDIC code pages) and then you just go on from there. Connecting to the MVS file system "under the covers" is the runtime implementor's job, not the application developer's job. As to providing a package to specifically access MVS datasets, my first question is why IBM didn't provide one in the first place, or better yet build it into the python I/O subsystem from the start? Why does the community need to provide such a basic functionality? "zoautil_py" is one such implementation, but it is extraordinarily "heavyweight" and expensive to use, and is not an acceptable solution in a world where every second counts and costs real money. The lack of easy-to-use and efficient MVS file access may well discourage more widespread use of python on z/OS for everyday computing tasks. And yes, I can accept that my expectations are not realistic. It is just a shame that an easy-to-use and efficient access to MVS datasets isn't and won't be there unless the "community" provides it. IMHO it should have been part-and-parcel of the original language implementation, but I know what my opinions and expectations are worth. Peter -Original Message- From: IBM Mainframe Discussion List On Behalf Of David Crayford Sent: Monday, October 3, 2022 1:48 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: IBM python documentation? [was: RE: IBM SMF Export with Python] On 3/10/22 07:18, Farley, Peter x23353 wrote: > Where, for instance, in the documentation that you linked to is there > information on whether the python "open()" function for files will or will > not support direct access to MVS datasets without transferring them down to > the Unix file system? I.E., are either of these calls to the open() > function valid or not: > > Myfile = open("//'MVS.DATA.SET.NAME", "rt", ccsid="1047") > > Myfile = open("//:DDNAME", "rt", ccsid="1047") # This of course > assumes that a call to "bpxunix" (or by other means) has been used to > establish a DD allocation in the AS where the python program is > executing This will *never* happen. If you read the doc WRT tagging behaviors you will understand why https://urldefense.com/v3/__https://www.ibm.com/docs/en/python-zos/3.9?topic=translation-tagging-behaviors__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!O2czdpcmZt_bBNIuRr4UXedStf34QpBfTBkZ-wQNwNFosWZR2zrDSP1nat8ZFfmZnRA9s7TVhDN6jaUYFy-CaGg$ . Rocket's Python port supported reading MVS data sets but it was not a documented feature and was used with great care. Patching the Python core I/O libraries to support MVS data sets is not a great idea. It's much better to write a specific library that *only* support MVS data sets and not z/OS UNIX files. A good reference implementation is the JZOS ZFile class. If you open an RFE IBM may do this. Or maybe the community can contribute? Either way it's going to be implemented as a specific package. Porting tools to support MVS data sets in trivial. They use open() because it's a low-level API and read/write support sockets, FIFO's, pipes etc. It's not just a case of replacing open with fopen(). Python implements a fdopen() https://urldefense.com/v3/__https://docs.python.org/3/library/os.html*os.fdopen__;Iw!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!O2czdpcmZt_bBNIuRr4UXedStf34QpBfTBkZ-wQNwNFosWZR2zrDSP1nat8ZFfmZnRA9s7TVhDN6jaUY6MzWLWc$ method for wrapping file descriptors. -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: IBM python documentation? [was: RE: IBM SMF Export with Python]
On 3/10/22 13:47, David Crayford wrote: Porting tools to support MVS data sets in trivial. I meant Porting tools to support MVS data sets is non trivial. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: IBM python documentation? [was: RE: IBM SMF Export with Python]
On 3/10/22 07:18, Farley, Peter x23353 wrote: Where, for instance, in the documentation that you linked to is there information on whether the python "open()" function for files will or will not support direct access to MVS datasets without transferring them down to the Unix file system? I.E., are either of these calls to the open() function valid or not: Myfile = open("//'MVS.DATA.SET.NAME", "rt", ccsid="1047") Myfile = open("//:DDNAME", "rt", ccsid="1047") # This of course assumes that a call to "bpxunix" (or by other means) has been used to establish a DD allocation in the AS where the python program is executing This will *never* happen. If you read the doc WRT tagging behaviors you will understand why https://www.ibm.com/docs/en/python-zos/3.9?topic=translation-tagging-behaviors. Rocket's Python port supported reading MVS data sets but it was not a documented feature and was used with great care. Patching the Python core I/O libraries to support MVS data sets is not a great idea. It's much better to write a specific library that *only* support MVS data sets and not z/OS UNIX files. A good reference implementation is the JZOS ZFile class. If you open an RFE IBM may do this. Or maybe the community can contribute? Either way it's going to be implemented as a specific package. Porting tools to support MVS data sets in trivial. They use open() because it's a low-level API and read/write support sockets, FIFO's, pipes etc. It's not just a case of replacing open with fopen(). Python implements a fdopen() https://docs.python.org/3/library/os.html#os.fdopen method for wrapping file descriptors. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: IBM python documentation? [was: RE: IBM SMF Export with Python]
py" functions to access the same data. Nowhere in the IBM-provided ZOAU documentation did I find any reference to performance characteristics or performance-enhancing advice. That separate python libraries for z/OS maintain separate python documentation of their functionality is OK, but then the IBM-specific library websites need to provide the same sort of automatic linking to the library documentation that the standard python documentation websites use. I.E., the IBM-specific libraries need to have their documentation directly available on the ".readthedocs.io" website and linked to from their specific page on the Pypi indexing website and not just at their github location. What IBM-specific library? There isn't one! ISTM that IBM is a day late and a dollar short on application-level documentation for this language on their systems. I disagree. Maybe your expectations are unrealistic. Peter -Original Message- From: IBM Mainframe Discussion List On Behalf Of David Crayford Sent: Saturday, October 1, 2022 9:35 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: IBM SMF Export with Python On 2/10/22 01:31, Charles Mills wrote: We LOVE Python. When is IBM going to love it enough to ship it and document it as though they loved it? You can order Python via zShop and install it via SMP/E or download a pax file. That's choice right there. What's wrong with the doc? https://www.ibm.com/docs/en/python-zos/3.10 has all the information I need to install, customize and build extensions. For z/OS specific Python libraries the doc is either shipped with the product so as ZAOU or for open source projects with the code in a repository. For example, the Python DB2 library and doc can be found on Github https://github.com/ibmdb/python-ibmdb . z/OS specific doc here https://github.com/ibmdb/node-ibm_db*configure-odbc-driver-on-zos . This is standard for any open source project. I understand that may be uncomfortable to folks that have no experience working with open source. Another example is https://github.com/ambitus/pyzkiln which is maintained by IBMers from Poughkeepsie but part of the OMP. Anybody interested in writing Python libraries that include calling HLASM should take a look at that repo. Honestly, I don't see what the issue is with doc? It's no different to Java. If I need doc for the JRE I read the Oracle doc. If I need doc for Python I head on over to the Python doc. Everything I need for z/OS is covered. (Rhetorical question. I know that David does not know the answer.) Charles -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of David Crayford Sent: Friday, September 30, 2022 11:09 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: IBM SMF Export with Python I know Python is not well received on this forum but as an example of IBMs investment they have announced a new Python toolkit to process SMF data using Jupyter notesbooks, an industry standard visualization tool for analytics and machine learning. I understand many people consider this stuff buzz words and fads but if you dig in a bit it's very cool. https://urldefense.com/v3/__https://zos-hot-topics.com/2022/SMF-Explor er/__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!MOPAkGKkKJZy2370bbp_YxYtD_tyt93 x3LCeTXq7oDQMU30Ek0NS5KXliytjmf7z0s0cSc5sVoDMlT8NCCU5Lyk$ -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: IBM python documentation? [was: RE: IBM SMF Export with Python]
On 3/10/22 07:18, Farley, Peter x23353 wrote: As the OP of the other thread on python documentation, let me answer that what I expected from IBM for python documentation was the same sort of Language Reference and Programmers Guide manuals as they provide for all of the other major programming languages that they support. My understanding at the moment is that that hasn't been done and is not likely to be done. Where, for instance, in the documentation that you linked to is there information on whether the python "open()" function for files will or will not support direct access to MVS datasets without transferring them down to the Unix file system? I.E., are either of these calls to the open() function valid or not: Where is the language reference and programming language for Java on z/OS? There isn't one, so why would Python be any different? The only language references and guides you will find for Java is for products such as DB2 or for z/OS specific toolkits such as JZOS. There are no z/OS specific modifications to Python other than patches for the build. If you want to open an MVS data set then write a package using cutils to directly reference the C RTL without writing a single line of C code. Once you have a library put it on pypi so everybody can download it using "pip". Why do you expect IBM to provide the function? They are working on a completely new set of tooling and have limited resources. https://medium.com/python-supply/working-with-foreign-functions-ee3b5e40f2ca Myfile = open("//'MVS.DATA.SET.NAME", "rt", ccsid="1047") Myfile = open("//:DDNAME", "rt", ccsid="1047") # This of course assumes that a call to "bpxunix" (or by other means) has been used to establish a DD allocation in the AS where the python program is executing In other words, where are the z/OS-specific enhancements to the standard python functions and libraries documented (assuming there are any such enhancements, which I am beginning to doubt)? For other IBM-supported programming languages, that information would probably be available in the Language Reference manual and practical examples and performance-enhancing advice would be provided in the Programmers Guide manual. The first link that you provided is fine for the systems programmer who installs the language on z/OS, but has basically nothing relevant for the python application programmer. The "zoautil_py" library does provide some "direct access" to MVS datasets for python programs, but the package is quite "heavyweight" and consumes a large amount of elapsed and CPU time to use, as recent experimentation I have performed clearly showed. I found that downloading an MVS dataset to the Unix file system with the "zowe" CLI interface and accessing that file with "normal" python file access functions proved far less expensive than using the "zoautil_py" functions to access the same data. Nowhere in the IBM-provided ZOAU documentation did I find any reference to performance characteristics or performance-enhancing advice. That separate python libraries for z/OS maintain separate python documentation of their functionality is OK, but then the IBM-specific library websites need to provide the same sort of automatic linking to the library documentation that the standard python documentation websites use. I.E., the IBM-specific libraries need to have their documentation directly available on the ".readthedocs.io" website and linked to from their specific page on the Pypi indexing website and not just at their github location. What IBM-specific library? There isn't one! ISTM that IBM is a day late and a dollar short on application-level documentation for this language on their systems. I disagree. Maybe your expectations are unrealistic. Peter -----Original Message- From: IBM Mainframe Discussion List On Behalf Of David Crayford Sent: Saturday, October 1, 2022 9:35 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: IBM SMF Export with Python On 2/10/22 01:31, Charles Mills wrote: We LOVE Python. When is IBM going to love it enough to ship it and document it as though they loved it? You can order Python via zShop and install it via SMP/E or download a pax file. That's choice right there. What's wrong with the doc? https://www.ibm.com/docs/en/python-zos/3.10 has all the information I need to install, customize and build extensions. For z/OS specific Python libraries the doc is either shipped with the product so as ZAOU or for open source projects with the code in a repository. For example, the Python DB2 library and doc can be found on Github https://github.com/ibmdb/python-ibmdb . z/OS specific doc here https://github.com/ibmdb/node-ibm_db*configure-odbc-driver-on-zos . This is standard for any open sour
Re: IBM python documentation? [was: RE: IBM SMF Export with Python]
As the OP of the other thread on python documentation, let me answer that what I expected from IBM for python documentation was the same sort of Language Reference and Programmers Guide manuals as they provide for all of the other major programming languages that they support. My understanding at the moment is that that hasn't been done and is not likely to be done. Where, for instance, in the documentation that you linked to is there information on whether the python "open()" function for files will or will not support direct access to MVS datasets without transferring them down to the Unix file system? I.E., are either of these calls to the open() function valid or not: Myfile = open("//'MVS.DATA.SET.NAME", "rt", ccsid="1047") Myfile = open("//:DDNAME", "rt", ccsid="1047") # This of course assumes that a call to "bpxunix" (or by other means) has been used to establish a DD allocation in the AS where the python program is executing In other words, where are the z/OS-specific enhancements to the standard python functions and libraries documented (assuming there are any such enhancements, which I am beginning to doubt)? For other IBM-supported programming languages, that information would probably be available in the Language Reference manual and practical examples and performance-enhancing advice would be provided in the Programmers Guide manual. The first link that you provided is fine for the systems programmer who installs the language on z/OS, but has basically nothing relevant for the python application programmer. The "zoautil_py" library does provide some "direct access" to MVS datasets for python programs, but the package is quite "heavyweight" and consumes a large amount of elapsed and CPU time to use, as recent experimentation I have performed clearly showed. I found that downloading an MVS dataset to the Unix file system with the "zowe" CLI interface and accessing that file with "normal" python file access functions proved far less expensive than using the "zoautil_py" functions to access the same data. Nowhere in the IBM-provided ZOAU documentation did I find any reference to performance characteristics or performance-enhancing advice. That separate python libraries for z/OS maintain separate python documentation of their functionality is OK, but then the IBM-specific library websites need to provide the same sort of automatic linking to the library documentation that the standard python documentation websites use. I.E., the IBM-specific libraries need to have their documentation directly available on the ".readthedocs.io" website and linked to from their specific page on the Pypi indexing website and not just at their github location. ISTM that IBM is a day late and a dollar short on application-level documentation for this language on their systems. Peter -Original Message----- From: IBM Mainframe Discussion List On Behalf Of David Crayford Sent: Saturday, October 1, 2022 9:35 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: IBM SMF Export with Python On 2/10/22 01:31, Charles Mills wrote: > We LOVE Python. > > When is IBM going to love it enough to ship it and document it as though they > loved it? You can order Python via zShop and install it via SMP/E or download a pax file. That's choice right there. What's wrong with the doc? https://www.ibm.com/docs/en/python-zos/3.10 has all the information I need to install, customize and build extensions. For z/OS specific Python libraries the doc is either shipped with the product so as ZAOU or for open source projects with the code in a repository. For example, the Python DB2 library and doc can be found on Github https://github.com/ibmdb/python-ibmdb . z/OS specific doc here https://github.com/ibmdb/node-ibm_db*configure-odbc-driver-on-zos . This is standard for any open source project. I understand that may be uncomfortable to folks that have no experience working with open source. Another example is https://github.com/ambitus/pyzkiln which is maintained by IBMers from Poughkeepsie but part of the OMP. Anybody interested in writing Python libraries that include calling HLASM should take a look at that repo. Honestly, I don't see what the issue is with doc? It's no different to Java. If I need doc for the JRE I read the Oracle doc. If I need doc for Python I head on over to the Python doc. Everything I need for z/OS is covered. > > (Rhetorical question. I know that David does not know the answer.) > > Charles > > > -----Original Message----- > From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] > On Behalf Of David Crayford > Sent: Friday, September 30, 2022 11:09 PM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: IBM SMF Export w
Re: IBM SMF Export with Python
On 2/10/22 01:31, Charles Mills wrote: We LOVE Python. When is IBM going to love it enough to ship it and document it as though they loved it? You can order Python via zShop and install it via SMP/E or download a pax file. That's choice right there. What's wrong with the doc? https://www.ibm.com/docs/en/python-zos/3.10 has all the information I need to install, customize and build extensions. For z/OS specific Python libraries the doc is either shipped with the product so as ZAOU or for open source projects with the code in a repository. For example, the Python DB2 library and doc can be found on Github https://github.com/ibmdb/python-ibmdb. z/OS specific doc here https://github.com/ibmdb/node-ibm_db#configure-odbc-driver-on-zos. This is standard for any open source project. I understand that may be uncomfortable to folks that have no experience working with open source. Another example is https://github.com/ambitus/pyzkiln which is maintained by IBMers from Poughkeepsie but part of the OMP. Anybody interested in writing Python libraries that include calling HLASM should take a look at that repo. Honestly, I don't see what the issue is with doc? It's no different to Java. If I need doc for the JRE I read the Oracle doc. If I need doc for Python I head on over to the Python doc. Everything I need for z/OS is covered. (Rhetorical question. I know that David does not know the answer.) Charles -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of David Crayford Sent: Friday, September 30, 2022 11:09 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: IBM SMF Export with Python I know Python is not well received on this forum but as an example of IBMs investment they have announced a new Python toolkit to process SMF data using Jupyter notesbooks, an industry standard visualization tool for analytics and machine learning. I understand many people consider this stuff buzz words and fads but if you dig in a bit it's very cool. https://zos-hot-topics.com/2022/SMF-Explorer/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: IBM SMF Export with Python
We LOVE Python. When is IBM going to love it enough to ship it and document it as though they loved it? (Rhetorical question. I know that David does not know the answer.) Charles -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of David Crayford Sent: Friday, September 30, 2022 11:09 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: IBM SMF Export with Python I know Python is not well received on this forum but as an example of IBMs investment they have announced a new Python toolkit to process SMF data using Jupyter notesbooks, an industry standard visualization tool for analytics and machine learning. I understand many people consider this stuff buzz words and fads but if you dig in a bit it's very cool. https://zos-hot-topics.com/2022/SMF-Explorer/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
IBM SMF Export with Python
I know Python is not well received on this forum but as an example of IBMs investment they have announced a new Python toolkit to process SMF data using Jupyter notesbooks, an industry standard visualization tool for analytics and machine learning. I understand many people consider this stuff buzz words and fads but if you dig in a bit it's very cool. https://zos-hot-topics.com/2022/SMF-Explorer/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN