David, my meaning was that timed comparisons like yours should use only python facilities available in the standard library, with NO PYPI packages installed. The python standard library includes no facilities for accessing z/OS files, hence my use of subprocess.run to accomplish that access using available and standard z/OS Unix shell commands (cat and cp) which do provide documented access to z/OS files. The subprocess.run facility out-performs the older os.system() call by a goodly margin in the testing I was able to perform.
In my test comparison, Rexx wins because of its native access to z/OS files via EXECIO, but not by a huge margin. WRT pyzfile source, the PYPI page for pyzfile provides downloads only for a python wheel, and specifically states that no source code is available. Thank you for the github address though. Peter -----Original Message----- From: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> On Behalf Of David Crayford Sent: Tuesday, February 28, 2023 3:10 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: zOSMF and zOWE for non-mainframers The pyzfile module source code has been available on Github [1] since its inception, and it can be considered the benchmark implementation for MVS data set I/O in Python. Many people, including IBM employees, have expressed interest in pyzfile [2]. Regarding your mention of standard Python modules, I'm not sure what you mean. While Python does have a large collection of modules, not all of them are part of the standard library. For instance, there is no standard YAML library in Python, but it's easy to install one using pip. This is commonplace in languages like Python, Java, and Perl. [1] https://urldefense.com/v3/__https://github.com/daveyc/pyzfile__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!ONuSOYkmdaXS_aeXM5blMXvO7VEVMO9OjwxiaLTibjqgqcjdKHRzuyjnDudiUkcCn3tj2Uixwi_gx-HZGDUZYZU$ [2] https://urldefense.com/v3/__https://community.ibm.com/community/user/ibmz-and-linuxone/discussion/reading-an-mvs-dataset-using-z-open-automation-utility__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!ONuSOYkmdaXS_aeXM5blMXvO7VEVMO9OjwxiaLTibjqgqcjdKHRzuyjnDudiUkcCn3tj2Uixwi_gx-HZxqVomhQ$ On 28/2/23 15:26, Farley, Peter wrote: > David, I will have to complain that your python benchmark is not a fair > comparison. Your python script uses a module named pyzfile to access z/OS > files (which I see from PYPI is authored by you but for which you have > published no source yet). A fairer comparison would be a python script that > only used standard python modules and shell commands. My python-vs-Rexx > testing was done using python's subprocess.run to execute the "cat" command > to copy data from z/OS files (PDS, QSAM, VSAM) to STDOUT captured by the > subprocess.run routine and then using the captured STDOUT data for the > processing. Writing to z/OS files (PDS member and QSAM only) was > accomplished by first writing the output file data to the Unix file system > (with encoding 1047 to write in EBCDIC) and then again using subprocess.run > to execute "cp" to copy the written Unix file to the z/OS file. In both read > and write cases I used the "//'DSN'" file name format for the z/OS files, > supported by both "cat" and "cp". > > Rexx using EXECIO or RXVSAM from CBT beats that type of python script by a > small margin but not by a lot -- the process I was measuring averaged 23-24 > "real" seconds per test for the python version and 19-20 "real" seconds per > test for the Rexx version. > > This was all done on the IBM Zxplore z/OS platform, which is x86 under the > covers rather than "real iron", so probably zPDT. In any case, "students" on > Zxplore aren't permitted to install any python packages and venv/virtualenv > are not available for the same reason (DASD-and-CPU-constrained system). The > platform only permits you to use standard python packages or one of the few > non-standard ones pre-installed by the admins there. > > Peter > > -----Original Message----- > From: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> On > Behalf Of David Crayford > Sent: Tuesday, February 28, 2023 1:53 AM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Re: zOSMF and zOWE for non-mainframers > > On 28/2/23 13:47, David Crayford wrote: >> On 25/2/23 01:23, Farley, Peter wrote: >>> Python on the mainframe is pretty good, but still can't beat out >>> Rexx in performance even when the Rex script needs to use BPXWUNIX >>> and friends to access z/OS Unix file systems, >>> >> I have conducted a series of benchtests, and the results suggest that >> REXX is not as fast as Python. In my testing, I compare the >> performance of C, Lua, Python, and REXX, and the results are clear: C >> is the fastest, followed by Lua, which is within an order of >> magnitude of C. Python comes next, within an order of magnitude of >> Lua, and REXX consistently performs the poorest. In addition to the >> performance factor, the vast Python ecosystem compared to the limited >> options available for REXX also make it an easy decision. Python is >> also simpler to extend with packages, while REXX requires more effort >> and potentially complex steps, such as using modern libraries that >> require Language Environment (LE). > > My benchtests > > Lua > > local file = assert(io.open(arg[1], "rb, type=record, noseek")) while true do > local rec = file:read() > if not rec then break end > end > > Python > > import sys > from pyzfile import * > try: > with ZFile(sys.argv[1], "rb,type=record,noseek") as file: > for rec in file: > pass > except ZFileError as e: > print(e) > > REXX > > /* REXX */ > arg dsname > address MVS > call bpxwdyn "ALLOC FI(INPUT) DA("dsname") SHR" > do until eof > "EXECIO 10000 DISKR INPUT ( STEM rec." > eof = (rc > 0) > end > "EXECIO 0 DISKR INPUT ( FINIS" > > The results: Add user+system to get total CPU time > > > time lua benchio.lua "//'CPA000.QADATA.PMR99999.SSA.HR1315PM'" && time > python3 benchio.py "//'CPA000.QADATA.PMR99999.SSA.HR1315PM'" && time > ./benchio.rex "CPA000.QADATA.PMR99999.SSA.HR1315PM" > real 0m47.019s > user 0m3.255s > sys 0m1.097s > > real 1m0.710s > user 0m8.001s > sys 0m2.678s > > real 1m17.772s > user 0m13.575s > sys 0m4.536s > -- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
