On 1/3/23 00:48, Farley, Peter wrote:
David, my meaning was that timed comparisons like yours should use only python
facilities available in the standard library, with NO PYPI packages installed.
I frequently work with Apache Kafka, and if I aimed to evaluate the
performance of producing Kafka records using Python and Java, I would
have to rely on open-source libraries since neither language
incorporates Kafka clients in their standard libraries. Utilizing
package managers to install packages is a fundamental aspect of
utilizing contemporary programming languages. Given that pyzfile has
been made available on PyPi, it should be deemed as the standard
implementation for accessing MVS data sets from Python.
In my test comparison, Rexx wins because of its native access to z/OS files via
EXECIO, but not by a huge margin.
WRT pyzfile source, the PYPI page for pyzfile provides downloads only for a
python wheel, and specifically states that no source code is available. Thank
you for the github address though.
Peter
-----Original Message-----
From: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> On Behalf Of
David Crayford
Sent: Tuesday, February 28, 2023 3:10 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: zOSMF and zOWE for non-mainframers
The pyzfile module source code has been available on Github [1] since its
inception, and it can be considered the benchmark implementation for MVS data
set I/O in Python. Many people, including IBM employees, have expressed
interest in pyzfile [2]. Regarding your mention of standard Python modules, I'm
not sure what you mean. While Python does have a large collection of modules,
not all of them are part of the standard library. For instance, there is no
standard YAML library in Python, but it's easy to install one using pip. This
is commonplace in languages like Python, Java, and Perl.
[1]
https://urldefense.com/v3/__https://github.com/daveyc/pyzfile__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!ONuSOYkmdaXS_aeXM5blMXvO7VEVMO9OjwxiaLTibjqgqcjdKHRzuyjnDudiUkcCn3tj2Uixwi_gx-HZGDUZYZU$
[2]
https://urldefense.com/v3/__https://community.ibm.com/community/user/ibmz-and-linuxone/discussion/reading-an-mvs-dataset-using-z-open-automation-utility__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!ONuSOYkmdaXS_aeXM5blMXvO7VEVMO9OjwxiaLTibjqgqcjdKHRzuyjnDudiUkcCn3tj2Uixwi_gx-HZxqVomhQ$
On 28/2/23 15:26, Farley, Peter wrote:
David, I will have to complain that your python benchmark is not a fair comparison. Your python script uses a module named
pyzfile to access z/OS files (which I see from PYPI is authored by you but for which you have published no source yet). A fairer
comparison would be a python script that only used standard python modules and shell commands. My python-vs-Rexx testing was
done using python's subprocess.run to execute the "cat" command to copy data from z/OS files (PDS, QSAM, VSAM) to
STDOUT captured by the subprocess.run routine and then using the captured STDOUT data for the processing. Writing to z/OS files
(PDS member and QSAM only) was accomplished by first writing the output file data to the Unix file system (with encoding 1047 to
write in EBCDIC) and then again using subprocess.run to execute "cp" to copy the written Unix file to the z/OS file.
In both read and write cases I used the "//'DSN'" file name format for the z/OS files, supported by both
"cat" and "cp".
Rexx using EXECIO or RXVSAM from CBT beats that type of python script by a small margin but not by
a lot -- the process I was measuring averaged 23-24 "real" seconds per test for the
python version and 19-20 "real" seconds per test for the Rexx version.
This was all done on the IBM Zxplore z/OS platform, which is x86 under the covers rather than
"real iron", so probably zPDT. In any case, "students" on Zxplore aren't
permitted to install any python packages and venv/virtualenv are not available for the same reason
(DASD-and-CPU-constrained system). The platform only permits you to use standard python packages
or one of the few non-standard ones pre-installed by the admins there.
Peter
-----Original Message-----
From: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> On
Behalf Of David Crayford
Sent: Tuesday, February 28, 2023 1:53 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: zOSMF and zOWE for non-mainframers
On 28/2/23 13:47, David Crayford wrote:
On 25/2/23 01:23, Farley, Peter wrote:
Python on the mainframe is pretty good, but still can't beat out
Rexx in performance even when the Rex script needs to use BPXWUNIX
and friends to access z/OS Unix file systems,
I have conducted a series of benchtests, and the results suggest that
REXX is not as fast as Python. In my testing, I compare the
performance of C, Lua, Python, and REXX, and the results are clear: C
is the fastest, followed by Lua, which is within an order of
magnitude of C. Python comes next, within an order of magnitude of
Lua, and REXX consistently performs the poorest. In addition to the
performance factor, the vast Python ecosystem compared to the limited
options available for REXX also make it an easy decision. Python is
also simpler to extend with packages, while REXX requires more effort
and potentially complex steps, such as using modern libraries that
require Language Environment (LE).
My benchtests
Lua
local file = assert(io.open(arg[1], "rb, type=record, noseek")) while true do
local rec = file:read()
if not rec then break end
end
Python
import sys
from pyzfile import *
try:
with ZFile(sys.argv[1], "rb,type=record,noseek") as file:
for rec in file:
pass
except ZFileError as e:
print(e)
REXX
/* REXX */
arg dsname
address MVS
call bpxwdyn "ALLOC FI(INPUT) DA("dsname") SHR"
do until eof
"EXECIO 10000 DISKR INPUT ( STEM rec."
eof = (rc > 0)
end
"EXECIO 0 DISKR INPUT ( FINIS"
The results: Add user+system to get total CPU time
> time lua benchio.lua "//'CPA000.QADATA.PMR99999.SSA.HR1315PM'" && time python3 benchio.py
"//'CPA000.QADATA.PMR99999.SSA.HR1315PM'" && time ./benchio.rex "CPA000.QADATA.PMR99999.SSA.HR1315PM"
real 0m47.019s
user 0m3.255s
sys 0m1.097s
real 1m0.710s
user 0m8.001s
sys 0m2.678s
real 1m17.772s
user 0m13.575s
sys 0m4.536s
--
This message and any attachments are intended only for the use of the addressee
and may contain information that is privileged and confidential. If the reader
of the message is not the intended recipient or an authorized representative of
the intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication
in error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
