Re: TRSMAIN question
McKown, John wrote: Is there any documentation on the compression algorithm used by TRSMAIN? The terse algorithm is explained in IBM's US patent 4814746 from 1989, easily viewable at http://www.google.com/patents?vid=USPAT4814746 . The patent contains a PL/I program that is claimed to implement the invention, though I doubt that it would directly interoperate with implementations like TRSMAIN. And since, of course, this is a patent, presumably in the US and perhaps other countries where software patents are granted, there would be constraints if you did write something using the information therein. It isn't clear to me, however, that the patent has anything to say about decompression, so perhaps implementing that would be OK. But I'm not a patent attorney, etc. etc. so please don't bet your company on this... There are also claims out there on the net that the terse algorithm is really the same as LZW, which is also patented, and which I understand is the algorithm that got the GIF graphics format into much trouble some years ago. There are implementations of terse for all sorts of platforms, but IBM doesn't seem to distribute them for much beyond MVS and VM. If you do some severe Googling, you may find, of all things, an OS/2 16-bit version (that will run under Windows), and perhaps even a 32-bit Windows version. There appear to be several flavours of terse, which do or don't handle things like ASCII-EBCDIC and codepage issues, mainframe-only things like PDSs, and various other options. And is there anyway to ftp that to an ASCII based server and uncompress it? Yes - this relates to my previous question about RACF IRRADU00 reformatted records. If you can find an implementation for your platform of choice that understands the mainframe version's options, then sure. But then you still have to process those mainframe records on that platform, which may well be the harder part. Tony H. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TRSMAIN question
On Tue, 30 Jan 2007 15:35:07 -0600, McKown, John [EMAIL PROTECTED] wrote: And is there anyway to ftp that to an ASCII based server and uncompress it? Yes - this relates to my previous question about RACF IRRADU00 reformatted records. John, I suggest you convert the tapes to AWS, ftp to a PC and zip them there. You may use GZIP on the mainframe to reduce the network load, which I guess was your original intention. If you are not bothered about the network but instead about storage costs, go with WinZip and copy the images onto a DVD. Better yet, convert to HET and send to a PC running MVS3.8 under Hercules to be printed. Dave (Super post from Barry Merril, as usual). -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TRSMAIN question
-Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of Dave Cartwright Sent: Wednesday, January 31, 2007 3:36 AM To: IBM-MAIN@BAMA.UA.EDU Subject: Re: TRSMAIN question On Tue, 30 Jan 2007 15:35:07 -0600, McKown, John [EMAIL PROTECTED] wrote: And is there anyway to ftp that to an ASCII based server and uncompress it? Yes - this relates to my previous question about RACF IRRADU00 reformatted records. John, I suggest you convert the tapes to AWS, ftp to a PC and zip them there. You may use GZIP on the mainframe to reduce the network load, which I guess was your original intention. If you are not bothered about the network but instead about storage costs, go with WinZip and copy the images onto a DVD. Better yet, convert to HET and send to a PC running MVS3.8 under Hercules to be printed. Dave (Super post from Barry Merril, as usual). The reason to compress on the mainframe was to reduce the time needed to ftp. Trying to ftp 21 MEDIA2 tapes (3490E) worth of data to my PC (over 100Mb ethernet) scares me. I was hoping that since IRRADU00 data is all character that it would compress very effectively and that I would, overall, save time. Likely a vain hope. -- John McKown Senior Systems Programmer HealthMarkets Keeping the Promise of Affordable Coverage Administrative Services Group Information Technology The information contained in this e-mail message may be privileged and/or confidential. It is for intended addressee(s) only. If you are not the intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication is strictly prohibited and could, in certain circumstances, be a criminal offense. If you have received this e-mail in error, please notify the sender by reply and delete this message without copying or disclosing it. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TRSMAIN question
I believe TRSMAIN uses an LZ (Lempel-Ziv?) or LZW (add Welch) algorithm of sorts, but of course the algorithm matters less than the archive format in your case. IBM's Unix Tools Toys page (I believe) has GZIP ported for Unix Systems Services. I got this to work for me: cat //'dataset_name' | gzip -c archive_name.gz Yes that is double quotes around the double slash entity, and single quotes around the fully-qualified dataset name, I am sure there may be better syntax(?) but it worked. So, if your tape data is cataloged, and you have mount authority you might be able to issue a command like that for your tapes, sit back, and wait. Once you're in gzip you can decompress it on a PC I'm sure. Oh - you might have to pipe it through iconv to get it into ASCII / Unicode before zipping it. Tim Hare Senior Systems Programmer Florida Department of Transportation (850) 414-4209 -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TRSMAIN question
-Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of Tim Hare Sent: Wednesday, January 31, 2007 8:50 AM To: IBM-MAIN@BAMA.UA.EDU Subject: Re: TRSMAIN question I believe TRSMAIN uses an LZ (Lempel-Ziv?) or LZW (add Welch) algorithm of sorts, but of course the algorithm matters less than the archive format in your case. IBM's Unix Tools Toys page (I believe) has GZIP ported for Unix Systems Services. I got this to work for me: cat //'dataset_name' | gzip -c archive_name.gz Yes that is double quotes around the double slash entity, and single quotes around the fully-qualified dataset name, I am sure there may be better syntax(?) but it worked. So, if your tape data is cataloged, and you have mount authority you might be able to issue a command like that for your tapes, sit back, and wait. Once you're in gzip you can decompress it on a PC I'm sure. Oh - you might have to pipe it through iconv to get it into ASCII / Unicode before zipping it. Tim Hare Thanks for the idea. It may be easier than trying to reverse engineer TRSMAIN. Assuming that I had the talent to do so and it is not forbidden by IBM. -- John McKown Senior Systems Programmer HealthMarkets Keeping the Promise of Affordable Coverage Administrative Services Group Information Technology The information contained in this e-mail message may be privileged and/or confidential. It is for intended addressee(s) only. If you are not the intended recipient, you are hereby notified that any disclosure, reproduction, distribution or other use of this communication is strictly prohibited and could, in certain circumstances, be a criminal offense. If you have received this e-mail in error, please notify the sender by reply and delete this message without copying or disclosing it. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TRSMAIN question
In a recent note, Tim Hare said: Date: Wed, 31 Jan 2007 09:49:31 -0500 IBM's Unix Tools Toys page (I believe) has GZIP ported for Unix Systems Services. I got this to work for me: cat //'dataset_name' | gzip -c archive_name.gz Yes that is double quotes around the double slash entity, and single quotes around the fully-qualified dataset name, I am sure there may be better syntax(?) but it worked. I have found no better syntax. cat is not among the utilities documented as supporting Classic data sets; use at your own risk. It will likely not produce expected results for both character and binary files. A form using a supported utility is: cp //'dataset_name' /dev/fd/1 | gzip -c archive_name.gz Additional options to cp can select character or binary mode, etc. So, if your tape data is cataloged, and you have mount authority you might be able to issue a command like that for your tapes, sit back, and wait. In an earlier submission to this thread I suggested using IEBGENER instead of cat or cp. I formed the IEBGENER habit before Classic data set support in cp was announced; it still retains the value of supporting uncatalogued data sets and concatenations. You'd need to write a wrapper (I use Rexx) to connect IEBGENER to gzip. Then you might have the added value or connecting gzip directly to FTP with no need for a temporary archive-name.gx. -- gil -- StorageTek INFORMATION made POWERFUL -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TRSMAIN question
- snip - Once you're in gzip you can decompress it on a PC I'm sure. Oh - you might have to pipe it through iconv to get it into ASCII / Unicode before zipping it. Thanks for the idea. It may be easier than trying to reverse engineer TRSMAIN. Assuming that I had the talent to do so and it is not forbidden by IBM. - snip - What about infozip URL: www.info-zip.org ? Knowing nothing about RACF IRRADU00 reformatted records, a question: Is it possible to transfer less data using DFSORT/ICETOOL to extract what you need or even to found the problematic records (or the number of them)? Zaromil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TRSMAIN question
In a recent note, McKown, John said: Date: Tue, 30 Jan 2007 15:35:07 -0600 Is there any documentation on the compression algorithm used by TRSMAIN? Or how effective it is? I.e. if I have 21 MEDIA2 (3490E) tapes worth of printable data, can I estimate how many compressed tapes that will take? And is there anyway to ftp that to an ASCII based server and uncompress it? Yes - this relates to my previous question about RACF IRRADU00 reformatted records. The documentation on TRSMAIN is, well, terse. Many years ago, I did a comparison and some flavor of Terse was the best compressor I found; better than zip, better than UNIX compress. I didn't compare it to bzip2; for all I know Terse may be similar to bzip2. On an ASCII platform you'd have better chance of finding uncompress or gunzip. It's a shame TRSMAIN refuses to write to a POSIX pipe which could eliminate the need for intermediate storage on z/OS. How does one submit a Requirement against TRSMAIN? If you can mount the the USB drive on an FTP server, I'd suggest: MEDIA2 - IEBGENER | compress | FTP //DD: - network - USB with no intermediate storage on z/OS. This should work if IBM has fixed the various APARs I've started against FTP. Your report size appears to be growing exponentially. -- gil -- StorageTek INFORMATION made POWERFUL -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: TRSMAIN question
1. We create both Windows Zipped and z/OS Tersed distribution files for MXG Software, which is a single sequential pure text file, currently 2,119,181 lines of text; the lines are FB 80 on z/OS, but are not numbered, so the file is smaller as a variable-length ASCII file. Our current version's stored sizes are: Size of FB 80 EBCDIC file, z/OS 169,534,800 bytes Size of PC ASCII variable length104,353,987 bytes Zipped PC file 17,589,006 bytes Tersed FB 8021,653,504 bytes Terse reduced the z/OS file by a factor of 7.82. Zip reduced the ASCII file by a factor of 5.93. But, the 8-bit z/OS file is 62% larger than the ASCII file; not only is there the 8-bit EBDCIC vs 7-bit ASCII, but the ASCII file lines are the actual length of text, while each line of the z/OS file is 80 bytes long. But the 169:21 reduction, almost 8:1 reduction of the 80-byte EBCDIC text to its TERSEd equivalent is very consistent with my experience with not only text files, but also z/OS customer's SMF data files. 2. For Windows-to-Windows ftp with compression, we use Serv-U as our ftp server and Voyager ftp clients, and consistently see the same 8:1 compression, i.e. reduced transfer time to 1/8th; sure would be nice if z/OS ftp programs would support Serv-U's compression for our customer's ftp of our product. But even without compression, with a single T1, it only takes 15 minutes to download the 160MB file, which is a whole lot faster than even overnight shipment. 3. Unfortunately, after writing paragraph 2, I realize you want to unterse on the PC, so that information is of no use to you, (but, having been written it's still worth sharing that experience with this august group). I'm not aware of any un-Terse on ASCII platforms, so I still have to ship customer's Tersed datas to a z/OS box for Un-tersing. Barry Merrill -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
