John, It sounds to me like splitting this up into several concurrent sorts, followed by a merge, may be the right way to do this from the get go. I'm guessing you are sorting a file in the 70-100GB range. Maybe larger, and one of the problems I've come across with very large sorts like this is the sort method has to do an intermediate merge causing a huge growth in IO and SORTWK space.
Having many sorts - 5, 10 , etc - running in parallel against the subsets of the input file will probably be much faster than a single huge sort. There are several ways to subset based on key or number of records, but it always means having several jobs reading the same dataset from the same volume(s), which leads to big time IOSQ and you aren't much better off. There's a few ways to solve this: 1) PAV, but you already said that this is not an options 2) HIPERBATCH, with an E15 exit to get sort to read the file with QSAM or BSAM, you will have just one job reading the file and the rest reading from the DLF buffer. Can be very fast and effective if you keep all the jobs swapped in and dispatched at the same rate. 32MB would be plenty for this case. 3) DFSMS data striping, with the dataset in RAID-0 across as many volumes as possible you can mitigate the IOSQ problem and get read rates close to HIPERBATCH. Because all the jobs will read the whole input file, the idea is optimize reading the SORTIN. After the SORTIN is read you will have 5, 10, etc concurrent sorts operating in parallel, followed by a simple merge; or concatenation if the sort keys will let you. My preference is for HIPERBATCH, but whatever is best for your site is what counts. Batchpipes may also be something to consider. Ron > -----Original Message----- > From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On > Behalf Of McKown, John > Sent: Monday, January 14, 2008 10:34 AM > To: IBM-MAIN@BAMA.UA.EDU > Subject: Re: [IBM-MAIN] DFSORT question > > > -----Original Message----- > > From: IBM Mainframe Discussion List > > [mailto:[EMAIL PROTECTED] On Behalf Of Rick Fochtman > > Sent: Monday, January 14, 2008 12:30 PM > > To: IBM-MAIN@BAMA.UA.EDU > > Subject: Re: DFSORT question > > > > > > --------------------<snip>---------------------- > > > > >This just came up. Which is better: to allocate more, > > smaller, SORTWKnn > > >DD statements, or fewer, larger, SORTWKnn DD statements, or > > does it not > > >really matter. By "better", I mean: "Which will result in a > > shorter run > > >time?". > > > > > >The question came up due to a huge sort this weekend. DFSORT > > wanted to > > >allocate the SORTWKnn space > 3390-3 volume. We only have 3390-3 > > >volumes. So the question is, do I update defaults to have > > more than 32 > > >volumes or do we consider creating 3390-9 volumes for SORTWKnn? > > > > > >This is IBM's DFSORT on z/OS 1.8. Yes, I am aware of > > Syncsort's MAXSORT > > >capability. No, it will not be considered. End of discussion on that > > >point, please. > > > > > > > > --------------------------<unsnip>---------------------- > > My thought would be to allow more SORTWKxx datasets. Just my > > $.02 worth. > > That is likely what will happen. Expecting the programmers to know when > and how to properly allocate SORTWKnn is likely expecting far too much > in this day and age. [sigh] > > -- > John McKown > Senior Systems Programmer > HealthMarkets > Keeping the Promise of Affordable Coverage > Administrative Services Group > Information Technology > > The information contained in this e-mail message may be privileged > and/or confidential. It is for intended addressee(s) only. If you are > not the intended recipient, you are hereby notified that any > disclosure, > reproduction, distribution or other use of this communication is > strictly prohibited and could, in certain circumstances, be a criminal > offense. If you have received this e-mail in error, please notify the > sender by reply and delete this message without copying or disclosing > it. > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO > Search the archives at http://bama.ua.edu/archives/ibm-main.html ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
