Re: z/OS Tag Sort
Hello Yifet, Sorry for the delayed reply but I was on vacation. I really can't go into specifics of the algorithm since that is part of our proprietary internals. We generally recommend that our customers use MAINSIZE=MAX and let DFSORT's dynamic storage adjustment calculate the optimal amount of virtual storage. I can give you some general ranges of what DFSORT would calculate as optimal for various file sizes. File SizeOptimal virtual storage 32GB ~ 64MB 64GB ~100MB 128GB ~150MB 256GB ~200MB 512GB ~256MB 1TB~350MB These are just approximations of what you'd observe if you executed sorts of these sizes with no restrictions on region, MAINSIZE=MAX and DSA large enough not to limit. Have a nice day, Dave Betten DFSORT Development, Performance Lead IBM Corporation email: bet...@us.ibm.com DFSORT/MVSontheweb at http://www.ibm.com/storage/dfsort/ IBM Mainframe Discussion List IBM-MAIN@bama.ua.edu wrote on 10/18/2011 07:39:10 AM: [image removed] Re: z/OS Tag Sort Yifat Oren to: IBM-MAIN 10/18/2011 07:50 AM Sent by: IBM Mainframe Discussion List IBM-MAIN@bama.ua.edu Please respond to IBM Mainframe Discussion List Hi David, You wrote: .. That exit is limiting the sort to just 24MB of virtual storage when the optimum amount would be closer to 200MB. The data size was 360 GB (336m records). Can you please share the formula you've used to determine the optimum amount is around 200MB? The DFSORT Tuning Guide (1.12) seems to think 2GB is the upper limit when it comes to recommending minimum virtual storage settings :) Thanks, Yifat Oren -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
Hi David, You wrote: .. That exit is limiting the sort to just 24MB of virtual storage when the optimum amount would be closer to 200MB. The data size was 360 GB (336m records). Can you please share the formula you've used to determine the optimum amount is around 200MB? The DFSORT Tuning Guide (1.12) seems to think 2GB is the upper limit when it comes to recommending minimum virtual storage settings :) Thanks, Yifat Oren -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
Thanks Elardus. Here are those messages you mentioned: ICE092I 0 MAIN STORAGE = (25165824,25165824,25147542) ICE156I 0 MAIN STORAGE ABOVE 16MB = (25083888,25083888) And here are the OPTIONS messages: ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256 ICE128I 0 OPTIONS: SIZE=25165824,MAXLIM=786432,MINLIM=786432,EQUALS=Y,LIST=Y,ERET=ABEND,MSGDDN=SYSOUT ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=FULL ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=(SYSALLDA,099),ABCODE=MSG ICE130I 0 OPTIONS: RESALL=32768,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2 ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=0,CINV=Y,CFW=N,DSA=0 ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=Y,TEXIT=Y,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N ICE133I 0 OPTIONS: HIPRMAX=8 ,DSPSIZE=20 ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=0 ICE235I 0 OPTIONS: NULLOUT=RC0 And David, thank you very much for your help. I wondered why it was only using 24MB. That did sound pretty small for a sort of this size. I didn't realize that any exits were in effect. I had suggested that they give it more virtual storage but they said they had tried it with REGION=0M and it made no difference. The fact that there's an exit explains why. So now it's a question of convincing the group responsible to permit more virtual storage to be used, at least for this particular sort. My original question about tag sorting doesn't apply in this case, it's clearly a matter of sort optimization. Thank you all for your help in this. I greatly appreciate it. Bye for now, John. On 14 October 2011 15:35, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: David Betten wrote: ICE162I 0 ICEIEXIT CHANGED ONE OR MORE OPTIONS IN EFFECT Unfortunately we don't display what options were changed. Also in the ICE132I Options message I saw IEXIT=Y, TEXIT=Y which indicate both an installation and a termination exit are in use on this system. Thanks very much, David, for pointing this out. Have a very great weekend! :-D Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- John Blythe Reid, Técnico de Sistemas de z/OS y de Sistemas Transaccionales, Barcelona, España. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
John Blythe Reid wrote: Thanks Elardus. Here are those messages you mentioned: It is a pleasure! :-D I know why you came at about 24MB. ... it's clearly a matter of sort optimization. If you are successfull, could you be kind to post the elapsed time used to sort your big dataset? Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
Here's a bit more information. What caught our eye was this line in one of the performance reports: NUM. OF BYTES JOB ELAPSED SORTWKXX TOTAL RECORDSMBYTESASSIGNED TRACKS JOBNAME STEP EXECUTIONS TIME DD EXCPS SORTEDSORTED WORK ASSIGNED XXXMA611P001 1 14:36:14 32.0 188078 3.3672E8378660.2 299102.1 14030580 This is the SORT statement: SORT FIELDS=(17,2,BI,A,19,2,CH,A,21,16,CH,A,37,4,BI,A,5,2,CH,A, 41,4,BI,A,45,4,BI,A) Here's a selection of the SORT messages: ICE247I 0 INTERMEDIATE MERGE ENTERED - PERFORMANCE MAY BE DEGRADED ICE751I 1 D8-BASE D4-K59451 EA-K57947 E1-BASE DF-BASE EA-K57947 F1-K38900 E8-K61438 ICE900I 0 CON=2,MUV=1,VOL=44,ENU=0,SBK=0,SRC=0,VEM=0 ICE999I 0 PWK=98 PSP=9069214 SWK=0 SSP=0 TWK=0 TSP=0 RWK=0 RSP=0 AWK=98 AWP=9069214 ICE090I 0 OUTPUT LRECL = 2100, BLKSIZE = 32760, TYPE = VB (SDB) ICE055I 0 INSERT 0, DELETE 0 ICE054I 0 RECORDS - IN: 336718372, OUT: 336718372 ICE134I 0 NUMBER OF BYTES SORTED: 378660163010 ICE253I 0 RECORDS SORTED - PROCESSED: 336718372, EXPECTED: 367987636 ICE098I 0 AVERAGE RECORD LENGTH - PROCESSED: 1124, EXPECTED: 1050 ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 14030580 , TRACKS USED: 14022900 ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES Bye for now, John. On 13 October 2011 20:09, Linda Mooney linda.lst...@comcast.net wrote: Hi John, When I was a student, I worked an internship with a city shop that was running a very small mainframe. Their machine was so small that they had to shutdown everything else to run payroll. Even then, they could not handle processing the entire file at once. What they called a tag sort, was a technique, not a sort feature. In this case the only truly unique field was the employee id, so that and only the fields necessary to run payroll would be pulled out to another file. There were several of these 'extract' files. Payroll processes would run through the smaller files. After that had been done for all of the 'extract' files they would be sorted back to their original sequence and the program would update the employee master file with the new field values one record at a time. This whole process took so long that the batch always started after 5PM on Friday so that it could (usually) finish in time for Monday. I don't remember what day they paid folks (I was on an unpaid internship). The city had no money for a bigger box or more memory, and the weekend was long enough for all of the I/O. Spending 14 hours in a sort seems like a very long time - how big, what kind of file is that? How long has this been a problem? Any recent changes? As others have suggested/offered, it and/or the sort and/or the job mix seem ripe for tuning. I would suggest that WLM settings, dataset placement on DASD - what kind of disk, have you checked to be sure that cache is on, do you have PAV and is it working properly, is the disk array healthy or running in a degraded mode, are there competing higher priority jobs or onlines; any of these issues can hurt performance. I have tuned poor performing batch by addressing problems with their VSAM files - took over a third of the wall clock time off of one process simply by tuning 5 VSAM files and adding some buffering, no changes to the sort or programs. What I'm saying is that, even though you are seeing a very long run time for the sort step, it is very possible that factors other than the sort are causing or contributing to the problem. If you have Omegamon, or another performance monitoring tool, it would be a good idea to monitor and/or run a collector for this job. RMF reports can also provide helpful information. HTH, Linda - Original Message - From: John Blythe Reid johnblyther...@gmail.com To: IBM-MAIN@bama.ua.edu Sent: Thursday, October 13, 2011 6:29:16 AM Subject: Re: z/OS Tag Sort Thanks, Lizette. We're using DFSORT with z/OS V1.11. Bye for now, John On 13 October 2011 13:22, Lizette Koehler stars...@mindspring.com wrote: We are sorting a very large file and it's taking about fourteen hours. I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Does anyone know if there is still a way of doing this ? Thanks Could you provide the z/OS level you are running? Sort product? Syncosrt release, CA SORT release, or DFSORT And whether or not you are doing your own sort processing rather than using DFSORT, SYNCSORT, or CA Sort? Each product might have a slight (very slight) different way
Re: z/OS Tag Sort
Thanks for your reply Elardus, I've already posted some details which answer some of your questions but here are more details: The input file is a variable length sequential dataset not an OMVS file. There are no sort exits being used. There are no specific limitations on sorting by WLM or any other process. Bye for now, John. On 13 October 2011 19:27, Elardus Engelbrecht elardus.engelbre...@sita.co.za wrote: John Blythe Reid wrote: We are sorting a very large file and it's taking about fourteen hours. After reviewing all those replies, I have four questions if you don't mind, please. 1. Please define 'very large file'. Is it a dataset or OMVS file? Is that containing long records or just that many millions of records? 2. 14 hours? I'm swallowing this very hard. Could you be kind to post your sort options, usage of memory and work datasets? Also post your sort criterias too. (If I have my way, I could sort it using an easy to use/process sort criterias and run my input again using another sort criteria...) 3. Is your sort job limited by WLM or by some other process? 4. Is your sort process having any usage of exits? I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Some solutions of DFSORT team suggested adding a number in and for each record. Perhaps you could search IBM-MAIN archives... But then I don't know of any such trick. Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- John Blythe Reid, Técnico de Sistemas de z/OS y de Sistemas Transaccionales, Barcelona, España. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
John, AFAIK, your problemis tracked by this message: ICE2471 INTERMEDIATE MERGE ENTERED - PERFORMANCE MAY BE DEGRADED. To avoid this situation, increase the DFSORT installation default DSA from it's shipped default of DSA=64 to DSA=500 to allow these larger sorts to allocate up to 500MB of virtual storage if needed. For very large sorts, consider increasing it up to 2000, which would allow the sort to use as much region as is available if needed. Note that DFSORT's dynamic storage adjustment calculates the optimum amount of virtual storage for each sort and the DSA parameter is a limit. So increasing DSA will not cause every sort to use more virutal storage but instead allow sorts that need a larger amount of virtual storage to do so. Hope this helps Paolo Cacciari - IBM Italy - BCRS IBM Italia S.p.A. Sede Legale: Circonvallazione Idroscalo - 20090 Segrate (MI) Cap. Soc. euro 384.747.048,00 C. F. e Reg. Imprese MI 01442240030 - Partita IVA 10914660153 Società soggetta all?attività di direzione e coordinamento di International Business Machines Corporation (Salvo che sia diversamente indicato sopra / Unless stated otherwise above) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
John Blythe Reid wrote: ICE247I 0 INTERMEDIATE MERGE ENTERED - PERFORMANCE MAY BE DEGRADED Good Eye Catcher. Thanks for stating it here. ICE054I 0 RECORDS - IN: 336718372, OUT: 336718372 ICE134I 0 NUMBER OF BYTES SORTED: 378660163010 ICE253I 0 RECORDS SORTED - PROCESSED: 336718372, EXPECTED: 367987636 Oh, that's just a few records... :-D No seriously, that is a very serious lot of records. I really doubt that I even reached that quantity in ONE sort job in my whole career... ICE098I 0 AVERAGE RECORD LENGTH - PROCESSED: 1124, EXPECTED: 1050 ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 14030580 , TRACKS USED: 14022900 Are you using static or dynamic work datasets? In my humble opinion, I would go to static work datasets (SORTWKxx DD), but then this is my opinion and preference. Perhaps it is not suitable for you. ICE090I 0 OUTPUT LRECL = 2100, BLKSIZE = 32760, TYPE = VB (SDB) Perhaps you could try out this system determined blocksize of 27998? ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES Zeroes only? Where are your ICE093I and ICE156I messages? Can't you post them here too, please? I would like to add to Paolo Cacciari's reply (increase DSA): Could you perhaps increase your job REGION, say to 512M or so? Also consider adding these options, which helped me to sort out big sort jobs: OPTION DYNSPC=512,SIZE=E9,MAINSIZE=MAX The DFSORT Team may add better recommendations. There are no sort exits being used. There are no specific limitations on sorting by WLM or any other process. Good. I'm glad. Hopes that helps much to SORT out your problem!!! :-D Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
John sent me his complete sysout. The Intermediate Merge is the main cause of the high elapsed time. However, in John's case, just increasing DSA is not going to help because there is an installation exit (ICEIEXIT) controlling the amount of virtual storage DFSORT can allocate. That exit is limiting the sort to just 24MB of virtual storage when the optimum amount would be closer to 200MB. As for the size of the sort, 336 million records (360GB), I have seen much larger sorts than that so I know DFSORT can handle it. We've actually executed sorts with record counts in the billions. Have a nice day, Dave Betten DFSORT Development, Performance Lead IBM Corporation email: bet...@us.ibm.com DFSORT/MVSontheweb at http://www.ibm.com/storage/dfsort/ IBM Mainframe Discussion List IBM-MAIN@bama.ua.edu wrote on 10/14/2011 06:59:11 AM: [image removed] Re: z/OS Tag Sort Elardus Engelbrecht to: IBM-MAIN 10/14/2011 07:23 AM Sent by: IBM Mainframe Discussion List IBM-MAIN@bama.ua.edu Please respond to IBM Mainframe Discussion List John Blythe Reid wrote: ICE247I 0 INTERMEDIATE MERGE ENTERED - PERFORMANCE MAY BE DEGRADED Good Eye Catcher. Thanks for stating it here. ICE054I 0 RECORDS - IN: 336718372, OUT: 336718372 ICE134I 0 NUMBER OF BYTES SORTED: 378660163010 ICE253I 0 RECORDS SORTED - PROCESSED: 336718372, EXPECTED: 367987636 Oh, that's just a few records... :-D No seriously, that is a very serious lot of records. I really doubt that I even reached that quantity in ONE sort job in my whole career... ICE098I 0 AVERAGE RECORD LENGTH - PROCESSED: 1124, EXPECTED: 1050 ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 14030580 , TRACKS USED: 14022900 Are you using static or dynamic work datasets? In my humble opinion, I would go to static work datasets (SORTWKxx DD), but then this is my opinion and preference. Perhaps it is not suitable for you. ICE090I 0 OUTPUT LRECL = 2100, BLKSIZE = 32760, TYPE = VB (SDB) Perhaps you could try out this system determined blocksize of 27998? ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES Zeroes only? Where are your ICE093I and ICE156I messages? Can't you post them here too, please? I would like to add to Paolo Cacciari's reply (increase DSA): Could you perhaps increase your job REGION, say to 512M or so? Also consider adding these options, which helped me to sort out big sort jobs: OPTION DYNSPC=512,SIZE=E9,MAINSIZE=MAX The DFSORT Team may add better recommendations. There are no sort exits being used. There are no specific limitations on sorting by WLM or any other process. Good. I'm glad. Hopes that helps much to SORT out your problem!!! :-D Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
David Betten bet...@us.ibm.com wrote: The Intermediate Merge is the main cause of the high elapsed time. However, in John's case, just increasing DSA is not going to help because there is an installation exit (ICEIEXIT) controlling the amount of virtual storage DFSORT can allocate. That exit is limiting the sort to just 24MB of virtual storage when the optimum amount would be closer to 200MB. Interesting. Are there any ICE message(s) showing that an exit is really in use/called? Just curious if you don't mind, please. As for the size of the sort, 336 million records (360GB), I have seen much larger sorts than that so I know DFSORT can handle it. We've actually executed sorts with record counts in the billions. Uhhh, I'm feeling SORTa small... :-D Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
ICE162I 0 ICEIEXIT CHANGED ONE OR MORE OPTIONS IN EFFECT Unfortunately we don't display what options were changed. Also in the ICE132I Options message I saw IEXIT=Y, TEXIT=Y which indicate both an installation and a termination exit are in use on this system. Have a nice day, Dave Betten DFSORT Development, Performance Lead IBM Corporation email: bet...@us.ibm.com 1-301-240-3809 DFSORT/MVSontheweb at http://www.ibm.com/storage/dfsort/ IBM Mainframe Discussion List IBM-MAIN@bama.ua.edu wrote on 10/14/2011 08:35:35 AM: [image removed] Re: z/OS Tag Sort Elardus Engelbrecht to: IBM-MAIN 10/14/2011 08:37 AM Sent by: IBM Mainframe Discussion List IBM-MAIN@bama.ua.edu Please respond to IBM Mainframe Discussion List David Betten bet...@us.ibm.com wrote: The Intermediate Merge is the main cause of the high elapsed time. However, in John's case, just increasing DSA is not going to help because there is an installation exit (ICEIEXIT) controlling the amount of virtual storage DFSORT can allocate. That exit is limiting the sort to just 24MB of virtual storage when the optimum amount would be closer to 200MB. Interesting. Are there any ICE message(s) showing that an exit is really in use/called? Just curious if you don't mind, please. As for the size of the sort, 336 million records (360GB), I have seen much larger sorts than that so I know DFSORT can handle it. We've actually executed sorts with record counts in the billions. Uhhh, I'm feeling SORTa small... :-D Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
David Betten wrote: ICE162I 0 ICEIEXIT CHANGED ONE OR MORE OPTIONS IN EFFECT Unfortunately we don't display what options were changed. Also in the ICE132I Options message I saw IEXIT=Y, TEXIT=Y which indicate both an installation and a termination exit are in use on this system. Thanks very much, David, for pointing this out. Have a very great weekend! :-D Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
z/OS Tag Sort
We are sorting a very large file and it's taking about fourteen hours. I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Does anyone know if there is still a way of doing this ? Thanks. Regards, John. -- John Blythe Reid, Técnico de Sistemas de z/OS y de Sistemas Transaccionales, Barcelona, España. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
We are sorting a very large file and it's taking about fourteen hours. I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Does anyone know if there is still a way of doing this ? Thanks Could you provide the z/OS level you are running? Sort product? Syncosrt release, CA SORT release, or DFSORT And whether or not you are doing your own sort processing rather than using DFSORT, SYNCSORT, or CA Sort? Each product might have a slight (very slight) different way of doing this. Or did you write your own sorting process? Not using a vendor product. Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
Thanks, Lizette. We're using DFSORT with z/OS V1.11. Bye for now, John On 13 October 2011 13:22, Lizette Koehler stars...@mindspring.com wrote: We are sorting a very large file and it's taking about fourteen hours. I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Does anyone know if there is still a way of doing this ? Thanks Could you provide the z/OS level you are running? Sort product? Syncosrt release, CA SORT release, or DFSORT And whether or not you are doing your own sort processing rather than using DFSORT, SYNCSORT, or CA Sort? Each product might have a slight (very slight) different way of doing this. Or did you write your own sorting process? Not using a vendor product. Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- John Blythe Reid, Técnico de Sistemas de z/OS y de Sistemas Transaccionales, Barcelona, España. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
Can you define what 'large' is. The best description would be: 1) Number of records - a ballpark figure will do 2) The average length of the records 3) The size of the key I don't know of any currently supported tag sort, especially of the type you are talking about. The main reason for this is that unless the input file is highly biased, or presorted, the random I/O access patterns will kill the elapsed time. It is far more efficient to read sequentially than to read random blocks. Chris Blaicher Senior Software Engineer, Software Services Syncsort Incorporated 50 Tice Boulevard, Woodcliff Lake, NJ 07677 P: 201-930-8260 | M: 512-627-3803 E: cblaic...@syncsort.com www.syncsort.com RETHINK THE ECONOMICS OF DATA -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of John Blythe Reid Sent: Thursday, October 13, 2011 4:54 AM To: MVS List Server 1 Subject: z/OS Tag Sort We are sorting a very large file and it's taking about fourteen hours. I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Does anyone know if there is still a way of doing this ? Thanks. Regards, John. -- John Blythe Reid, Técnico de Sistemas de z/OS y de Sistemas Transaccionales, Barcelona, España. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
sssnbsp; ---Please excuse my brevity. I'm trying to win The Email Game!-Original Message- Re: z/OS Tag Sort From: John Blythe Reid lt;johnblyther...@gmail.comgt;To: ibm-m...@bama.ua.eduDate: Thursday, October 13, 2011 at 6:59PM Thanks, Lizette.We're using DFSORT with z/OS V1.11.Bye for now,JohnOn 13 October 2011 13:22, Lizette Koehler wrote:gt; gt;gt; gt; We are sorting a very large file and it's taking about fourteen hours. Igt; remember thatgt; gt; there used to be a technique whereby the sort wrote out a list of recordgt; addresses andgt; gt; an application program could read the list to access the records in thegt; desiredgt; gt; sequence.gt; gt;gt; gt; Does anyone know if there is still a way of doing this ?gt; gt;gt; gt; Thanksgt;gt; Could you provide the z/OS level you are running?gt; Sort product? Syncosrt release, CA SORT release, or DFSORTgt;gt; And whether or not you are doing your own sort processing rather than usinggt; DFSORT, SYNCSORT, or CA Sort?gt;gt; Each product might have a slight (very slight) different way of doing this.gt;gt; Or did you write your own sorting process? Not using a vendor product.gt;gt; Lizettegt;gt; --gt; For IBM-MAIN subscribe / signoff / archive access instructions,gt; send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFOgt; Search the archives at http://bama.ua.edu/archives/ibm-main.htmlgt;-- John Blythe Reid,Teacute;cnico de Sistemas de z/OS y de Sistemas Transaccionales,Barcelona,Espantilde;a.--For IBM-MAIN subscribe / signoff / archive access instructions,send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFOSearch the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
Either: (1) You have a virus or (2) you're being a PITA. In either case, you're now in my autodelete file. John McKown Systems Engineer IV IT Administrative Services Group HealthMarkets(r) 9151 Boulevard 26 * N. Richland Hills * TX 76010 (817) 255-3225 phone * john.mck...@healthmarkets.com * www.HealthMarkets.com Confidentiality Notice: This e-mail message may contain confidential or proprietary information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. HealthMarkets(r) is the brand name for products underwritten and issued by the insurance subsidiaries of HealthMarkets, Inc. -The Chesapeake Life Insurance Company(r), Mid-West National Life Insurance Company of TennesseeSM and The MEGA Life and Health Insurance Company.SM -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of Arun Shanmugasundaram Sent: Thursday, October 13, 2011 8:44 AM To: IBM-MAIN@bama.ua.edu Subject: Re: z/OS Tag Sort sssnbsp; ---Please excuse my brevity. I'm trying to win The Email Game!-Original Message- Re: z/OS Tag Sort From: John Blythe Reid lt;johnblyther...@gmail.comgt;To: ibm-m...@bama.ua.eduDate: Thursday, October 13, 2011 at 6:59PM Thanks, Lizette.We're using DFSORT with z/OS V1.11.Bye for now,JohnOn 13 October 2011 13:22, Lizette Koehler wrote:gt; gt;gt; gt; We are sorting a very large file and it's taking about fourteen hours. Igt; remember thatgt; gt; there used to be a technique whereby the sort wrote out a list of recordgt; addresses andgt; gt; an application program could read the list to access the records in thegt; desiredgt; gt; sequence.gt; gt;gt; gt; Does anyone know if there is still a way of doing this ?gt; gt;gt; gt; Thanksgt;gt; Could you provide the z/OS level you are running?gt; Sort product? Syncosrt release, CA SORT release, or DFSORTgt;gt; And whether or not you are doing your own sort processing rather than usinggt; DFSORT, SYNCSORT, or CA Sort?gt;gt; Each product might have a slight (very slight) different way of doing this.gt;gt; Or did you write your own sorting process? Not using a vendor product.gt;gt; Lizettegt;gt; -- gt; For IBM-MAIN subscribe / signoff / archive access instructions,gt; send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFOgt; Search the archives at http://bama.ua.edu/archives/ibm-main.htmlgt;-- John Blythe Reid,Teacute;cnico de Sistemas de z/OS y de Sistemas Transaccionales,Barcelona,Espantilde;a.-- For IBM-MAIN subscribe / signoff / archive access instructions,send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFOSearch the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
Send the full joblog/sysout from the sort to dfs...@us.ibm.com and we can take a look to see if there are any ways to tune the large sort for you. Have a nice day, Dave Betten DFSORT Development, Performance Lead IBM Corporation email: bet...@us.ibm.com DFSORT/MVSontheweb at http://www.ibm.com/storage/dfsort/ IBM Mainframe Discussion List IBM-MAIN@bama.ua.edu wrote on 10/13/2011 05:54:04 AM: [image removed] z/OS Tag Sort John Blythe Reid to: IBM-MAIN 10/13/2011 05:54 AM Sent by: IBM Mainframe Discussion List IBM-MAIN@bama.ua.edu Please respond to IBM Mainframe Discussion List We are sorting a very large file and it's taking about fourteen hours. I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Does anyone know if there is still a way of doing this ? Thanks. Regards, John. -- John Blythe Reid, Técnico de Sistemas de z/OS y de Sistemas Transaccionales, Barcelona, España. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
John, Could you send us some more information on the size of this sort? Or perhaps include the JCL and DFSORT output from this run (suitably altered to protect sensitive information, of course)? Perhaps someone can find a way to tune and improve your SORT performance. Or, might there be an easy way to break the input into a number of smaller input files? Those could then run in parallel, thus shortening the elapsed time. Then, all you need is a final MERGE of all intermediate SORT outputs to create the final output file. Regards, Ulrich Krueger -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of John Blythe Reid Sent: Thursday, October 13, 2011 2:54 AM To: IBM-MAIN@bama.ua.edu Subject: z/OS Tag Sort We are sorting a very large file and it's taking about fourteen hours. I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Does anyone know if there is still a way of doing this ? Thanks. Regards, John. -- John Blythe Reid, Técnico de Sistemas de z/OS y de Sistemas Transaccionales, Barcelona, España. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
Ulrich, We had similar issues 40 years ago. We had to sort 14 million records (magazine subscriber base) the decision was made to break it up to 5 jobs it cost a lot in sort work areas and then a final merge. One other huge sort did a little larger sort and it used checkpoint restart. We did a lot of restarts (groan). My memory is weak here but the elapsed time was around 14 hours for each job. We did not need to gain access to the sorted output files the way this person does though. Ed -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
John Blythe Reid wrote: We are sorting a very large file and it's taking about fourteen hours. After reviewing all those replies, I have four questions if you don't mind, please. 1. Please define 'very large file'. Is it a dataset or OMVS file? Is that containing long records or just that many millions of records? 2. 14 hours? I'm swallowing this very hard. Could you be kind to post your sort options, usage of memory and work datasets? Also post your sort criterias too. (If I have my way, I could sort it using an easy to use/process sort criterias and run my input again using another sort criteria...) 3. Is your sort job limited by WLM or by some other process? 4. Is your sort process having any usage of exits? I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Some solutions of DFSORT team suggested adding a number in and for each record. Perhaps you could search IBM-MAIN archives... But then I don't know of any such trick. Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: z/OS Tag Sort
Hi John, When I was a student, I worked an internship with a city shop that was running a very small mainframe. Their machine was so small that they had to shutdown everything else to run payroll. Even then, they could not handle processing the entire file at once. What they called a tag sort, was a technique, not a sort feature. In this case the only truly unique field was the employee id, so that and only the fields necessary to run payroll would be pulled out to another file. There were several of these 'extract' files. Payroll processes would run through the smaller files. After that had been done for all of the 'extract' files they would be sorted back to their original sequence and the program would update the employee master file with the new field values one record at a time. This whole process took so long that the batch always started after 5PM on Friday so that it could (usually) finish in time for Monday. I don't remember what day they paid folks (I was on an unpaid internship). The city had no money for a bigger box or more memory, and the weekend was long enough for all of the I/O. Spending 14 hours in a sort seems like a very long time - how big, what kind of file is that? How long has this been a problem? Any recent changes? As others have suggested/offered, it and/or the sort and/or the job mix seem ripe for tuning. I would suggest that WLM settings, dataset placement on DASD - what kind of disk, have you checked to be sure that cache is on, do you have PAV and is it working properly, is the disk array healthy or running in a degraded mode, are there competing higher priority jobs or onlines; any of these issues can hurt performance. I have tuned poor performing batch by addressing problems with their VSAM files - took over a third of the wall clock time off of one process simply by tuning 5 VSAM files and adding some buffering, no changes to the sort or programs. What I'm saying is that, even though you are seeing a very long run time for the sort step, it is very possible that factors other than the sort are causing or contributing to the problem. If you have Omegamon, or another performance monitoring tool, it would be a good idea to monitor and/or run a collector for this job. RMF reports can also provide helpful information. HTH, Linda - Original Message - From: John Blythe Reid johnblyther...@gmail.com To: IBM-MAIN@bama.ua.edu Sent: Thursday, October 13, 2011 6:29:16 AM Subject: Re: z/OS Tag Sort Thanks, Lizette. We're using DFSORT with z/OS V1.11. Bye for now, John On 13 October 2011 13:22, Lizette Koehler stars...@mindspring.com wrote: We are sorting a very large file and it's taking about fourteen hours. I remember that there used to be a technique whereby the sort wrote out a list of record addresses and an application program could read the list to access the records in the desired sequence. Does anyone know if there is still a way of doing this ? Thanks Could you provide the z/OS level you are running? Sort product? Syncosrt release, CA SORT release, or DFSORT And whether or not you are doing your own sort processing rather than using DFSORT, SYNCSORT, or CA Sort? Each product might have a slight (very slight) different way of doing this. Or did you write your own sorting process? Not using a vendor product. Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- John Blythe Reid, Técnico de Sistemas de z/OS y de Sistemas Transaccionales, Barcelona, España. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
