Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
Staffen Tylen at IBM Mainframe Discussion List wrote on 02/22/2012 12:34:17 PM: > Frank, you did it again! I start to believe that I had not tried the > MOSIZE parameter together with all the rest, so when it's there - no > more dynamic allocations :) But as expected I got sort capacity > exceeded situations, which I only could solve by adding a high > MAINSIZE value like 1000M together with REGION=0M. With all this in > place in a separate test job that just ran a standard PGM=SORT with > no sort work files it worked like a charm and with an incredible > performance, less than 20 secs to sort almost 750,000 records :) > > Again, many thanks Frank and all others involved. I'm glad to hear you got what you wanted, and glad I could help. Frank Yaeger - DFSORT Development Team (IBM) - yae...@us.ibm.com Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration => DFSORT/MVS is on the Web at http://www.ibm.com/storage/dfsort -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
Steve, the original code was making calls to an internal QuickSort routine written in assembler but I got to a point where the code abended when the input to the sort was too large so I decided to look at using DFSORT instead. I now have a combination of the two in place where small sorts use the internal QuickSort and bigger ones triggers calls to DFSORT. Not very elegant maybe but the next generation programmers need something to do as well ;) Thanks anyway. Cheers, Staffan -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
Frank, you did it again! I start to believe that I had not tried the MOSIZE parameter together with all the rest, so when it's there - no more dynamic allocations :) But as expected I got sort capacity exceeded situations, which I only could solve by adding a high MAINSIZE value like 1000M together with REGION=0M. With all this in place in a separate test job that just ran a standard PGM=SORT with no sort work files it worked like a charm and with an incredible performance, less than 20 secs to sort almost 750,000 records :) Again, many thanks Frank and all others involved. Staffan -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
On 2/22/2012 12:27 PM, Ulrich Krueger wrote: Thank you for your response, Staffan. To the best of my knowledge (and, being retired, I cannot experiment with a current release of DFSORT), there's no way I can think of to make DFSORT _not_ allocate SORTWKs for every invocation. So I had been thinking along the lines of "How can I avoid the _entire_ overhead of invoking SORT multiple times?" With or without SORTWKs, the cumulative overhead of invoking SORT (as you said) 1000's of times ... that's going to kill your performance. If there's any chance to rework the program's logic, similar to what I said in my previous post, and call SORT only once ... that's going to give you the best bang for your buck. I hope, you can find a solution to your problem. Perhaps posting the SORT messages for one of the invocations might help shed some more light on the issue. Regards, Ulrich Krueger So, DFSORT is being invoked from a program, as I recall. Here's an idea: have the program call the C qsort function instead of DFSORT. I have a simple COBOL program that does this: 01 no-rowspic s9(9) binary value 0. 01 row-size pic s9(9) binary value +60. 01 compdesc function-pointer. ... 01 parts-table. 05 parts occurs 300 times indexed by part-index. 10 parts-no pic x(9). 10 parts-desc pic x(30). 10 parts-on-hand pic 9. 10 parts-on-ord pic 999. 10 parts-pricepic v999. 10 parts-reordpic 999. 10 pic xxx. ... procedure division using parms. mainline. set compdesc to entry 'compdesc' ... call 'qsort' using parts-table, by value no-rows, row-size, compdesc The routine 'compdesc' is where the actual compare takes place. Since I am comparing the descrption field (parts-desc in the table above), I wrote the following Assembler code: COMPDESC CSECT COMPDESC AMODE 31 COMPDESC RMODE ANY * Copyright (C) 2004 by Steven H. Comstock * * code to be called as a C function to compare two consecutive *entries in a table of Inventory items, in process of *sorting by description field, which is 9 bytes in * with a length of 30; no save area is used or *needed, and code returns zero if two items * equal, positive number if first is high, *negative number if first is low * l 15,0(1) l 1,4(,1) clc9(30,15),9(1) je zero jh high lnr15,15 high ds 0h br 14 zero ds 0h sr 15,15 br 14 dc c'compdesc ver1' END COMPDESC So, could you code a similar program to do the compare between two entries then invoke the C qsort using your routine for the compare? -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of Staffan Tylen Sent: Wednesday, February 22, 2012 11:08 AM To: IBM-MAIN@bama.ua.edu Subject: Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort David, Ulrich, Many thanks for your comments but I think we are drifting away from the original issue, namely that there seems to be no way to prevent dynamic allocation of sort work files. I wish to be able to sort records in storage without "risking" that work files are dynamically allocated. I know this may sound silly but that's not the point. The point I'm making is that the documentation for DFSORT seems to show ways to do this using various parameters such as DYNALLOC, FILESZ, etc. but I can't make it work. Many thanks for any continued input to this. Staffan -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN -- Kind regards, -Steve Comstock The Trainer's Friend, Inc. 303-355-2752 http://www.trainersfriend.com * To get a good Return on your Investment, first make an investment! + Training your people is an excellent investment * Try our tool for calculating your Return On Investment for training dollars at http://www.trainersfriend.com/ROI/roi.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
Staffan Tylen at IBM Mainframe Discussion List wrote on 02/22/2012 11:08:10 AM: >I wish to be able to sort > records in storage without "risking" that work files are dynamically > allocated Staffan, I don't know exactly what your job looks like so it's difficult to tell you how to do what you want. Can you try doing the following with your job: Remove any SORTWKdd DD statements you have coded. Add: //DFSPARM DD * OPTION MOSIZE=0,HIPRMAX=0,DSPSIZE=0,DYNALLOC=OFF /* If the job still dynamically allocates work data sets, then send me your complete JES log offline (yae...@us.ibm.com) and I'll take a look. Frank Yaeger - DFSORT Development Team (IBM) - yae...@us.ibm.com Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration => DFSORT/MVS is on the Web at http://www.ibm.com/storage/dfsort -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
Thank you for your response, Staffan. To the best of my knowledge (and, being retired, I cannot experiment with a current release of DFSORT), there's no way I can think of to make DFSORT _not_ allocate SORTWKs for every invocation. So I had been thinking along the lines of "How can I avoid the _entire_ overhead of invoking SORT multiple times?" With or without SORTWKs, the cumulative overhead of invoking SORT (as you said) 1000's of times ... that's going to kill your performance. If there's any chance to rework the program's logic, similar to what I said in my previous post, and call SORT only once ... that's going to give you the best bang for your buck. I hope, you can find a solution to your problem. Perhaps posting the SORT messages for one of the invocations might help shed some more light on the issue. Regards, Ulrich Krueger -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of Staffan Tylen Sent: Wednesday, February 22, 2012 11:08 AM To: IBM-MAIN@bama.ua.edu Subject: Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort David, Ulrich, Many thanks for your comments but I think we are drifting away from the original issue, namely that there seems to be no way to prevent dynamic allocation of sort work files. I wish to be able to sort records in storage without "risking" that work files are dynamically allocated. I know this may sound silly but that's not the point. The point I'm making is that the documentation for DFSORT seems to show ways to do this using various parameters such as DYNALLOC, FILESZ, etc. but I can't make it work. Many thanks for any continued input to this. Staffan -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
David, Ulrich, Many thanks for your comments but I think we are drifting away from the original issue, namely that there seems to be no way to prevent dynamic allocation of sort work files. I wish to be able to sort records in storage without "risking" that work files are dynamically allocated. I know this may sound silly but that's not the point. The point I'm making is that the documentation for DFSORT seems to show ways to do this using various parameters such as DYNALLOC, FILESZ, etc. but I can't make it work. Many thanks for any continued input to this. Staffan -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
Staffan, I'm still thinking about ways to eliminate the multiple SORTs ... there should be a way to achieve this, but, not having the full details about your program, I might me totally off base. So, my thinking is: If you currently sort by fields A, B, C in each SORT invocation ... Wouldn't adding a sequence number to the records solve the problem? For example, all records belonging to SORT #1 have SEQNUM=1, all records for SORT #2 have SEQNUM=2, etc., incrementing SEQNUM by 1 for what is now each individual SORT. When all records have been generated, run one SORT and sort by SEQNUM, A, B, C. Now process the output in groups by SEQNUM. Hot, warm, cold, not even close? How am I doing? Regards, Ulrich Krueger -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of Staffan Tylen Sent: Wednesday, February 22, 2012 1:12 AM To: IBM-MAIN@bama.ua.edu Subject: Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort Yes, unfortunately, because the records are dynamically created and could not be put in a single file for a one-time sort. Why should life be simple? :) I still find it strange that there is no way to prevent dynamic allocation 100%, even though the documentation hints about ways to do so, for example DYNALLOC=OFF,FILESZ=U0,DSPSIZE=0. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
I think you really need to figure out why your JCL sortwork is being directed to VIO. Have you tried using a large space allocation. Usually ACS routines set a MAXSIZE threshold for directing to a VIO storage group. If you can stop the SORT work data set from being directed to VIO, then DFSORT will not try to re-allocate it to SORTDK. Have a nice day, Dave Betten DFSORT Development, Performance Lead IBM Corporation email: bet...@us.ibm.com 1-301-240-3809 DFSORT/MVSontheweb at http://www.ibm.com/storage/dfsort/ IBM Mainframe Discussion List wrote on 02/22/2012 12:17:33 PM: > [image removed] > > Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort > > Staffan Tylen > > to: > > IBM-MAIN > > 02/22/2012 12:19 PM > > Sent by: > > IBM Mainframe Discussion List > > Please respond to IBM Mainframe Discussion List > > Sorry Otto, SORTWK01 in the JCL doesn't help. DFSORT then starts to > allocate SORTDKnn files as real files, not VIO as is the case with > dynamic allocation. > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
Sorry Otto, SORTWK01 in the JCL doesn't help. DFSORT then starts to allocate SORTDKnn files as real files, not VIO as is the case with dynamic allocation. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
Try specifying one a sortwrk dd with space=(cyl,(1,1) by do so sort should not use dynamic allocation. Please let me know your results. Regards Otto H Schumacher Transaction and Database Systems - CICS Specialist U. S. Mainframe HP Enterprise Services Telephone +1 864 987 1417 Mobile +1 864 569 5338 Email otto.schumac...@hp.com -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of Staffan Tylen Sent: Wednesday, February 22, 2012 4:12 AM To: IBM-MAIN@bama.ua.edu Subject: Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort Yes, unfortunately, because the records are dynamically created and could not be put in a single file for a one-time sort. Why should life be simple? :) I still find it strange that there is no way to prevent dynamic allocation 100%, even though the documentation hints about ways to do so, for example DYNALLOC=OFF,FILESZ=U0,DSPSIZE=0. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
Yes, unfortunately, because the records are dynamically created and could not be put in a single file for a one-time sort. Why should life be simple? :) I still find it strange that there is no way to prevent dynamic allocation 100%, even though the documentation hints about ways to do so, for example DYNALLOC=OFF,FILESZ=U0,DSPSIZE=0. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
> ... DFSORT is invoked 1000's of times by the program ... Once upon a time, at a previous job, I ran into a similar situation. The job ran for hours and hours. Analysis showed that SORT was invoked to sort 1 record (yes, that's ONE record!) most of the time and between 2 and about 100 records the rest of the time. A programmer had tried to cobble something together, quick-and-dirty-like, and overlooked the fact that his approach would require several thousands of SORT invocations. SORT invocations take time, especially DYNALLOC, and there's no way (that I know of) around it. You may be able reduce the overhead a bit by adding a //DFSPARM DD * OPTION DYNALLOC=(3390,2) to the JCL, as recommended earlier. The programmer eventually changed the program logic to output all the records to be sorted to a file, then called SORT and finally processed all sorted records ... the job now ran in under 5 minutes. So, my question to the original poster is: Is it really necessary to invoke SORT thousands of times or could the program logic be changed to avoid or reduce the number of SORT invocations? Regards, Ulrich Krueger -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: Avoiding DFSORT dynamic allocatoin, was: Abend S0C4 in an internal sort
I can't give specific recommendations without knowing more about these sorts being executed and the environment. But perhaps I can offer a little info that will be useful. The reason the JCL SORTWKxx data sets get reallocated to SORTDKxx is because VIO=NO is in effect. This is the installation default we recommend. So when the work data sets are allocated to VIO, DFSORT attempts to re-allocate them to SORTDKxx data sets using a different unit name in hopes they will be directed to disk instead of VIO. Unfortunately, VIO cannot be specified as a run time option so you cannot change it for just this job. But perhaps you can code JCL SORTWKs using a unit that is not directed to VIO. I don't know how your ACS routines are set up but usually a VIO storage class is assigned based on the primary space allocation size. So perhaps if you allocated 1 or 2 SORTWK dds with a larger storage size, they'd be assigned to disk and thus DFSORT would not try to re-allocate them. When no JCL sortwks are present, then DFSORT's dynamic allocation is used. Usually, even if the sort is expected to complete in memory, work data sets are still allocated since we can't be entirely sure of the file size being passed. You could at least reduce the number of work data sets being passed by using an OPTION statement with DYNALLOC=(,2) which would reduce the number of work data sets allocated. There might be some other options we can come up with if we had more info. Perhaps you can send the entire joblog from an execution of this program to our DFSORT hot line, dfs...@us.ibm.com and we can take a look. Have a nice day, Dave Betten DFSORT Development, Performance Lead IBM Corporation email: bet...@us.ibm.com DFSORT/MVSontheweb at http://www.ibm.com/storage/dfsort/ IBM Mainframe Discussion List wrote on 02/20/2012 12:22:44 PM: > [image removed] > > was: Abend S0C4 in an internal sort > > Staffan Tylen > > to: > > IBM-MAIN > > 02/20/2012 12:23 PM > > Sent by: > > IBM Mainframe Discussion List > > Please respond to IBM Mainframe Discussion List > > I wish to continue this thread as it's related. I'm now running my > program with internal calls to DFSORT and it works fine except for > one issue that I have, namely that I get a lot of job log messages > generated as a result of dynamic allocation of VIO sortwork files. > DFSORT is invoked 1000's of times by the program (yes, I know but > there is nothing I can do about that) and each time it creates a > bunch of job log messages for VIO allocations. For performance I > want all sorting to be performed in memory, and I don't really care > at the moment whether it's in hiperspace or dataspace or whatever. I > successfully used the FILSZ=Ennn parameter until I ran out of space, > so I increased the value to match the reality and since then I > cannot get rid of the dynamic allocation messages. I've tried > DYNALLOC=OFF, HIPRMAX=0, DSPSIZE=0, DSA=0, FILSZ=U0, you name it! in > all sorts of combinations in an attempt to prevent dynamic > allocation of work files, all in vain. I've tried region sizes far! > too big including REGION=0M. I've also tried preallocating > SORTWKxx to VIO data sets, but DFSORT then allocates its own > SORTDKxx data sets instead :( > > So how can I successfully sort my records in storage without using > dynamic allocation with all the rubbish job log messages as a result? > > Many thanks, > Staffan > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: was: Abend S0C4 in an internal sort
On 2/20/2012 12:32 PM, John Gilmore wrote: There may well be a facility for suppressing unwanted DFSORT messages, particularly informational ones. If there is, FY will provide you and the rest of us with information about it here (unless perchance he is in Timbuktu today). I don't think this is what he is asking to suppress. I think he wants to eliminate the allocation messages for the dynamically allocated SORTWKxx DD's. Lat time we had SYNCSORT, it didn't allocate SORTWK's unless they were needed, so an in-core sort never allocated them. Last time I checked, DFSORT seems to allocate them when it starts, so you can't eliminate them. One option may be to allocate the SORTWKxx in JCL, so there is only one set of them. Make sure DFSORT is not setup to dynamically re-allocate them. This assume you do not have multiple sorts running at the same time. Maybe Frank knows some way to cause DFSORT to not allocate them for in-core sorts. -- Richard -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: was: Abend S0C4 in an internal sort
There may well be a facility for suppressing unwanted DFSORT messages, particularly informational ones. If there is, FY will provide you and the rest of us with information about it here (unless perchance he is in Timbuktu today). John Gilmore, Ashland, MA 01721 - USA -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
was: Abend S0C4 in an internal sort
I wish to continue this thread as it's related. I'm now running my program with internal calls to DFSORT and it works fine except for one issue that I have, namely that I get a lot of job log messages generated as a result of dynamic allocation of VIO sortwork files. DFSORT is invoked 1000's of times by the program (yes, I know but there is nothing I can do about that) and each time it creates a bunch of job log messages for VIO allocations. For performance I want all sorting to be performed in memory, and I don't really care at the moment whether it's in hiperspace or dataspace or whatever. I successfully used the FILSZ=Ennn parameter until I ran out of space, so I increased the value to match the reality and since then I cannot get rid of the dynamic allocation messages. I've tried DYNALLOC=OFF, HIPRMAX=0, DSPSIZE=0, DSA=0, FILSZ=U0, you name it! in all sorts of combinations in an attempt to prevent dynamic allocation of work files, all in vain. I've tried region sizes far! too big including REGION=0M. I've also tried preallocating SORTWKxx to VIO data sets, but DFSORT then allocates its own SORTDKxx data sets instead :( So how can I successfully sort my records in storage without using dynamic allocation with all the rubbish job log messages as a result? Many thanks, Staffan -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
