Re: jBASE 4.1.5.17 - does anyone face "out of memory" problems after SSELECT?

[Jim Idle]

Jim Idle wrote: Pawel (privately) wrote: Hi, I will post more here if I work out how to guarantee that your SELECT and SSELECT will execute in an external process. Before I answer this, I meant to say that all evidence tells me that you should immediately switch to the Watson allocator - the issue may well be alleviated, or even fixed by that. Don't worry about the free(NULL), but do report it. So, the problem with just removing the .so version of SELECT and SSELECT is that the incoming and outgoing SELECT lists that these programs generate can ONLY be passed around in memory. Which means that just renaming the .so will mean the SSELECT is executed out of process but the resulting select list will be discarded. So, first try the Watson allocator (it almost certainly will not be worse), and if that does not help, try this. Until TEMENOS can fix the memory leak (assuming for the moment that there is one), we have to get very tricky, but this will immediately cure your problem if jQL memory leaks are the issue. Make sure you try this in a test installation of course as a mistake would screw up badly. What we need to do is get SELECT and SSELECT to execute externally from the batch program. Now if the real problem is just that your SSELECT is just running out of memory because it is just too damned big, then this won't help at all, or it might help short term because there is a bit more memory available in the process when it is tand alone, but will run out anyway later when the file gets bigger. If it is just too big a query, then you will have to stop doing SSELECT or get to jBASE 5 and use 64 bits and so on (though that may not help). Basically, if the SSELECT accumulates 2GB of memory allocations, then it will just die with out of memory anyway. This is why the Watson allocator may just stop the problem. So, to do this, you need the following: Create a new SSELECT and SELECT command as BASIC programs and make sure that wherever you put the resulting programs, they are found in the PATH before the real SELECT and SSELECT. Create secondary SSELECT-2 and SELECT-2 programs, that will really run SELECT and SSELECT commands - delete the .so versions of these programs after they are compiled (note we do not want to use EB.COMPILE and so on for these programs, but the jcompile command); These programs should work in conjunction with a UNIX directory, where they can store temporary lists, choose a directory that is backed by reasonably well performing disks; The replacement SELECT and SSELECT (note it can be one program that is symbolically linked to the other and uses SENTENCE() to work out which it is), picks up the command that was EXECUTED via the SENTENCE() command so it knows what to pass on; The new SELECT and SSELECT should also look to see if there is an incoming select list from the batch program and if there is, it should use WRITELIST to save it in the temporary Unix directory; Unfortunately, the replacement programs may have to look at stacked DATA too and deal with that The replacement program now does EXECUTE "SSELECT-2 ":restofcommand  (or SELECT-2) Because there are no .so versions of SSELECT-2 and SELECT-2, they will execute in a new process; These programs read in any incoming select list using READLIST and make it active They also stack any incoming DATA Then they change their PATH (might also need LD_LIBRARY_PATH or LIBPATH) environment variable so that the real SSELECT and SELECT now come first in the list They execute SSELECT with the original command line They pick up the generated select list and WRITELIST it to the temporary Unix directory, then stop gracefully The replacement SSELECT takes over again, uses READLIST to pick up the resulting list and make it active, then exits gracefully. OK, so that sounds complicated and of course this is just a work around to try and prove things one way or another. However, it isn't very many more lines of jBC code than the description above and can be implemented without disturbing the real SELECT and SSELECT. It has the advantage that if you are not leaking memory but the query is too big for in process, then the external one will probably work and will always tart with clean memory, but if you ARE leaking memory, AND it is jQL that is doing it, then you will no longer leak it. I realize that that program is probably a lot easier for me to write, but I am sure you can do it. Just test it a lot in your own login before changing the .profile for everyone on the test machine, then get everyone to try it until you are happy with putting it live. Of course, whether this helps or not, you need to be getting TEMENOS to help you fix any problem; I am sure that they will. Jim --~--~---------~--~----~------------~-------~--~----~ Please read the posting guidelines at: http://groups.google.com/group/jBASE/web/Posting%20Guidelines IMPORTANT: Type T24: at the start of the subject line for questions specific to Globus/T24 To post, send email to jBASE@googlegroups.com To unsubscribe, send email to jbase-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/jBASE?hl=en -~----------~----~----~----~------~----~------~--~---