Re: jBASE 4.1.5.17 - does anyone face "out of memory" problems after SSELECT?

[Jim Idle]

Pawel (privately) wrote: Hi, Story continues... We have found out that during night/batch processing memory utilisation grows significantly after SSELECT is performed by the session. Moreover these memory blocks seem not to be reclaimed until process dies. There are only two conclusions: 1) You get back the SELECT list of 2 million items and use it in some way but you never get rid of it in the application. If you repeat this, the select lists (or rather, what you are doing with them) will eventually eat all the memory; 2) It is a bug in jQL, which is not releasing memory blocks and you are EXECUTING the SELECT in the same process space as the calling program, which means that the calling program is accumulating the lost memory; While 1) is definitely a possibility, my guess at the moment is at 2). We did a simple test after my colleague suggestion. Here are details: a) session A: login to jshell, check your port via WHO command b) session B: run SSELECT on large table with many IDs, preferably right justified or with some non-standard characters like asterrisk, comma, dot, question mark, etc. c) session A: quickly, meanwhile, run MW42 -p<port_number_from_session_A_WHO> and observe memory allocation Process will hit some number and stay at this level, say 200 MB which is still quite too much for me. That's how much it takes if you are doing a SORT - you must accumulate all the keys and essentially make a second pass. That's one reason that I continuously tell people not to use SSELECT unless there really is some logical reason why the records must be processed in this sort Given the sort algorithm used though, this should grow somewhat linearly. Run a SSELECT on smaller table with lower number of IDs and observe memory allocation. Memory usage will stay at the same level, say 200 MB. Run a SSELECT on largest table you have with many IDs. You will see that additional memory is allocated at some point - say you will finally reach 400 MB. This sorting buffer seems never to shrink. Now, you could be seeing one of two things here, so don't jump to conclusions yet. 1) As above, the items are sorted, but now you have created a SELECT list which contains all those items, so of course the memory does not shrink. Use CLEARSELECT or runa command to utilize this list and the memory shoudl reduce (note, because of the way allocators work, it might not come back to where you think it would, but it should come back quite a bit; 2) The SSELECT, or the select list management code in jBASE is not working correctly and is simply not releasing space. If you try using the list and virtually nothing changes in memory usage, then this may be the issue. I know that previously I suggested that perhaps "transaction buffer" is not downsized, but we can easily tell that session that runs SSELECTs can allocate more and more memory. Well, we can get to this conclusion, but we have not yet done so as your methodology isn't quite correct. I am confused and still does not know wheter memory leaks or not. I understand, it is a difficult thing to pin down is memory (pun intended). We have also quickly changed to "MALLOCTYPE=watson" on test area and here is interesting output: jsh techuser ~ -->SSELECT <largetable> jBASE: Attempting to free NULL pointer at jQLSortProcessor.cpp,721(select.b,145) jBASE: Attempting to free NULL pointer at jQLSortProcessor.cpp,721(select.b,145) 1496042 Records selected CLEARSELECT Please remember that nothing was displayed on "default" allocator. Are we right that something is wrong and memory leaks? Why errors are thrown on non-standard memory allocator? And herein lies the rub. I think that you may well have uncovered a problem in jQL. The reason that using a different allocator shows this issue is that it is probably behaving better. NOw, don;t worry so much about that as jBASE is intercepting the bug and preventing the free(NULL) from causing any problems. However, this is very suspicious and would tend to give you the feeling that it is trying to free the WRONG pointer and therefore not freeing the correct one! The two things that MAY not be being freed might be huge memory allocations. Of course, it could just be that the program is trying to release something that it never allocated in the first place. There is no way to know without source code. Perhaps someone like Pat, who reads this list, could take a look? Here is how to proceed: 1) If at all possible and this is the case in ALL situations for everyone, NEVER use SSELECT to generate a processing list UNLESS the records you need to process MUST be processed in sorted order (there are of course some times when this is necessary), for the following reasons: a) It takes a lot more memory to do the sort; b) And more importantly, assuming that you are generating keys from the same file that you are going to process, you have just changed the disk access pattern from sequential (as in group has order) to random (as in wherever the primary keys happen to has in the file). This will have a huge impact on the READ times for the records and the issue will increase quadratically as the key set gets larger. I cannot emphasize enough what the performance impact of this is. 2) Now, in the past, the {S}SELECT would be performed in a separate process to the program that does the EXECUTE. This would mask any issues of not releasing memory because at process end, UNIX will reclaim the memory heap anyway and you will just be left with the memory usage for the select list, which will be given up when the batch process ends. However, in 4.1, for performance reasons, these programs are all executed within the same process space, so any memory that is not freed will never be reclaimed and will gradually make a process run out of memory. To test whether this is happening, we need to force the SSELECT to EXECUTE in a new process. If this is your program, then you can use EXECUTE CHAR(255):'k':'SSELECT .... but if it is not your process, then we will have to get tricky. However, you can redo your SSELECT testing described above by running a program that does an EXECUTE as shown here rather than the SSELECT command at jSHELL. If there is a SSELECT memory leak, then the memory used by the calling program will not be expanded by the SSELECT program, which will return the select list and the calling program will use that. Make the program repeat the SSELECT 10 times or as many as is needed to show that memory remains stable. Then, either take off the CHAR(255):'k' piece, or run SSELECT from the jSHELL lots of times and show that memory increases and never reduces. Report the findings to TEMENOS. Because you are in a very problematic situation, you should force the SSELECT (or SELECT if you can change it to use this) to execute in a separate process all the time. In jBASE 3.x there was an environment variable you could use to force a list of commands to execute in a new process all the time, but in 4.1, I do not know if this is the case. I will try to find out in the docs, but as they are now ONLY in stupid PDF and some .DOC formats, they cannot be searched on the web site. TEMENOS help desk should be able to tell you this, or maybe the jBASE guys on here already know that you can do this (sorry that my memory fails me here). The envvar for 3.1 was: http://www.jbase.com/knowledgebase/manuals/3.0/30manpages/man/env2_JBCNOINTERNAL.htm I think that it is also possible to delete the .so version of the command and this will force EXECUTE to run the executable version. However, if the executable version only calls the .so version then this will not work. There was some vacillation on whether it should do one or the other. Find the SELECT binary and find the SELECT.so and see if they are of similar size (make sure you find real binaries and not any symbolic links). However, if you have a test machine, just rename the .so versions of SELECT and SSELECT and see if they SELECT and SSELECT commands still work. If they do, then rerun on your test machine with these .so file renamed. You should find that your problems go away if the issues is SSELECT not freeing memory. Finally, if you have remote access available and you really are that in need of help, you could always, err, PAY <gasp> me to come in and at least diagnose the problem properly for you! Finally, finally, on your test machine, make sure you have the latest AIX 5.3 patches loaded (see my past diatribes about the illogic of not applying patches to operating systems) and the latest version of jBASE 4.1 and see if this helps in any way at all! I will post more here if I work out how to guarantee that your SELECT and SSELECT will execute in an external process. Jim --~--~---------~--~----~------------~-------~--~----~ Please read the posting guidelines at: http://groups.google.com/group/jBASE/web/Posting%20Guidelines IMPORTANT: Type T24: at the start of the subject line for questions specific to Globus/T24 To post, send email to jBASE@googlegroups.com To unsubscribe, send email to jbase-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/jBASE?hl=en -~----------~----~----~----~------~----~------~--~---