Hi ApacheDS / MINA-ers, I am currently doing some load and memory testing against an extension made to early version of ApacheDS (after it became 1.5 but before a repository restructure that occurred not long after the rename from 1.0). What I am finding is that all search result entries are cached in memory due to ExecutorFilter.fireEvent() queing Events on the session buffer's event queue that refer to them. Only after the SearchResponseIterator queues the final SearchResponseDone message and it is encoded and flushed, do any of the search results actually get flushed out to my JNDI client (JMeter as it happens). As the searches concerned are attempting to return 100k results this behaviour leads to a huge heap requirement, and in fact imposes a seemingly articificial limit on the total number of results which can be returned by a search (I have gone to great lengths to ensure a "streaming" approach is used at the level of my custom partitions through the use of lazy custom NamingEnumerations).
This seems quite bizarre to me as I can see repeated calls to flush encoded search results but these don't seem to have the effect of streaming search results incrementally to the client as they become available, which was my hope / expectation. The streaming behaviour of MINA / ApacheDS was the subject of a query I posed many months ago and the response from the dev list at that time was that individual search results would be streamed fine but that large attributes like JPEGs may pose a problem as they are not currently streamed. I've spent a full day trying to debug what is going on between the thread doing the encoding and the thread which recieved the original search request and then iterated through the search results, but I still can't see how the SearchResponseDone causes all of the results to finally be sent when it is eventually written and flushed (and the synchronize block on the queue is exited) as the same NIO actions of writing and flushing seem to be applied for all the previous search result messages too. Also it seems strange to me that the ExecutorFilter's SessionBuffer continues to hold references to an Event matching each search result long after that result was encoded and flushed out. Once the search is 100% completed and the final search response done message is written, then the ProcessEventsRunnable.processEvents() method is called for each of these queued events but actually does nothing with them. I was just wanting to check if: a) this behaviour in the processing of search results has been noticed before b) if anyone has any tips on debugging the MINA stack in this particular case as I've found it extremely hard to get my mind around what is actually going on re the individual search results which seem to have been flushed out to NIO but then don't get passed on to the client until the search is 100% finished as the MINA code concerned is very subtle and I've not tried to groc it before. Any help / opinions / pointers gratefully accepted. Thanks, Norval
