Some ideas: - Make sure you HAVE set the largest JVM memory option you can get away with (duh.) - Try "-server" JVM option - Use a profiler to see what is truly using memory - Look on your physical DB for buffer size settings (MySql / Mongo?) - Try closing session after every 100 or 1000 saves, and starting fresh, to make sure everything is flushed to disk - Use log statements to write to log file free VM memory after each commit, to check how fast it goes down and find any jumps in memory use - I have never used a Profiler but a lot of people use them to find memory hogging things - are you sure you are closing all streams and resources in finally blocks, and not leaking resources related to the actual image processing? - are there some GC options you can tweak to help
Best regards, Clay Ferguson [email protected] On Mon, Nov 23, 2015 at 11:13 AM, Roll, Kevin <[email protected]> wrote: > Our use case is the following: an external process generates 70 images, > each around ~700k in size. These are uploaded as sub-nodes under a master > node that encapsulates the run. There are are also some sister nodes that > contain a modest amount of metadata about each image and the run that > generated it. In general most of the writing consists of a client POSTing > these images into the repository via Sling; there are then some event > handlers and tasks that look at the data that arrived. The only subsequent > writes at present are some properties that are set after these images are > examined and replicated into another system. So, I don't expect much at all > in the way of concurrent read/write; it's mainly write a bunch and then > read it back later. > > By heavy pressure what I mean is that we have a test lab running > continuously against this system. It's a lot more traffic than can be > expected in the real world, but it is good for shaking out problems. What > concerns me is that according to the documentation an entire Lucene index > is kept in memory. Right now we don’t do any pruning - our repository only > grows larger. This implies to me that the index will only grow as well and > we will ultimately run out of memory no matter how big the heap is. > Hopefully I'm wrong about that. > > At the moment we have no JVM flags set. The SearchIndex configuration is > also default (by default I mean what came with Sling), although I am > looking at turning off supportHighlighting and putting a small value for > resultFetchSize, say 100. > > -----Original Message----- > From: Ben Frisoni [mailto:[email protected]] > Sent: Monday, November 23, 2015 11:55 AM > To: [email protected] > Subject: Re: Memory usage > > A little bit of description on the term heavy pressure might help? Does > this involve concurrent read operations/ write operations or both? > > Also some other things that effect performance: > 1. What jvm parameters are set? > 2. Do you have any custom index configurations set? > 3. What does you repostiory.xml look like? > > This background info might help with answering your question. > > On Mon, Nov 23, 2015 at 8:13 AM, Roll, Kevin <[email protected]> wrote: > > > We have started to encounter OutOfMemory errors on Jackrabbit under heavy > > pressure (it's worth noting that we are using the full Sling stack). I've > > discovered that Lucene keeps a full index of the repository in memory, > and > > this terrifies me because we are already having problems just in a test > > scenario and the repository will only grow. Unfortunately we are forced > to > > run this system on older 32-bit hardware in the field that does not have > > any room to expand memory-wise. Are there any options I can tweak to > reduce > > the memory footprint? Any other things I can disable that will cut down > on > > memory usage? Is Oak better in this regard? Thanks! > > > > >
