No problem David, these things happen, and I'm glad to hear you've found the culprit. Thanks for doing all the digging.
Max On Mon, Mar 15, 2010 at 2:19 PM, David Fuelling <sappe...@gmail.com> wrote: > Hey Max, > > I'm eating crow now. Turns out this is *not* a bug in the datastore > code. I discovered that one of my DAO parent classes was doing a > lookup before it actually created the Entity, and this lookup was > getting an EntityManager but not closing it. Turns out my own code > was the leak. > > DOH! Appologies for the false alarm and the waste of time.... > > david > > On Mar 15, 8:00 pm, David Fuelling <sappe...@gmail.com> wrote: > > Hey Max, > > > > Took a heap dump using two different scenarios. In test 1, I set the - > > Xmx to 100M, and in test 2 I set the max to 512M, and am comparing the > > difference between the number of objects across the two memory > > scenarios. In my code, I'm running a for-loop that executes 5000 > > times, and most of the objects present in JHat show up no more than > > 500 times, and in equal amounts, in either memory scenario (leads me > > to believe these are not the culprits). However, there are a handful > > of datastore classes that show a 1:1 relationship between the amount > > of memory I allow in each test. Following are the outputs from JHat: > > > > TEST #1 (100M Max) > > 101489 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $PropertyValue > > 101488 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $Property > > 52846 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $Path$Element > > 43623 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $Path > > 29782 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $Reference > > 15944 instances of class com.google.apphosting.api.DatastorePb$Query > > 13839 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $EntityProto > > 9237 instances of class > > com.google.appengine.api.datastore.dev.LocalCompositeIndexManager > > $IndexComponentsOnlyQuery > > 6705 instances of class > > com.google.appengine.api.datastore.dev.LocalDatastoreService > > $LiveQuery > > 4613 instances of class > > com.google.appengine.api.datastore.dev.LocalDatastoreService$Profile > > $EntityGroup > > > > TEST #2 (512M Max) > > 579558 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $Property > > 579558 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $PropertyValue > > 25863 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $Path$Element > > 24805 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $Path > > 17683 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $Reference > > 10564 instances of class com.google.apphosting.api.DatastorePb$Query > > 8441 instances of class > > com.google.appengine.api.datastore.dev.LocalDatastoreService > > $LiveQuery > > 6065 instances of class com.google.storage.onestore.v3.OnestoreEntity > > $EntityProto > > 5002 instances of class > > com.google.appengine.api.datastore.dev.LocalDatastoreService$Profile > > $EntityGroup > > > > From what I can tell, the OnestoreEntity$Property and OnestoreEntity > > $PropertyValue classes are growing the most (and somewhat linearly) > > with the number of Datastore operations I execute (capped only by the > > amount of memory I allow the JVM). Now, my first inclination is to > > explain my error away on the grounds that the local datastore is "in- > > memory", and so if I don't allow enough JVM memory for my datastore, > > this is expected behavior. > > > > However, I'm only persisting 5,000 User entities (and not even getting > > that far). My "User" is not terribly complicated (6 properties, 4 > > Lists, and a Preferences object). The lists and preferences are Lazy > > and empty -- I'm not creating anything for these in my test -- just a > > User....plus I'm only populating one of the field properties > > (username) with a very short string. > > > > For me to run out of 512MB of memory after creating only 4k-ish users > > means that each of my User entities is taking up around 125k each. > > Accounting for other datastore components taking up memory, we could > > say that my entities are only taking up 100k each, leaving the > > datastore the other 25k, or 100MB of memory. That still seems like my > > entities are taking up way too much memory (hence some sort of leak?), > > though I guess I don't know what's going on with the OnestoreEntity > > class to simulate the datastore, so I can't say for sure. > > > > Any thoughts here would be much appreciated. If I can provide any > > more data for you, please let me know. > > > > Thanks! > > > > David > > > > On Mar 12, 9:37 pm, "Max Ross (Google)" > > <maxr+appeng...@google.com<maxr%2bappeng...@google.com> > > > > wrote: > > > > > > > > > Thanks for the report David, this certainly seems suspicious. There is > at > > > least one memory leak I'm aware of but it's related to transactions so > > > that's probably not what you're bumping into. Have you tried taking a > heap > > > dump to see what exactly is building up? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
