You are reading too far into my analysis. The leak is NOT EntityManagerFactoryImpl instances, it is a data structure (MappingRepository) that gets created as a result of creating an EMF. The MappingRepository is statically referenced elsewhere in our code and if you fail to close all EntityManagerFactoryImpl instances, the MappingRepository instances don't get cleaned up.
Thanks, Rick On Tue, May 28, 2013 at 11:24 AM, Alejandro Abdelnur <[email protected]>wrote: > Rick, > > Thanks for following up. > > After you email I've looked again at the hprof dump and the number > of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is 1. > I've also noticed the heap has > 500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but they > are referenced from a finalizer; thus in queue to be GC. So, it would > argue, the leak does not come from there. > > I'll give it a ry using 2.2.0 to see if that makes a difference. > > Thanks again. > > Alejandro > > > > On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <[email protected]> wrote: > > > After digging into the heapdump, I'm fairly certain that somewhere in > your > > application EntityManagerFactories are being created, yet they are never > > closed. I know you pointed me at your JPA service source code, but there > > must be something else going on. For giggles you could move to version > > 2.2.0, where we made some changes[1] to better tolerate applications that > > fail to close EMFs. The JIRA talks about additional diagnostics, but we > > just changed a bit of our code to use WeakReferences in hopes of avoiding > > these sorts of OOMs. If moving to 2.2.0 makes your problem go away, > you'll > > need to look elsewhere in your code for EMF creation. Hope this helps. > > > > Thanks, > > Rick > > > > > > [1] https://issues.apache.org/jira/browse/OPENJPA-2042 > > > > > > On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <[email protected] > > >wrote: > > > > > Hi Rick, a hprof dump is available at > > > http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz > > > > > > Thank you very much for looking into this. > > > > > > Alejandro > > > > > > > > > On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <[email protected]> > wrote: > > > > > > > Can you post a heapdump somewhere for review? > > > > > > > > Thanks, > > > > Rick > > > > > > > > > > > > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur < > [email protected] > > > > >wrote: > > > > > > > > > Rick, > > > > > > > > > > We create a single EMF: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160 > > > > > > > > > > And we use create/use/dispose all EMs using a command patter: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209 > > > > > > > > > > Any other idea/hint? > > > > > > > > > > Thanks > > > > > > > > > > > > > > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <[email protected]> > > > > wrote: > > > > > > > > > > > Alejandro - > > > > > > > > > > > > It appears that your application is creating > > EntityManagerFactories, > > > > yet > > > > > > they are never getting closed. > > > > > > > > > > > > Thanks, > > > > > > Rick > > > > > > > > > > > > > > > > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur < > > > [email protected] > > > > > > >wrote: > > > > > > > > > > > > > Rick, by looking at the other thread memory analyzer image, the > > > > issues > > > > > > seem > > > > > > > to be different. > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > Alejandro -- > > > > > > > > > > > > > > > > Is the problem that you're asking about the same as Hovan? If > > so, > > > > > lets > > > > > > > > consolidate this conversation down to a single thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Rick > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Alejandro, > > > > > > > > > It looks like something is holding onto references... How > > are > > > > you > > > > > > > > > enhancing your classes? At build time? Or, some other > > > > mechanism? > > > > > > > > > > > > > > > > > > Does this memory leak happen with any of your configured > > > > databases? > > > > > > > Or, > > > > > > > > > just select ones? > > > > > > > > > > > > > > > > > > This shouldn't make a difference, but specifying your > Entity > > > > > classes > > > > > > > via > > > > > > > > > both <class> and the openjpa.MetaDataFactory property is > > > > overkill. > > > > > > The > > > > > > > > > <class> elements are sufficient and is the standard, so I'd > > > > suggest > > > > > > > > > sticking with that mechanism and removing the > > > > > openjpa.MetaDataFactory > > > > > > > > > property. > > > > > > > > > > > > > > > > > > We do considerable memory leak testing both as a standalone > > JPA > > > > > > > provider > > > > > > > > as > > > > > > > > > well as the JPA provider for WebSphere Application Server. > > We > > > > > could > > > > > > > have > > > > > > > > > missed something due to your configuration requirements, > but > > > this > > > > > > looks > > > > > > > > to > > > > > > > > > be a pretty major leak... > > > > > > > > > > > > > > > > > > Can you narrow down or simplify your scenario to the point > of > > > > > > sharing a > > > > > > > > > project that demonstrates the leak? Or, maybe there is > > memory > > > > data > > > > > > > dumps > > > > > > > > > that can be shared to help diagnose the issue? > > > > > > > > > > > > > > > > > > Kevin > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur < > > > > > > [email protected] > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > And .... I'm back..... > > > > > > > > > > > > > > > > > > > > It seems that removing runtime enhancement has some > effect, > > > but > > > > > the > > > > > > > > heap > > > > > > > > > > keeps building up in the same spot. > > > > > > > > > > > > > > > > > > > > The following link is show where is the memory sink > > > > > > > > > > > > > > > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png > > > > > > > > > > > > > > > > > > > > I'd appreciate any ideas/suggestions. > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > Alejandro > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > It shoudn't... But, with that flag set as "supported", > > we > > > > > can't > > > > > > be > > > > > > > > > sure > > > > > > > > > > > that the entities were actually enhanced at build time. > > By > > > > > > setting > > > > > > > > > this > > > > > > > > > > > property to "unsupported" or removing it altogether, > then > > > if > > > > an > > > > > > > > > > unenhanced > > > > > > > > > > > entity class is loaded, we'll log a message and quit > > > > > processing. > > > > > > > So, > > > > > > > > > > let's > > > > > > > > > > > go this route first and then we'll deal with any > > potential > > > > > memory > > > > > > > > leak. > > > > > > > > > > > > > > > > > > > > > > Thanks, Kevin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur < > > > > > > > > [email protected] > > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > > > I'm in the process of verifying if Kevin's suggestion > > > fixes > > > > > the > > > > > > > > > > problem. > > > > > > > > > > > > > > > > > > > > > > > > While making the necessary changes, I've notice we > were > > > > > already > > > > > > > > > > enhancing > > > > > > > > > > > > the classes at build time. > > > > > > > > > > > > > > > > > > > > > > > > If the classes were enhanced at build time, still > would > > > be > > > > we > > > > > > see > > > > > > > > the > > > > > > > > > > > leak > > > > > > > > > > > > if the runtime enhancement flag is enabled? > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Kevin, will try that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> Don't use this: > > > > > > > > > > > > >> > > > > > > > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses" > > > > > > > > > > value="supported"/> > > > > > > > > > > > > >> > > > > > > > > > > > > >> That is probably your issue. Reference this page > > for > > > > more > > > > > > > > > > > information: > > > > > > > > > > > > >> http://openjpa.apache.org/entity-enhancement.html > > > > > > > > > > > > >> > > > > > > > > > > > > >> Good luck, Kevin > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro > Abdelnur > > < > > > > > > > > > > [email protected] > > > > > > > > > > > > >> >wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > >> > Hi Rick, > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Thanks for looking into this, the > persistence.xml > > > file > > > > > we > > > > > > > are > > > > > > > > > > using > > > > > > > > > > > > is: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > The EMF is instatiated here: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156 > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Thanks again. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Alejandro > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Can we see the contents of your > persistence.xml > > > > file? > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro > > > Abdelnur < > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in > > Oozie. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be holding > a > > > > large > > > > > > > amount > > > > > > > > > of > > > > > > > > > > > > memory > > > > > > > > > > > > >> > via > > > > > > > > > > > > >> > > > its dataCacheTimeout (then listeners) > instance > > > > > > variable > > > > > > > > > > keeping > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Class # Objects > Used > > > > Heap > > > > > > > > > Retained > > > > > > > > > > > > Heap > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 > > > 304,320 > > > > > > > > > > > 904,410,184 > > > > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 > > > > 1,223,680 > > > > > > > > > > > 105,073,624 > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > We are using a single EntityManagerFactory > and > > > we > > > > > are > > > > > > > > > closing > > > > > > > > > > > all > > > > > > > > > > > > >> > > > EntityManager instances correctly (via a > > command > > > > > > > pattern). > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > All caching settings are default ones. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Any hint where to look would be appreciated > > it. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Thanks > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Alejandro > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > -- > > > > > > > > > > > > >> > > *Rick Curtis* > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *Rick Curtis* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > *Rick Curtis* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Rick Curtis* > > > > > > > > > > > > > > > -- > > *Rick Curtis* > > > -- *Rick Curtis*
