Hi Jon Excellent stuff. Yes it's an ear with about 10 skinny wars inside and all shared libs in /libs in the ear . Each war is a microservice but we deploy as an ear in environments where it's too much of a hassle to split the services up and deploy individually.
Thanks so much for your effort on this! On Mon, 16 Dec 2019, 16:48 Jonathan Gallimore, <[email protected]> wrote: > Further update - I think my changes work with respect to this issue, but I > do have a bunch of unit test failures which I need to look into. For info, > here's the diff: > > diff --git > > a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java > > b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java > index a6e2bdc4f5..c0d1a8c98f 100644 > --- > > a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java > +++ > > b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java > @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService; > import org.apache.openejb.classloader.ClassLoaderConfigurer; > import org.apache.openejb.classloader.CompositeClassLoaderConfigurer; > import org.apache.openejb.component.ClassLoaderEnricher; > -import org.apache.openejb.config.ConfigurationFactory; > -import org.apache.openejb.config.NewLoaderLogic; > -import org.apache.openejb.config.QuickJarsTxtParser; > -import org.apache.openejb.config.TldScanner; > +import org.apache.openejb.config.*; > import org.apache.openejb.core.ConnectorReference; > import org.apache.openejb.core.CoreContainerSystem; > import org.apache.openejb.core.CoreUserTransaction; > @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool > implements org.apache.openejb.spi.A > > final AppContext appContext = new > AppContext(appInfo.appId, SystemInstance.get(), classLoader, > globalJndiContext, appJndiContext, appInfo.standaloneModule); > appContext.getProperties().putAll(appInfo.properties); > + > + final Set<Object> keys = > appContext.getProperties().keySet(); > + for (final Object key : keys) { > + if (! Module.class.isInstance(key)) { > + appContext.getProperties().remove(key); > + } > + } > + > appContext.getInjections().addAll(injections); > appContext.getBindings().putAll(globalBindings); > appContext.getBindings().putAll(appBindings); > diff --git > > a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java > > b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java > index feff954d40..aed0fda941 100644 > --- > > a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java > +++ > > b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java > @@ -384,12 +384,7 @@ public class JndiEncBuilder { > reference = new LinkRef(jndiName); > > } else if (BeanManager.class.equals(type)) { > - reference = new LazyObjectReference<BeanManager>(new > Callable<BeanManager>() { > - @Override > - public BeanManager call() throws Exception { > - return new > > InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl()); > - } > - }); > + reference = new LazyObjectReference<>(new > BeanManagerLazyReference()); > > } else if (UserTransaction.class.equals(type)) { > reference = new > IntraVmJndiReference("comp/UserTransaction"); > @@ -684,4 +679,11 @@ public class JndiEncBuilder { > } > } > } > + > + public static class BeanManagerLazyReference implements > Callable<BeanManager> { > + @Override > + public BeanManager call() throws Exception { > + return new > > InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl()); > + } > + } > } > diff --git > > a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java > > b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java > index 2cf387e112..285007925f 100644 > --- > > a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java > +++ > > b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java > @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader { > private final ClassLoader system; > private final boolean embedded; > private final boolean parentURLClassLoader; > + private final StackTraceElement[] created; > > // 80% of class files are smaller then 6k > private final ByteArrayOutputStream bout = new ByteArrayOutputStream(6 > * 1024); > @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader { > this.system = ClassLoader.getSystemClassLoader(); > this.embedded = this.getClass().getClassLoader() == this.system; > this.parentURLClassLoader = > URLClassLoader.class.isInstance(parent); > + this.created = new Throwable().getStackTrace(); > } > > /* > > In essence, this does 2 things: > > 1. Defines the Callable for the LazyObjectReference for BeanManager as a > static class. This stops that object from having a reference to > JndiEncBuilder and everything that has attached to it. > 2. Strips off the EjbModule / WebModule from the SuperProperties on > AppContext. This is set when deploying through the apps folder and lives > forever. It has the TempClassLoader attached, and therefore all the > AnnotationFinder stuff. > > I suspect (2) is too aggressive and is causing the test failures, but will > confirm. > > Jon > > On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore < > [email protected]> wrote: > > > Thanks for sending those over. I'm digging through this - I *think* I > have > > pinned down a specific reference that stops that classloader from being > > released. The behaviour for a war/ear deployed in apps/ seems different > to > > deploying in webapps/ - I'm assuming that's what you're doing. If you can > > confirm, that would help. I'll test out a patch with some bigger .ear > files > > today. > > > > Thanks > > > > Jon > > > > On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore < > > [email protected]> wrote: > > > >> The mailing list seems to be stripping off your images - can you resend > >> and include my email address as well? I'd be keen to dig into this a > little > >> more. > >> > >> Thanks > >> > >> Jon > >> > >> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown > >> <[email protected]> wrote: > >> > >>> Hi, > >>> > >>> The code does seem to use org.apache.openejb.core.TempClassLoader. I > can > >>> see one instance in the heap for every war in my ear. + 1. > >>> > >>> The TempClassLoader however still has 100's of strong references to it. > >>> E.g: > >>> > >>> [image: image.png] > >>> > >>> [image: image.png] > >>> > >>> > >>> Paul Carter-Brown > >>> Director > >>> Jini Guru > >>> m: +27 (0) 83 442 7179 <+27834427179> > >>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie > >>> Johannesburg, South Africa > >>> w: jini.guru e: [email protected] > >>> > >>> Disclaimer: This message and/or attachment(s) may contain > >>> privileged, confidential and/or personal information. If you are not > the > >>> intended recipient you may not disclose or distribute any of > >>> the information contained within this message. In such case you must > >>> destroy this message and inform the sender of the error. Jini Guru may > not > >>> accept liability for any errors, omissions, information and viruses > >>> contained in the transmission of this message. Any opinions, > conclusions > >>> and other information contained within this message not related to Jini > >>> Guru official business is deemed to be that of the individual only and > is > >>> not endorsed by Jini Guru. > >>> > >>> > >>> > >>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg > <[email protected]> > >>> wrote: > >>> > >>>> Hi! > >>>> > >>>> All this is just a workaround imo. > >>>> > >>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for > >>>> scanning. That means after doing all the class scans we only keep the > >>>> extracted information but do not keep the Classes in memory. Doesn't > this > >>>> work anymore? > >>>> > >>>> LieGrue, > >>>> strub > >>>> > >>>> > >>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown > >>>> <[email protected]>: > >>>> > > >>>> > FYI. Graph of the change in heap size on a prod instance after this > >>>> change: > >>>> > > >>>> > > >>>> > Paul Carter-Brown > >>>> > Director > >>>> > Jini Guru > >>>> > m: +27 (0) 83 442 7179 > >>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and > >>>> Leslie > >>>> > Johannesburg, South Africa > >>>> > w: jini.guru e: [email protected] > >>>> > > >>>> > Disclaimer: This message and/or attachment(s) may contain > privileged, > >>>> confidential and/or personal information. If you are not the intended > >>>> recipient you may not disclose or distribute any of the information > >>>> contained within this message. In such case you must destroy this > message > >>>> and inform the sender of the error. Jini Guru may not accept > liability for > >>>> any errors, omissions, information and viruses contained in the > >>>> transmission of this message. Any opinions, conclusions and other > >>>> information contained within this message not related to Jini Guru > official > >>>> business is deemed to be that of the individual only and is not > endorsed by > >>>> Jini Guru. > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown > >>>> <[email protected]> wrote: > >>>> > Hi Jon, > >>>> > > >>>> > I took a look at the source and worked out that I could add my own > >>>> filters using openejb.additional.exclude. > >>>> > > >>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM > >>>> heap now sits at around 150MB whereas it would never go below 450MB > >>>> previously!!! > >>>> > > >>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence > >>>> they are not filtered out and the system works 100%. > >>>> > > >>>> > My sense is that there should be an easier way to specify what jars > >>>> and/or packages to process for annotations. I have come across the > >>>> following settings but can't really work out what fits where: > >>>> > > >>>> > openejb.additional.exclude > >>>> > openejb.additional.include > >>>> > openejb.deployments.classpath.filter.descriptors > >>>> > openejb.deployments.classpath.filter.systemapps > >>>> > openejb.deployments.classpath.include > >>>> > openejb.deployments.classpath.exclude > >>>> > openejb.deployments.package.include > >>>> > openejb.deployments.package.exclude > >>>> > > >>>> > > >>>> > P.S. Perhaps also add these to the standard exclusion list as they > >>>> are common and yet won't need to be processed for annotations I guess? > >>>> > com.amazonaws > >>>> > com.fasterxml > >>>> > com.google > >>>> > com.hazelcast > >>>> > io.grpc > >>>> > io.netty > >>>> > org.docx4j > >>>> > org.mongodb > >>>> > org.rocksdb > >>>> > org.asynchttpclient > >>>> > org.apache > >>>> > org.aspectj > >>>> > > >>>> > But then maybe some are too broad and I don't really understand what > >>>> the annotation index/cache is really needed for at runtime? Can it > not be > >>>> lazy loaded or discarded if unused? Just thinking out loud here :-( > Seems > >>>> like the cache uses a significant amount of heap when considering the > drive > >>>> towards tiny micro services. > >>>> > > >>>> > Paul Carter-Brown > >>>> > Director > >>>> > Jini Guru > >>>> > m: +27 (0) 83 442 7179 > >>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and > >>>> Leslie > >>>> > Johannesburg, South Africa > >>>> > w: jini.guru e: [email protected] > >>>> > > >>>> > Disclaimer: This message and/or attachment(s) may contain > privileged, > >>>> confidential and/or personal information. If you are not the intended > >>>> recipient you may not disclose or distribute any of the information > >>>> contained within this message. In such case you must destroy this > message > >>>> and inform the sender of the error. Jini Guru may not accept > liability for > >>>> any errors, omissions, information and viruses contained in the > >>>> transmission of this message. Any opinions, conclusions and other > >>>> information contained within this message not related to Jini Guru > official > >>>> business is deemed to be that of the individual only and is not > endorsed by > >>>> Jini Guru. > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown > >>>> <[email protected]> wrote: > >>>> > Hi Jon, > >>>> > > >>>> > Unfortunately the snapshot behaves exactly the same way > >>>> > > >>>> > Paul Carter-Brown > >>>> > Director > >>>> > Jini Guru > >>>> > m: +27 (0) 83 442 7179 > >>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and > >>>> Leslie > >>>> > Johannesburg, South Africa > >>>> > w: jini.guru e: [email protected] > >>>> > > >>>> > Disclaimer: This message and/or attachment(s) may contain > privileged, > >>>> confidential and/or personal information. If you are not the intended > >>>> recipient you may not disclose or distribute any of the information > >>>> contained within this message. In such case you must destroy this > message > >>>> and inform the sender of the error. Jini Guru may not accept > liability for > >>>> any errors, omissions, information and viruses contained in the > >>>> transmission of this message. Any opinions, conclusions and other > >>>> information contained within this message not related to Jini Guru > official > >>>> business is deemed to be that of the individual only and is not > endorsed by > >>>> Jini Guru. > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore < > >>>> [email protected]> wrote: > >>>> > Hi Paul > >>>> > > >>>> > Would you mind trying your application with a recent snapshot? > >>>> > > >>>> > https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/ > >>>> . > >>>> > I recently updated the exclude list. > >>>> > > >>>> > Many thanks > >>>> > > >>>> > Jon > >>>> > > >>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown > >>>> > <[email protected]> wrote: > >>>> > > >>>> > > Hi, > >>>> > > > >>>> > > I've been trying to lower the memory footprint of an EAR deployed > >>>> to TomEE > >>>> > > 8.0.0. > >>>> > > > >>>> > > Here is a screenshot of the heap histogram. The total Old Gen is > >>>> about > >>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my > EAR > >>>> then > >>>> > > the old gen is about 6MB. > >>>> > > > >>>> > > [image: Screenshot from 2019-12-11 12-53-12.png] > >>>> > > > >>>> > > The entire ear is 140MB zip, most of which is in the ears /lib > >>>> directory > >>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google > >>>> cloud > >>>> > > APIs, AWS client APIs etc etc. In total our code has about 100 > >>>> actual > >>>> > > EJB's. If i remove files from the lib folder in the ear then I can > >>>> see that > >>>> > > the memory used by the annotation finder is lowered. > >>>> > > > >>>> > > Is there any way I can tell TomEE that it need not bother scanning > >>>> > > everything in the /lib folder of my EAR for annotations and > fulling > >>>> up the > >>>> > > heap. I already > >>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to > >>>> scan > >>>> > > only the one jar in /lib which does have EJB's in it and it seems > >>>> to obey > >>>> > > this property but it doesn't seem to mean that annotation > >>>> processing is > >>>> > > skipped for all these other jars in /lib > >>>> > > > >>>> > > Thanks! > >>>> > > > >>>> > > Paul Carter-Brown > >>>> > > Director > >>>> > > Jini Guru > >>>> > > m: +27 (0) 83 442 7179 <+27834427179> > >>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and > >>>> Leslie > >>>> > > Johannesburg, South Africa > >>>> > > w: jini.guru e: [email protected] > >>>> > > > >>>> > > Disclaimer: This message and/or attachment(s) may contain > >>>> > > privileged, confidential and/or personal information. If you are > >>>> not the > >>>> > > intended recipient you may not disclose or distribute any of > >>>> > > the information contained within this message. In such case you > must > >>>> > > destroy this message and inform the sender of the error. Jini Guru > >>>> may not > >>>> > > accept liability for any errors, omissions, information and > viruses > >>>> > > contained in the transmission of this message. Any opinions, > >>>> conclusions > >>>> > > and other information contained within this message not related to > >>>> Jini > >>>> > > Guru official business is deemed to be that of the individual only > >>>> and is > >>>> > > not endorsed by Jini Guru. > >>>> > > > >>>> > > > >>>> > >>>> >
