I think I know now what's going on. The XPathBuilder.class is using a pool to store all @XPath() expressions and it keeps adds to the pool but it never clears the pool.
In class there's a method doStop() which runs pool.clear() and poolLogNamespaces.clear(); but it is never called, afaik That's the leak, imo /M ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, November 9th, 2021 at 07:15, Mikael Andersson Wigander <mikael.andersson.wigan...@pm.me.INVALID> wrote: > Hi > > I see what you mean but still there must be a leak somewhere… > > The payload is small > > <RefData> > > <FinInstrmGnlAttrbts> > > <Id>DE000C3JALS0</Id> > > <FullNm>EAA3 SI 20211220 CS</FullNm> > > <ShrtNm>EEX/EUAA P AUCTION SPOT</ShrtNm> > > <ClssfctnTp>ITNXXX</ClssfctnTp> > > <NtnlCcy>EUR</NtnlCcy> > > <CmmdtyDerivInd>false</CmmdtyDerivInd> > > </FinInstrmGnlAttrbts> > > <Issr>529900J0JGLSFDWNFC20</Issr> > > <TradgVnRltdAttrbts> > > <Id>XEER</Id> > > <IssrReq>false</IssrReq> > > <FrstTradDt>2015-03-25T06:00:00Z</FrstTradDt> > > <TermntnDt>2031-12-30T23:59:59Z</TermntnDt> > > </TradgVnRltdAttrbts> > > <DerivInstrmAttrbts> > > <PricMltplr>500</PricMltplr> > > <AsstClssSpcfcAttrbts> > > <Cmmdty> > > <Pdct> > > <Envttl> > > <Emssns> > > <BasePdct>ENVR</BasePdct> > > <SubPdct>EMIS</SubPdct> > > <AddtlSubPdct>EUAA</AddtlSubPdct> > > </Emssns> > > </Envttl> > > </Pdct> > > <TxTp>OTHR</TxTp> > > <FnlPricTp>EXOF</FnlPricTp> > > </Cmmdty> > > </AsstClssSpcfcAttrbts> > > </DerivInstrmAttrbts> > > <TechAttrbts> > > <RlvntCmptntAuthrty>DE</RlvntCmptntAuthrty> > > <PblctnPrd> > > <FrDt>2021-03-18</FrDt> > > </PblctnPrd> > > <RlvntTradgVn>XEER</RlvntTradgVn> > > </TechAttrbts> > > </RefData> > > Is there a way to control the behaviour of @Xpath() as with .xpath() ? > > /M > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > On Tuesday, November 9th, 2021 at 06:40, Siano, Stephan > stephan.si...@sap.com.INVALID wrote: > > > Hi > > > > I don’t think that this issue is related to thread safety. XPath as such is > > a very expensive operation as it requires parsing the document into a DOM. > > You have 10 of those XPath parameters and the heap dump shows 10 XPath > > builders that are consuming a lot of memory. You’d probably better pass the > > payload only once (maybe as a Node or Document) and then execute the XPath > > expressions on it inside the bean (then you will only parse your document > > once and have only one DOM tree). > > > > Best regards > > > > Stephan > > > > From: Mikael Andersson Wigander mikael.andersson.wigan...@pm.me.INVALID > > > > Sent: Monday, 8 November 2021 20:30 > > > > To: Camel Mail List users@camel.apache.org > > > > Subject: Sv: How to make a bean thread safe? > > > > There’s a typo in the code sample. > > > > The processing SHOULD be parallel, not sequential as in the snippet. > > > > /M > > > > På mån, nov. 8, 2021 vid 20:11, Mikael Andersson Wigander > > mikael.andersson.wigan...@pm.me.INVALID skrev: > > > > > Hi > > > > > > With the risk of being seen as a n00b (again)… > > > > > > We are processing large XML files (0.5GB/~500.000 records). > > > > > > To process them we use stream caching, spit, parallel processing, xpath > > > and a bean. > > > > > > We get a lot of OutOfMemoryExceptions and after analysing we see that the > > > call to the bean method is the villain. > > > > > > The process is to split() using tokenizeXML() on a tag that makes up one > > > record in the XML. > > > > > > For each of these records we call a bean where the method utilises > > > @Xpath() on the method parameters. > > > > > > We see in the heap dump that these calls are never GC'd, we have 90% > > > leftovers > > > > > > The question is: is this related to a not thread safe bean/method or what > > > could be the reason? > > > > > > The documentation states the default behaviour is a Signleton and when > > > used in concurrent processing it must be thread safe… > > > > > > https://camel.apache.org/components/3.11.x/bean-component.html#_options > > > > > > Running as a war under Tomcat 9 on Windows using Camel 3.11.3 and Spring > > > Boot 2.5.6. > > > > > > Server has 32GB of RAM… > > > > > > Route: > > > > > > from(file("Full")) > > > > > > .streamCaching() > > > > > > .unmarshal() > > > > > > .zipFile() > > > > > > .split() > > > > > > .tokenizeXML("RefData") > > > > > > .streaming() > > > > > > .parallelProcessing(false) > > > > > > .bean(XmlToSqlBean.class) > > > > > > .to(jdbc("default")) > > > > > > .end(); > > > > > > Bean: > > > > > > public class XmlToSqlBean { > > > > > > public String toSql(@XPath("//FinInstrmGnlAttrbts/Id") final String isin, > > > > > > @XPath("//NtnlCcy") final String currency, > > > > > > @XPath("//FullNm") final String fullName, > > > > > > @XPath("//TradgVnRltdAttrbts/Id") final String venue, > > > > > > @XPath("//ClssfctnTp") final String classification, > > > > > > @XPath("//TradgVnRltdAttrbts/TermntnDt") final String terminationDate, > > > > > > @XPath("//Issr") final String issuer, > > > > > > @XPath("//MtrtyDt") String maturityDate, > > > > > > @XPath("//TermntdRcrd") final String termnRecord, > > > > > > @XPath("//NewRcrd") final String newRecord) { > > > > > > … > > > > > > } > > > > > > } > > > > > > Thanks > > > > > > /M
