Hi all, I have filed a critical JIRA Bug Ticket: https://issues.apache.org/jira/browse/CAMEL-18255 The case can easily be reproduced and risks a lot of our splitting routes.
Best Michael Am Fr., 1. Juli 2022 um 16:06 Uhr schrieb ski n <raymondmees...@gmail.com>: > Hi Florian, > > Did you also set a specific Strategy for the cache of the streaming: > > https://camel.apache.org/manual/stream-caching.html > > There you can set the threshold so that the stream is spooled to a disk > instead of kept in memory. > > Raymond > > > > > > On Fri, Jul 1, 2022 at 3:49 PM EIBL Florian (ICS460-BAI) > <florian.e...@spar-ics.com.invalid> wrote: > >> Hi, >> >> >> >> we currently use Camel 3.14. >> >> >> >> We identified a memory problem while using Camel SQL component with >> outputType=StreamList and Camel split in streaming mode. >> >> We select lots of rows from a database table and the app slows down >> (probably due to increased garbage collection) until we receive an OoM from >> the JVM. >> >> The expected behavior is that Camel split streams one record from the >> database table, handles the record and releases the exchange at the end of >> the split (before the next split iteration). >> >> Unfortunately it seems that Camel split does not dereference everything >> properly, memory is built up and this leads to an OoM at the end. >> >> >> >> I have developed a very simple test route which demonstrates my problem >> (without SQL, but it shows the mem problem of the Splitter): >> >> >> >> from("scheduler:testScheduler?repeatCount=1") >> >> .routeId("test-route") >> >> .log("Starting route test-route") >> >> .process(exchange -> { >> >> Iterator<String> infiniteIter = new Iterator<>() { >> >> private int integer = 0; >> >> >> >> @Override public boolean hasNext() { >> >> return true; >> >> } >> >> @Override public String next() { >> >> return String.valueOf(integer++); >> >> } >> >> }; >> >> exchange.getMessage().setBody(infiniteIter); >> >> }) >> >> .split().body().streaming() >> >> .log("inside split: ${body}") >> >> .end() >> >> .log("test-route never finishes"); >> >> >> >> >> >> (I know the split streaming option does not make much sense in this >> sample route, but we face the same mem problem with real streaming (from: >> SQL StreamList) in the above mentioned use case) >> >> Am I missing something? For me it seems like a bug in the Splitter. I use >> SQL with outputType=StreamList and split with the streaming option to >> reduce memory consumption, but it does not help when the Splitter builds up >> all the memory. >> >> >> >> Thanks in advance for your help. >> >> >> >> >> >> Regards, >> >> Flo >> Sollten Sie diese E-Mail unbeabsichtigt bzw. irrtümlich erhalten haben, >> so weisen wir Sie darauf hin, dass gemäß § 161 Abs 4 TKG 2021 der Inhalt >> sowie die Tatsache des Empfangs dieser E-Mail weder aufgezeichnet noch >> verwertet oder Unbefugten mitgeteilt werden dürfen. Wir ersuchen Sie, die >> Nachricht von Ihrem System zu löschen und sich mit uns in Verbindung zu >> setzen. If you have received this email accidentally or in error, we point >> out that, in accordance with § 161 para. 4 TKG 2021 (Telecommunications >> Act), the contents of this email and the fact of its receipt must not be >> recorded, exploited or communicated to unauthorized persons. We ask you to >> delete the message from your system and to contact us. >> >> Intern I Internal >> <https://www.spar-ics.com/> >> >> Florian EIBL >> Software Engineer >> Business Automation & Integration >> *Phone:* +43 664 6259733 >> *Mobile:* +43 664 6259733 >> *E-Mail:* florian.e...@spar-ics.com >> >> *SPAR Business Services GmbH*Information & Communication Services >> Europastrasse 3, 5015 Salzburg, Austria >> >> >
