Hi Florian, Did you also set a specific Strategy for the cache of the streaming:
https://camel.apache.org/manual/stream-caching.html There you can set the threshold so that the stream is spooled to a disk instead of kept in memory. Raymond On Fri, Jul 1, 2022 at 3:49 PM EIBL Florian (ICS460-BAI) <florian.e...@spar-ics.com.invalid> wrote: > Hi, > > > > we currently use Camel 3.14. > > > > We identified a memory problem while using Camel SQL component with > outputType=StreamList and Camel split in streaming mode. > > We select lots of rows from a database table and the app slows down > (probably due to increased garbage collection) until we receive an OoM from > the JVM. > > The expected behavior is that Camel split streams one record from the > database table, handles the record and releases the exchange at the end of > the split (before the next split iteration). > > Unfortunately it seems that Camel split does not dereference everything > properly, memory is built up and this leads to an OoM at the end. > > > > I have developed a very simple test route which demonstrates my problem > (without SQL, but it shows the mem problem of the Splitter): > > > > from("scheduler:testScheduler?repeatCount=1") > > .routeId("test-route") > > .log("Starting route test-route") > > .process(exchange -> { > > Iterator<String> infiniteIter = new Iterator<>() { > > private int integer = 0; > > > > @Override public boolean hasNext() { > > return true; > > } > > @Override public String next() { > > return String.valueOf(integer++); > > } > > }; > > exchange.getMessage().setBody(infiniteIter); > > }) > > .split().body().streaming() > > .log("inside split: ${body}") > > .end() > > .log("test-route never finishes"); > > > > > > (I know the split streaming option does not make much sense in this sample > route, but we face the same mem problem with real streaming (from: SQL > StreamList) in the above mentioned use case) > > Am I missing something? For me it seems like a bug in the Splitter. I use > SQL with outputType=StreamList and split with the streaming option to > reduce memory consumption, but it does not help when the Splitter builds up > all the memory. > > > > Thanks in advance for your help. > > > > > > Regards, > > Flo > Sollten Sie diese E-Mail unbeabsichtigt bzw. irrtümlich erhalten haben, so > weisen wir Sie darauf hin, dass gemäß § 161 Abs 4 TKG 2021 der Inhalt sowie > die Tatsache des Empfangs dieser E-Mail weder aufgezeichnet noch verwertet > oder Unbefugten mitgeteilt werden dürfen. Wir ersuchen Sie, die Nachricht > von Ihrem System zu löschen und sich mit uns in Verbindung zu setzen. If > you have received this email accidentally or in error, we point out that, > in accordance with § 161 para. 4 TKG 2021 (Telecommunications Act), the > contents of this email and the fact of its receipt must not be recorded, > exploited or communicated to unauthorized persons. We ask you to delete the > message from your system and to contact us. > > Intern I Internal > <https://www.spar-ics.com/> > > Florian EIBL > Software Engineer > Business Automation & Integration > *Phone:* +43 664 6259733 > *Mobile:* +43 664 6259733 > *E-Mail:* florian.e...@spar-ics.com > > *SPAR Business Services GmbH*Information & Communication Services > Europastrasse 3, 5015 Salzburg, Austria > >
