In addition to Simo I also encountered sometimes FileNotFoundException and other I think disk related issues like:
org.apache.camel.RuntimeCamelException: Cannot reset stream from file tmp/camelcontext-ID_5ffdbadf852c8f0010034cb9/cos4845255624781072102.tmp What maybe also is possible is to set it by default to true, but that there are some checks like if there is enough space (for example there must be 10% available) and the correct file permissions. If they are not the case then print a warning (instead of failing) that spooling is set to false with the reason why this is the case. I think ActiveMQ Artemis uses this kind of check for its message store. Raymond On Fri, Nov 25, 2022 at 11:40 AM Simo Hakanen <simo.haka...@axelhealth.com> wrote: > On Fri, Nov 25, 2022 at 11:09 AM Michael Rambichler <mich...@rambichler.at > > > wrote: > > > Hi Claus, Bakab > > > > Thanks for your feedback. > > The argument with the explicit decision is valid. But on the other hand > > IMHO its better that the route fails immediately then running > uncontrolled > > OOM later in production. > > > > If you agree with me I will make the JIRA/Pull Request for setting the > > default to true. > > > > @Claus. We have some routes with more then 1MB XML messages which has to > be > > streamed and parsed. > > > > BR > > Michael > > > > Am Do., 24. Nov. 2022 um 17:11 Uhr schrieb Babak Vahdat > > <babak.vah...@swissonline.ch.invalid>: > > > > > > > > > > > > On 24 Nov 2022, at 16:48, Claus Ibsen <claus.ib...@gmail.com> wrote: > > > > > > > > On Thu, Nov 24, 2022 at 3:27 PM Babak Vahdat > > > > <babak.vah...@swissonline.ch.invalid> wrote: > > > > > > > >> Hi > > > >> > > > >> I remember there was a rational behind that: > > > >> > > > > > > Sorry I made a tiny typo above, I meant “rationale” and not “rational”. > > > > > > Babak > > > > > > >> https://issues.apache.org/jira/browse/CAMEL-18098 < > > > >> https://issues.apache.org/jira/browse/CAMEL-18098> > > > >> > > > >> "As spooling to disk requires that the volume have space and the > user > > > has > > > >> permission to write to disk etc. For container workloads this is not > > > always > > > >> the case." > > > >> > > > >> So better would be to explicitly enable if *really* required for > large > > > >> data streams which is not always the case? > > > >> > > > > Babak > > > >> > > > > > > > > > > > > Ah yeah, good point Babak. Users that work with large streams needs > to > > > > configure for such use-cases. > > > > You may also want to configure which temporary directory to use for > > > > spooling, and what byte limit to overflow etc. > > > > > > > > > > > > > > > >> > > > >>> On 24 Nov 2022, at 14:59, Claus Ibsen <claus.ib...@gmail.com> > wrote: > > > >>> > > > >>> Hi > > > >>> > > > >>> Yeah that is a good idea to overflow to disk. > > > >>> You are welcome to create a JIRA and send a PR against main branch. > > > >>> > > > >>> How big streams are you processing btw ? > > > >>> > > > >>> On Thu, Nov 24, 2022 at 2:37 PM Michael Rambichler < > > > >> mich...@rambichler.at> > > > >>> wrote: > > > >>> > > > >>>> Hi all, > > > >>>> > > > >>>> with camel 3.18. stream caching is enabled by default. That's > good. > > > >>>> > > > >>>> But we have realized that spooling is disabled by default. > > > >>>> IMHO this can lead to unwanted OOM situations. Because from now on > > (>= > > > >>>> 3.18.x) a big stream will be cached by default in memory. > > > >>>> Or am I wrong? > > > >>>> > > > >>>> I would strongly suggest that we set the default of spoolEnabled > > also > > > to > > > >>>> true. > > > >>>> > > > >>>> What do you think? > > > >>>> > > > >>>> BR > > > >>>> Michael > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> Claus Ibsen > > > >>> ----------------- > > > >>> @davsclaus > > > >>> Camel in Action 2: https://www.manning.com/ibsen2 > > > >> > > > >> > > > > > > > > -- > > > > Claus Ibsen > > > > ----------------- > > > > @davsclaus > > > > Camel in Action 2: https://www.manning.com/ibsen2 > > > > > > > > > > Hi all > > Not to hijack the thread but to point out that there's possible problems > with streamcaching and spooling, related to InOut-routes. > When UnitOfWork ends, the spooled file will be deleted and that happens > before the Out-response is created, leading to > FileNotFoundException. So defaulting spooling enabled might lead to > unexpected problems elsewhere. > > -- > Simo Hakanen >
