[ https://issues.apache.org/jira/browse/CAMEL-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Anpilov closed CAMEL-20556. ------------------------------------- Resolution: Fixed BacklogTracing was tried to log file content. You should disable tracing or disable tracing file content with properties > FileConsumer OutOfMemory for big files in Unix > ---------------------------------------------- > > Key: CAMEL-20556 > URL: https://issues.apache.org/jira/browse/CAMEL-20556 > Project: Camel > Issue Type: Bug > Components: camel-file > Affects Versions: 3.20.9, 3.22.1 > Reporter: Alexander Anpilov > Priority: Minor > > For clarity, I have tested and described issue with simple routes: > > 1. *FileConsumer* > ** > {code:java} > from("file:/path") > .log("Received file: ${file:name}"); {code} > > > 2. *File pollEnrich* > {code:java} > from("direct:start") > .setProperty("SOURCE_URI", simple("{{path_to_folder}}")) > .pollEnrich().exchangeProperty("SOURCE_URI").timeout(60000) > .log("Received file: ${file:name}"); {code} > My process receive big files (more than 4x of Java XMX) and running as > spring-boot app on Kubernetes. > After upgrading from camel-3.20.2 to camel-3.20.9, the OutOfMemory error > occurs on consume. I have rolled back and checked with camel-3.20.2 again on > the same files - everything OK. > It seems, that FileConsumer publish file content into memory, instead of > using streams. > The problem starts from version camel-3.20.3 and reproduced up to 3.20.9. > *BUT:* I couldn't reproduce problem on Windows with Camel Test, only in Unix > prod env. Maybe, there are some camel-test effects... > *UPD:* I have tested camel-3.22.1 and received OutOfMemory > *UPD* > With some route modifications: > {code:java} > from("amq:start") > .pollEnrich().simple("{{path_to_folder}}").timeout(5000) > .log("Received file: ${file:name}"); {code} > I have collected HeapDump and as I can see, Camel tries to convert file > content to String > {code:java} > at java.lang.OutOfMemoryError.<init>(<unresolved string 0x0>) > at java.util.Arrays.copyOf(<unresolved string 0x0>) > local variable: byte[]#206926 > at java.lang.AbstractStringBuilder.ensureCapacityInternal(<unresolved > string 0x0>) > local variable: java.lang.StringBuilder#70 > at java.lang.AbstractStringBuilder.append(<unresolved string 0x0>) > local variable: java.lang.StringBuilder#70 > local variable: char[]#3191 > at java.lang.StringBuilder.append(<unresolved string 0x0>) > local variable: java.lang.StringBuilder#70 > at org.apache.camel.util.IOHelper.toString(IOHelper.java:142) > local variable: java.io.BufferedReader#1 > local variable: java.lang.StringBuilder#70 > local variable: char[]#3191 > at org.apache.camel.util.IOHelper.toString(IOHelper.java:125) > local variable: java.io.BufferedReader#1 > at org.apache.camel.util.IOHelper.toString(IOHelper.java:121) > local variable: java.io.BufferedReader#1 > at > org.apache.camel.component.file.GenericFileConverter.genericFileToString(GenericFileConverter.java:149) > local variable: org.apache.camel.component.file.GenericFile#1 > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: java.io.BufferedReader#1 > at > org.apache.camel.component.file.GenericFileConverterLoader.lambda$registerConverters$3(GenericFileConverterLoader.java:52) > local variable: class java.lang.String > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: org.apache.camel.component.file.GenericFile#1 > at > org.apache.camel.component.file.GenericFileConverterLoader$$Lambda$1150+0x00007f36829dbc80.doConvert(<unresolved > string 0x0>) > local variable: > org.apache.camel.component.file.GenericFileConverterLoader$$Lambda$1150+0x00007f36829dbc80#1 > local variable: class java.lang.String > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: org.apache.camel.component.file.GenericFile#1 > at > org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConverter.java:101) > local variable: class java.lang.String > local variable: org.apache.camel.component.file.GenericFile#1 > at > org.apache.camel.support.SimpleTypeConverter.tryConvertTo(SimpleTypeConverter.java:90) > at > org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:477) > local variable: org.apache.camel.impl.converter.DefaultTypeConverter#1 > local variable: class java.lang.String > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: org.apache.camel.component.file.GenericFile#1 > local variable: org.apache.camel.support.SimpleTypeConverter#26 > at > org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:358) > local variable: org.apache.camel.impl.converter.DefaultTypeConverter#1 > local variable: class java.lang.String > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: org.apache.camel.component.file.GenericFile#1 > at > org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryConvertTo(CoreTypeConverterRegistry.java:347) > at > org.apache.camel.support.MessageHelper.extractValueForLogging(MessageHelper.java:386) > local variable: org.apache.camel.component.file.GenericFile#1 > local variable: org.apache.camel.component.jms.JmsMessage#1 > at > org.apache.camel.support.MessageHelper.extractValueForLogging(MessageHelper.java:316) > local variable: org.apache.camel.component.file.GenericFile#1 > local variable: org.apache.camel.component.jms.JmsMessage#1 > at > org.apache.camel.support.MessageHelper.extractBodyForLogging(MessageHelper.java:265) > local variable: org.apache.camel.component.jms.JmsMessage#1 > at > org.apache.camel.support.MessageHelper.dumpAsXml(MessageHelper.java:528) > local variable: org.apache.camel.component.jms.JmsMessage#1 > local variable: java.lang.StringBuilder#71 > local variable: java.lang.StringBuilder#72 > local variable: java.lang.String#70214 > at > org.apache.camel.impl.engine.CamelInternalProcessor$BacklogTracerAdvice.before(CamelInternalProcessor.java:587) > local variable: > org.apache.camel.impl.engine.CamelInternalProcessor$BacklogTracerAdvice#71 > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: java.lang.String#45325 > local variable: java.lang.String#189314 > at > org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:335) > local variable: org.apache.camel.impl.engine.DefaultChannel#70 > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: org.apache.camel.processor.Pipeline$PipelineTask#1 > local variable: java.lang.Object[]#28303 > local variable: > org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask#1 > local variable: > org.apache.camel.impl.engine.CamelInternalProcessor$BacklogTracerAdvice#71 > at org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:104) > local variable: org.apache.camel.processor.Pipeline$PipelineTask#1 > local variable: org.apache.camel.impl.engine.DefaultChannel#70 > at > org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:181) > local variable: > org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker#7 > at > org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) > local variable: org.apache.camel.impl.engine.DefaultReactiveExecutor#1 > local variable: org.apache.camel.processor.Pipeline$PipelineTask#1 > at org.apache.camel.processor.Pipeline.process(Pipeline.java:165) > local variable: org.apache.camel.processor.RoutePipeline#14 > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: > org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask#2 > local variable: org.apache.camel.processor.Pipeline$PipelineTask#1 > at > org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:392) > local variable: org.apache.camel.impl.engine.CamelInternalProcessor#19 > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: > org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager$$Lambda$1773+0x00007f3683025fc8#1 > local variable: java.lang.Object[]#28304 > local variable: > org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask#2 > local variable: org.apache.camel.impl.engine.DefaultUnitOfWork#1 > local variable: > org.apache.camel.impl.engine.CamelInternalProcessor$AsyncAfterTask#2 > at > org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83) > local variable: > org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager#1 > local variable: org.apache.camel.impl.engine.CamelInternalProcessor#19 > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: java.util.concurrent.CountDownLatch#29 > at > org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:41) > local variable: org.apache.camel.impl.engine.CamelInternalProcessor#19 > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: > org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager#1 > at > org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:132) > local variable: > org.apache.camel.component.jms.EndpointMessageListener#6 > local variable: org.apache.activemq.command.ActiveMQTextMessage#2 > local variable: org.apache.activemq.jms.pool.PooledSession#79 > local variable: org.apache.activemq.command.ActiveMQQueue#83 > local variable: org.apache.camel.support.DefaultExchange#1 > local variable: > org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback#1 > at > org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736) > local variable: > org.apache.camel.component.jms.DefaultJmsMessageListenerContainer#6 > local variable: > org.apache.camel.component.jms.EndpointMessageListener#6 > local variable: org.apache.activemq.jms.pool.PooledSession#79 > local variable: org.apache.activemq.command.ActiveMQTextMessage#2 > local variable: org.apache.activemq.jms.pool.PooledSession#79 > at > org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696) > local variable: > org.apache.camel.component.jms.DefaultJmsMessageListenerContainer#6 > local variable: org.apache.activemq.jms.pool.PooledSession#79 > local variable: org.apache.activemq.command.ActiveMQTextMessage#2 > local variable: > org.apache.camel.component.jms.EndpointMessageListener#6 > at > org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674) > local variable: > org.apache.camel.component.jms.DefaultJmsMessageListenerContainer#6 > local variable: org.apache.activemq.jms.pool.PooledSession#79 > local variable: org.apache.activemq.command.ActiveMQTextMessage#2 > at > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:331) > local variable: > org.apache.camel.component.jms.DefaultJmsMessageListenerContainer#6 > at > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:270) > at > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1237) > local variable: > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker#79 > at > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1227) > local variable: > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker#79 > local variable: java.lang.Object#11095 > at > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1120) > local variable: > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker#79 > at java.util.concurrent.ThreadPoolExecutor.runWorker(<unresolved string > 0x0>) > local variable: java.util.concurrent.ThreadPoolExecutor#7 > local variable: java.util.concurrent.ThreadPoolExecutor$Worker#119 > local variable: java.lang.Thread#134 > local variable: > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker#79 > at java.util.concurrent.ThreadPoolExecutor$Worker.run(<unresolved string > 0x0>) > local variable: java.util.concurrent.ThreadPoolExecutor$Worker#119 > at java.lang.Thread.run(<unresolved string 0x0>) > local variable: java.lang.Thread#134 > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)