[
https://issues.apache.org/jira/browse/LOG4J2-414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785231#comment-13785231
]
Remko Popma commented on LOG4J2-414:
------------------------------------
Good point.
I've done some back-of-the-napkin calculations, and assuming that every log
message is 50 characters long and is parameterized with two integers, each
ParameterizedMessage object in the RingBufferLogEvent would take up about 419
bytes.
The FQN, logger name and thread name (attributes of RingBufferLogEvent ) are
all cached (references to the same single String object) so I think we can
discount the memory they occupy.
For the default ring buffer size of 256*1024 slots, I calculate 30MB for the
(empty) RingBufferLogEvents, and an additional 110MB if the ring buffer is
fully filled with these 50-char + 2 integers log messages. This assumes:
* none of the log events have any exceptions (Exceptions and the stack trace
they contain will take up a lot of memory)
* the application does *not* use the ThreadContext map or stack (otherwise
every RingBufferEvent will have a copy of the map/stack)
Yiru, can you confirm that your log files have (almost) no exceptions and that
your application does not use the ThreadContext map/stack?
What puzzles me is the error message: " SEVERE: Exception processing: 781134 /
775221 org.apache.logging.log4j.core.async.RingBufferLogEvent@xxxxxx
java.lang.OutOfMemoryError: Java heap space / GC overhead limit exceeded"
This sounds like RingBufferLogEvents are being allocated and garbage collected.
This should not happen!
There is only one static ring buffer, and it is fully populated with
RingBufferLogEvent objects when Log4J initializes. After initialization new
RingBufferLogEvents are never created and the existing RingBufferLogEvents are
never released. So I don't understand why the error message mentions
RingBufferLogEvent...
> Async all loggers cause OutOfMemory error in log4j-2.0-beta9
> ------------------------------------------------------------
>
> Key: LOG4J2-414
> URL: https://issues.apache.org/jira/browse/LOG4J2-414
> Project: Log4j 2
> Issue Type: Bug
> Components: API, Core, log4j 1.2 emulation, SLF4J Bridge
> Affects Versions: 2.0-beta9
> Environment: linux core-4.0, java version: 1.7.0_17, memory: 8G,
> CPU: 2 cores, Intel (R) Xeon(R), startup options: -Xms64m -Xmx2048m
> -XX:MaxPermSixe=256m
> Reporter: Yiru Li
>
> 1. Problem description:
> The main function of my company's system is to read a file and then do
> calculation. The system has been using log4j1.2. We intend to switch to
> log4j2. The problem is found when we are doing evaluation on log4j2.
>
> Using the log4j2.xml described below , setting all loggers being synchronous,
> there is no problem to ran a 30k-row-long file through the system.
> With setting all loggers being asynchronous, there is no problem to run a
> small file (10-row-long) through the system, but the OutOfMemory error (heap
> space) is caused when the system runs a little larger file (3k-row-long). The
> error message is:" SEVERE: Exception processing: 781134
> org.apache.logging.log4j.core.async.RingBufferLogEvent@4f6f399c
> java.lang.OutOfMemoryError: Java heap space"
> Then I increased Xmx to 4048m, the error message shown is little different:
> "SEVERE: Exception processing: 775221
> org.apache.logging.log4j.core.async.RingBufferLogEvent@1c6b80a9
> java.lang.OutOfMemoryError: GC overhead limit exceeded"
> The same issue is repeated whenever the system runs a 3k-long file. I don't
> know how you can repeat this issue in your site.
>
> 2. start-up options
> -Xms64m -Xmx4048m -XX:MaxPermSize=256m -server $PARAM
> -Djava.security.egd=file:/dev/./urandom
> -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
> -Dlog4j.debug
> 3. The relevant jars which is deployed to our system:
> slf4j-api-1.6.6.jar
> log4j-slf4j-impl-2.0-beta9.jar
> log4j-1.2-api-2.0-beta9.jar
> log4j-api-2.0-beta9.jar
> log4j-core-2.0-beta9.jar
> disruptor-3.2.0.jar
> 4. the content of log4j2.xml is copied below:
> {code}
> <?xml version="1.0" encoding="UTF-8"?>
> <Configuration>
> <Appenders>
> <!-- Appender R -->
> <RollingFile name="R"
> fileName="/logs4j2/tpaeventsystem/log4j/STACBatch-Common.log"
>
> filePattern="/logs-4j2/tpaeventsystem/log4j/$${date:yyyy-MM}/STACBatch-Common-%d{MM-dd-yyyy}-%i.log.gz"
> append="true" bufferedIO="true" immediateFlush="true" >
> <PatternLayout pattern="(%d), %X{FILEID}, %X{HostName},
> STACBatch-Common, %m %t %n"/>
> <Policies>
> <SizeBasedTriggeringPolicy size="10 MB"/>
> </Policies>
> <DefaultRolloverStrategy max="50"/>
> </RollingFile>
>
> <!-- Appender FileProcessor -->
> <RollingFile name="FileProcessor"
> fileName="/logs-4j2/tpaeventsystem/log4j/STACBatch-FileProcessor.log"
>
> filePattern="/logs-4j2/tpaeventsystem/log4j/$${date:yyyy-MM}/STACBatch-FileProcessor-%d{MM-dd-yyyy}-%i.log.gz"
> append="true" bufferedIO="true" immediateFlush="true">
> <PatternLayout pattern="(%d), %X{FILEID}, %X{HostName},
> STACBatch-FileProcessor, %m %t %n"/>
> <Policies>
> <SizeBasedTriggeringPolicy size="10 MB"/>
> </Policies>
> <DefaultRolloverStrategy max="50"/>
>
> </RollingFile>
>
> <!-- Appender FTPProcessor -->
> <RollingFile name="FTPProcessor"
> fileName="/logs-4j2/tpaeventsystem/log4j/STACatch-FTPProcessor.log"
>
> filePattern="/logs-4j2/tpaeventsystem/log4j/$${date:yyyy-MM}/STACBatch-FTPProcessor-%d{MM-dd-yyyy}-%i.log.gz"
> append="true" bufferedIO="true" immediateFlush="true">
> <PatternLayout pattern="(%d), %X{HostName},
> STACBatch-FTPProcessor, %m %t %n"/>
> <Policies>
> <SizeBasedTriggeringPolicy size="10 MB"/>
> </Policies>
> <DefaultRolloverStrategy max="50"/>
> </RollingFile>
>
> <!-- Appender Email -->
> <RollingFile name="Email"
> fileName="/logs-4j2/tpaeventsystem/log4j/STACBatch-Email.log"
>
> filePattern="/logs-4j2/tpaeventsystem/log4j/$${date:yyyy-MM}/STACBatch-Email-%d{MM-dd-yyyy}-%i.log.gz"
> append="true" bufferedIO="true" immediateFlush="true">
> <PatternLayout pattern="(%d), %X{FILEID}, %X{HostName},
> STACBatch-Email, %m %t %n"/>
> <Policies>
> <SizeBasedTriggeringPolicy size="10 MB"/>
> </Policies>
> <DefaultRolloverStrategy max="50"/>
> </RollingFile>
>
> <!-- Appender Security -->
> <RollingFile name="Security"
> fileName="/logs-4j2/tpaeventsystem/log4j/STACBatch-Security.log"
>
> filePattern="/logs-4j2/tpaeventsystem/log4j/$${date:yyyy-MM}/STACBatch-Security-%d{MM-dd-yyyy}-%i.log.gz"
> append="true" bufferedIO="true" immediateFlush="true">
> <PatternLayout pattern="(%d), %X{FILEID}, %X{HostName},
> STACBatch-Security, %m %t %n"/>
> <Policies>
> <SizeBasedTriggeringPolicy size="10 MB"/>
> </Policies>
> <DefaultRolloverStrategy max="50"/>
> </RollingFile>
>
> <!-- Appender PerformanceStatus -->
> <RollingFile name="PerformanceStatus"
> fileName="/logs-4j2/tpaeventsystem/log4j/STACBatch-PerfMonitoring.log"
>
> filePattern="/logs-4j2/tpaeventsystem/log4j/$${date:yyyy-MM}/STACBatch-PerfMonitoring-%d{MM-dd-yyyy}-%i.log.gz"
> append="true" bufferedIO="true" immediateFlush="true">
> <PatternLayout pattern="(%d)|%m%n"/>
> <Policies>
> <SizeBasedTriggeringPolicy size="10 MB"/>
> </Policies>
> <DefaultRolloverStrategy max="50"/>
> </RollingFile>
>
> <!-- Appender EventSystem -->
> <RollingFile name="EventSystem"
> fileName="/logs-4j2/tpaeventsystem/log4j/STACBatch-EventSystem.log"
>
> filePattern="/logs-4j2/tpaeventsystem/log4j/$${date:yyyy-MM}/STACBatch-EventSystem-%d{MM-dd-yyyy}-%i.log.gz"
> append="true" bufferedIO="true" immediateFlush="true">
> <PatternLayout pattern="(%d), %X{FILEID}, %X{HostName},
> STACBatch-EventSystem, %m %t %n"/>
> <Policies>
> <SizeBasedTriggeringPolicy size="10 MB"/>
> </Policies>
> <DefaultRolloverStrategy max="50"/>
> </RollingFile>
> </Appenders>
>
> <Loggers>
> <Logger name="com.envisagesystems.database" level="ERROR" >
> <AppenderRef ref="R" />
> </Logger>
>
> <Logger name="com.envisagesystems.util.fileprocessor"
> level="DEBUG" >
> <AppenderRef ref="FileProcessor" />
> </Logger>
>
> <Logger
> name="com.envisagesystems.cloud.workflow.eventsystem.ftp" level="DEBUG">
> <AppenderRef ref="FTPProcessor" />
> </Logger>
>
> <Logger name="com.envisagesystems.util.ftp" level="DEBUG">
> <AppenderRef ref="FTPProcessor" />
> </Logger>
>
> <Logger name="com.envisagesystems.util.email" level="DEBUG">
> <AppenderRef ref="Email" />
> </Logger>
>
> <Logger name="com.envisagesystems.cloud.workflow.eventsystem"
> level="DEBUG" >
> <AppenderRef ref="EventSystem" />
> </Logger>
>
> <Logger name="com.envisagesystems.cloud.workflow.eventsystemmq"
> level="DEBUG" >
> <AppenderRef ref="EventSystem" />
> </Logger>
>
> <Logger
> name="com.envisagesystems.cloud.workflow.eventsystem.tiaa" level="DEBUG">
> <AppenderRef ref="EventSystem" />
> </Logger>
>
> <Logger name="com.envisagesystems.security" level="DEBUG">
> <AppenderRef ref="Security" />
> </Logger>
>
> <Logger name="com.envisagesystems.stac" level="INFO">
> <AppenderRef ref="R" />
> </Logger>
>
> <Logger name="STATUS_UPDATES" level="INFO">
> <AppenderRef ref="PerformanceStatus" />
> </Logger>
> <!-- ROOT CONFIG -->
> <Root level="DEBUG">
> <AppenderRef ref="R" />
> </Root>
> </Loggers>
> </Configuration>
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-dev-h...@logging.apache.org
