RE: Re: RE: Re: performance problem in multithread environment
The only improvement that can be added beside what everybody mentioned is to
improve the locking mechanism in Category to use instead a ReadWriteLock
<http://download.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks
/ReadWriteLock.html> . However, it’s hard to justify the change. It should
be a bottleneck only in an application logging like 50+ log statements per
second, which is likely only happening when debugging, which by definition
do not care much about performance.
In your real application, you likely do one of the following:
- Your application do almost nothing else than logging
- Your application do not use async appender and/or buffering
>From the information you provided it seems you execute on a single threaded
machine, something like 100 threads, since we see in your screen shot
“http8080-Processor99”. It’s not the optimum configuration to get the
best throughput. To give you an example, if you use asychnonous socket IO,
and you estimate that 10% of the time your application is waiting for IO
(disk, socket), so mainly CPU bound, then you just need 2 threads, then
obviously you will have lot less synchronization contention.
From: 刘东 [mailto:betterjo...@hotmail.com]
Sent: Friday, December 17, 2010 1:40 AM
To: Log4J Users List
Subject: Re: Re: RE: Re: performance problem in multithread environment
Hi,I use jprofiler to monitor appliaction,the result shows that,at the same
time only on thread is runing,the others are blocked,why?
pls see attachment,thanks.
2010-12-17
_
刘东
_
发件人： 刘东
发送时间： 2010-12-17 10:51:48
收件人： Log4J Users List
抄送：
主题： Re: RE: Re: performance problem in multithread environment
Hi,thank you very much.
I test asyncappender again,the result is as follows:
thread number |time cost |asyn BufferSize
1 |28282 |1024
2 |56359 |1024
4 |107188 |1024
16 |421547 |1024
my log4j.xml is:
http://jakarta.apache.org/log4j/";>
I turned asyn BufferSize to 10240,but the test result is no changing.
why?
2010-12-17
刘东
发件人： Sebastien Tardif
发送时间： 2010-12-17 02:32:38
收件人： 'Log4J Users List'
抄送：
主题： RE: Re: performance problem in multithread environment
File IO is slow, you need to have a asyncappender between your application
and file i/o.
-Original Message-
From: 刘东 [mailto:betterjo...@hotmail.com]
Sent: Thursday, December 16, 2010 10:02 AM
To: Log4J Users List
Subject: Re: Re: performance problem in multithread environment
tanks for you reply.
I write test program to show my appliaction case.
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerTest
{
Logger logger = LoggerFactory.getLogger(LoggerTest.class);
public static void main(String[] args)
{
int i = 0;
while (i < 2)
{
LoggerThread t = new LoggerThread();
t.start();
i++;
}
}
}
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerThread extends Thread
{
Logger logger = LoggerFactory.getLogger(LoggerThread.class);
@Override
public void run()
{
int i = 0;
long start = System.currentTimeMillis();
while (i < 100)
{
logger.debug("i = " + i);
i++;
}
long ecl = System.currentTimeMillis() - start;
System.out.println(this.getName() + "-" + this.getId() + " spend "
+ ecl);
}
}
The test reslut is
thread number | cost
1| 46859
2| 96593
4| 196921
16 | 717703
I use fileappender.
why?
2010-12-16
刘东
发件人： Curt Arnold
发送时间： 2010-12-16 14:15:03
收件人： Log4J Users List
抄送：
主题： Re: performance problem in multithread environment
On Dec 13, 2010, at 10:54 AM, Jacob Kjome wrote:
> You will get better performance by not logging at all, no question.
However, there are tuning possibilities. I see you mention "logger.info()".
In production, I generally only have "warn()" and above for the vast
majority of loggers. In fact, I configure the root logger up with the
"WARN" level and selectively set other loggers to something less than WARN
(if need be).
>
> The other thing you might look at is whether you are concatenating strings
in your logging statements. For instance the following will incur an
unnecessary cost in concatenating strings even when the "DEBUG" level is not
enabled
>
> logger.debug("pr
Re: Re: RE: Re: performance problem in multithread environment
Hi,I use jprofiler to monitor appliaction,the result shows that,at the same
time only on thread is runing,the others are blocked,why?
pls see attachment,thanks.
2010-12-17
刘东
发件人： 刘东
发送时间： 2010-12-17 10:51:48
收件人： Log4J Users List
抄送：
主题： Re: RE: Re: performance problem in multithread environment
Hi,thank you very much.
I test asyncappender again,the result is as follows:
thread number |time cost |asyn BufferSize
1 |28282 |1024
2 |56359 |1024
4 |107188 |1024
16 |421547 |1024
my log4j.xml is:
http://jakarta.apache.org/log4j/";>
I turned asyn BufferSize to 10240,but the test result is no changing.
why?
2010-12-17
刘东
发件人： Sebastien Tardif
发送时间： 2010-12-17 02:32:38
收件人： 'Log4J Users List'
抄送：
主题： RE: Re: performance problem in multithread environment
File IO is slow, you need to have a asyncappender between your application
and file i/o.
-Original Message-
From: 刘东 [mailto:betterjo...@hotmail.com]
Sent: Thursday, December 16, 2010 10:02 AM
To: Log4J Users List
Subject: Re: Re: performance problem in multithread environment
tanks for you reply.
I write test program to show my appliaction case.
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerTest
{
Logger logger = LoggerFactory.getLogger(LoggerTest.class);
public static void main(String[] args)
{
int i = 0;
while (i < 2)
{
LoggerThread t = new LoggerThread();
t.start();
i++;
}
}
}
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerThread extends Thread
{
Logger logger = LoggerFactory.getLogger(LoggerThread.class);
@Override
public void run()
{
int i = 0;
long start = System.currentTimeMillis();
while (i < 100)
{
logger.debug("i = " + i);
i++;
}
long ecl = System.currentTimeMillis() - start;
System.out.println(this.getName() + "-" + this.getId() + " spend "
+ ecl);
}
}
The test reslut is
thread number | cost
1| 46859
2| 96593
4| 196921
16 | 717703
I use fileappender.
why?
2010-12-16
刘东
发件人： Curt Arnold
发送时间： 2010-12-16 14:15:03
收件人： Log4J Users List
抄送：
主题： Re: performance problem in multithread environment
On Dec 13, 2010, at 10:54 AM, Jacob Kjome wrote:
> You will get better performance by not logging at all, no question.
However, there are tuning possibilities. I see you mention "logger.info()".
In production, I generally only have "warn()" and above for the vast
majority of loggers. In fact, I configure the root logger up with the
"WARN" level and selectively set other loggers to something less than WARN
(if need be).
>
> The other thing you might look at is whether you are concatenating strings
in your logging statements. For instance the following will incur an
unnecessary cost in concatenating strings even when the "DEBUG" level is not
enabled
>
> logger.debug("product: " + someProduct + ", price: " + somePrice);
>
> A more efficient way to define this in your code is
>
> if (logger.isDebugEnabled()) {
> logger.debug("product: " + someProduct + ", price: " + somePrice);
> }
>
alternatively use LogMF or LogSF (in the extras companion or in the SVN
HEAD)
LogMF.debug(logger, "product: {0}, price: {1}", price,somePrice);
will have performance generally comparable to using logger.isDebugEnabled
when logging is disabled since any conversion and concatenation is deferred
until after the logging level is checked.
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
Re: RE: Re: performance problem in multithread environment
Hi,thank you very much.
I test asyncappender again,the result is as follows:
thread number |time cost |asyn BufferSize
1 |28282 |1024
2 |56359 |1024
4 |107188 |1024
16 |421547 |1024
my log4j.xml is:
http://jakarta.apache.org/log4j/";>
I turned asyn BufferSize to 10240,but the test result is no changing.
why?
2010-12-17
刘东
发件人： Sebastien Tardif
发送时间： 2010-12-17 02:32:38
收件人： 'Log4J Users List'
抄送：
主题： RE: Re: performance problem in multithread environment
File IO is slow, you need to have a asyncappender between your application
and file i/o.
-Original Message-
From: 刘东 [mailto:betterjo...@hotmail.com]
Sent: Thursday, December 16, 2010 10:02 AM
To: Log4J Users List
Subject: Re: Re: performance problem in multithread environment
tanks for you reply.
I write test program to show my appliaction case.
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerTest
{
Logger logger = LoggerFactory.getLogger(LoggerTest.class);
public static void main(String[] args)
{
int i = 0;
while (i < 2)
{
LoggerThread t = new LoggerThread();
t.start();
i++;
}
}
}
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerThread extends Thread
{
Logger logger = LoggerFactory.getLogger(LoggerThread.class);
@Override
public void run()
{
int i = 0;
long start = System.currentTimeMillis();
while (i < 100)
{
logger.debug("i = " + i);
i++;
}
long ecl = System.currentTimeMillis() - start;
System.out.println(this.getName() + "-" + this.getId() + " spend "
+ ecl);
}
}
The test reslut is
thread number | cost
1| 46859
2| 96593
4| 196921
16 | 717703
I use fileappender.
why?
2010-12-16
刘东
发件人： Curt Arnold
发送时间： 2010-12-16 14:15:03
收件人： Log4J Users List
抄送：
主题： Re: performance problem in multithread environment
On Dec 13, 2010, at 10:54 AM, Jacob Kjome wrote:
> You will get better performance by not logging at all, no question.
However, there are tuning possibilities. I see you mention "logger.info()".
In production, I generally only have "warn()" and above for the vast
majority of loggers. In fact, I configure the root logger up with the
"WARN" level and selectively set other loggers to something less than WARN
(if need be).
>
> The other thing you might look at is whether you are concatenating strings
in your logging statements. For instance the following will incur an
unnecessary cost in concatenating strings even when the "DEBUG" level is not
enabled
>
> logger.debug("product: " + someProduct + ", price: " + somePrice);
>
> A more efficient way to define this in your code is
>
> if (logger.isDebugEnabled()) {
> logger.debug("product: " + someProduct + ", price: " + somePrice);
> }
>
alternatively use LogMF or LogSF (in the extras companion or in the SVN
HEAD)
LogMF.debug(logger, "product: {0}, price: {1}", price,somePrice);
will have performance generally comparable to using logger.isDebugEnabled
when logging is disabled since any conversion and concatenation is deferred
until after the logging level is checked.
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
Re: RE: Re: performance problem in multithread environment
Hi,thank you very much.
I test asyncappender again,the result is as follows:
thread numbertime costasyn BufferSize
1282821024
2563591024
41071881024
164215471024
my log4j.xml is:
http://jakarta.apache.org/log4j/";>
I turned asyn BufferSize to 10240,but the test result is no changing.
why?
2010-12-17
刘东
发件人： Sebastien Tardif
发送时间： 2010-12-17 02:32:38
收件人： 'Log4J Users List'
抄送：
主题： RE: Re: performance problem in multithread environment
File IO is slow, you need to have a asyncappender between your application
and file i/o.
-Original Message-
From: 刘东 [mailto:betterjo...@hotmail.com]
Sent: Thursday, December 16, 2010 10:02 AM
To: Log4J Users List
Subject: Re: Re: performance problem in multithread environment
tanks for you reply.
I write test program to show my appliaction case.
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerTest
{
Logger logger = LoggerFactory.getLogger(LoggerTest.class);
public static void main(String[] args)
{
int i = 0;
while (i < 2)
{
LoggerThread t = new LoggerThread();
t.start();
i++;
}
}
}
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerThread extends Thread
{
Logger logger = LoggerFactory.getLogger(LoggerThread.class);
@Override
public void run()
{
int i = 0;
long start = System.currentTimeMillis();
while (i < 100)
{
logger.debug("i = " + i);
i++;
}
long ecl = System.currentTimeMillis() - start;
System.out.println(this.getName() + "-" + this.getId() + " spend "
+ ecl);
}
}
The test reslut is
thread number | cost
1| 46859
2| 96593
4| 196921
16 | 717703
I use fileappender.
why?
2010-12-16
刘东
发件人： Curt Arnold
发送时间： 2010-12-16 14:15:03
收件人： Log4J Users List
抄送：
主题： Re: performance problem in multithread environment
On Dec 13, 2010, at 10:54 AM, Jacob Kjome wrote:
> You will get better performance by not logging at all, no question.
However, there are tuning possibilities. I see you mention "logger.info()".
In production, I generally only have "warn()" and above for the vast
majority of loggers. In fact, I configure the root logger up with the
"WARN" level and selectively set other loggers to something less than WARN
(if need be).
>
> The other thing you might look at is whether you are concatenating strings
in your logging statements. For instance the following will incur an
unnecessary cost in concatenating strings even when the "DEBUG" level is not
enabled
>
> logger.debug("product: " + someProduct + ", price: " + somePrice);
>
> A more efficient way to define this in your code is
>
> if (logger.isDebugEnabled()) {
> logger.debug("product: " + someProduct + ", price: " + somePrice);
> }
>
alternatively use LogMF or LogSF (in the extras companion or in the SVN
HEAD)
LogMF.debug(logger, "product: {0}, price: {1}", price,somePrice);
will have performance generally comparable to using logger.isDebugEnabled
when logging is disabled since any conversion and concatenation is deferred
until after the logging level is checked.
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
RE: Re: performance problem in multithread environment
File IO is slow, you need to have a asyncappender between your application
and file i/o.
-Original Message-
From: 刘东 [mailto:betterjo...@hotmail.com]
Sent: Thursday, December 16, 2010 10:02 AM
To: Log4J Users List
Subject: Re: Re: performance problem in multithread environment
tanks for you reply.
I write test program to show my appliaction case.
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerTest
{
Logger logger = LoggerFactory.getLogger(LoggerTest.class);
public static void main(String[] args)
{
int i = 0;
while (i < 2)
{
LoggerThread t = new LoggerThread();
t.start();
i++;
}
}
}
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerThread extends Thread
{
Logger logger = LoggerFactory.getLogger(LoggerThread.class);
@Override
public void run()
{
int i = 0;
long start = System.currentTimeMillis();
while (i < 100)
{
logger.debug("i = " + i);
i++;
}
long ecl = System.currentTimeMillis() - start;
System.out.println(this.getName() + "-" + this.getId() + " spend "
+ ecl);
}
}
The test reslut is
thread number | cost
1| 46859
2| 96593
4| 196921
16 | 717703
I use fileappender.
why?
2010-12-16
刘东
发件人： Curt Arnold
发送时间： 2010-12-16 14:15:03
收件人： Log4J Users List
抄送：
主题： Re: performance problem in multithread environment
On Dec 13, 2010, at 10:54 AM, Jacob Kjome wrote:
> You will get better performance by not logging at all, no question.
However, there are tuning possibilities. I see you mention "logger.info()".
In production, I generally only have "warn()" and above for the vast
majority of loggers. In fact, I configure the root logger up with the
"WARN" level and selectively set other loggers to something less than WARN
(if need be).
>
> The other thing you might look at is whether you are concatenating strings
in your logging statements. For instance the following will incur an
unnecessary cost in concatenating strings even when the "DEBUG" level is not
enabled
>
> logger.debug("product: " + someProduct + ", price: " + somePrice);
>
> A more efficient way to define this in your code is
>
> if (logger.isDebugEnabled()) {
> logger.debug("product: " + someProduct + ", price: " + somePrice);
> }
>
alternatively use LogMF or LogSF (in the extras companion or in the SVN
HEAD)
LogMF.debug(logger, "product: {0}, price: {1}", price,somePrice);
will have performance generally comparable to using logger.isDebugEnabled
when logging is disabled since any conversion and concatenation is deferred
until after the logging level is checked.
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
Re: Re: performance problem in multithread environment
tanks for you reply.
I write test program to show my appliaction case.
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerTest
{
Logger logger = LoggerFactory.getLogger(LoggerTest.class);
public static void main(String[] args)
{
int i = 0;
while (i < 2)
{
LoggerThread t = new LoggerThread();
t.start();
i++;
}
}
}
package com.iss.cnooc.test.ebank;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoggerThread extends Thread
{
Logger logger = LoggerFactory.getLogger(LoggerThread.class);
@Override
public void run()
{
int i = 0;
long start = System.currentTimeMillis();
while (i < 100)
{
logger.debug("i = " + i);
i++;
}
long ecl = System.currentTimeMillis() - start;
System.out.println(this.getName() + "-" + this.getId() + " spend "
+ ecl);
}
}
The test reslut is
thread number | cost
1| 46859
2| 96593
4| 196921
16 | 717703
I use fileappender.
why?
2010-12-16
刘东
发件人： Curt Arnold
发送时间： 2010-12-16 14:15:03
收件人： Log4J Users List
抄送：
主题： Re: performance problem in multithread environment
On Dec 13, 2010, at 10:54 AM, Jacob Kjome wrote:
> You will get better performance by not logging at all, no question. However,
> there are tuning possibilities. I see you mention "logger.info()". In
> production, I generally only have "warn()" and above for the vast majority of
> loggers. In fact, I configure the root logger up with the "WARN" level and
> selectively set other loggers to something less than WARN (if need be).
>
> The other thing you might look at is whether you are concatenating strings in
> your logging statements. For instance the following will incur an
> unnecessary cost in concatenating strings even when the "DEBUG" level is not
> enabled
>
> logger.debug("product: " + someProduct + ", price: " + somePrice);
>
> A more efficient way to define this in your code is
>
> if (logger.isDebugEnabled()) {
> logger.debug("product: " + someProduct + ", price: " + somePrice);
> }
>
alternatively use LogMF or LogSF (in the extras companion or in the SVN HEAD)
LogMF.debug(logger, "product: {0}, price: {1}", price,somePrice);
will have performance generally comparable to using logger.isDebugEnabled when
logging is disabled since any conversion and concatenation is deferred until
after the logging level is checked.
-
To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
For additional commands, e-mail: log4j-user-h...@logging.apache.org
Re: Re: Re: performance problem in multithread environment
I need your help!!! Please help me ,thanks. 2010-12-13 刘东 发件人： 刘东 发送时间： 2010-12-11 01:52:03 收件人： Log4J Users List 抄送： 主题： Re: Re: performance problem in multithread environment hi, through jprofiler I found that many thread are blocked at logger.info(message).I saw that each logger.info() call will cost 2,200 us,the application are running at a machine which has one phyical cpu(64 virtual cpu),1.5Ghz,32G memory,solaris 10. If I comment all logger message(means no logging message),the application performance will be faster more than ten times. If I crate new logger instance for each log message,the application performance will be faster more than four times. Could tell me the difference between category and logger. Api shows that logger extends category,logger.info() just is category.info()。 The code show that synchronization is by category,but code category = this,shows that at this time category just is logger. BTW: In my application,when process one order,it will log about 700 messages. With log,application will process three orders one second,But without log,application will process 40 orders one second. Is there any mistakes by using log4j?need more config? thanks. 2010-12-11 刘东 发件人： Sebastien Tardif 发送时间： 2010-12-11 01:10:40 收件人： log4j-user 抄送： 主题： Re: performance problem in multithread environment The code show that synchronization is by category. If appendLoopOnAppenders is faster, like if the underline code use asynchronous appender, that should not be a problem. So you should investigate what the thread having the lock is doing. - To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org For additional commands, e-mail: log4j-user-h...@logging.apache.org
Re: Re: performance problem in multithread environment
hi, through jprofiler I found that many thread are blocked at logger.info(message).I saw that each logger.info() call will cost 2,200 us,the application are running at a machine which has one phyical cpu(64 virtual cpu),1.5Ghz,32G memory,solaris 10. If I comment all logger message(means no logging message),the application performance will be faster more than ten times. If I crate new logger instance for each log message,the application performance will be faster more than four times. Could tell me the difference between category and logger. Api shows that logger extends category,logger.info() just is category.info()。 The code show that synchronization is by category,but code category = this,shows that at this time category just is logger. BTW: In my application,when process one order,it will log about 700 messages. With log,application will process three orders one second,But without log,application will process 40 orders one second. Is there any mistakes by using log4j?need more config? thanks. 2010-12-11 刘东 发件人： Sebastien Tardif 发送时间： 2010-12-11 01:10:40 收件人： log4j-user 抄送： 主题： Re: performance problem in multithread environment The code show that synchronization is by category. If appendLoopOnAppenders is faster, like if the underline code use asynchronous appender, that should not be a problem. So you should investigate what the thread having the lock is doing. - To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org For additional commands, e-mail: log4j-user-h...@logging.apache.org
