RE: Performance between Log4j and custom in-house logger
Comments below. At 18:05 04.09.2002 -0700, sk k wrote: Hi, I did the test with AsyncAppender using an xml config file(set the buff size to 1000 even though default size of 128 should be sufficient. My logger uses a size of like 60 ) and it seems the performance isn't any better. Infact the performance dropped a little and it seems to vaildate the AsyncAppender performance numbers shown in the javadocs. Pls let me know if anything is wrong or needs to be tuned. config file: = log4j:configuration xmlns:log4j=http://jakarta.apache.org/log4j/; debug=true appender name=dest1 class=org.apache.log4j.RollingFileAppender param name=File value=c:/skk/cps/logger/test/test1.log/ param name=MaxFileSize value=4MB/ param name=MaxBackupIndex value=2/ layout class=org.apache.log4j.TTCCLayout/ /appender appender name=dest2 class=org.apache.log4j.RollingFileAppender param name=File value=c:/skk/cps/logger/test/test2.log/ param name=MaxFileSize value=4MB/ param name=MaxBackupIndex value=2/ param name=BufferSize value=1000/ layout class=org.apache.log4j.TTCCLayout/ /appender category name=ut.log.Log4jPerfTest additivity=false priority value=debug/ appender-ref ref=dest2/ /category root priority value=DEBUG/ appender-ref ref=dest1/ /root /log4j:configuration I am measuring the time when the method starts and and when it returns and it seems that the background task time is not accounted for using my logger and it is accounted for while using log4j. It seems that application shouldn't wait till the log message has been written as it would degrade peformance . The above configuration XML script does not use nor mention AsyncAppender. Try the following ?xml version=1.0 encoding=UTF-8 ? !DOCTYPE log4j:configuration SYSTEM log4j.dtd log4j:configuration debug=true xmlns:log4j=http://jakarta.apache.org/log4j/; appender name=ASYNC class=org.apache.log4j.AsyncAppender param name=BufferSize value=2/ appender-ref ref=dest1 / /appender appender name=dest1 class=org.apache.log4j.FileAppender param name=File value=c:/skk/cps/logger/test/test1.log/ layout class=org.apache.log4j.TTCCLayout/ /appender root priority value=DEBUG/ appender-ref ref=ASYNC/ /root /log4j:configuration You may want to post the Java code that takes the measurements. -- Ceki TCP implementations will follow a general principle of robustness: be conservative in what you do, be liberal in what you accept from others. -- Jon Postel, RFC 793 -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Performance between Log4j and custom in-house logger
SPECIFICALLY: As Ceki mentioned you currently have logging statements being output to both files... this will hurt performance in this test. Simply setting additivity to false for the two loggers should work wonders for this simple test. Try this for your config: ### # Define root logger/category ### log4j.rootCategory=DEBUG,dest1 ### # Define non-root loggers/categories ### log4j.category.ut.log.Log4jPerfTest=DEBUG,dest2 log4j.additivity.ut.log.Log4jPerfTest=false ### # Define appenders ### log4j.appender.dest1=org.apache.log4j.RollingFileAppender log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log log4j.appender.dest1.MaxFileSize=4MB log4j.appender.dest1.MaxBackupIndex=60 log4j.appender.dest2=org.apache.log4j.RollingFileAppender log4j.appender.dest2.layout=org.apache.log4j.TTCCLayout log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log log4j.appender.dest2.MaxFileSize=4MB log4j.appender.dest2.MaxBackupIndex=60 GENERALLY: The way to mimic your setup in log4j would be to use AsyncAppender as your attached appender. This will create a background thread to handle appending the actual logging events. AsyncAppender can only be created from config files using the XML config. Or you can do it programatically for your tests. See http://jakarta.apache.org/log4j/docs/api/org/apache/log4j/AsyncAppender.html for its description. Also, be aware that the async logging in log4j does still strive to maintain delivery of generated LoggingEvents to its attached appenders in the chronological order in which those events were created. To this end, it uses a bounded buffer as the queue for these events. This slows performance in simple, non-intensive apps; however it does ensure delivery of the logging events in the correct order. For a discussion of this see the section on AsyncAppender at http://jakarta.apache.org/log4j/docs/api/org/apache/log4j/performance/Loggin g.html HTH |-Original Message- |From: sk k [mailto:[EMAIL PROTECTED]] |Sent: Tuesday, September 03, 2002 9:37 PM |To: Log4J Users List |Subject: Re: Performance between Log4j and custom in-house logger | | | |Hi, | |Any thoughts on how I can improve the performance |numbers for logging. | | |Thanks. | | |--- Ceki Gülcü [EMAIL PROTECTED] wrote: | | What is the difference between Strings, taking the | values 2,4 and 8, | and Request, taking the values 100, 1000 and 2000? | | Are you sure you are waiting for the background | thread to finish? | | Your log4j configuration file suggests that your | tests use two | appenders dest1 and dest2. You are aware that | appenders are additive, | right? Does logging output go both | c:/skk/cps/logger/test/test1.log | and c:/skk/cps/logger/test/test.log? Given that | writing to a file is | the most time consuming task, is it fait to say that | log4j is doing | twice the work roughly at the same cost? | | At 07:29 03.09.2002 -0700, you wrote: | Hi, | | We are planning to move from a custom in house | logging | framework (MyLogger) to using Log4j and have been | doing some performance comparison between log4j and | MyLogger. | | The numbers seem to be favoring mylogger over | log4j. | | | Sample Numbers | == | Note: Timing includes logj startup time, mylogger | startup time and junitperf startup time. | | 1) Logging 2 strings, each of size: 128bytes | | Request MyLogger(sec) Log4j(sec) | == === = | 100 1.221.29 | 10001.671.71 | 20002.122.01 | | | | 2) Logging 4 strings, each of size: 128bytes | | Request MyLogger(sec) Log4j(sec) | == === = | 100 1.211.25 | 10001.653 1.82 | 20002.242.56 | | | 3) Logging 8 strings, each of size: 128bytes | | Request MyLogger(sec) Log4j(sec) | == === = | 100 1.351.35 | 400 1.561.71 | 900 1.8 2.35 | | 4) Logging 80 strings, each of size: 128bytes | | Request MyLogger(sec) Log4j(sec) | == === = | 100 1.251 1.7 | 10001.816.1 | 20002.1511.2 | | | Sample code
RE: Performance between Log4j and custom in-house logger
I made the changes mentioned and still the performance has increased by only 10-50% but still my custom logger results are ahead of log4j. Also measured the cpu and memory and it seems my custom logger is consuming lesser values since formatting and writing to file is done in the background. I did some more testing on writting to multiple appenders and my custom logger is like 10-300% faster than log4j. Are there any plans to re-write log4j (any next generation version). It seems that log4j needs an architectural change rather than anything else. Any ideas? --- Ebersole, Steven [EMAIL PROTECTED] wrote: SPECIFICALLY: As Ceki mentioned you currently have logging statements being output to both files... this will hurt performance in this test. Simply setting additivity to false for the two loggers should work wonders for this simple test. Try this for your config: ### # Define root logger/category ### log4j.rootCategory=DEBUG,dest1 ### # Define non-root loggers/categories ### log4j.category.ut.log.Log4jPerfTest=DEBUG,dest2 log4j.additivity.ut.log.Log4jPerfTest=false ### # Define appenders ### log4j.appender.dest1=org.apache.log4j.RollingFileAppender log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log log4j.appender.dest1.MaxFileSize=4MB log4j.appender.dest1.MaxBackupIndex=60 log4j.appender.dest2=org.apache.log4j.RollingFileAppender log4j.appender.dest2.layout=org.apache.log4j.TTCCLayout log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log log4j.appender.dest2.MaxFileSize=4MB log4j.appender.dest2.MaxBackupIndex=60 GENERALLY: The way to mimic your setup in log4j would be to use AsyncAppender as your attached appender. This will create a background thread to handle appending the actual logging events. AsyncAppender can only be created from config files using the XML config. Or you can do it programatically for your tests. See http://jakarta.apache.org/log4j/docs/api/org/apache/log4j/AsyncAppender.html for its description. Also, be aware that the async logging in log4j does still strive to maintain delivery of generated LoggingEvents to its attached appenders in the chronological order in which those events were created. To this end, it uses a bounded buffer as the queue for these events. This slows performance in simple, non-intensive apps; however it does ensure delivery of the logging events in the correct order. For a discussion of this see the section on AsyncAppender at http://jakarta.apache.org/log4j/docs/api/org/apache/log4j/performance/Loggin g.html HTH |-Original Message- |From: sk k [mailto:[EMAIL PROTECTED]] |Sent: Tuesday, September 03, 2002 9:37 PM |To: Log4J Users List |Subject: Re: Performance between Log4j and custom in-house logger | | | |Hi, | |Any thoughts on how I can improve the performance |numbers for logging. | | |Thanks. | | |--- Ceki Gülcü [EMAIL PROTECTED] wrote: | | What is the difference between Strings, taking the | values 2,4 and 8, | and Request, taking the values 100, 1000 and 2000? | | Are you sure you are waiting for the background | thread to finish? | | Your log4j configuration file suggests that your | tests use two | appenders dest1 and dest2. You are aware that | appenders are additive, | right? Does logging output go both | c:/skk/cps/logger/test/test1.log | and c:/skk/cps/logger/test/test.log? Given that | writing to a file is | the most time consuming task, is it fait to say that | log4j is doing | twice the work roughly at the same cost? | | At 07:29 03.09.2002 -0700, you wrote: | Hi, | | We are planning to move from a custom in house | logging | framework (MyLogger) to using Log4j and have been | doing some performance comparison between log4j and | MyLogger. | | The numbers seem to be favoring mylogger over | log4j. | | | Sample Numbers | == | Note: Timing includes logj startup time, mylogger | startup time and junitperf startup time. | | 1) Logging 2 strings, each of size: 128bytes | | Request MyLogger(sec) Log4j(sec) | == === = | 100 1.221.29
RE: Performance between Log4j and custom in-house logger
At 11:13 04.09.2002 -0700, you wrote: I made the changes mentioned and still the performance has increased by only 10-50% but still my custom logger results are ahead of log4j. Do the test with a AsyncAppender BufferSize of 20'000. You'll be logging at no time at all. (This statement is very misleading, read below.) Also measured the cpu and memory and it seems my custom logger is consuming lesser values since formatting and writing to file is done in the background. As others explained previously, back grounding tasks does not necessarily result in better performance. To give you an example, when Alice asks her secretary, Bob, to type a letter, Alice's job is done after she delegates to Bob. However, that does not mean that the letter has been typed. Are you sure you are not measuring the time it takes Alice to delegate as opposed to Bob actually typing the letter? I did some more testing on writting to multiple appenders and my custom logger is like 10-300% faster than log4j. Where is the code? Are there any plans to re-write log4j (any next generation version). It seems that log4j needs an architectural change rather than anything else. I wouldn't jump the gun if I were you. :-) Any ideas? --- Ebersole, Steven [EMAIL PROTECTED] wrote: SPECIFICALLY: As Ceki mentioned you currently have logging statements being output to both files... this will hurt performance in this test. Simply setting additivity to false for the two loggers should work wonders for this simple test. Try this for your config: ### # Define root logger/category ### log4j.rootCategory=DEBUG,dest1 ### # Define non-root loggers/categories ### log4j.category.ut.log.Log4jPerfTest=DEBUG,dest2 log4j.additivity.ut.log.Log4jPerfTest=false ### # Define appenders ### log4j.appender.dest1=org.apache.log4j.RollingFileAppender log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log log4j.appender.dest1.MaxFileSize=4MB log4j.appender.dest1.MaxBackupIndex=60 log4j.appender.dest2=org.apache.log4j.RollingFileAppender log4j.appender.dest2.layout=org.apache.log4j.TTCCLayout log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log log4j.appender.dest2.MaxFileSize=4MB log4j.appender.dest2.MaxBackupIndex=60 GENERALLY: The way to mimic your setup in log4j would be to use AsyncAppender as your attached appender. This will create a background thread to handle appending the actual logging events. AsyncAppender can only be created from config files using the XML config. Or you can do it programatically for your tests. See http://jakarta.apache.org/log4j/docs/api/org/apache/log4j/AsyncAppender.html for its description. Also, be aware that the async logging in log4j does still strive to maintain delivery of generated LoggingEvents to its attached appenders in the chronological order in which those events were created. To this end, it uses a bounded buffer as the queue for these events. This slows performance in simple, non-intensive apps; however it does ensure delivery of the logging events in the correct order. For a discussion of this see the section on AsyncAppender at http://jakarta.apache.org/log4j/docs/api/org/apache/log4j/performance/Loggin g.html HTH |-Original Message- |From: sk k [mailto:[EMAIL PROTECTED]] |Sent: Tuesday, September 03, 2002 9:37 PM |To: Log4J Users List |Subject: Re: Performance between Log4j and custom in-house logger | | | |Hi, | |Any thoughts on how I can improve the performance |numbers for logging. | | |Thanks. | | |--- Ceki Gülcü [EMAIL PROTECTED] wrote: | | What is the difference between Strings, taking the | values 2,4 and 8, | and Request, taking the values 100, 1000 and 2000? | | Are you sure you are waiting for the background | thread to finish? | | Your log4j configuration file suggests that your | tests use two | appenders dest1 and dest2. You are aware that | appenders are additive, | right? Does logging output go both | c:/skk/cps/logger/test/test1.log | and c:/skk/cps/logger/test/test.log? Given that | writing to a file is | the most time consuming task, is it fait to say that | log4j is doing
RE: Performance between Log4j and custom in-house logger
Are there any plans to re-write log4j (any next generation version). It seems that log4j needs an architectural change rather than anything else. I wouldn't jump the gun if I were you. :-) Hi, Though you have to love it when someone uses a mature, popular system for a few days, then promptly claims his/her system is better and (instead of just using his/her) demands a rewrite of the other system ;) I find it amusing ;) Yoav Shapira Millennium ChemInformatics This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Performance between Log4j and custom in-house logger
It's probably counter-productive, but I just have to chime in here. I too have been highly entertained by this exchange. My compliments to those who manage to respond in a level-headed fashion! -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED]] Sent: Wednesday, September 04, 2002 2:45 PM To: Log4J Users List Subject: RE: Performance between Log4j and custom in-house logger Are there any plans to re-write log4j (any next generation version). It seems that log4j needs an architectural change rather than anything else. I wouldn't jump the gun if I were you. :-) Hi, Though you have to love it when someone uses a mature, popular system for a few days, then promptly claims his/her system is better and (instead of just using his/her) demands a rewrite of the other system ;) I find it amusing ;) Yoav Shapira Millennium ChemInformatics -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Performance between Log4j and custom in-house logger
### # Define appenders ### log4j.appender.dest1=org.apache.log4j.RollingFileAppender log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log log4j.appender.dest1.MaxFileSize=4MB log4j.appender.dest1.MaxBackupIndex=60 log4j.appender.dest2=org.apache.log4j.RollingFileAppender log4j.appender.dest2.layout=org.apache.log4j.TTCCLayout log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log log4j.appender.dest2.MaxFileSize=4MB log4j.appender.dest2.MaxBackupIndex=60 GENERALLY: The way to mimic your setup in log4j would be to use AsyncAppender as your attached appender. This will create a background thread to handle appending the actual logging events. AsyncAppender can only be created from config files using the XML config. Or you can do it programatically for your tests. See http://jakarta.apache.org/log4j/docs/api/org/apache/log4j/AsyncAppender.html for its description. Also, be aware that the async logging in log4j does still strive to maintain delivery of generated LoggingEvents to its attached appenders in the chronological order in which those events were created. To this end, it uses a bounded buffer as the queue for these events. This slows performance in simple, non-intensive apps; however it does ensure delivery of the logging events in the correct order. For a discussion of this see the section on AsyncAppender at http://jakarta.apache.org/log4j/docs/api/org/apache/log4j/performance/Loggin g.html HTH |-Original Message- |From: sk k [mailto:[EMAIL PROTECTED]] |Sent: Tuesday, September 03, 2002 9:37 PM |To: Log4J Users List |Subject: Re: Performance between Log4j and custom in-house logger | | | |Hi, | |Any thoughts on how I can improve the performance |numbers for logging. | | |Thanks. | | |--- Ceki Gülcü [EMAIL PROTECTED] wrote: | | What is the difference between Strings, taking the | values 2,4 and 8, | and Request, taking the values 100, 1000 and 2000? | | Are you sure you are waiting for the background | thread to finish? | | Your log4j configuration file suggests that your | tests use two | appenders dest1 and dest2. You are aware that | appenders are additive, | right? Does logging output go both | c:/skk/cps/logger/test/test1.log | and c:/skk/cps/logger/test/test.log? Given that | writing to a file is | the most time consuming task, is it fait to === message truncated === __ Do You Yahoo!? Yahoo! Finance - Get real-time stock quotes http://finance.yahoo.com -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Performance between Log4j and custom in-house logger
Hi,
There are always going to be difference between any two systems designed
to do the same thing. Your system was obviously designed for your
needs, and probably not much more. Log4j is designed to be generic,
useful to a very broad range of people / applications, easily
extensible, and yet fast.
At my organization, we carefully compared five or six in-house custom
logging systems and a couple of 3rd party ones. We included very
detailed performance comparisons and profilings. We found log4j to be
as fast or faster than nearly all the other implementation, while
providing the best range of features. As we need a lot of its features
(JDBC logging, SMTP logging, runtime configurability, JMS logging, daily
rolling, appender thresholds, object renderers, etc), we picked log4j,
and have been very very happy since.
So it's a question of what you need from your logging system. If your
system did everything you need, you probably wouldn't even be
considering log4j, right? ;) Maybe it';; easier to add just the
features you need to your own system, instead of migrating to log4j
completely?
The point is, if you want to help tune log4j you're welcome to. It's
open source and contributors are always welcome, right Ceki? ;)
However, having used log4j in numerous apps from very small to very
large, we've never had any complaints about its performance. There was
always a lot of tuning to be done on our own apps before logging
performance showed up as a blip on the profiler screen...
Yoav Shapira
Millennium ChemInformatics
-Original Message-
From: sk k [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 03, 2002 10:30 AM
To: [EMAIL PROTECTED]
Subject: Performance between Log4j and custom in-house logger
Hi,
We are planning to move from a custom in house logging
framework (MyLogger) to using Log4j and have been
doing some performance comparison between log4j and
MyLogger.
The numbers seem to be favoring mylogger over log4j.
Sample Numbers
==
Note: Timing includes logj startup time, mylogger
startup time and junitperf startup time.
1) Logging 2 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
1001.221.29
1000 1.671.71
2000 2.122.01
2) Logging 4 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
1001.211.25
1000 1.653 1.82
2000 2.242.56
3) Logging 8 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
1001.351.35
4001.561.71
9001.8 2.35
4) Logging 80 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
1001.251 1.7
1000 1.816.1
2000 2.1511.2
Sample code and configuration
===
log4j.properties
==
log4j.rootCategory=DEBUG,dest1
log4j.appender.dest1=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout
log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log
log4j.appender.dest1.MaxFileSize=4MB
log4j.appender.dest1.MaxBackupIndex=60
log4j.category.ut.log.Log4jPerfTest=DEBUG,dest2
log4j.additivity.Log4jPerfTest=false
log4j.appender.dest2=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.TTCCLayout
log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log
log4j.appender.dest2.MaxFileSize=4MB
log4j.appender.dest2.MaxBackupIndex=60
Test Logger code
===
Note: logMsgs is an array with strings of size 128
bytes.
public void test_Log4j() throws Exception {
int size = logMsgs.size();
for( int i=0; isize; i++ ) {
log4jLogger.info( (String)logMsgs.get(i)
);
}
}
Some of my questions
1) Is there anything that I can do to increase the
performance of log4j on my box( Pentium-3, 800Mhz,
512MB RAM)
2) I see some architectural difference between the way
mylogger and log4j is implemented namely
My logger does formattingwriting to log file in a
background thread whereas it seems that log4j doesn't
seem do the same. Is this true.
My logger uses the concept of observer and
observable.
Thanks.
__
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com
--
To unsubscribe, e-mail: mailto:log4j-user-
[EMAIL PROTECTED]
For additional commands, e-mail: mailto:log4j-user-
[EMAIL PROTECTED]
This e-mail, including any attachments, is a confidential business communication, and
may contain information that is confidential, proprietary and/or privileged. This
e-mail is intended only for the individual(s) to whom it is addressed, and may not be
saved, copied, printed, disclosed or used by anyone else. If you are not the(an)
intended recipient, please immediately delete this e-mail from your computer system
and notify the sender. Thank you.
--
To unsubscribe, e-mail:
RE: Performance between Log4j and custom in-house logger
Perhaps one telling comment is My logger does formattingwriting to log
file in a background thread. My question is, do your measurements
account for all background processing having been completed? If not,
this may explain why log4j performance appears to degrade in comparison
with mylogger as logging load increases -- assuming that I am reading
your results correctly.
Jim
-Original Message-
From: Shapira, Yoav [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 03, 2002 9:40 AM
To: Log4J Users List
Subject: RE: Performance between Log4j and custom in-house logger
Hi,
There are always going to be difference between any two systems designed
to do the same thing. Your system was obviously designed for your
needs, and probably not much more. Log4j is designed to be generic,
useful to a very broad range of people / applications, easily
extensible, and yet fast.
At my organization, we carefully compared five or six in-house custom
logging systems and a couple of 3rd party ones. We included very
detailed performance comparisons and profilings. We found log4j to be
as fast or faster than nearly all the other implementation, while
providing the best range of features. As we need a lot of its features
(JDBC logging, SMTP logging, runtime configurability, JMS logging, daily
rolling, appender thresholds, object renderers, etc), we picked log4j,
and have been very very happy since.
So it's a question of what you need from your logging system. If your
system did everything you need, you probably wouldn't even be
considering log4j, right? ;) Maybe it';; easier to add just the
features you need to your own system, instead of migrating to log4j
completely?
The point is, if you want to help tune log4j you're welcome to. It's
open source and contributors are always welcome, right Ceki? ;)
However, having used log4j in numerous apps from very small to very
large, we've never had any complaints about its performance. There was
always a lot of tuning to be done on our own apps before logging
performance showed up as a blip on the profiler screen...
Yoav Shapira
Millennium ChemInformatics
-Original Message-
From: sk k [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 03, 2002 10:30 AM
To: [EMAIL PROTECTED]
Subject: Performance between Log4j and custom in-house logger
Hi,
We are planning to move from a custom in house logging
framework (MyLogger) to using Log4j and have been
doing some performance comparison between log4j and
MyLogger.
The numbers seem to be favoring mylogger over log4j.
Sample Numbers
==
Note: Timing includes logj startup time, mylogger
startup time and junitperf startup time.
1) Logging 2 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
1001.221.29
1000 1.671.71
2000 2.122.01
2) Logging 4 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
1001.211.25
1000 1.653 1.82
2000 2.242.56
3) Logging 8 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
1001.351.35
4001.561.71
9001.8 2.35
4) Logging 80 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
1001.251 1.7
1000 1.816.1
2000 2.1511.2
Sample code and configuration
===
log4j.properties
==
log4j.rootCategory=DEBUG,dest1
log4j.appender.dest1=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout
log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log
log4j.appender.dest1.MaxFileSize=4MB
log4j.appender.dest1.MaxBackupIndex=60
log4j.category.ut.log.Log4jPerfTest=DEBUG,dest2
log4j.additivity.Log4jPerfTest=false
log4j.appender.dest2=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.TTCCLayout
log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log
log4j.appender.dest2.MaxFileSize=4MB
log4j.appender.dest2.MaxBackupIndex=60
Test Logger code
===
Note: logMsgs is an array with strings of size 128
bytes.
public void test_Log4j() throws Exception {
int size = logMsgs.size();
for( int i=0; isize; i++ ) {
log4jLogger.info( (String)logMsgs.get(i)
);
}
}
Some of my questions
1) Is there anything that I can do to increase the
performance of log4j on my box( Pentium-3, 800Mhz,
512MB RAM)
2) I see some architectural difference between the way
mylogger and log4j is implemented namely
My logger does formattingwriting to log file in a
background thread whereas it seems that log4j doesn't
seem do the same. Is this true.
My logger uses the concept of observer and
observable.
Thanks.
__
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com
--
To unsubscribe, e-mail: mailto:log4j-user-
[EMAIL
Re: Performance between Log4j and custom in-house logger
What is the difference between Strings, taking the values 2,4 and 8,
and Request, taking the values 100, 1000 and 2000?
Are you sure you are waiting for the background thread to finish?
Your log4j configuration file suggests that your tests use two
appenders dest1 and dest2. You are aware that appenders are additive,
right? Does logging output go both c:/skk/cps/logger/test/test1.log
and c:/skk/cps/logger/test/test.log? Given that writing to a file is
the most time consuming task, is it fait to say that log4j is doing
twice the work roughly at the same cost?
At 07:29 03.09.2002 -0700, you wrote:
Hi,
We are planning to move from a custom in house logging
framework (MyLogger) to using Log4j and have been
doing some performance comparison between log4j and
MyLogger.
The numbers seem to be favoring mylogger over log4j.
Sample Numbers
==
Note: Timing includes logj startup time, mylogger
startup time and junitperf startup time.
1) Logging 2 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.221.29
10001.671.71
20002.122.01
2) Logging 4 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.211.25
10001.653 1.82
20002.242.56
3) Logging 8 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.351.35
400 1.561.71
900 1.8 2.35
4) Logging 80 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.251 1.7
10001.816.1
20002.1511.2
Sample code and configuration
===
log4j.properties
==
log4j.rootCategory=DEBUG,dest1
log4j.appender.dest1=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout
log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log
log4j.appender.dest1.MaxFileSize=4MB
log4j.appender.dest1.MaxBackupIndex=60
log4j.category.ut.log.Log4jPerfTest=DEBUG,dest2
log4j.additivity.Log4jPerfTest=false
log4j.appender.dest2=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.TTCCLayout
log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log
log4j.appender.dest2.MaxFileSize=4MB
log4j.appender.dest2.MaxBackupIndex=60
Test Logger code
===
Note: logMsgs is an array with strings of size 128
bytes.
public void test_Log4j() throws Exception {
int size = logMsgs.size();
for( int i=0; isize; i++ ) {
log4jLogger.info( (String)logMsgs.get(i)
);
}
}
Some of my questions
1) Is there anything that I can do to increase the
performance of log4j on my box( Pentium-3, 800Mhz,
512MB RAM)
2) I see some architectural difference between the way
mylogger and log4j is implemented namely
My logger does formattingwriting to log file in a
background thread whereas it seems that log4j doesn't
seem do the same. Is this true.
My logger uses the concept of observer and
observable.
Thanks.
__
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com
--
To unsubscribe, e-mail: mailto:[EMAIL PROTECTED]
For additional commands, e-mail: mailto:[EMAIL PROTECTED]
--
Ceki
TCP implementations will follow a general principle of robustness: be
conservative in what you do, be liberal in what you accept from
others. -- Jon Postel, RFC 793
--
To unsubscribe, e-mail: mailto:[EMAIL PROTECTED]
For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Re: Performance between Log4j and custom in-house logger
Hi,
We want to move to log4j and that is the reason why
we are doing the POC.
I really like log4j(features and functionality ) and
wanted to push the usage of log4j in the entire
organization.
I need some input from guru's to see how I can improve
the performance numbers since the decision is not made
by me. I need to put together the pro's, con's and
performance numbers and present it to somebody who
will go over the facts and make a decision.
Is there any changes in code or config settings that
will enable me to get better performance numbers. The
functionality and features offered by log4j are
awesome.
Let me give some explaination of the perf numbers with
2 strings( array size = 2).
1) The contents of the string is as shown below
My Sample String:
String msg[] = {2-Aug-2002:08:21:30.568 xx
136.503debuguseThread: idle count is 19,
12-Aug-2002:08:21:30.568 xxx 136.483
debugstats: intf FCOPingInterface };
Pls note that the strings are some arbitrary values.
2) Now I call test_log4j method 100 times ( so it is
100 req) and gather the time
Repeat the above test with 1000 requests and
gather the time
Repeat the above test with 2000 requests and
gather the time
Thats how the timing stats are gathered.
Appender additivity is turned off as per by config
setting and I don't see
any log messages written to test1.log either.
Sorry, I didn't understand your question about
waiting for background thread.
--- Ceki Gülcü [EMAIL PROTECTED] wrote:
What is the difference between Strings, taking the
values 2,4 and 8,
and Request, taking the values 100, 1000 and 2000?
Are you sure you are waiting for the background
thread to finish?
Your log4j configuration file suggests that your
tests use two
appenders dest1 and dest2. You are aware that
appenders are additive,
right? Does logging output go both
c:/skk/cps/logger/test/test1.log
and c:/skk/cps/logger/test/test.log? Given that
writing to a file is
the most time consuming task, is it fait to say that
log4j is doing
twice the work roughly at the same cost?
At 07:29 03.09.2002 -0700, you wrote:
Hi,
We are planning to move from a custom in house
logging
framework (MyLogger) to using Log4j and have been
doing some performance comparison between log4j and
MyLogger.
The numbers seem to be favoring mylogger over
log4j.
Sample Numbers
==
Note: Timing includes logj startup time, mylogger
startup time and junitperf startup time.
1) Logging 2 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.221.29
10001.671.71
20002.122.01
2) Logging 4 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.211.25
10001.653 1.82
20002.242.56
3) Logging 8 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.351.35
400 1.561.71
900 1.8 2.35
4) Logging 80 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.251 1.7
10001.816.1
20002.1511.2
Sample code and configuration
===
log4j.properties
==
log4j.rootCategory=DEBUG,dest1
log4j.appender.dest1=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout
log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log
log4j.appender.dest1.MaxFileSize=4MB
log4j.appender.dest1.MaxBackupIndex=60
log4j.category.ut.log.Log4jPerfTest=DEBUG,dest2
log4j.additivity.Log4jPerfTest=false
log4j.appender.dest2=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.TTCCLayout
log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log
log4j.appender.dest2.MaxFileSize=4MB
log4j.appender.dest2.MaxBackupIndex=60
Test Logger code
===
Note: logMsgs is an array with strings of size 128
bytes.
public void test_Log4j() throws Exception {
int size = logMsgs.size();
for( int i=0; isize; i++ ) {
log4jLogger.info(
(String)logMsgs.get(i)
);
}
}
Some of my questions
1) Is there anything that I can do to increase the
performance of log4j on my box( Pentium-3, 800Mhz,
512MB RAM)
2) I see some architectural difference between the
way
mylogger and log4j is implemented namely
My logger does formattingwriting to log file in
a
background thread whereas it seems that log4j
doesn't
seem do the same. Is this true.
My logger uses the concept of observer and
observable.
Thanks.
__
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com
--
To unsubscribe, e-mail:
mailto:[EMAIL PROTECTED]
For additional
RE: Performance between Log4j and custom in-house logger
Yes, you are right. My measurements doesn't account
for the background thread processing time since it use
the pattern observer/observable(more like publish and
subscribe but in-process).
I thought that Log4j would also be doing the same but
once I started digging through the source code(to get
better perf #'s ), it didn't seem to do the same.
Let me know, if I can make some settings change and
get a better perf numbers.
--- Cakalic, James [EMAIL PROTECTED] wrote:
Perhaps one telling comment is My logger does
formattingwriting to log
file in a background thread. My question is, do
your measurements
account for all background processing having been
completed? If not,
this may explain why log4j performance appears to
degrade in comparison
with mylogger as logging load increases --
assuming that I am reading
your results correctly.
Jim
-Original Message-
From: Shapira, Yoav [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 03, 2002 9:40 AM
To: Log4J Users List
Subject: RE: Performance between Log4j and custom
in-house logger
Hi,
There are always going to be difference between any
two systems designed
to do the same thing. Your system was obviously
designed for your
needs, and probably not much more. Log4j is
designed to be generic,
useful to a very broad range of people /
applications, easily
extensible, and yet fast.
At my organization, we carefully compared five or
six in-house custom
logging systems and a couple of 3rd party ones. We
included very
detailed performance comparisons and profilings. We
found log4j to be
as fast or faster than nearly all the other
implementation, while
providing the best range of features. As we need a
lot of its features
(JDBC logging, SMTP logging, runtime
configurability, JMS logging, daily
rolling, appender thresholds, object renderers,
etc), we picked log4j,
and have been very very happy since.
So it's a question of what you need from your
logging system. If your
system did everything you need, you probably
wouldn't even be
considering log4j, right? ;) Maybe it';; easier to
add just the
features you need to your own system, instead of
migrating to log4j
completely?
The point is, if you want to help tune log4j you're
welcome to. It's
open source and contributors are always welcome,
right Ceki? ;)
However, having used log4j in numerous apps from
very small to very
large, we've never had any complaints about its
performance. There was
always a lot of tuning to be done on our own apps
before logging
performance showed up as a blip on the profiler
screen...
Yoav Shapira
Millennium ChemInformatics
-Original Message-
From: sk k [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 03, 2002 10:30 AM
To: [EMAIL PROTECTED]
Subject: Performance between Log4j and custom
in-house logger
Hi,
We are planning to move from a custom in house
logging
framework (MyLogger) to using Log4j and have been
doing some performance comparison between log4j and
MyLogger.
The numbers seem to be favoring mylogger over
log4j.
Sample Numbers
==
Note: Timing includes logj startup time, mylogger
startup time and junitperf startup time.
1) Logging 2 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.221.29
1000 1.671.71
2000 2.122.01
2) Logging 4 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.211.25
1000 1.653 1.82
2000 2.242.56
3) Logging 8 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.351.35
400 1.561.71
900 1.8 2.35
4) Logging 80 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.251 1.7
1000 1.816.1
2000 2.1511.2
Sample code and configuration
===
log4j.properties
==
log4j.rootCategory=DEBUG,dest1
log4j.appender.dest1=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout
log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log
log4j.appender.dest1.MaxFileSize=4MB
log4j.appender.dest1.MaxBackupIndex=60
log4j.category.ut.log.Log4jPerfTest=DEBUG,dest2
log4j.additivity.Log4jPerfTest=false
log4j.appender.dest2=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.TTCCLayout
log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log
log4j.appender.dest2.MaxFileSize=4MB
log4j.appender.dest2.MaxBackupIndex=60
Test Logger code
===
Note: logMsgs is an array with strings of size 128
bytes.
public void test_Log4j() throws Exception {
int size = logMsgs.size();
for( int i=0; isize; i++ ) {
log4jLogger.info(
(String)logMsgs.get(i
RE: Performance between Log4j and custom in-house logger
Yes, you are right. My measurements doesn't account for the background thread processing time since it use the pattern observer/observable(more like publish and subscribe but in-process). Performance has many dimensions. You're talking about absolute intervals for a single operation. If you care about scalability, you definitely want to account for the resources used by the background threads. At some point, the machine has to do the work, one way or another. Also, you cited measurements that included start-up costs. Unless you have a fairly trivial application which doesn't live long, you might want to consider factoring out the start-up costs as not mattering much in the steady state. (If you do have such an application, you're probably being killed by JVM start-up overhead in the first place.) Anyhow, maybe you already know all this and you're measuring things just the way you want to measure them. For a pretty significant percentage of developers and applications, the above two points are pretty pertinent, though. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Re: Performance between Log4j and custom in-house logger
Hi,
Any thoughts on how I can improve the performance
numbers for logging.
Thanks.
--- Ceki Gülcü [EMAIL PROTECTED] wrote:
What is the difference between Strings, taking the
values 2,4 and 8,
and Request, taking the values 100, 1000 and 2000?
Are you sure you are waiting for the background
thread to finish?
Your log4j configuration file suggests that your
tests use two
appenders dest1 and dest2. You are aware that
appenders are additive,
right? Does logging output go both
c:/skk/cps/logger/test/test1.log
and c:/skk/cps/logger/test/test.log? Given that
writing to a file is
the most time consuming task, is it fait to say that
log4j is doing
twice the work roughly at the same cost?
At 07:29 03.09.2002 -0700, you wrote:
Hi,
We are planning to move from a custom in house
logging
framework (MyLogger) to using Log4j and have been
doing some performance comparison between log4j and
MyLogger.
The numbers seem to be favoring mylogger over
log4j.
Sample Numbers
==
Note: Timing includes logj startup time, mylogger
startup time and junitperf startup time.
1) Logging 2 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.221.29
10001.671.71
20002.122.01
2) Logging 4 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.211.25
10001.653 1.82
20002.242.56
3) Logging 8 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.351.35
400 1.561.71
900 1.8 2.35
4) Logging 80 strings, each of size: 128bytes
Request MyLogger(sec) Log4j(sec)
== === =
100 1.251 1.7
10001.816.1
20002.1511.2
Sample code and configuration
===
log4j.properties
==
log4j.rootCategory=DEBUG,dest1
log4j.appender.dest1=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.SimpleLayout
log4j.appender.dest1.File=c:/skk/cps/logger/test/test1.log
log4j.appender.dest1.MaxFileSize=4MB
log4j.appender.dest1.MaxBackupIndex=60
log4j.category.ut.log.Log4jPerfTest=DEBUG,dest2
log4j.additivity.Log4jPerfTest=false
log4j.appender.dest2=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.layout=org.apache.log4j.TTCCLayout
log4j.appender.dest2.File=c:/skk/cps/logger/test/test.log
log4j.appender.dest2.MaxFileSize=4MB
log4j.appender.dest2.MaxBackupIndex=60
Test Logger code
===
Note: logMsgs is an array with strings of size 128
bytes.
public void test_Log4j() throws Exception {
int size = logMsgs.size();
for( int i=0; isize; i++ ) {
log4jLogger.info(
(String)logMsgs.get(i)
);
}
}
Some of my questions
1) Is there anything that I can do to increase the
performance of log4j on my box( Pentium-3, 800Mhz,
512MB RAM)
2) I see some architectural difference between the
way
mylogger and log4j is implemented namely
My logger does formattingwriting to log file in
a
background thread whereas it seems that log4j
doesn't
seem do the same. Is this true.
My logger uses the concept of observer and
observable.
Thanks.
__
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com
--
To unsubscribe, e-mail:
mailto:[EMAIL PROTECTED]
For additional commands, e-mail:
mailto:[EMAIL PROTECTED]
--
Ceki
TCP implementations will follow a general principle
of robustness: be
conservative in what you do, be liberal in what you
accept from
others. -- Jon Postel, RFC 793
--
To unsubscribe, e-mail:
mailto:[EMAIL PROTECTED]
For additional commands, e-mail:
mailto:[EMAIL PROTECTED]
__
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com
--
To unsubscribe, e-mail: mailto:[EMAIL PROTECTED]
For additional commands, e-mail: mailto:[EMAIL PROTECTED]
