RE: logging during multithreaded builds (follow-up)
Hi, I dug a little (really, just a little) further, and I'm wondering what is missing when I would just set org.slf4j.simpleLogger.showThreadName=true in maven/conf/logging/simplelogger.properties? It seems too simple :) But it seems to be showing me thread names to distinguish things by: [main] [BuilderThread 0] [BuilderThread 1] Et cetera... Thanks, Sander. Sander Verhagen [ san...@sanderverhagen.net ] NOTICE: my e-mail address has changed. Please remove verha...@sander.com now and start using san...@sanderverhagen.net from now on. Please update your address book. Thank you! -Original Message- From: Igor Fedorenko [mailto:i...@ifedorenko.com] Sent: Tuesday, June 09, 2015 12:15 To: Maven Developers List; Sander Verhagen Subject: Re: logging during multithreaded builds (follow-up) Based on the feedback I got on this list I did not include the logger extension in maven itself but kept it in a separate repository. I may opensource that repository some time in the future but it is not a priority for me right now. -- Regards, Igor On June 9, 2015 1:33:46 PM Sander Verhagen san...@sanderverhagen.net wrote: Hi group, Sorry if it's inappropriate for me to ask this in the dev list, but this is where this was previously discussed. I'm interested in the work that Igor mentioned around logging during multithreaded builds. It sounded like there was an addition to Maven being built (whether it be in the core or some extension). Is that somehow generally available, be it in released form or as (alpha) source code that I can build myself? Much appreciated, Sander Verhagen [ san...@sanderverhagen.netmailto:san...@sanderverhagen.net ] - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: logging during multithreaded builds (follow-up)
For my needs logging thread names was not enough. I needed to capture logging using other logging frameworks (commons-logging, java.util.logging, log4j, logback, System.out/err). I also wanted to be able to see individual project build messages grouped together, either in separate files on filesystem or in console. I also wanted ability to do custom routing of logging messages, which I use in m2e maven dev tools to capture and display build logs. Not saying everyone needs all of these, of course, you don't need anything fancier if thread names are good enough for you. -- Regards, Igor On Fri, Jul 17, 2015, at 12:16 PM, Sander Verhagen wrote: Hi, I dug a little (really, just a little) further, and I'm wondering what is missing when I would just set org.slf4j.simpleLogger.showThreadName=true in maven/conf/logging/simplelogger.properties? It seems too simple :) But it seems to be showing me thread names to distinguish things by: [main] [BuilderThread 0] [BuilderThread 1] Et cetera... Thanks, Sander. Sander Verhagen [ san...@sanderverhagen.net ] NOTICE: my e-mail address has changed. Please remove verha...@sander.com now and start using san...@sanderverhagen.net from now on. Please update your address book. Thank you! -Original Message- From: Igor Fedorenko [mailto:i...@ifedorenko.com] Sent: Tuesday, June 09, 2015 12:15 To: Maven Developers List; Sander Verhagen Subject: Re: logging during multithreaded builds (follow-up) Based on the feedback I got on this list I did not include the logger extension in maven itself but kept it in a separate repository. I may opensource that repository some time in the future but it is not a priority for me right now. -- Regards, Igor On June 9, 2015 1:33:46 PM Sander Verhagen san...@sanderverhagen.net wrote: Hi group, Sorry if it's inappropriate for me to ask this in the dev list, but this is where this was previously discussed. I'm interested in the work that Igor mentioned around logging during multithreaded builds. It sounded like there was an addition to Maven being built (whether it be in the core or some extension). Is that somehow generally available, be it in released form or as (alpha) source code that I can build myself? Much appreciated, Sander Verhagen [ san...@sanderverhagen.netmailto:san...@sanderverhagen.net ] - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
logging during multithreaded builds (follow-up)
Hi group, Sorry if it's inappropriate for me to ask this in the dev list, but this is where this was previously discussed. I'm interested in the work that Igor mentioned around logging during multithreaded builds. It sounded like there was an addition to Maven being built (whether it be in the core or some extension). Is that somehow generally available, be it in released form or as (alpha) source code that I can build myself? Much appreciated, Sander Verhagen [ san...@sanderverhagen.netmailto:san...@sanderverhagen.net ]
Re: logging during multithreaded builds (follow-up)
Based on the feedback I got on this list I did not include the logger extension in maven itself but kept it in a separate repository. I may opensource that repository some time in the future but it is not a priority for me right now. -- Regards, Igor On June 9, 2015 1:33:46 PM Sander Verhagen san...@sanderverhagen.net wrote: Hi group, Sorry if it's inappropriate for me to ask this in the dev list, but this is where this was previously discussed. I'm interested in the work that Igor mentioned around logging during multithreaded builds. It sounded like there was an addition to Maven being built (whether it be in the core or some extension). Is that somehow generally available, be it in released form or as (alpha) source code that I can build myself? Much appreciated, Sander Verhagen [ san...@sanderverhagen.netmailto:san...@sanderverhagen.net ] - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: logging during multithreaded builds
I agree with Jason, it would be better to keep this outside of core (the
core distro).
/Anders
On Thu, May 28, 2015 at 5:21 AM, Jason van Zyl ja...@takari.io wrote:
I honestly don't think an optional feature relying on an optional
dependency belongs in the core itself.
On May 27, 2015, at 10:34 PM, Igor Fedorenko i...@ifedorenko.com wrote:
There are three semi-related parts to my implementation
1. SLF4J MDC management, basically setting and removing project
attributes in a thread-local map. Truly reliable implementation will
need to be coded in all Builders. Alternatively, it should be possible
to use existing lifecycle callbacks to implement mostly reliable lib/ext
extension.
2. Install and configure System out/err and java.util.logging bridges.
This can be implemented as lib/ext extension using existing callbacks
3. Custom logback appenders and configuration that provide interesting
behaviour using SLF4J MDC. These too can be implemented of the core.
Keeping all this in the core will allow marginally more reliable SLF4J
MDC management implementation and I also think will make it more likely
for other developers to help with the implementation. It will also serve
as a working example for adding support for other logging framework. But
I agree, this does not have to be implemented in the core so I'll try to
rework my implementation as lib/ext extension.
--
Regards,
Igor
On Wed, May 27, 2015, at 06:40 PM, Jason van Zyl wrote:
On May 27, 2015, at 3:55 PM, Igor Fedorenko i...@ifedorenko.com
wrote:
So I went ahead and implemented these changes, including working (but
not terribly well tested) logback appenders to buffer-and-group project
console log messages and create per-project build.log files.
What changes were required in the core?
Does anyone see a problem if I check in these appenders in maven core
source tree or you prefer me to keep them elsewhere?
If it's required to alter the distribution to install logback in order
to
use the appenders then I think elsewhere is more appropriate. As they
won't be used by the standard version of Maven so I don't think it
belongs in core along with the other modules. Maven shared is probably
fine.
Just to be clear, I do not propose to hardware maven to logback and I
do not propose to include logback support in maven distribution. I want
to introduce new maven-ext-logback module, which users will be able to
use to customize their maven distributions very much the same way they
need to do it now to use any of the advanced logging frameworks.
--
Regards,
Igor
On Tue, May 26, 2015, at 03:38 PM, Igor Fedorenko wrote:
I spent some time looking into this, and I think project-level logging
will require several semi-related changes.
* As Ralph pointed out, Maven needs to use SLF4J MDC to associate log
messages with individual projects being built. This is required to
enable any project-related logging approach and I plan to submit this
change in next couple of days.
* Another problem is use of non-slf4j logging techniques by Maven
plugins. Direct use of System out/err and java.util.logging are two
obvious problems and I plan to change maven to bridge these to
slf4j.
* The old log4j 1.x logging and commons-logging need to be bridged to
slf4j too. I can make this change in maven, but this is not strictly
necessary because it can be done from lib/ext extension too.
The rest really depends on whether we can agree on single advanced
logging backend and if we need to support several logging
configurations. I think we've found three viable project-level logging
approaches: buffered console output, better logging pattern and my
original target/build.log idea. Unless somebody really wants to
restart
logback-vs-log4j discussion, I suggest we postpone this decision and
instead implement project-level logging as out-of-core extensions.
Does this make sense?
--
Regards,
Igor
On Tue, May 26, 2015, at 01:13 PM, Ralph Goers wrote:
If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x
and
2.x - you can include anything stored in the MDC on every line of log
output. Just use %X to include all MDC items or %MDC{key} to
include the
specific key. This would require storing the value(s) at the
beginning
of every thread.
If you include %t in the pattern than every log line should include
the
threadId.
If you are saying that the log lines include newlines in them that
should
still be OK. The only way the lines should be mangled is if each
thread
somehow has its own instance of the logging framework and they are
all
configured to write to the same file.
Ralph
On May 25, 2015, at 7:28 AM, Igor Fedorenko i...@ifedorenko.com
wrote:
Yes, thread-id will help to some degree, but maven uses multiline
log
messages quite often and these will still be
Re: logging during multithreaded builds
I know it's not directly related, but any general purpose algorithm that
captures to memory buffers needs an overflow to disk mechanism - every time
a ByteArrayOutputStream is used for this, some guy with a *huge* output
from his build gets an OOM.
It's nice you're looking into this issue. As for the actual question of
logger implementations, I prefer trivial problems like trying to determine
that P=NP.
Kristian
2015-05-28 8:39 GMT+02:00 Anders Hammar and...@hammar.net:
I agree with Jason, it would be better to keep this outside of core (the
core distro).
/Anders
On Thu, May 28, 2015 at 5:21 AM, Jason van Zyl ja...@takari.io wrote:
I honestly don't think an optional feature relying on an optional
dependency belongs in the core itself.
On May 27, 2015, at 10:34 PM, Igor Fedorenko i...@ifedorenko.com
wrote:
There are three semi-related parts to my implementation
1. SLF4J MDC management, basically setting and removing project
attributes in a thread-local map. Truly reliable implementation will
need to be coded in all Builders. Alternatively, it should be possible
to use existing lifecycle callbacks to implement mostly reliable
lib/ext
extension.
2. Install and configure System out/err and java.util.logging bridges.
This can be implemented as lib/ext extension using existing callbacks
3. Custom logback appenders and configuration that provide
interesting
behaviour using SLF4J MDC. These too can be implemented of the core.
Keeping all this in the core will allow marginally more reliable SLF4J
MDC management implementation and I also think will make it more likely
for other developers to help with the implementation. It will also
serve
as a working example for adding support for other logging framework.
But
I agree, this does not have to be implemented in the core so I'll try
to
rework my implementation as lib/ext extension.
--
Regards,
Igor
On Wed, May 27, 2015, at 06:40 PM, Jason van Zyl wrote:
On May 27, 2015, at 3:55 PM, Igor Fedorenko i...@ifedorenko.com
wrote:
So I went ahead and implemented these changes, including working (but
not terribly well tested) logback appenders to buffer-and-group
project
console log messages and create per-project build.log files.
What changes were required in the core?
Does anyone see a problem if I check in these appenders in maven core
source tree or you prefer me to keep them elsewhere?
If it's required to alter the distribution to install logback in order
to
use the appenders then I think elsewhere is more appropriate. As they
won't be used by the standard version of Maven so I don't think it
belongs in core along with the other modules. Maven shared is probably
fine.
Just to be clear, I do not propose to hardware maven to logback
and I
do not propose to include logback support in maven distribution. I
want
to introduce new maven-ext-logback module, which users will be able
to
use to customize their maven distributions very much the same way
they
need to do it now to use any of the advanced logging frameworks.
--
Regards,
Igor
On Tue, May 26, 2015, at 03:38 PM, Igor Fedorenko wrote:
I spent some time looking into this, and I think project-level
logging
will require several semi-related changes.
* As Ralph pointed out, Maven needs to use SLF4J MDC to associate
log
messages with individual projects being built. This is required to
enable any project-related logging approach and I plan to submit
this
change in next couple of days.
* Another problem is use of non-slf4j logging techniques by Maven
plugins. Direct use of System out/err and java.util.logging are two
obvious problems and I plan to change maven to bridge these to
slf4j.
* The old log4j 1.x logging and commons-logging need to be bridged
to
slf4j too. I can make this change in maven, but this is not strictly
necessary because it can be done from lib/ext extension too.
The rest really depends on whether we can agree on single advanced
logging backend and if we need to support several logging
configurations. I think we've found three viable project-level
logging
approaches: buffered console output, better logging pattern and my
original target/build.log idea. Unless somebody really wants to
restart
logback-vs-log4j discussion, I suggest we postpone this decision and
instead implement project-level logging as out-of-core extensions.
Does this make sense?
--
Regards,
Igor
On Tue, May 26, 2015, at 01:13 PM, Ralph Goers wrote:
If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x
and
2.x - you can include anything stored in the MDC on every line of
log
output. Just use %X to include all MDC items or %MDC{key} to
include the
specific key. This would require storing the value(s) at the
Re: logging during multithreaded builds
So I went ahead and implemented these changes, including working (but
not terribly well tested) logback appenders to buffer-and-group project
console log messages and create per-project build.log files.
Does anyone see a problem if I check in these appenders in maven core
source tree or you prefer me to keep them elsewhere?
Just to be clear, I do not propose to hardware maven to logback and I
do not propose to include logback support in maven distribution. I want
to introduce new maven-ext-logback module, which users will be able to
use to customize their maven distributions very much the same way they
need to do it now to use any of the advanced logging frameworks.
--
Regards,
Igor
On Tue, May 26, 2015, at 03:38 PM, Igor Fedorenko wrote:
I spent some time looking into this, and I think project-level logging
will require several semi-related changes.
* As Ralph pointed out, Maven needs to use SLF4J MDC to associate log
messages with individual projects being built. This is required to
enable any project-related logging approach and I plan to submit this
change in next couple of days.
* Another problem is use of non-slf4j logging techniques by Maven
plugins. Direct use of System out/err and java.util.logging are two
obvious problems and I plan to change maven to bridge these to slf4j.
* The old log4j 1.x logging and commons-logging need to be bridged to
slf4j too. I can make this change in maven, but this is not strictly
necessary because it can be done from lib/ext extension too.
The rest really depends on whether we can agree on single advanced
logging backend and if we need to support several logging
configurations. I think we've found three viable project-level logging
approaches: buffered console output, better logging pattern and my
original target/build.log idea. Unless somebody really wants to restart
logback-vs-log4j discussion, I suggest we postpone this decision and
instead implement project-level logging as out-of-core extensions.
Does this make sense?
--
Regards,
Igor
On Tue, May 26, 2015, at 01:13 PM, Ralph Goers wrote:
If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x and
2.x - you can include anything stored in the MDC on every line of log
output. Just use %X to include all MDC items or %MDC{key} to include the
specific key. This would require storing the value(s) at the beginning
of every thread.
If you include %t in the pattern than every log line should include the
threadId.
If you are saying that the log lines include newlines in them that should
still be OK. The only way the lines should be mangled is if each thread
somehow has its own instance of the logging framework and they are all
configured to write to the same file.
Ralph
On May 25, 2015, at 7:28 AM, Igor Fedorenko i...@ifedorenko.com wrote:
Yes, thread-id will help to some degree, but maven uses multiline log
messages quite often and these will still be mangled and unreadable.
Per-project build log files is the only solution I found to preserve
readable logs. Also, each project build is mostly independent from the
rest and I find reading self-contain per-project log files much easier.
--
Regards,
Igor
On Mon, May 25, 2015, at 10:21 AM, Sean Busbey wrote:
In multithreaded builds we could add a thread ID to each output line.
That
would make it easier to read and filter in different files in post
processing.
--
Sean
On May 25, 2015 6:30 AM, Igor Fedorenko i...@ifedorenko.com wrote:
I had to troubleshoot a large multithreaded build last week and that
proved to be rather difficult mostly because build log was a jumble of
messages produced by concurrently running threads. It was not possible
to tell which message came from which thread, which made the build log
more or less useless.
What I ended up doing was to write per-module log message to individual
${project.build.directory}/build.log log files.
That was kinda tricky to implement because log files were opened very
early during module build and were subsequently deleted by
maven-clean-plugin (I tried on Linux and OSX, and I assume the build
will fail on Windows). I had to modify maven-clean-plugin configuration
in the project pom.xml to retain ${project.build.directory}/build.log
files.
I felt my solution required too my effort and I wonder what others do to
capture logs during multithreaded builds. Can we come up with a
recommended way of doing this?
--
Regards,
Igor
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional
Re: logging during multithreaded builds
There are three semi-related parts to my implementation
1. SLF4J MDC management, basically setting and removing project
attributes in a thread-local map. Truly reliable implementation will
need to be coded in all Builders. Alternatively, it should be possible
to use existing lifecycle callbacks to implement mostly reliable lib/ext
extension.
2. Install and configure System out/err and java.util.logging bridges.
This can be implemented as lib/ext extension using existing callbacks
3. Custom logback appenders and configuration that provide interesting
behaviour using SLF4J MDC. These too can be implemented of the core.
Keeping all this in the core will allow marginally more reliable SLF4J
MDC management implementation and I also think will make it more likely
for other developers to help with the implementation. It will also serve
as a working example for adding support for other logging framework. But
I agree, this does not have to be implemented in the core so I'll try to
rework my implementation as lib/ext extension.
--
Regards,
Igor
On Wed, May 27, 2015, at 06:40 PM, Jason van Zyl wrote:
On May 27, 2015, at 3:55 PM, Igor Fedorenko i...@ifedorenko.com wrote:
So I went ahead and implemented these changes, including working (but
not terribly well tested) logback appenders to buffer-and-group project
console log messages and create per-project build.log files.
What changes were required in the core?
Does anyone see a problem if I check in these appenders in maven core
source tree or you prefer me to keep them elsewhere?
If it's required to alter the distribution to install logback in order to
use the appenders then I think elsewhere is more appropriate. As they
won't be used by the standard version of Maven so I don't think it
belongs in core along with the other modules. Maven shared is probably
fine.
Just to be clear, I do not propose to hardware maven to logback and I
do not propose to include logback support in maven distribution. I want
to introduce new maven-ext-logback module, which users will be able to
use to customize their maven distributions very much the same way they
need to do it now to use any of the advanced logging frameworks.
--
Regards,
Igor
On Tue, May 26, 2015, at 03:38 PM, Igor Fedorenko wrote:
I spent some time looking into this, and I think project-level logging
will require several semi-related changes.
* As Ralph pointed out, Maven needs to use SLF4J MDC to associate log
messages with individual projects being built. This is required to
enable any project-related logging approach and I plan to submit this
change in next couple of days.
* Another problem is use of non-slf4j logging techniques by Maven
plugins. Direct use of System out/err and java.util.logging are two
obvious problems and I plan to change maven to bridge these to slf4j.
* The old log4j 1.x logging and commons-logging need to be bridged to
slf4j too. I can make this change in maven, but this is not strictly
necessary because it can be done from lib/ext extension too.
The rest really depends on whether we can agree on single advanced
logging backend and if we need to support several logging
configurations. I think we've found three viable project-level logging
approaches: buffered console output, better logging pattern and my
original target/build.log idea. Unless somebody really wants to restart
logback-vs-log4j discussion, I suggest we postpone this decision and
instead implement project-level logging as out-of-core extensions.
Does this make sense?
--
Regards,
Igor
On Tue, May 26, 2015, at 01:13 PM, Ralph Goers wrote:
If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x and
2.x - you can include anything stored in the MDC on every line of log
output. Just use %X to include all MDC items or %MDC{key} to include the
specific key. This would require storing the value(s) at the beginning
of every thread.
If you include %t in the pattern than every log line should include the
threadId.
If you are saying that the log lines include newlines in them that should
still be OK. The only way the lines should be mangled is if each thread
somehow has its own instance of the logging framework and they are all
configured to write to the same file.
Ralph
On May 25, 2015, at 7:28 AM, Igor Fedorenko i...@ifedorenko.com wrote:
Yes, thread-id will help to some degree, but maven uses multiline log
messages quite often and these will still be mangled and unreadable.
Per-project build log files is the only solution I found to preserve
readable logs. Also, each project build is mostly independent from the
rest and I find reading self-contain per-project log files much easier.
--
Regards,
Igor
On Mon, May 25, 2015, at 10:21 AM, Sean Busbey wrote:
In multithreaded builds we could add a thread ID to each output
Re: logging during multithreaded builds
I honestly don't think an optional feature relying on an optional dependency
belongs in the core itself.
On May 27, 2015, at 10:34 PM, Igor Fedorenko i...@ifedorenko.com wrote:
There are three semi-related parts to my implementation
1. SLF4J MDC management, basically setting and removing project
attributes in a thread-local map. Truly reliable implementation will
need to be coded in all Builders. Alternatively, it should be possible
to use existing lifecycle callbacks to implement mostly reliable lib/ext
extension.
2. Install and configure System out/err and java.util.logging bridges.
This can be implemented as lib/ext extension using existing callbacks
3. Custom logback appenders and configuration that provide interesting
behaviour using SLF4J MDC. These too can be implemented of the core.
Keeping all this in the core will allow marginally more reliable SLF4J
MDC management implementation and I also think will make it more likely
for other developers to help with the implementation. It will also serve
as a working example for adding support for other logging framework. But
I agree, this does not have to be implemented in the core so I'll try to
rework my implementation as lib/ext extension.
--
Regards,
Igor
On Wed, May 27, 2015, at 06:40 PM, Jason van Zyl wrote:
On May 27, 2015, at 3:55 PM, Igor Fedorenko i...@ifedorenko.com wrote:
So I went ahead and implemented these changes, including working (but
not terribly well tested) logback appenders to buffer-and-group project
console log messages and create per-project build.log files.
What changes were required in the core?
Does anyone see a problem if I check in these appenders in maven core
source tree or you prefer me to keep them elsewhere?
If it's required to alter the distribution to install logback in order to
use the appenders then I think elsewhere is more appropriate. As they
won't be used by the standard version of Maven so I don't think it
belongs in core along with the other modules. Maven shared is probably
fine.
Just to be clear, I do not propose to hardware maven to logback and I
do not propose to include logback support in maven distribution. I want
to introduce new maven-ext-logback module, which users will be able to
use to customize their maven distributions very much the same way they
need to do it now to use any of the advanced logging frameworks.
--
Regards,
Igor
On Tue, May 26, 2015, at 03:38 PM, Igor Fedorenko wrote:
I spent some time looking into this, and I think project-level logging
will require several semi-related changes.
* As Ralph pointed out, Maven needs to use SLF4J MDC to associate log
messages with individual projects being built. This is required to
enable any project-related logging approach and I plan to submit this
change in next couple of days.
* Another problem is use of non-slf4j logging techniques by Maven
plugins. Direct use of System out/err and java.util.logging are two
obvious problems and I plan to change maven to bridge these to slf4j.
* The old log4j 1.x logging and commons-logging need to be bridged to
slf4j too. I can make this change in maven, but this is not strictly
necessary because it can be done from lib/ext extension too.
The rest really depends on whether we can agree on single advanced
logging backend and if we need to support several logging
configurations. I think we've found three viable project-level logging
approaches: buffered console output, better logging pattern and my
original target/build.log idea. Unless somebody really wants to restart
logback-vs-log4j discussion, I suggest we postpone this decision and
instead implement project-level logging as out-of-core extensions.
Does this make sense?
--
Regards,
Igor
On Tue, May 26, 2015, at 01:13 PM, Ralph Goers wrote:
If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x and
2.x - you can include anything stored in the MDC on every line of log
output. Just use %X to include all MDC items or %MDC{key} to include the
specific key. This would require storing the value(s) at the beginning
of every thread.
If you include %t in the pattern than every log line should include the
threadId.
If you are saying that the log lines include newlines in them that should
still be OK. The only way the lines should be mangled is if each thread
somehow has its own instance of the logging framework and they are all
configured to write to the same file.
Ralph
On May 25, 2015, at 7:28 AM, Igor Fedorenko i...@ifedorenko.com wrote:
Yes, thread-id will help to some degree, but maven uses multiline log
messages quite often and these will still be mangled and unreadable.
Per-project build log files is the only solution I found to preserve
readable logs. Also, each project build is mostly independent from the
rest and I find reading self-contain per-project log files much easier.
--
Regards,
Igor
Re: logging during multithreaded builds
On May 27, 2015, at 3:55 PM, Igor Fedorenko i...@ifedorenko.com wrote:
So I went ahead and implemented these changes, including working (but
not terribly well tested) logback appenders to buffer-and-group project
console log messages and create per-project build.log files.
What changes were required in the core?
Does anyone see a problem if I check in these appenders in maven core
source tree or you prefer me to keep them elsewhere?
If it's required to alter the distribution to install logback in order to use
the appenders then I think elsewhere is more appropriate. As they won't be used
by the standard version of Maven so I don't think it belongs in core along with
the other modules. Maven shared is probably fine.
Just to be clear, I do not propose to hardware maven to logback and I
do not propose to include logback support in maven distribution. I want
to introduce new maven-ext-logback module, which users will be able to
use to customize their maven distributions very much the same way they
need to do it now to use any of the advanced logging frameworks.
--
Regards,
Igor
On Tue, May 26, 2015, at 03:38 PM, Igor Fedorenko wrote:
I spent some time looking into this, and I think project-level logging
will require several semi-related changes.
* As Ralph pointed out, Maven needs to use SLF4J MDC to associate log
messages with individual projects being built. This is required to
enable any project-related logging approach and I plan to submit this
change in next couple of days.
* Another problem is use of non-slf4j logging techniques by Maven
plugins. Direct use of System out/err and java.util.logging are two
obvious problems and I plan to change maven to bridge these to slf4j.
* The old log4j 1.x logging and commons-logging need to be bridged to
slf4j too. I can make this change in maven, but this is not strictly
necessary because it can be done from lib/ext extension too.
The rest really depends on whether we can agree on single advanced
logging backend and if we need to support several logging
configurations. I think we've found three viable project-level logging
approaches: buffered console output, better logging pattern and my
original target/build.log idea. Unless somebody really wants to restart
logback-vs-log4j discussion, I suggest we postpone this decision and
instead implement project-level logging as out-of-core extensions.
Does this make sense?
--
Regards,
Igor
On Tue, May 26, 2015, at 01:13 PM, Ralph Goers wrote:
If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x and
2.x - you can include anything stored in the MDC on every line of log
output. Just use %X to include all MDC items or %MDC{key} to include the
specific key. This would require storing the value(s) at the beginning
of every thread.
If you include %t in the pattern than every log line should include the
threadId.
If you are saying that the log lines include newlines in them that should
still be OK. The only way the lines should be mangled is if each thread
somehow has its own instance of the logging framework and they are all
configured to write to the same file.
Ralph
On May 25, 2015, at 7:28 AM, Igor Fedorenko i...@ifedorenko.com wrote:
Yes, thread-id will help to some degree, but maven uses multiline log
messages quite often and these will still be mangled and unreadable.
Per-project build log files is the only solution I found to preserve
readable logs. Also, each project build is mostly independent from the
rest and I find reading self-contain per-project log files much easier.
--
Regards,
Igor
On Mon, May 25, 2015, at 10:21 AM, Sean Busbey wrote:
In multithreaded builds we could add a thread ID to each output line.
That
would make it easier to read and filter in different files in post
processing.
--
Sean
On May 25, 2015 6:30 AM, Igor Fedorenko i...@ifedorenko.com wrote:
I had to troubleshoot a large multithreaded build last week and that
proved to be rather difficult mostly because build log was a jumble of
messages produced by concurrently running threads. It was not possible
to tell which message came from which thread, which made the build log
more or less useless.
What I ended up doing was to write per-module log message to individual
${project.build.directory}/build.log log files.
That was kinda tricky to implement because log files were opened very
early during module build and were subsequently deleted by
maven-clean-plugin (I tried on Linux and OSX, and I assume the build
will fail on Windows). I had to modify maven-clean-plugin configuration
in the project pom.xml to retain ${project.build.directory}/build.log
files.
I felt my solution required too my effort and I wonder what others do to
capture logs during multithreaded builds. Can we come up with a
recommended way of doing this?
--
Regards,
Igor
Re: logging during multithreaded builds
I spent some time looking into this, and I think project-level logging
will require several semi-related changes.
* As Ralph pointed out, Maven needs to use SLF4J MDC to associate log
messages with individual projects being built. This is required to
enable any project-related logging approach and I plan to submit this
change in next couple of days.
* Another problem is use of non-slf4j logging techniques by Maven
plugins. Direct use of System out/err and java.util.logging are two
obvious problems and I plan to change maven to bridge these to slf4j.
* The old log4j 1.x logging and commons-logging need to be bridged to
slf4j too. I can make this change in maven, but this is not strictly
necessary because it can be done from lib/ext extension too.
The rest really depends on whether we can agree on single advanced
logging backend and if we need to support several logging
configurations. I think we've found three viable project-level logging
approaches: buffered console output, better logging pattern and my
original target/build.log idea. Unless somebody really wants to restart
logback-vs-log4j discussion, I suggest we postpone this decision and
instead implement project-level logging as out-of-core extensions.
Does this make sense?
--
Regards,
Igor
On Tue, May 26, 2015, at 01:13 PM, Ralph Goers wrote:
If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x and
2.x - you can include anything stored in the MDC on every line of log
output. Just use %X to include all MDC items or %MDC{key} to include the
specific key. This would require storing the value(s) at the beginning
of every thread.
If you include %t in the pattern than every log line should include the
threadId.
If you are saying that the log lines include newlines in them that should
still be OK. The only way the lines should be mangled is if each thread
somehow has its own instance of the logging framework and they are all
configured to write to the same file.
Ralph
On May 25, 2015, at 7:28 AM, Igor Fedorenko i...@ifedorenko.com wrote:
Yes, thread-id will help to some degree, but maven uses multiline log
messages quite often and these will still be mangled and unreadable.
Per-project build log files is the only solution I found to preserve
readable logs. Also, each project build is mostly independent from the
rest and I find reading self-contain per-project log files much easier.
--
Regards,
Igor
On Mon, May 25, 2015, at 10:21 AM, Sean Busbey wrote:
In multithreaded builds we could add a thread ID to each output line.
That
would make it easier to read and filter in different files in post
processing.
--
Sean
On May 25, 2015 6:30 AM, Igor Fedorenko i...@ifedorenko.com wrote:
I had to troubleshoot a large multithreaded build last week and that
proved to be rather difficult mostly because build log was a jumble of
messages produced by concurrently running threads. It was not possible
to tell which message came from which thread, which made the build log
more or less useless.
What I ended up doing was to write per-module log message to individual
${project.build.directory}/build.log log files.
That was kinda tricky to implement because log files were opened very
early during module build and were subsequently deleted by
maven-clean-plugin (I tried on Linux and OSX, and I assume the build
will fail on Windows). I had to modify maven-clean-plugin configuration
in the project pom.xml to retain ${project.build.directory}/build.log
files.
I felt my solution required too my effort and I wonder what others do to
capture logs during multithreaded builds. Can we come up with a
recommended way of doing this?
--
Regards,
Igor
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: logging during multithreaded builds
Igor Fedorenko i...@ifedorenko.com wrote on 05/25/2015 07:28:45 AM:
I had to troubleshoot a large multithreaded build last week and that
proved to be rather difficult mostly because build log was a jumble of
messages produced by concurrently running threads. It was not possible
to tell which message came from which thread, which made the build log
more or less useless.
What I ended up doing was to write per-module log message to individual
${project.build.directory}/build.log log files.
That was kinda tricky to implement because log files were opened very
early during module build and were subsequently deleted by
maven-clean-plugin (I tried on Linux and OSX, and I assume the build
will fail on Windows). I had to modify maven-clean-plugin configuration
in the project pom.xml to retain ${project.build.directory}/build.log
files.
I felt my solution required too my effort and I wonder what others do to
capture logs during multithreaded builds. Can we come up with a
recommended way of doing this?
In my Ant days we had a similar problem. I ended up writing a thread-aware
logging implementation that would buffer log messages in memory, then
flush meaningful sections when a logging context ended. The logs ended up
being out-of-order when viewed strictly as a time sequence, but the
sections did make sense and allowed someone to follow in a coherent manner
what happened in a particular context during the build.
I tried to do something similar when I changed teams and we were using
Maven, but frankly the Ant API for plugging in loggers listeners was
easier to work with. I'm currently working on thread-aware event spies for
performance profiling would be happy to work on a thread aware logger as
well if there is interest.
Re: logging during multithreaded builds
If you use the SLF4J MDC - which is supported by Logback, Log4j 1.x and 2.x -
you can include anything stored in the MDC on every line of log output. Just
use %X to include all MDC items or %MDC{key} to include the specific key. This
would require storing the value(s) at the beginning of every thread.
If you include %t in the pattern than every log line should include the
threadId.
If you are saying that the log lines include newlines in them that should still
be OK. The only way the lines should be mangled is if each thread somehow has
its own instance of the logging framework and they are all configured to write
to the same file.
Ralph
On May 25, 2015, at 7:28 AM, Igor Fedorenko i...@ifedorenko.com wrote:
Yes, thread-id will help to some degree, but maven uses multiline log
messages quite often and these will still be mangled and unreadable.
Per-project build log files is the only solution I found to preserve
readable logs. Also, each project build is mostly independent from the
rest and I find reading self-contain per-project log files much easier.
--
Regards,
Igor
On Mon, May 25, 2015, at 10:21 AM, Sean Busbey wrote:
In multithreaded builds we could add a thread ID to each output line.
That
would make it easier to read and filter in different files in post
processing.
--
Sean
On May 25, 2015 6:30 AM, Igor Fedorenko i...@ifedorenko.com wrote:
I had to troubleshoot a large multithreaded build last week and that
proved to be rather difficult mostly because build log was a jumble of
messages produced by concurrently running threads. It was not possible
to tell which message came from which thread, which made the build log
more or less useless.
What I ended up doing was to write per-module log message to individual
${project.build.directory}/build.log log files.
That was kinda tricky to implement because log files were opened very
early during module build and were subsequently deleted by
maven-clean-plugin (I tried on Linux and OSX, and I assume the build
will fail on Windows). I had to modify maven-clean-plugin configuration
in the project pom.xml to retain ${project.build.directory}/build.log
files.
I felt my solution required too my effort and I wonder what others do to
capture logs during multithreaded builds. Can we come up with a
recommended way of doing this?
--
Regards,
Igor
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
logging during multithreaded builds
I had to troubleshoot a large multithreaded build last week and that
proved to be rather difficult mostly because build log was a jumble of
messages produced by concurrently running threads. It was not possible
to tell which message came from which thread, which made the build log
more or less useless.
What I ended up doing was to write per-module log message to individual
${project.build.directory}/build.log log files.
That was kinda tricky to implement because log files were opened very
early during module build and were subsequently deleted by
maven-clean-plugin (I tried on Linux and OSX, and I assume the build
will fail on Windows). I had to modify maven-clean-plugin configuration
in the project pom.xml to retain ${project.build.directory}/build.log
files.
I felt my solution required too my effort and I wonder what others do to
capture logs during multithreaded builds. Can we come up with a
recommended way of doing this?
--
Regards,
Igor
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: logging during multithreaded builds
Yes, thread-id will help to some degree, but maven uses multiline log
messages quite often and these will still be mangled and unreadable.
Per-project build log files is the only solution I found to preserve
readable logs. Also, each project build is mostly independent from the
rest and I find reading self-contain per-project log files much easier.
--
Regards,
Igor
On Mon, May 25, 2015, at 10:21 AM, Sean Busbey wrote:
In multithreaded builds we could add a thread ID to each output line.
That
would make it easier to read and filter in different files in post
processing.
--
Sean
On May 25, 2015 6:30 AM, Igor Fedorenko i...@ifedorenko.com wrote:
I had to troubleshoot a large multithreaded build last week and that
proved to be rather difficult mostly because build log was a jumble of
messages produced by concurrently running threads. It was not possible
to tell which message came from which thread, which made the build log
more or less useless.
What I ended up doing was to write per-module log message to individual
${project.build.directory}/build.log log files.
That was kinda tricky to implement because log files were opened very
early during module build and were subsequently deleted by
maven-clean-plugin (I tried on Linux and OSX, and I assume the build
will fail on Windows). I had to modify maven-clean-plugin configuration
in the project pom.xml to retain ${project.build.directory}/build.log
files.
I felt my solution required too my effort and I wonder what others do to
capture logs during multithreaded builds. Can we come up with a
recommended way of doing this?
--
Regards,
Igor
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
Re: logging during multithreaded builds
In multithreaded builds we could add a thread ID to each output line. That
would make it easier to read and filter in different files in post
processing.
--
Sean
On May 25, 2015 6:30 AM, Igor Fedorenko i...@ifedorenko.com wrote:
I had to troubleshoot a large multithreaded build last week and that
proved to be rather difficult mostly because build log was a jumble of
messages produced by concurrently running threads. It was not possible
to tell which message came from which thread, which made the build log
more or less useless.
What I ended up doing was to write per-module log message to individual
${project.build.directory}/build.log log files.
That was kinda tricky to implement because log files were opened very
early during module build and were subsequently deleted by
maven-clean-plugin (I tried on Linux and OSX, and I assume the build
will fail on Windows). I had to modify maven-clean-plugin configuration
in the project pom.xml to retain ${project.build.directory}/build.log
files.
I felt my solution required too my effort and I wonder what others do to
capture logs during multithreaded builds. Can we come up with a
recommended way of doing this?
--
Regards,
Igor
-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org
