[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118829#comment-17118829 ] ASF subversion and git services commented on SOLR-11934: Commit 5d4b0e3d11db7126784d3cb7c20232f6fce6c0d4 in lucene-solr's branch refs/heads/branch_8x from David Smiley [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5d4b0e3 ] SOLR-11934: REVERT addition of collection to log message This reverts commit e4dc9e94 (cherry picked from commit 596c64a01ea21b92f4b9218427e3223c72e40b55) > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118830#comment-17118830 ] ASF subversion and git services commented on SOLR-11934: Commit 5d4b0e3d11db7126784d3cb7c20232f6fce6c0d4 in lucene-solr's branch refs/heads/branch_8x from David Smiley [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5d4b0e3 ] SOLR-11934: REVERT addition of collection to log message This reverts commit e4dc9e94 (cherry picked from commit 596c64a01ea21b92f4b9218427e3223c72e40b55) > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118827#comment-17118827 ] ASF subversion and git services commented on SOLR-11934: Commit 596c64a01ea21b92f4b9218427e3223c72e40b55 in lucene-solr's branch refs/heads/master from David Smiley [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=596c64a ] SOLR-11934: REVERT addition of collection to log message This reverts commit e4dc9e94 > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118710#comment-17118710 ] Joel Bernstein commented on SOLR-11934: --- This particular record type didn't include the MDC information. So Erick added it a different way. Let's use the MDC approach to so we can track new searchers by collection as well. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118650#comment-17118650 ] David Smiley commented on SOLR-11934: - [~jbernste] can you please explain why it's insufficient/problematic to parse out the collection from the MDC brackets? If it isn't or if I don't hear from you, then I will revert Erick's last commit here. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106463#comment-17106463 ] David Smiley commented on SOLR-11934: - I don't agree with that change; it's redundant with what MDC is already providing and is there for. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106447#comment-17106447 ] Erick Erickson commented on SOLR-11934: --- Done. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106445#comment-17106445 ] ASF subversion and git services commented on SOLR-11934: Commit d992d0a059e1a5ff52b27c4c20af90a386e2727c in lucene-solr's branch refs/heads/branch_8x from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=d992d0a ] SOLR-11934: Visit Solr logging, it's too noisy. (added collection to log messages 'Registered new searcher...' (cherry picked from commit e4dc9e9401ed077101672b19171304e59bb7b4f6) > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106443#comment-17106443 ] ASF subversion and git services commented on SOLR-11934: Commit e4dc9e9401ed077101672b19171304e59bb7b4f6 in lucene-solr's branch refs/heads/master from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e4dc9e9 ] SOLR-11934: Visit Solr logging, it's too noisy. (added collection to log messages 'Registered new searcher...' > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106356#comment-17106356 ] Joel Bernstein commented on SOLR-11934: --- Looks good! > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106330#comment-17106330 ] Erick Erickson commented on SOLR-11934: --- Joel: Yeah I wondered about that too. I was going to get snarky about anyone who named their collection "mycolllection_shard32_notmyproblem" ;) Cassandra: OK, I'll make that happen. Both, how does this look? 2020-05-13 14:02:54.898 INFO (searcherExecutor-27-thread-1-processing-n:localhost:8981_solr x:eoe_shard1_replica_n1 c:eoe s:shard1 r:core_node2) [c:eoe s:shard1 r:core_node2 x:eoe_shard1_replica_n1] o.a.s.c.SolrCore [eoe_shard1_replica_n1] Registered new searcher autowarm time: 0 ms: Collection: 'eoe' > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106319#comment-17106319 ] Joel Bernstein commented on SOLR-11934: --- I'm also wondering if there are situations where the core name might not conform to this pattern? > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106310#comment-17106310 ] Cassandra Targett commented on SOLR-11934: -- bq. Couldn't the log analytics cut off everything after the _shard* when doing its analysis? The log analytics happen after the parsing of the logs and indexing of the records with bin/postlogs (analytics is done by essentially querying indexed log records in various ways) and it's the parsing that needs to be able to separate the elements of each log record into the different fields. Making the parser create a new field by cutting part of another for this particular record type is likely pretty do-able but a better alternative would be to avoid parser logic complication by just printing the helpful fields in the log record whenever possible. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106230#comment-17106230 ] Erick Erickson commented on SOLR-11934: --- [~jbernste] So you need *production_cv_month_201912* in your example broken out separately, right? Looking at that log line, there's no real purpose served by printing out *production_cv_month_201912_shard35_replica_n1* twice, although altering either one might pop out weirdly. Couldn't the log analytics cut off everything after the *_shard** when doing its analysis? But it would be trivial to add the collection name with: *{color:#00}newSearcher{color}.getCore().getCoreDescriptor().getCollectionName()* > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106207#comment-17106207 ] Joel Bernstein commented on SOLR-11934: --- Here is a sample log record for a new searcher: {code:java} 2019-12-16 19:00:23.931 INFO (searcherExecutor-66-thread-1) [ ] o.a.s.c.SolrCore [production_cv_month_201912_shard35_replica_n1] Registered new searcher Searcher@16ef5fac[production_cv_month_201912_shard35_replica_n1] ... {code} > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105900#comment-17105900 ] David Smiley commented on SOLR-11934: - {quote}tie new searchers to a collection as well as the core {quote} Can you please elaborate on that? Logs have MDC with collection, core etc. That said there are some bugs where some logs are missing MDC but I've been addressing that lately; I'm nearly done. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105777#comment-17105777 ] Joel Bernstein commented on SOLR-11934: --- Just took a look and seems fine. The postlogs tool will still pickup new searchers as it was looking at the new searcher line that you left in. One thing that would be nice would be to tie new searchers to a collection as well as the core. Right now when doing log analytics you can't include newSearcher events with collection level reports. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103371#comment-17103371 ] ASF subversion and git services commented on SOLR-11934: Commit 67be31cdcf147630835ac9346ee72e934b124f39 in lucene-solr's branch refs/heads/branch_8x from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=67be31c ] SOLR-11934: Visit Solr logging, it's too noisy. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103370#comment-17103370 ] ASF subversion and git services commented on SOLR-11934: Commit 15be0db58696d379c6f7e1a6d0afa18dd7cdd43d in lucene-solr's branch refs/heads/master from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=15be0db ] SOLR-11934: Visit Solr logging, it's too noisy. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102025#comment-17102025 ] Erick Erickson commented on SOLR-11934: --- {quote} I wish it had timing info and ditched the too-detailed low {quote} I can make that happen. It's a bit ugly, but looks something like this: {code} if (log.isdebugEnabled()) { // This form dumps a bunch of messages like // Uninverting(_6(9.0.0):C177290:[diagnostics={lucene.version=9.0.0, java.vm.version=11.0.5+10, // java.version=11.0.5, timestamp=1588849987808, os=Mac OS X, // java.vendor=AdoptOpenJDK, os.version=10.15.4, java.runtime.version=11.0.5+10, // os.arch=x86_64, source=flush}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] // :id=ce2v4okod0tsdz9wxrl628t9s) // which I've never found particularly useful. // I'll take this comment out before pushing, this comment is here for discussion. // nocommit log.debug("{} Registered new searcher {} autowarm time: {} ms", logid, newSearcher, newSearcher.getWarmupTime()); } else if (log.isInfoEnabled()) { log.info("{} Registered new searcher autowarm time: {} ms", logid, newSearcher.getWarmupTime()); } {code} Or just get rid of all the low-level stuff altogether? Personally, I've never really found the low-level stuff useful and would happily get rid of it altogether. Once upon a time the Uninverting message was a red flag indicating that a field should have docvalues, but that disappeared. bq. See getInstance() and notice that... Ah, I was looking in a totally different place, I'll add that in. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102019#comment-17102019 ] David Smiley commented on SOLR-11934: - {quote}The one in SolrCore comes after autowarming and can trivially include the autowarm time with each message. WDYT? {quote} Agreed; keep the last one (SolrCore "Registered new searcher"). I wish it had timing info and ditched the too-detailed low level Reader metadata; alas. I could file an issue for that or preferrably just share with you a little bit of code. RE LogUpdateProcessorFactory and WARN See getInstance() and notice that it only looks at the log level to skip the URP altogether. It ought to check slowUpdateThresholdMillis >= 0 _as well_ but does not. I suppose another Jira might be proper but it's not a big deal to do here if you want because it's minor; avoids the weight of our process. RE the request log: Nevermind; the current logs there are fine. I just did a bit of experimenting to see. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101784#comment-17101784 ] Erick Erickson commented on SOLR-11934: --- {quote}I can live with losing the inner details of commit but need that one SolrIndexSearcher for opening it. We agree on this I think. {quote} It Depends (tm) on how valuable warmup times would be. The log message in SolrIndexSearcher comes very early in the process, long before autowarming is done. The one in SolrCore comes after autowarming and can trivially include the autowarm time with each message. WDYT? {quote}{quote}I really value LogUpdateProcessorFactory; I insist we keep it at INFO by default. {quote}{quote} I looked some more to see how much I disagreed and... well I don't disagree any more. Turns out that both the client app that started this Jira and the logs I have to evaluate are ill-behaved. The overwhelming number of updates are a single document, not to mention external commits and the like. So if people insist on following this pattern, they can bump the log level to WARN for LogUpdateProcessorFactory in the log4j2 config files. It won't be nearly as egregious if sane patterns are followed. {quote} BTW if you set it to WARN threshold, I can see a small bug where it skips the URP altogether and thus also misses logging the slow queries at WARN. {quote} I don't quite follow. Are you talking about what's done in {code:java} LogUpdateProcessorFactory.getLogStringAndClearRspToLog()?{code} where the response.toLog is cleared? That looks...confused, perhaps another JIRA? The response.toLog is referenced in the LogUpdateProcessorFactory and cleared in the method I just mentioned (if called). Then, back in SolrCore the information is (potentially) logged in requestLog and/or slowLog, both of which are bypassed if either of the calls in LogUpdateProcessorFactory are called since the response.toLog is cleared. Which means that slow logging and request logging aren't working at all. Which isn't true AFAIK, so I'd rather not try to address it in this JIRA. {quote} Lets also ensure that this log category is easily separated out so that it's easy to manipulate without messing with other logs in whatever class this happens to be in. {quote} Please raise another Jira if for that, I don't want too much feature creep here. I'm not quite sure what you mean by "this log category" here anyway, the requestLog? > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101692#comment-17101692 ] David Smiley commented on SOLR-11934: - Ugh. Trade-offs. I really value LogUpdateProcessorFactory; I insist we keep it at INFO by default. It goes to great lengths to generate a single succinct log message that is useful. If we keep anything related to indexing (includes commit), it needs to be this and I'm a big fan of it. BTW if you set it to WARN threshold, I can see a small bug where it skips the URP altogether and thus also misses logging the slow queries at WARN. I can live with losing the inner details of commit but need that one SolrIndexSearcher for opening it. We agree on this I think. Also I insist that by default, every query is logged. Lets also ensure that this log category is easily separated out so that it's easy to manipulate without messing with other logs in whatever class this happens to be in. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101305#comment-17101305 ] Erick Erickson commented on SOLR-11934: --- I was able to look at a few more logs and... it's ambiguous of course. For instance, I don't see _any_ commit_flush messages. I believe that's because there was no indexing going on. Looking at the data set I posted earlier, some interesting things: I checked locally, and a commit (triggered from autocommit settings that open a new searcher) logs the following: * SolrCore logging that it's registered a new searcher * SolrIndexWriter logging that it's calling setCommitData * QuerySenderListener with a message about "QuerySenderListener sending requests to..." * DirectUpdateHandler2 logging it's started and ended a commit_flush * SolrIndexSearcher telling us it's opened a new searcher So the above run had somewhere in the neighborhood of 140K commits (whether external or internal I don't think matters) that generated on the order of 840K messages. Oh, none of the searcher opening messages tell us anything about how long it took to open the searcher, that would be good to add as part of another JIRA. So by setting the log level to debug for all the messages relating to opening a searcher except one (I think the one in SolrIndexSearcher is the logical one, I'll beef it up a little) and setting the call in LogUpdateProcessorFactory to debug I can drop the number of log messages from this kind of app by roughly 2/3, which accomplishes the original intent. Note that the original problem was the observation that, in an app that indexed one doc at a time and committed it (yeah, I know how bad that is and I told them so) the logging was verbose, which squares with the observations above. This looks like an index-heavy kind of application with either external commits or short autocommit intervals. I'd _really_ like to see a query-heavy type application. My first impulse when I thought of dropping logging the query to debug rather than INFO was NOO!. But with the slow query log would that be so bad? Tossing that out for discussion, my personal feeling is that analyzing how Solr is performing often depends s much on query response time that logging what we do now for the query results is a must. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101220#comment-17101220 ] David Smiley commented on SOLR-11934: - Fair point about metrics; no need to log the particular timings I mentioned. As long as we log a searcher was opened -- might as well say how long it took. Arguably the flush of index writing buffers might be deemed too much of an internal detail as it is not in and of itself a visible thing; it's more of an internal event. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101206#comment-17101206 ] Erick Erickson commented on SOLR-11934: --- Why did the Zookeeper call get my attention? How about because my bifocals made me think I was on the line below? Or, I threw that one in to see if anyone was paying attention? Yeah, that's the ticket! That's totally unnecessary to change, sorry for the confusion. The thing I'm wondering about with HttpSolrCall is why there are so many admin/metrics calls. One of the reasons I'd _really_ like some more logs is to see if this is just an anomaly with this particular data set. For all I know, an external client is hammering Solr for dashboard reasons, in which case these are fine. And about commits. Again, I'd really like some more logs as it's quite possible that this is a result of commits being fired externally, in which case there's no reason to change the level. Let's claim this is internally generated for a minute. Do metrics give you the information you'd need? Or are you looking for individual anomalies/spikes and aggregated information isn't useful? Although the fact that it's in the handler implies that it's coming in externally, in which case it should stay at INFO. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101127#comment-17101127 ] David Smiley commented on SOLR-11934: - The asterisked ones to remove seem mostly reasonable at a glance. * Updated live nodes from ZooKeeper... This one is important to see when there is a change of nodes in the cluster. "35" is a small number; why did it get your attention? Just please ensure that if there is a commit, that there is a log message about this and how long it took. I know there's a distinction between the physical flush aspect of the commit and opening of the searcher; ideally timing is available for both and even separately logged. RE HttpSolrCall: as long as we log per HTTP request somehow. For SolrCore handlers, I know the SolrCore will do so but for node level handlers, we want to ensure there is a log as well. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101036#comment-17101036 ] Erick Erickson commented on SOLR-11934: --- Alas, the second set of logs does not have INFO enabled, so it's not what I need. > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I think we have way too much INFO level logging. Or, perhaps more correctly, > Solr logging needs to be examined and messages logged at an appropriate level. > We log every update at an INFO level for instance. But I think we log LIR at > INFO as well. As a sysadmin I don't care to have my logs polluted with a > message for every update, but if I'm trying to keep my system healthy I want > to see LIR messages and try to understand why. > Plus, in large installations logging at INFO level is creating a _LOT_ of > files. > What I want to discuss on this JIRA is > 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE > levels? > 2> Who's the audience at each level? For a running system that's functioning, > sysops folks would really like WARN messages that mean something need > attention for instance. If I'm troubleshooting should I turn on INFO? DEBUG? > TRACE? > So let's say we get some kind of agreement as to the above. Then I propose > three things > 1> Someone (and probably me but all help gratefully accepted) needs to go > through our logging and assign appropriate levels. This will take quite a > while, I intend to work on it in small chunks. > 2> Actually answer whether unnecessary objects are created when something > like log.info("whatever {}", someObjectOrMethodCall); is invoked. Is this > independent on the logging implementation used? The SLF4J and log4j seem a > bit contradictory. > 3> Maybe regularize log, logger, LOG as variable names, but that's a nit. > As a tactical approach, I suggest we tag each LoggerFactory.getLogger in > files we work on with //SOLR-(whatever number is assigned when I create > this). We can remove them all later, but since I expect to approach this > piecemeal it'd be nice to keep track of which files have been done already. > Finally, I really really really don't want to do this all at once. There are > 5-6 thousand log messages. Even at 1,000 a week that's 6 weeks, even starting > now it would probably span the 7.3 release. > This will probably be an umbrella issue so we can keep all the commits > straight and people can volunteer to "fix the files in core" as a separate > piece of work (hint). > There are several existing JIRAs about logging in general, let's link them in > here as well. > Let the discussion begin! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.
[ https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100889#comment-17100889 ] Erick Erickson commented on SOLR-11934: --- OK, getting back to this finally. I've analyzed one set of log files, with all the caveats that there may be special circumstances here. This is Solr 7x. I still have another set to go but wanted to start the discussion. Here's the summary of the classes that account for over 95% of the log messages in this sample. I categorized these mostly by taking fragments of text from the logging calls, but for some I had to do something else, for instance HttpSolrCall just logs the call, there's no surrounding text so I bucketed on path instead. It looks like this cluster was having some issues, so I'm not worried too much about SolrClientNodeStateProvider or RecoveryStrategy. But some of the others are...interesting. I propose all the ones I've marked with an asterisk be moved to DEBUG level. The alternative is to set the entire class to default to WARN level in the logging config files. Ones I'd like some comment on are marked with a ?. for instance, the ones in HttpSolrCall may be coming from outside? [~ab] do you have any off-the-top-of-your-head comments about the admin/metrics calls? {code:java} # calls %Level class 17,504 (pct: 0.010756) occurrences of WARN: o.a.s.c.s.i.SolrClientNodeStateProvider 22,528 (pct: 0.013843) occurrences of INFO: o.a.s.c.s.i.SolrClientNodeStateProvider 22,827 (pct: 0.014027) occurrences of INFO: o.a.s.c.c.ZkStateReader 24,330 (pct: 0.014950) occurrences of INFO: o.a.s.c.RecoveryStrategy 25,227 (pct: 0.015501) occurrences of INFO: o.a.s.c.S.Request 74,128 (pct: 0.045549) occurrences of INFO: o.a.s.s.HttpSolrCall 139,203 (pct: 0.085536) occurrences of INFO: o.a.s.u.SolrIndexWriter 140,589 (pct: 0.086388) occurrences of INFO: o.a.s.s.SolrIndexSearcher 142,723 (pct: 0.087699) occurrences of INFO: o.a.s.c.SolrCore 281,150 (pct: 0.172758) occurrences of INFO: o.a.s.c.QuerySenderListener 284,356 (pct: 0.174728) occurrences of INFO: o.a.s.u.DirectUpdateHandler2 381,468 (pct: 0.234401) occurrences of INFO: o.a.s.u.p.LogUpdateProcessorFactory Classes Of Interest (95% of log messages come from these classes) Lines in SolrCore 2, msg: ERROR 3, msg: core reload 4, msg: CLOSING SolrCore 15, msg: Updating index properties... * 1,047, msg: config update listener called for core * 1,078, msg: Opening new SolrCore at * 140,576, msg: Registered new searcher Lines in LogUpdateProcessorFactory * 381,468, msg: There's only one line Lines in SolrIndexWriter * 139,203, msg: Calling setCommitData with IW: ? All of theseLines in HttpSolrCall 1, msg: path=/admin/configs 2, msg: path=/admin/zookeeper 5, msg: path=/admin/info/logging 9, msg: Unable to write response, client closed connection or we are shutting down 20, msg: path=/admin/info/system 63, msg: path=/cluster/autoscaling/suggestions 78, msg: path=/admin/collections 90, msg: ERROR ? 2,122, msg: path=/admin/cores ? 71,828, msg: path=/admin/metrics Lines in ZkStateReader * 35, msg: Updated live nodes from ZooKeeper... * 22,792, msg: A cluster state change: Lines in RecoveryStrategy 1, msg: seconds before trying to recover again 4, msg: Replaying buffered documents. 287, msg: ERROR 289, msg: Updating version bucket highest from index after successful recovery. 290, msg: PeerSync Recovery was not successful - trying replication. 291, msg: Starting Replication Recovery. 560, msg: currentVersions size= 1,428, msg: startupVersions size= 1,672, msg: Failed to connect leader 1,814, msg: Replaying updates buffered during PeerSync. 2,101, msg: No replay needed. 2,105, msg: Starting recovery process. recoveringAfterStartup= 2,106, msg: Sending prep recovery command to Lines in QuerySenderListener * 140,575, msg: QuerySenderListener sending requests to Lines in DirectUpdateHandler2 3, msg: WARN * 1,983, msg: No uncommitted changes. Skipping IW.commit. * 141,186, msg: start (note, probably mostly commit flush, but may be multiple) * 141,187, msg: end_commit_flush Lines in SolrIndexSearcher *140,589, msg: Opening Lines in SolrClientNodeStateProvider 17,504, msg: WARN 22,528, msg: Error on getting remote info, trying again: {code} > Visit Solr logging, it's too noisy. > --- > > Key: SOLR-11934 > URL: https://issues.apache.org/jira/browse/SOLR-11934 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I think we have way too much INFO level logging. Or, perhaps