Re: Proposal for 1.2: built-in logging with django.core.log
On Oct 11, 7:57 pm, Hanne Moawrote: > On Sun, Oct 11, 2009 at 04:42, Simon Willison wrote: > > I'm keen to receive as much feedback on the implementation as possible. > > It is perfectly possible to set up logging in such a way that all the > logs for several programs wind up in the same file, or there are > duplicate entries. The first happens/happened if/when several programs > used the same interpreter. The second is user error. Been there been > stumped by both. Users need to be protected from both, somehow, but > how... Might be that a standard way for doing it in Django will > suffice for case 2. Will a warning/docs be needed for case 1? "Logs for several programs wind up in the same file": this could happen if multiple programs (scripts) use a FileHandler for file "foo.log" with mode "a" (append). Then the logs will end up in the same file because that's what's been asked for. I'm not sure what you mean when you say "several programs used the same interpreter" - as normally you have one Python interpreter per process. So if you are talking about one invocation of a Python executable, then I'm not sure what you mean by "programs", other than the executable running several scripts in turn, in a way such as: for scriptname in scriptnames: mod = __import__(scriptname) mod.main() In that case, since the Python logging system is per-process, it will be there across these multiple invocations and behave as if all of the scriptnames main() calls are part of the single process - which they are. Typically, duplicate messages are caused when multiple handlers are added to loggers. For example if one handler was added to both "foo" and the root logger, and if the "foo" logger propagates to its parent (the default setting) then you will get messages twice - once from "foo"'s handlers and once from the root's handlers. This often happens because people wrongly conflate loggers and handlers and think there has to be a one-to-one correspondence between them. Nowadays you only occasionally see questions on comp.lang.python about duplicate messages, so I'm not sure if this still causes confusion. I'm not sure what you can do to protect users from this, it's similar to how it's hard in Django to have code which is guaranteed to execute once and only once (see Simon's separate thread about this), but in logging it's easy to avoid. You just have to document how things work as best you can, and go from there. > When I first used the logging-library (2002?) I had a problem in > threaded programs, some exceptions were swallowed that shouldn't have > been, leading to rather mysterious behaviour. The exception-hierarchy > have been changed (a couple of times ;) ) since then so it might no > longer be an issue but I'd sleep better if there was tests for this > too. Vinay? What exception hierarchy are you talking about? Logging's policy on swallowing exceptions has been consistent throughout, and is that exceptions caused in logging (say, format string doesn't tie up with arguments) are always swallowed because you don't want an application to crash because of a logging error or misconfiguration. Exceptions such as SystemExit and KeyboardInterrupt should never be swallowed. Some other exceptions are handled by handleError() methods of handlers which sometimes retry (in the case of sockets) and at other times check the logging.raiseExceptions flag. If this is set to False (as it should be in production) then exceptions during handling are always swallowed - but these are exceptions due to the logging itself rather than exceptions in the application. Logging should be safe in threaded programs because threading.RLocks are used to synchronize access to module shared state and I/O between multiple threads. I don't know of any threading bugs - and remember, use of threading just by itself can sometimes lead to "mysterious behaviour" because it's easy to miss race conditions, deadlocks etc. One problem with writing tests for threading is that a threaded program can follow a whole bunch of execution paths, all of which are correct, because of OS scheduling being affected by other stuff running on the machine (e.g. backup jobs, network monitoring) which is outside your control. All of which means that having a defined outcome to compare against becomes problematic. Of course if you have any specific thread-safety issue relating to logging, then please raise an issue on the Python bug tracker and attach a small script which illustrates the problem. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at
Re: Proposal for 1.2: built-in logging with django.core.log
On Sun, Oct 11, 2009 at 04:42, Simon Willisonwrote: > I'm keen to receive as much feedback on the implementation as possible. It is perfectly possible to set up logging in such a way that all the logs for several programs wind up in the same file, or there are duplicate entries. The first happens/happened if/when several programs used the same interpreter. The second is user error. Been there been stumped by both. Users need to be protected from both, somehow, but how... Might be that a standard way for doing it in Django will suffice for case 2. Will a warning/docs be needed for case 1? When I first used the logging-library (2002?) I had a problem in threaded programs, some exceptions were swallowed that shouldn't have been, leading to rather mysterious behaviour. The exception-hierarchy have been changed (a couple of times ;) ) since then so it might no longer be an issue but I'd sleep better if there was tests for this too. Vinay? HM --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
I'm now tracking the logging implementation in ticket #12012: http://code.djangoproject.com/ticket/12012 I've started posting patches from my branch there. Again, this is just exploratory code at the moment but I'm keen to receive as much feedback on the implementation as possible. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sat, Oct 10, 2009 at 9:11 AM, Vinay Sajipwrote: > > > > On Oct 10, 1:56 pm, Simon Willison wrote: >> I just hacked in a logger that increments a counter for every message >> - it indicates that there are 158,527 SQL statements executed by the >> test suite, which means we should expect to add slightly over a second >> to the time taken for the overall unit tests based on the >> microbenchmark. > > You don't need to hack a logger to do this - you can just add a Filter > which passes everything but counts as it goes. > > Anyway, the numbers don't look too bad at the moment :-) > > Regards, > > Vinay Sajip > > > It's worth noting that though we should be considered with extreme effects were they to occur, over in Mountain View we have some friends working to make this not an issue for us :) . If Django is run under Unladen Swallow the result of all of these calls will be that both the logger function itself, and the code that calls it (for example the execute SQL query function) will be considered HOT by the interpreter and compiled down to machine code. There at that stage the compiler will be able to apply some nice heuristics and since this function never gets monkey patched or changed the compiler will be able to inline it. Since almost all log handlers will be trivial, the inlining will effectively give us 0-overhead logging. Compilers are awesome'ly yours, Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Oct 10, 1:56 pm, Simon Willisonwrote: > I just hacked in a logger that increments a counter for every message > - it indicates that there are 158,527 SQL statements executed by the > test suite, which means we should expect to add slightly over a second > to the time taken for the overall unit tests based on the > microbenchmark. You don't need to hack a logger to do this - you can just add a Filter which passes everything but counts as it goes. Anyway, the numbers don't look too bad at the moment :-) Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Oct 10, 1:41 pm, Simon Willisonwrote: > Oops, yes you're right - I misread the numbers. A 2.5 second different > would correspond to around 350,000 log messages (assuming the rate of > 130,000/second from my microbenchmark) - I have no idea how many > statements the unit tests execute in total. Once I've written a bit > more code of course I could always find out using a logger... I just hacked in a logger that increments a counter for every message - it indicates that there are 158,527 SQL statements executed by the test suite, which means we should expect to add slightly over a second to the time taken for the overall unit tests based on the microbenchmark. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Oct 10, 12:55 pm, Luke Plantwrote: > > master: 314.442s > > logging: 317.096s > > > Since there's nothing at all in my logging code that will speed > > anything up, I'm putting that down to chance (I did run the suite a > > few times first to "warm up" the VM I used, but like I said, this > > is pretty much the opposite of science). > > Those numbers are the time in seconds, right? In which case your > branch is slower, not faster, as I think you are implying. It could > still be down to chance, but its only 1% which would be an acceptable > hit for me anyway. Oops, yes you're right - I misread the numbers. A 2.5 second different would correspond to around 350,000 log messages (assuming the rate of 130,000/second from my microbenchmark) - I have no idea how many statements the unit tests execute in total. Once I've written a bit more code of course I could always find out using a logger... Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Saturday 10 October 2009 07:31:31 Simon Willison wrote: > master: 314.442s > logging: 317.096s > > Since there's nothing at all in my logging code that will speed > anything up, I'm putting that down to chance (I did run the suite a > few times first to "warm up" the VM I used, but like I said, this > is pretty much the opposite of science). Those numbers are the time in seconds, right? In which case your branch is slower, not faster, as I think you are implying. It could still be down to chance, but its only 1% which would be an acceptable hit for me anyway. Luke -- "Where a person wishes to attract, they should always be ignorant." (Jane Austen) Luke Plant || http://lukeplant.me.uk/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Oct 9, 4:54 pm, Jacob Kaplan-Mosswrote: > One big question I have is about performance: have you done any a/b > testing against trunk to see how much adding logging -- especially > upon every SQL query -- impacts performance? > > For me, at least, performance is going to be the critical factor. It's > clear that even a do-nothing logger will add some overhead. If we're > talking fractions of percents here, fine... but if there's a > measurable performance hit that's going to be a big problem, and it's > probably best to be tracking speed as we go along here. Nothing scientific yet - I've run the full unit test and got these results: master: 314.442s logging: 317.096s Since there's nothing at all in my logging code that will speed anything up, I'm putting that down to chance (I did run the suite a few times first to "warm up" the VM I used, but like I said, this is pretty much the opposite of science). Microbenchmarks I performed with the timeit module suggest we can log to a NullHandler logger at a rate of over 100,000 log messages a second, which is why I haven't paid performance much attention since running the microbenchmark. Here's how to replicate it (using Python 2.6, which lets you pass a callable to the timeit.timeit function): import logging class NullHandler(logging.Handler): def emit(self, record): pass logger = logging.getLogger('django') logger.addHandler(NullHandler()) logger.propagate = False time_for_a_million_messages = timeit.timeit(lambda: logger.info('a log message'), number=100) messages_per_second = 1 / (time_for_a_million_messages / 100) I ran that just now and got 137,223 messages per second. I figure even a really heavy Django page will execute maybe a couple of hundred SQL queries, which would add about 0.0015 of logging time if no handler was hooked up. Hence why I'm not too worried. Is there a reasonable benchmark I can be using for this? The test suite run time doesn't seem particularly finely grained. I know Jeremy Dunck was talking about this a while back. Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Fri, Oct 9, 2009 at 4:33 AM, Simon Willisonwrote: > For anyone interested in following this, I've just started a > (currently experimental) branch on GitHub: > > http://github.com/simonw/django/tree/logging One big question I have is about performance: have you done any a/b testing against trunk to see how much adding logging -- especially upon every SQL query -- impacts performance? For me, at least, performance is going to be the critical factor. It's clear that even a do-nothing logger will add some overhead. If we're talking fractions of percents here, fine... but if there's a measurable performance hit that's going to be a big problem, and it's probably best to be tracking speed as we go along here. Jacob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Fri, Oct 9, 2009 at 6:12 AM, Luke Plantwrote: > > On Friday 09 October 2009 10:33:28 Simon Willison wrote: >> For anyone interested in following this, I've just started a >> (currently experimental) branch on GitHub: >> >> http://github.com/simonw/django/tree/logging > > Is there some easy way to see the patch? Preferably a link which will > just show the diff between the latest of your branch and trunk, but > failing that, a recipe of commands using git (for those of us who > haven't bothered to learn git properly yet). > > Cheers, > > Luke > > -- > "We may not return the affection of those who like us, but we > always respect their good judgement." -- Libbie Fudim > > Luke Plant || http://lukeplant.me.uk/ > > > > Don't we wish it was possible for github to show a diff between two branches... :( Just venting'ly yours, Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Oct 9, 11:12 am, Luke Plantwrote: > >http://github.com/simonw/django/tree/logging > > Is there some easy way to see the patch? Preferably a link which will > just show the diff between the latest of your branch and trunk, but > failing that, a recipe of commands using git (for those of us who > haven't bothered to learn git properly yet). Once things are a bit further along I'll start posting patches to the ticket tracker, but for the moment you can see the changelog here: http://github.com/simonw/django/commits/logging There are only two relevant commits: http://github.com/simonw/django/commit/b5227e1ac1d70b1f936ee69d6e347d8148df461e http://github.com/simonw/django/commit/65ff505718538124ee2979fb60dfe1a37ca1b8bc But since that means you have to patch together what's going on from the two commits, I've pasted a full diff to trunk here: http://dpaste.com/hold/104862/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Friday 09 October 2009 10:33:28 Simon Willison wrote: > For anyone interested in following this, I've just started a > (currently experimental) branch on GitHub: > > http://github.com/simonw/django/tree/logging Is there some easy way to see the patch? Preferably a link which will just show the diff between the latest of your branch and trunk, but failing that, a recipe of commands using git (for those of us who haven't bothered to learn git properly yet). Cheers, Luke -- "We may not return the affection of those who like us, but we always respect their good judgement." -- Libbie Fudim Luke Plant || http://lukeplant.me.uk/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
For anyone interested in following this, I've just started a (currently experimental) branch on GitHub: http://github.com/simonw/django/tree/logging So far I've added a 'django' logger which is silent by default, and hooked up a new CursorLoggingWrapper to log all SQL statements from the ORM to it. I've also modified the runserver command so you can use the following to watch the execute SQL flow past on the console: ./manage.py runserver --loglevel=django.db.sql:info None of the implementation code is intended to be production quality yet (I'm just experimenting for the moment), but I'm very interested in feedback on the approach. Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 29, 2:25 pm, Ivan Sagalaevwrote: > LOGGING = { > 'errors': { > 'handler': 'django.logging.FileLogger', # WatchedFileLogger copy > 'filename': '...', > 'level': 'debug', > }, > 'maintenance': { > 'handler': 'logging.handlers.HTTPHandler', > 'host': '...', > 'url': '', > 'format': '' > }, > > } > On one of my projects, I had a YAML configuration which I loaded into a dict using PyYAML, then configured logging from that. I propose to generalise that solution to add a new function to logging.config called dictConfig() which takes a dict as an argument and uses that to configure logging. In practice the dict can of course come from a literal in the code (your example), or a JSON or YAML string. The important thing is to get the dict schema right and to make it easy to generalise for user-defined handlers etc. So I will be posting a proposal on python-dev to get comments and suggestions, and providing a standalone implementation so that people have something specific to comment on. Following a period for feedback (and assuming the python devs don't shoot the idea down in flames), I propose to actually implement this functionality in Python logging itself, perhaps in 2.7/3.2. However, to make it easy to use in Django and other systems which have to work with older versions of Python, the functionality will be in the form of a class (working title: DictConfigurator) which can just be copied into Django (as has been suggested for other bits of logging, such as WatchedFileHandler and NullHandler. Comments welcome. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Oct 4, 9:22 pm, Ivan Sagalaevwrote: > I see one problem with explicitly calling a function from settings.py. > This will execute instantiation of handlers and all the user staff very > early when settings aren't fully configured yet. This prevents a user to > have a handler that will use Django models for example. In general > settings should be as dumb as possible for easy bootstrapping. This is > why all those middleware/context processors/backends etc are strings and > not actual callables. I can't say I'm too fond of behind-the-scenes magic and that's why I'd prefer an explicit call to configure logging; otherwise users might run into limitations such as the fact that you already can't import certain things in settings.py without running into circular import errors. I don't know how common a use case it might be to use Django models in logging handlers. Regards, Vinay --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Oct 4, 8:55 pm, Ivan Sagalaevwrote: > Uhm... No this is not about hand-holding or something like that. I > propose a format that is shorter and more convenient than the > ConfigParser's. This is the point. > > > My reservation with Django's own take on it is simply > > that it goes against TOOWTDI and the Zen of Python, a little at least. > > Yes, this is the bad part about it. However the Zen of Python is not a > dogma and its application is subjective. I think my proposal still > stands and the decision is left for core devs. As for me I'll most > probably play with the implementation of parsing of this format for our > own projects anyway. I'll file it as a ticket if I would still like it > by that time :-). Fair enough, I don't really object to having a format defined in settings.py, but I think it's a bad idea if Django uses that configuration automagically, as I said elsewhere on this thread. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
Vinay Sajip wrote: > We should provide a configure_logging(dict) utility function which > takes a dict in an agreed format (such as Ivan's suggestion). Then in > settings.py users can invoke configure_logging(LOGGING) after > defining the configuration right there, in settings.py itself, or > obtaining the dict from any other source. This avoids any "magic" (by > LOGGING being automagically processed) and allows advanced users to > configure "by hand" if they so wish. I see one problem with explicitly calling a function from settings.py. This will execute instantiation of handlers and all the user staff very early when settings aren't fully configured yet. This prevents a user to have a handler that will use Django models for example. In general settings should be as dumb as possible for easy bootstrapping. This is why all those middleware/context processors/backends etc are strings and not actual callables. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
Sorry for taking a bit long to respond. Looks like we aren't disagree much anyway but I had to get some thoughts off my chest :-). Vinay Sajip wrote: > It's similar to Django's support for, say, simplejson. I think it's > reasonable for Django to alias WatchedFileHandler so that it's > available either bound to logging's own implementation (in > sufficiently recent versions of Python) or else a copy in Django's own > code. Then people can use it if they want to, even in older Python > versions. Agreed. > I have no big problem with a configuration scheme such as you suggest > - if it's felt that a lot of Django users are not Python-savvy enough > and need some hand-holding Uhm... No this is not about hand-holding or something like that. I propose a format that is shorter and more convenient than the ConfigParser's. This is the point. > My reservation with Django's own take on it is simply > that it goes against TOOWTDI and the Zen of Python, a little at least. Yes, this is the bad part about it. However the Zen of Python is not a dogma and its application is subjective. I think my proposal still stands and the decision is left for core devs. As for me I'll most probably play with the implementation of parsing of this format for our own projects anyway. I'll file it as a ticket if I would still like it by that time :-). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 30, 1:20 pm, Russell Keith-Mageewrote: > > I was thinking that a calming, motherly "there there, it's all right, > the boogeymonster isn't real" would do the trick :-) I'll see what I can do... > > I suppose this is a big part of the problem. The logging module isn't > a trivial API, and you really need to understand how it works before > you can use it effectively. What is needed is a really good tutorial; > what we have is a reference guide with a little bit of a tutorial > tacked on the front. The reference is needed, but the tutorial is much > more important from the point of view of getting the message out and > encouraging people to use the API. > You're absolutely right about this. Hopefully working on this with Doug Hellmann (from whose PyMOTW the initial tutorial bits came) will yield some improvements. > >http://plumberjack.blogspot.com/2009/09/python-logging-101.html > > I saw this, and yes, it's a good start. However, it (or a version of > it) needs to be formalized and in the official docs, not lingering on > a blog somewhere. You're right, and that will (hopefully) happen at some point not too far in the future. > Agreed, it's important that the simple use case is demonstrated. > However, in the docs, the simple example is then used as a staging > ground for demonstrating rotating file loggers and other things that a > simple example doesn't need. > > Logging is a complex topic. A simple example doesn't provide the > complexity that is required to demonstrate the strengths of the API. > The simple example is useful for demonstrating the "hello world" case, > but not for demonstrating more complex features. I get it now, thanks for clarifying. > On the subject of examples, another suggestion: I'm wary of examples > that try to be too smart. Take the logging level demonstration (the > 5th code snippet in 16.6.1.1). This code very cleverly writes a script > that can take a command line argument and turn it into a logging > level, but in being clever, it obscures the point of the example - > that you can output log messages at any level, but the configuration > determines which ones are output. When you're trying to demonstrate > the simple stuff, the less extraneous detail, the better. The next > example (which demonstrates multiple loggers) is much clearer - it's > obvious from first inspection what the code is doing. Okay. I'll see about sorting this out as part of a larger makeover. > To avoid a dependency on the logging module, thereby keeping Adrian happy :-) > > Seriously - I really do want to find out the nature of Adrian's > objection to logging (if it is even current). I'd rather have an > actual discussion than dance around a strawman. It would perhaps be a problem if it was an external dependency, rather than an integral part of Python. You're right, though - let's see what he has to say. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Tue, Sep 29, 2009 at 10:59 PM, Vinay Sajipwrote: > > On Sep 29, 1:53 pm, Russell Keith-Magee > wrote: >> >> However, since you're interested in feedback, my suggestion would be >> to look at every defense you've made of logging in this thread (and >> any other threads where you've had similar arguments), and work out >> why the current docs have allowed those viewpoints to be established >> as apparent fact. Some examples: >> >> * Acknowledge that there is some Java heritage, but point out that >> this doesn't mean it's a bad thing, and that there is a lot that >> _isn't_ Java based about Python's logger. >> > > That's easier to do when people raise specific points, rather than > talk about Java heritage in an arm-waving way, as if it's an offshoot > of the Black Death ;-) I was thinking that a calming, motherly "there there, it's all right, the boogeymonster isn't real" would do the trick :-) >> * Highlight the important architectural picture. As you noted in >> another reply - the logger and the handler are quite separate, and >> this gives a lot of power. However, the existence and significance of >> that architectural separation isn't really a major feature of the >> current docs. At present, the architectural bits are buried inside API > > That's because the docs are really pitched mainly as a a reference > guide. I suppose this is a big part of the problem. The logging module isn't a trivial API, and you really need to understand how it works before you can use it effectively. What is needed is a really good tutorial; what we have is a reference guide with a little bit of a tutorial tacked on the front. The reference is needed, but the tutorial is much more important from the point of view of getting the message out and encouraging people to use the API. >> discussion, but understanding this architecture is important if you're >> going to understand why logging works the way it does, and understand >> that logging isn't just putting lines into a file. > > I've recently created a blog about Python logging, where I talk about > logging from first principles and try to show why the design of Python > logging is as it is. It's not perfect, but it's a start. > > http://plumberjack.blogspot.com/2009/09/python-logging-101.html I saw this, and yes, it's a good start. However, it (or a version of it) needs to be formalized and in the official docs, not lingering on a blog somewhere. >> * Make the simple example actually simple. IMHO, a single-file simple >> logging example is good for exactly 2 things: >> - showing how to configure the simplest possible case of logging >> - explaining the "why don't I have any output" problem. > > I'm not sure what you're getting at. Sometimes, those two things is > all that people want to know at that time. Agreed, it's important that the simple use case is demonstrated. However, in the docs, the simple example is then used as a staging ground for demonstrating rotating file loggers and other things that a simple example doesn't need. Logging is a complex topic. A simple example doesn't provide the complexity that is required to demonstrate the strengths of the API. The simple example is useful for demonstrating the "hello world" case, but not for demonstrating more complex features. On the subject of examples, another suggestion: I'm wary of examples that try to be too smart. Take the logging level demonstration (the 5th code snippet in 16.6.1.1). This code very cleverly writes a script that can take a command line argument and turn it into a logging level, but in being clever, it obscures the point of the example - that you can output log messages at any level, but the configuration determines which ones are output. When you're trying to demonstrate the simple stuff, the less extraneous detail, the better. The next example (which demonstrates multiple loggers) is much clearer - it's obvious from first inspection what the code is doing. >> > They (logging and signals) are two different things. Python logging >> > allows you to consider the dimensions "What happened?", "Where did it >> > happen?", "How important is it?" and "Who wants to know?" >> > intelligently, and in particular it treats "What happened" and "Who >> > wants to know?" orthogonally. You get a lot of ways of getting >> > information to *people* whereas signals is more about letting *code* >> > know what's going on. >> >> Granted, although the two aren't mutually exclusive. After all, the >> way you let someone know "what happened" is with code; It isn't hard >> to think of a setup where we emit a signal everywhere that we might >> want to log, and then attach a logging signal handler to those signal. >> >> I'm not suggesting that this would be a good architecture - merely a >> possible one. > > I agree - it's possible, but perhaps not the most simple or intuitive. > If you can log something directly from where you are, why send a >
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 29, 11:29 pm, Simon Willisonwrote: > > Here's my understanding of what we need to do for Django. > > 1. Create a 'django' logger somewhere in the framework that will > always be executed if part of Django has been imported - would the > django/__init__.py file work? Configure that with a NullHandler, and > set it NOT to propagate events to the root logger. That way we can > start using 'django.*' loggers throughout the framework but none of > the messages will be sent anywhere without further configuration. The > code should look something like this: > > import logging > > class NullHandler(logging.Handler): > def emit(self, record): > pass > > logger = logging.getLogger('django') I tend to prefer the pattern logging.getLogger(__name__) in general, but this is equivalent. > logger.addHandler(NullHandler()) > logger.propagate = False I'm not sure about propagate=False here, though I can see why you might want to set it to this. Generally, many users set handlers up on the root logger and expect libraries they use to feed through to those handlers: setting propagate to False prevents this, and would require handlers to be set just for Django. This might not be a problem when Django is being run as a server, but if Django is being used as a library in any sense then this would not be the best thing to do. If you set the 'django' logger's level to e.g. WARNING, then DEBUG and INFO messages would never get through, but those >= WARNINGS would. You could of course set the bar higher, say to ERROR. This would prevent any e.g. DEBUG messages from Django code being generated, no matter what logger levels were set to elsewhere in the logger namespace. > NullHandler is a generally useful class (I'm surprised it's not part > of the core logging library), so we should probably put that in > django.utils.log. The NullHandler is there purely to prevent the one- > off "you haven't configured a handler yet" error that gets sent to > sys.stderr. Mea culpa. Leaving NullHandler out was an oversight originally, because I hadn't thought carefully enough about libraries that use logging working with applications that don't use or configure logging. It's now part of the stdlib, but it's not in Python 2.6 or earlier. > 2. Start using loggers within the Django framework itself. The most > obvious place to start is SQL queries - at the point where we execute > SQL, we should do this: > > logging.getLogger('django.db.sql').debug(sql_to_execute) Normally I do logger = logging.getLogger(__name__) followed by logger.debug(...) or similar. In this case, though, Perhaps the specific name is better because it's easier to remember than 'django.db.backends.XXX'. People familiar with Django internals would probably prefer the module name as it pinpoints where the event was raised. > > That logs the SQL to a 'django.db.sql' logger - which is a child of > the 'django' logger and hence will be swallowed by the NullHandler. > > Another place we could use logging is to record cache gets and misses: > > logging.getLogger('django.core.cache').debug('Cache miss for key %s', > key) > > And for sending e-mail: > > logging.getLogger('django.core.mail').info('Mail sent to %s', > to_email) > > The logger names reflect the structure of Django up to a point > (django.db) but I don't think there's anything wrong with straying > from that convention (django.db.sql for example) provided the prefixes > are sensible. > See my comments earlier about choice of logger names. > 3. Add a bunch of syntactic sugar for making log messages visible at > various points within Django. For example: > > ./manage.py runserver --log-level=django:info > > Would cause the runserver to first configure the 'django' logger to > print all messages of level 'info' and higher to stdout. > > ./manage.py runserver --log-level=django:info --log- > level=django.db:debug > > Same as above, but also configures django.db messages of debug or > higher to display (I'm sure the command line syntax could be > improved). > > By default, runserver would display no log messages at all. > > The same arguments could also work for ./manage.py test: > > ./manage.py test --log-level=django.db.sql:debug > > Hmm... maybe the --log-level argument could work for ALL management > commands. > > My philosophy here is that log messages should be ignored unless > explicitly told otherwise. Django gets run in various different ways - > runserver, test and in deployment under mod_wsgi/FastCGI/etc - and > it's not at all obvious what the default log output behaviour should > be. As long as we make it extremely easy to see log messages if we > want them I think having them off by default makes sense. It also > ensures no surprises for people upgrading from 1.1 to 1.2. > Agreed. > 4. Add a way to configure loggers in Django's settings.py file, in > particular for use in production. I'm not sure if these settings > should be ignored when running under
Re: Proposal for 1.2: built-in logging with django.core.log
So I've read through the Python Logging 101 (really useful) and the logging docs: http://plumberjack.blogspot.com/2009/09/python-logging-101.html http://docs.python.org/library/logging.html Here's my understanding of what we need to do for Django. 1. Create a 'django' logger somewhere in the framework that will always be executed if part of Django has been imported - would the django/__init__.py file work? Configure that with a NullHandler, and set it NOT to propagate events to the root logger. That way we can start using 'django.*' loggers throughout the framework but none of the messages will be sent anywhere without further configuration. The code should look something like this: import logging class NullHandler(logging.Handler): def emit(self, record): pass logger = logging.getLogger('django') logger.addHandler(NullHandler()) logger.propagate = False NullHandler is a generally useful class (I'm surprised it's not part of the core logging library), so we should probably put that in django.utils.log. The NullHandler is there purely to prevent the one- off "you haven't configured a handler yet" error that gets sent to sys.stderr. 2. Start using loggers within the Django framework itself. The most obvious place to start is SQL queries - at the point where we execute SQL, we should do this: logging.getLogger('django.db.sql').debug(sql_to_execute) That logs the SQL to a 'django.db.sql' logger - which is a child of the 'django' logger and hence will be swallowed by the NullHandler. Another place we could use logging is to record cache gets and misses: logging.getLogger('django.core.cache').debug('Cache miss for key %s', key) And for sending e-mail: logging.getLogger('django.core.mail').info('Mail sent to %s', to_email) The logger names reflect the structure of Django up to a point (django.db) but I don't think there's anything wrong with straying from that convention (django.db.sql for example) provided the prefixes are sensible. 3. Add a bunch of syntactic sugar for making log messages visible at various points within Django. For example: ./manage.py runserver --log-level=django:info Would cause the runserver to first configure the 'django' logger to print all messages of level 'info' and higher to stdout. ./manage.py runserver --log-level=django:info --log- level=django.db:debug Same as above, but also configures django.db messages of debug or higher to display (I'm sure the command line syntax could be improved). By default, runserver would display no log messages at all. The same arguments could also work for ./manage.py test: ./manage.py test --log-level=django.db.sql:debug Hmm... maybe the --log-level argument could work for ALL management commands. My philosophy here is that log messages should be ignored unless explicitly told otherwise. Django gets run in various different ways - runserver, test and in deployment under mod_wsgi/FastCGI/etc - and it's not at all obvious what the default log output behaviour should be. As long as we make it extremely easy to see log messages if we want them I think having them off by default makes sense. It also ensures no surprises for people upgrading from 1.1 to 1.2. 4. Add a way to configure loggers in Django's settings.py file, in particular for use in production. I'm not sure if these settings should be ignored when running under other circumstances (./manage.py test etc) in favour of the command line option, or if they should still take effect. I quite liked Ivan's suggestion: LOGGING = { 'django.db.sql': { 'handler': 'django.logging.FileLogger', # WatchedFileLogger copy 'filename': '/tmp/django-sql.log', 'level': 'debug', }, 'django.core.cache': { 'handler': 'logging.handlers.HTTPHandler', 'host': '...', 'url': '', 'format': '' }, 'django.errors': { 'handler': 'logging.handlers.SMTPHandler', ... } } django.errors is where we get to replace the current "e-mail 500 errors to an ADMIN" functionality. There should also be an option for advanced users to ignore the declarative syntax entirely and just provide a bunch of code that configures logging in the traditional way: import logging django_logger = logging.getLogger('django') django_logger.setLevel(...) django_logger.addHandler(...) # etc I'm not sure where that custom code should live - right there in settings.py? At any rate, I think it's vital that people can use the low level logging configuration API if they want to. I'd rather not encourage people to use the logging module's support for .ini style configuration, just because Django already has its own configuration standards. That's not to say people can't call that from their own code if they want to, but I don't think we should emphasise that ability. 5. Document how users should go about implementing their own loggers. I think we should tell people NOT to put them within the 'django' logger
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 29, 6:07 pm, Simon Willisonwrote: > Thanks for dropping by. Firstly, I'm sorry for disparaging of your > work. I mis-spoke when I said the logging module wasn't "nicely > designed" - I obviously don't believe that, since I'm very keen on > including it in Django. I should have picked my words much more > carefully. > > The point I intended to make is that using the logging module > correctly requires quite a lot of knowledge on the part of the > developer - there's a lot to understand in there. When I talk about a > "thin wrapper" for Django really I'm really just talking about smart > defaults - ensuring that Django users can start using logging without > needing to know anything more than "import this and call the log.error > () method", but that the full power of the logging library is still > available once they need it. > > So again, I apologise for my extremely poor choice of words - that was > decidedly unclassy. I'm excited that you're interested in helping us > integrate logging with Django in the best possible way. > No worries. Hope we can get something going soon. Best regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 29, 2:00 pm, Waylan Limbergwrote: > The hard part is that basicConfig only works like that back to Python > 2.4 yet Django supports 2.3. When I added logging to Python-Markdown, > this was the hardest part. Figuring out how to configure logging so > that it works in 2.3 as well. The documentation is not exactly helpful > in that regard. > Did you ask any questions on python-list? How were they responded to? The answer is pretty simple, just like in parts of Django - if it's not supported by an older version of Python, you just try and copy the relevant bits from the later version of Python (or simplejson, say). I generally monitor both python-list regularly for logging issues and questions, and numerous other people on the list also weigh in when people ask logging-related questions. So if in doubt, ask - you should generally get a friendly and hopefully helpful response. > In fact, it was for this very reason that we added our own wrapper > around logging. It didn't seem reasonable for our users to go through > the same pain that we did. Sure we got a few things wrong at first, > but with the help of a few people in the community we worked those out > and our wrapper seems to work ok now. Yes - ok - I get the sense it > could be better. > > Ever since then, any mention of logging leaves a bad taste in my > mouth. Perhaps if I was working only in 2.6 or such, this wouldn't be > an issue, but we have promised support back to 2.3. > > Of course, it is possible that I'm missing something obvious. If you had asked around on python-list and got fobbed off, perhaps a bad taste in your mouth would be understandable. But there's no reason for it - the Python developers didn't set out to make things difficult for you, and the community is generally very helpful. In fact it was the help I got when I started with Python that motivated me to contribute something back in the first place. I'm also responsive to feature enhancement requests added via bugs.python.org - while not every suggestion gets accepted, there's been a steady stream of incremental improvements to the logging package and documentation thanks to specific suggestions by members of the Python community (not to mention patches). Remember, these are all volunteer projects and while we all want to help, it's easier to latch onto specific problems and issues rather than work with an amorphous goal which begins and ends with "improve things". Notice how popular Stack Overflow is these days - that's an example of how you can get help by being specific and asking targeted questions. So please, raise any questions and/or issues on python- list, bugs.python.org, Stack Overflow, etc. Now I'll get off my soapbox. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 18, 5:21 pm, Eric Holscherwrote: > I have looked into Logging before for another project, and I found that > SQLAlchemy's support seemed to be a pretty good model to follow. They define > all of their loggers under the sqlalchemy namespace, and then you can > configure different handlers for different things[1]: > I think that this would be necessary to have in Django, so that for > instance, I could listen to the django.orm logs, and not the django.http, or > listen to them with different handlers/levels. > The approach I'd recommend is to have loggers at module level, and more granular than that if and where needed. We don't have to follow this blindly and tool up everywhere all at once to the same extent - it can progress naturally as and when we tackle issues and feel that logging is a part of continuous improvement. I think that's the spirit in which Simon initially posted. >From a user's point of view, it should be easy for them to turn the verbosity up or down at a module level at least (for example, an individual view module somewhere in Django). By which I mean, it'd be most natural to express "where" in the application you want verbosity turned up or down by thinking about apps and individual modules in those apps. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 29, 9:36 am, Vinay Sajipwrote: > Simon, I'm the author of Python's logging package. Sorry for the delay > in replying, I've been away from this list awhile. I think the "Java > heritage shines through" is just FUD. Hi Vinjay, Thanks for dropping by. Firstly, I'm sorry for disparaging of your work. I mis-spoke when I said the logging module wasn't "nicely designed" - I obviously don't believe that, since I'm very keen on including it in Django. I should have picked my words much more carefully. The point I intended to make is that using the logging module correctly requires quite a lot of knowledge on the part of the developer - there's a lot to understand in there. When I talk about a "thin wrapper" for Django really I'm really just talking about smart defaults - ensuring that Django users can start using logging without needing to know anything more than "import this and call the log.error () method", but that the full power of the logging library is still available once they need it. So again, I apologise for my extremely poor choice of words - that was decidedly unclassy. I'm excited that you're interested in helping us integrate logging with Django in the best possible way. Thanks, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 29, 1:53 pm, Russell Keith-Mageewrote: > > First off - let me reinforce that I'm in your camp here - I like > Python's logger, and I think we should be adding logging to Django. > Any hesitation I have expressed is mostly a function of institutional > inertia, especially with regards to Adrian's historical position on > logging. > That's great. I agree about adding logging to Django, and would like to help if I can. It would be good to understand what underlies Adrian's historical position on logging. > However, I would point out that IMHO, FUD is an accurate description > of the state of play - though probably not in the way you probably > meant. > > Python's logging api _looks_ a lot like log4j in parts. This is at > least partially because there's a limit to how many ways you can > express 'log.debug()' before you start to copy. However, as a result, > there's a lot of Fear, Uncertainty and Doubt as to whether a framework > that apparently has Java heritage is going to be any good in Python. > Don't forget that a lot of us (myself included) got into writing > Python to get away from the stupidities of the Java world. Those scars > are deep, and aren't going away in a hurry. Speaking personally, log4j > is responsible for a lot of those scars, due in no small part to the > abysmal documentation for that project. > We're on the same page, I think. Python's similarity to log4j is, I feel, skin deep. Just as our having features in common with monkeys and apes doesn't *make* us monkeys and apes, so also with Python logging and log4j. Compare and contrast: log4j is around 160 source files and 16K SLOC, whereas Python logging is 3 source files and under 1500 SLOC! Notice the order of magnitude difference. Functionally, Python logging pretty much provides the same functionality as log4j, but it's a lot simpler internally. Python logging is *not* a port of log4j, is written in as Pythonic a way as I know how (given that it was written when it was), and got a lot of peer review from the smart people on python-dev before going in (and got changed here and there to satisy concerns raised during the review process). Nevertheless, FUD (and I think I did mean it in that sense - Fear, Uncertainty and Doubt) needs to allayed. I'll be happy to try and do this, please feel free to ask any specific questions or make any specific criticisms and I'll do my best to deal with them. > My comment was actually directed at Django's documentation, which is > currently silent on the issue of logging - and probably shouldn't be. Right. > However, since you're interested in feedback, my suggestion would be > to look at every defense you've made of logging in this thread (and > any other threads where you've had similar arguments), and work out > why the current docs have allowed those viewpoints to be established > as apparent fact. Some examples: > > * Acknowledge that there is some Java heritage, but point out that > this doesn't mean it's a bad thing, and that there is a lot that > _isn't_ Java based about Python's logger. > That's easier to do when people raise specific points, rather than talk about Java heritage in an arm-waving way, as if it's an offshoot of the Black Death ;-) If you want an example of Python written in the Java style, look at Apache QPid. Python logging ain't that. > * Highlight the important architectural picture. As you noted in > another reply - the logger and the handler are quite separate, and > this gives a lot of power. However, the existence and significance of > that architectural separation isn't really a major feature of the > current docs. At present, the architectural bits are buried inside API That's because the docs are really pitched mainly as a a reference guide. > discussion, but understanding this architecture is important if you're > going to understand why logging works the way it does, and understand > that logging isn't just putting lines into a file. > I've recently created a blog about Python logging, where I talk about logging from first principles and try to show why the design of Python logging is as it is. It's not perfect, but it's a start. http://plumberjack.blogspot.com/2009/09/python-logging-101.html > * Make the simple example actually simple. IMHO, a single-file simple > logging example is good for exactly 2 things: >- showing how to configure the simplest possible case of logging >- explaining the "why don't I have any output" problem. I'm not sure what you're getting at. Sometimes, those two things is all that people want to know at that time. > Tasks like configuring the logger to use a rotating file handler are > important, but can wait for much later - once issues of basic usage > and architecture have been established. Sure, and I hope with Doug Hellmann's input we can get the Python logging docs to be laid out in a more logical order and imbued with more clarity. Doug's PyMOTW series sets a high bar for
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 29, 2:35 pm, Ivan Sagalaevwrote: > Forgot one thing. As it stands now Django has this "historic" behavior > when it imports and executes settings module twice. This results in > breakage when you setup loggers and handlers by hand. We now circumvent > this by having a helper function that memoizes configured loggers and > call it from settings.py. Having a declarative config we can hide this > inside of Django and not scare users. This is how the way basicConfig works actually helps - it's a no-op if you call it again. (It wasn't done that way for use with Django particularly - just to make life easier for newbies and casual users. Sure, an internal function would hide this from users. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 29, 2:25 pm, Ivan Sagalaevwrote: > > Using WatchedFileHandler is a safe default because it works as > FileHandler, just doesn't break with logrotate. I don't know of any > disadvantages of WatchedFileHandler before the old FileHandler. So I > don't think there's much value in giving people this choice in settings > because non-default behavior will be rare (and still possible anyway). > It's similar to Django's support for, say, simplejson. I think it's reasonable for Django to alias WatchedFileHandler so that it's available either bound to logging's own implementation (in sufficiently recent versions of Python) or else a copy in Django's own code. Then people can use it if they want to, even in older Python versions. > One of the reasons why I propose Django's own settings structure for > logging is because we can choose better defaults for logging and have > more compact syntax for them. Standard Python logging configuration has > a noticable gap between very simplistic basicConfig which configures > only a root channel and a verbose imperative definition of handler > objects, formatter objects and logger objects. I've found that my usage > of logging inevitably falls in between: I often need a few logging > channels but I almost never, say, reuse handler objects between them. > > Here's a variant of a simple config that I had in mind lately: > > LOGGING = { > 'errors': { > 'handler': 'django.logging.FileLogger', # WatchedFileLogger copy > 'filename': '...', > 'level': 'debug', > }, > 'maintenance': { > 'handler': 'logging.handlers.HTTPHandler', > 'host': '...', > 'url': '', > 'format': '' > }, > > } > > Top-level keys are logger names. Values are dicts describing handlers. > These dicts have several keys that Django knows about: > > - 'handler': a handler class. It's imported like any other stringified > classes in settings > > - 'level': a level keyword that is translated into logging.* constants. > This is done to not make users import logging by hand. > > - 'format': a format string for the logging.Formatter object. We can > have a more sensible default for this than the one in Python logging. Or > not :-) > > These keys are pop'd out of the dict and the rest is used as **kwargs to > the handler class instantiation. > > Django's default setup may look like this: > > LOGGING = { > '': {'handler': 'logging.StreamHandler'} > > } > > This has an advantage of always configuring a root logger to avoid an > infamous warning from Python logging when the logger doesn't have any > handlers defined. Users wanting to configure all the logging themselves > may null-out this using `LOGGING = {}`. I have no big problem with a configuration scheme such as you suggest - if it's felt that a lot of Django users are not Python-savvy enough and need some hand-holding, then you'd perhaps need something like this. I usually configure the logging system using its own configuration file format (ConfigParser based, and supported by the stdlib so no additional Django code required) or using YAML (where it's already being used for other configuration, and when having a PyYAML dependency is not a problem.) Either way it's declarative and not too painful. My reservation with Django's own take on it is simply that it goes against TOOWTDI and the Zen of Python, a little at least. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
Ivan Sagalaev wrote: > Standard Python logging configuration has > a noticable gap between very simplistic basicConfig which configures > only a root channel and a verbose imperative definition of handler > objects, formatter objects and logger objects. Forgot one thing. As it stands now Django has this "historic" behavior when it imports and executes settings module twice. This results in breakage when you setup loggers and handlers by hand. We now circumvent this by having a helper function that memoizes configured loggers and call it from settings.py. Having a declarative config we can hide this inside of Django and not scare users. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
Vinay Sajip wrote: > Actually you don't need much in settings.py, and Django doesn't need > to grow any code of its own to "wrap" logging. You can either > configure logging programmatically (for which I use basicConfig, in > simple setups) or using a configuration file (ConfigParser-based, > fully documented in Python docs) for more complex setups. Thanks! Didn't know that. However see my further comment. >> Now we've settled on a WatchedFileHandler ported from Python 2.6 logging >> module. It watches for file descriptor change and doesn't require >> SIG_HUP to pick up a new file. May be we should port it to Django and >> use it as a default handler for logging to file system. > > Why "port it to Django"? Do you mean, copy it into Django? I'm not > sure it should be the default - not everybody uses logrotate. I'd > leave this sort of decision for code in settings.py. Using WatchedFileHandler is a safe default because it works as FileHandler, just doesn't break with logrotate. I don't know of any disadvantages of WatchedFileHandler before the old FileHandler. So I don't think there's much value in giving people this choice in settings because non-default behavior will be rare (and still possible anyway). One of the reasons why I propose Django's own settings structure for logging is because we can choose better defaults for logging and have more compact syntax for them. Standard Python logging configuration has a noticable gap between very simplistic basicConfig which configures only a root channel and a verbose imperative definition of handler objects, formatter objects and logger objects. I've found that my usage of logging inevitably falls in between: I often need a few logging channels but I almost never, say, reuse handler objects between them. Here's a variant of a simple config that I had in mind lately: LOGGING = { 'errors': { 'handler': 'django.logging.FileLogger', # WatchedFileLogger copy 'filename': '...', 'level': 'debug', }, 'maintenance': { 'handler': 'logging.handlers.HTTPHandler', 'host': '...', 'url': '', 'format': '' }, } Top-level keys are logger names. Values are dicts describing handlers. These dicts have several keys that Django knows about: - 'handler': a handler class. It's imported like any other stringified classes in settings - 'level': a level keyword that is translated into logging.* constants. This is done to not make users import logging by hand. - 'format': a format string for the logging.Formatter object. We can have a more sensible default for this than the one in Python logging. Or not :-) These keys are pop'd out of the dict and the rest is used as **kwargs to the handler class instantiation. Django's default setup may look like this: LOGGING = { '': {'handler': 'logging.StreamHandler'} } This has an advantage of always configuring a root logger to avoid an infamous warning from Python logging when the logger doesn't have any handlers defined. Users wanting to configure all the logging themselves may null-out this using `LOGGING = {}`. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Tue, Sep 29, 2009 at 9:00 PM, Waylan Limbergwrote: > > On Tue, Sep 29, 2009 at 4:36 AM, Vinay Sajip wrote: >> >> >> >> On Sep 17, 9:25 am, Simon Willison wrote: >>> Problems and challenges >>> === >>> >>> 1. The Python logging module isn't very nicely designed - its Java >>> heritage shines through, and things like logging.basicConfig behave in >>> unintuitive ways (if you call basicConfig twice the second call fails >>> silently but has no effect). This is why I suggest wrapping it in our >>> own higher level interface. >> >> Simon, I'm the author of Python's logging package. Sorry for the delay >> in replying, I've been away from this list awhile. I think the "Java >> heritage shines through" is just FUD. basicConfig's behaviour is fully >> documented here: >> >> http://docs.python.org/library/logging.html#logging.basicConfig >> >> Including the fact that it sometimes (by design) has no effect. >> >> There are a lot of people for whom logging just means writing to a >> file, and that's why they have difficulty understanding why logging is >> designed as it is. I would suggest you take a quick look at >> >> http://plumberjack.blogspot.com/2009/09/python-logging-101.html >> >> and then tell me why you think Python logging isn't well designed for >> its purpose. You can do basic logging with two lines of setup (one >> line if you ignore the import): >> >> import logging >> logging.basicConfig(level=logging.DEBUG,filename='/path/to/my/log', >> format='%(asctime)s %(message)s') >> >> and then >> >> logging.getLogger(__name__).debug("Just checking this works") >> >> Not too sure where the Java heritage is there, or where the hard part >> is. >> > > The hard part is that basicConfig only works like that back to Python > 2.4 yet Django supports 2.3. When I added logging to Python-Markdown, > this was the hardest part. Figuring out how to configure logging so > that it works in 2.3 as well. The documentation is not exactly helpful > in that regard. ... > Of course, it is possible that I'm missing something obvious. As luck would have it, you are :-) Django 1.2 will drop formal support for Python 2.3. Yours, Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Tue, Sep 29, 2009 at 4:36 AM, Vinay Sajipwrote: > > > > On Sep 17, 9:25 am, Simon Willison wrote: >> Problems and challenges >> === >> >> 1. The Python logging module isn't very nicely designed - its Java >> heritage shines through, and things like logging.basicConfig behave in >> unintuitive ways (if you call basicConfig twice the second call fails >> silently but has no effect). This is why I suggest wrapping it in our >> own higher level interface. > > Simon, I'm the author of Python's logging package. Sorry for the delay > in replying, I've been away from this list awhile. I think the "Java > heritage shines through" is just FUD. basicConfig's behaviour is fully > documented here: > > http://docs.python.org/library/logging.html#logging.basicConfig > > Including the fact that it sometimes (by design) has no effect. > > There are a lot of people for whom logging just means writing to a > file, and that's why they have difficulty understanding why logging is > designed as it is. I would suggest you take a quick look at > > http://plumberjack.blogspot.com/2009/09/python-logging-101.html > > and then tell me why you think Python logging isn't well designed for > its purpose. You can do basic logging with two lines of setup (one > line if you ignore the import): > > import logging > logging.basicConfig(level=logging.DEBUG,filename='/path/to/my/log', > format='%(asctime)s %(message)s') > > and then > > logging.getLogger(__name__).debug("Just checking this works") > > Not too sure where the Java heritage is there, or where the hard part > is. > The hard part is that basicConfig only works like that back to Python 2.4 yet Django supports 2.3. When I added logging to Python-Markdown, this was the hardest part. Figuring out how to configure logging so that it works in 2.3 as well. The documentation is not exactly helpful in that regard. In fact, it was for this very reason that we added our own wrapper around logging. It didn't seem reasonable for our users to go through the same pain that we did. Sure we got a few things wrong at first, but with the help of a few people in the community we worked those out and our wrapper seems to work ok now. Yes - ok - I get the sense it could be better. Ever since then, any mention of logging leaves a bad taste in my mouth. Perhaps if I was working only in 2.6 or such, this wouldn't be an issue, but we have promised support back to 2.3. Of course, it is possible that I'm missing something obvious. -- \X/ /-\ `/ |_ /-\ |\| Waylan Limberg --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Tue, Sep 29, 2009 at 5:09 PM, Vinay Sajipwrote: > > On Sep 17, 3:04 pm, Russell Keith-Magee > wrote: >> In the absence of specifics, this makes me a little bit nervous. The >> Python logging interface may be very Java-heavy and complex, but it is >> a thoroughly known quantity, and it houses a lot of features. > > See my comment about Java-heavy being FUD in Simon's initial post. First off - let me reinforce that I'm in your camp here - I like Python's logger, and I think we should be adding logging to Django. Any hesitation I have expressed is mostly a function of institutional inertia, especially with regards to Adrian's historical position on logging. However, I would point out that IMHO, FUD is an accurate description of the state of play - though probably not in the way you probably meant. Python's logging api _looks_ a lot like log4j in parts. This is at least partially because there's a limit to how many ways you can express 'log.debug()' before you start to copy. However, as a result, there's a lot of Fear, Uncertainty and Doubt as to whether a framework that apparently has Java heritage is going to be any good in Python. Don't forget that a lot of us (myself included) got into writing Python to get away from the stupidities of the Java world. Those scars are deep, and aren't going away in a hurry. Speaking personally, log4j is responsible for a lot of those scars, due in no small part to the abysmal documentation for that project. >> I'm also not entirely convinced that the answer here isn't just >> documentation. The documentation for log4j has historically been >> pretty awful, and while Python's documentation is an improvement, it >> could certainly be better IMHO. Good documentation for how to use >> logging in the context of Django could go a long way. >> > > I'm working with Doug Hellmann (PyMOTW) to try and improve the layout > of the logging documentation in Python. I'm not asking for patches > (though it would be nice), but if you can give *specific* criticisms > (e.g. what you think is missing, or unclear) then that will focus our > efforts. My comment was actually directed at Django's documentation, which is currently silent on the issue of logging - and probably shouldn't be. However, since you're interested in feedback, my suggestion would be to look at every defense you've made of logging in this thread (and any other threads where you've had similar arguments), and work out why the current docs have allowed those viewpoints to be established as apparent fact. Some examples: * Acknowledge that there is some Java heritage, but point out that this doesn't mean it's a bad thing, and that there is a lot that _isn't_ Java based about Python's logger. * Highlight the important architectural picture. As you noted in another reply - the logger and the handler are quite separate, and this gives a lot of power. However, the existence and significance of that architectural separation isn't really a major feature of the current docs. At present, the architectural bits are buried inside API discussion, but understanding this architecture is important if you're going to understand why logging works the way it does, and understand that logging isn't just putting lines into a file. * Make the simple example actually simple. IMHO, a single-file simple logging example is good for exactly 2 things: - showing how to configure the simplest possible case of logging - explaining the "why don't I have any output" problem. Tasks like configuring the logger to use a rotating file handler are important, but can wait for much later - once issues of basic usage and architecture have been established. * Better examples of how logging works in the real world. All the examples focus on single file projects. Most of the complexities I've had with logging stem from how to use it in a multiple-file project, yet as far as I can make out, there is very little discussion of how logging should be used in a real multiple-file project. - Should I have one logger instance per module? One per conceptual "task"? - You've used "logging.getLogger(__name__)" in this thread, but this pattern isn't mentioned once in the docs. Is this best practice, or a quick-and-dirty hack? - When I have multiple loggers across multiple files, how do I configure logging? Should I be putting logging.config.fileConfig() at the start of every python file, or should I put the logging config into a single python file somewhere that configures logging, and import that module as needed? >> Details notwithstanding, I'm +1 to the idea of adding logging to the >> core framework - or, at least, making it easier to use logs for >> reporting internal state and error conditions instead of email). >> > > You can have your cake and eat it. It's perfectly feasible in Python > logging to send only certain events to nominated email addresses (all > configurable at run-time, so emails
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 9:41 pm, Simon Willisonwrote: > I should clarify - by "lightweight wrapper" I basically mean a pre- > configured log setup and a standard place to import the logger from - There's no "the logger". Each module should have its own logger, this allows you to control the verbosity of logging to at least the module level and potentially with finer granularity than this. > and maybe a tiny bit of syntactic sugar if it will make the common > case more palatable. I'm mostly just interested in making the logging > module an encouraged technique within the Django world. It should > definitely play nicely with any already-existant logging code. +1, there are already patterns for doing this which work, involve no need for django.contrib.log or similar and I'll happily work with the core devs to make this happen. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 3:04 pm, Russell Keith-Mageewrote: > To clarify - I think that documentation is the very least we should > do. As your comments indicate, there are a lot of things you need to > do in order to get logging right, so we should at the very least > provide some documentation on how to do it right. > As I posted in an earlier response to Simon, I'm happy to help with this. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 3:04 pm, Russell Keith-Mageewrote: > In the absence of specifics, this makes me a little bit nervous. The > Python logging interface may be very Java-heavy and complex, but it is > a thoroughly known quantity, and it houses a lot of features. See my comment about Java-heavy being FUD in Simon's initial post. > > I've seen several attempts to wrap Java loggers in a "nicer" > interface, and every one of them ended up hobbling some of the power > features of the logger. There is also the issue of our wrapper playing > nicely with the loggers already being used in the wild. > Absolutely agree. Wrapping is the wrong way to go, and not even needed. I use logging with Django all the time and see no need to have any special code in Django to support logging. Where necessary, I've patched my Django to include logging statements. > I'm also not entirely convinced that the answer here isn't just > documentation. The documentation for log4j has historically been > pretty awful, and while Python's documentation is an improvement, it > could certainly be better IMHO. Good documentation for how to use > logging in the context of Django could go a long way. > I'm working with Doug Hellmann (PyMOTW) to try and improve the layout of the logging documentation in Python. I'm not asking for patches (though it would be nice), but if you can give *specific* criticisms (e.g. what you think is missing, or unclear) then that will focus our efforts. > > Details notwithstanding, I'm +1 to the idea of adding logging to the > core framework - or, at least, making it easier to use logs for > reporting internal state and error conditions instead of email). > You can have your cake and eat it. It's perfectly feasible in Python logging to send only certain events to nominated email addresses (all configurable at run-time, so emails can be turned on/off, sent to different/additional destinations etc.) as well as e.g. logging tracebacks to file for the same events. > As for likely roadblocks: I've been led to believe that Adrian has > objections to framework-level logging. I have no idea as to the nature > of his objection, but ticket #5415 indicates that he is (or has been, > historically) in favor of adding signals that could be used for > logging or debugging purposes. > They (logging and signals) are two different things. Python logging allows you to consider the dimensions "What happened?", "Where did it happen?", "How important is it?" and "Who wants to know?" intelligently, and in particular it treats "What happened" and "Who wants to know?" orthogonally. You get a lot of ways of getting information to *people* whereas signals is more about letting *code* know what's going on. Unfortunately, a lot of people have got the impression that Python logging is "Java-like" and "not Pythonic" just because I acknowledged some good ideas in log4j. It's not as if Python people have a monopoly on good ideas, is it? This "Java heritage" perception sometimes leads to prejudice against logging, for no good reason that I can see. Of course there might be grievances - for example, people complain about "slow". As a single logging call which just has a file handler is of the order of some tens of microseconds, I don't know how bad this is - what do we compare against? The Tornado webserver (used by FriendFeed) is a high-performance solution which uses Python logging. SQLAlchemy uses Python logging. They are careful to consider performance and as a consequence logging doesn't present a problem in practice. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 11:53 am, Ivan Sagalaevwrote: > Talking about predictable and standard way I want to be sure that we > don't break existing logging. > > I.e. we at Yandex now have many reusable Django apps that don't setup > loggers themselves but just log things into named loggers and expect > them to be setup by a project that uses them. That's normal. The pattern I use is: In each module which needs to use logging, instantiate a module-global logger using logger = logging.getLogger(__name__) and log to it in the module's code. The configuration happens in settings.py. > What I gather from your proposal is that you want the same model ("an > app logs, a project setups") plus a nice declarative syntax in > settings.py instead of boring creation of handlers, formatters and > loggers. Right? > Actually you don't need much in settings.py, and Django doesn't need to grow any code of its own to "wrap" logging. You can either configure logging programmatically (for which I use basicConfig, in simple setups) or using a configuration file (ConfigParser-based, fully documented in Python docs) for more complex setups. > > - We could replace (or complement) django.connection.queries with a > > log of executed SQL. This would make the answer to the common question > > "how do I see what SQL is being executed" much more obvious. > Yes, I do this with a patched CursorDebugWrapper. You can direct the SQL to a separate file which contains only the SQL events and not other logging events. > > We had this problem with standard logging. Then we switched to a > RotatingFileHandler which wasn't very good however because its behavior > is simplistic and is not controllable by admins with an interface that > they know, namely logrotate. Setting up logrotate also wasn't without > problems. When it rotates a file it should let an app know about it and > it uses SIG_HUP for that. However this instantly terminates Django's > flup-based FastCGI server which we use. > > Now we've settled on a WatchedFileHandler ported from Python 2.6 logging > module. It watches for file descriptor change and doesn't require > SIG_HUP to pick up a new file. May be we should port it to Django and > use it as a default handler for logging to file system. Why "port it to Django"? Do you mean, copy it into Django? I'm not sure it should be the default - not everybody uses logrotate. I'd leave this sort of decision for code in settings.py. Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 10:37 am, Mat Claytonwrote: > +1 for this, another random thought which doesn't do your long post justice. > But what are everyone's thoughts about log aggregation, taking logs from X > app servers and combining them into a single location, something like > Facebook's Scribe. I assume this could be built in as a separate log > handler, but it would be nice for the guys who need this functionality to be > able to achieve it easily, probably not right for core though. Mat, logging's design already allows you to do this aggregation - you can have logging events collected in multiple locations. See the Python documentation here for pointers: http://docs.python.org/library/logging.html#sending-and-receiving-logging-events-across-a-network Regards, Vinay Sajip --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 9:25 am, Simon Willisonwrote: > Problems and challenges > === > > 1. The Python logging module isn't very nicely designed - its Java > heritage shines through, and things like logging.basicConfig behave in > unintuitive ways (if you call basicConfig twice the second call fails > silently but has no effect). This is why I suggest wrapping it in our > own higher level interface. Simon, I'm the author of Python's logging package. Sorry for the delay in replying, I've been away from this list awhile. I think the "Java heritage shines through" is just FUD. basicConfig's behaviour is fully documented here: http://docs.python.org/library/logging.html#logging.basicConfig Including the fact that it sometimes (by design) has no effect. There are a lot of people for whom logging just means writing to a file, and that's why they have difficulty understanding why logging is designed as it is. I would suggest you take a quick look at http://plumberjack.blogspot.com/2009/09/python-logging-101.html and then tell me why you think Python logging isn't well designed for its purpose. You can do basic logging with two lines of setup (one line if you ignore the import): import logging logging.basicConfig(level=logging.DEBUG,filename='/path/to/my/log', format='%(asctime)s %(message)s') and then logging.getLogger(__name__).debug("Just checking this works") Not too sure where the Java heritage is there, or where the hard part is. > > 2. There may be some performance overhead, especially if we replace > mechanisms like django.connection.queries with logging. This should be > negligble: here's a simple benchmark: > > # ("hello " * 100) gives a 600 char string, long enough for a SQL > statement>>> import timeit, logging > >>> t = timeit.Timer('logging.info("hello " * 100)', 'import logging') > >>> t.timeit(number=100) # one hundred statements > > 0.00061702728271484375>>> t.timeit(number=100) # one million statements > > 6.458014965057373 > > That's 0.0006 of a second overhead for a page logging 100 SQL > statements. The performance overhead will go up if you attach a > handler, but that's fine - the whole point of a framework like > 'logging' is that you can log as much as you like but only act on > messages above a certain logging level. A quick-and-dirty measurement showed me that a logging call (which writes to file) takes on the order of 57 microseconds, which can be reduced to around 50 microseconds if you forego collecting stack frame, thread and process informaion. Not too shabby, though perhaps not appropriate for extremely high-performance use cases. I hasten to add, it's not a scientific benchmark. > > 3. We risk people using logging where signals would be more > appropriate. > They're for entirely different purposes so I can't imagine this will happen too often. > 4. We might go too far, and make Django a "noisy" piece of software > which logs almost everything that happens within it. Let's be tasteful > about this. > One thing about the logging design (which perhaps makes it appear complicated) is that developers can shape the logging in such a way that the verbosity in different parts can be turned on and off pretty much at will, and even without restarting the server in some cases. So, with a little care in how things are arranged, this needn't happen. > 5. People might leave logging on, then find their server disk has > filled up with log files and caused their site to crash. > > 6. Logging levels are confusing - what exactly is the difference > between warn, info, error, debug, critical and fatal? We would need to > document this and make decisions on which ones get used for what > within the framework. DEBUG: Detailed information, of no interest when everything is working well but invaluable when diagnosing problems. INFO: Affirmations that things are working as expected, e.g. "service has started" or "indexing run complete". Often ignored. WARNING:There may be a problem in the near future, and this gives advance warning of it. But the application is able to proceed normally. ERROR: The application has been unable to proceed as expected, due to the problem being logged. CRITICAL: This is a serious error, and some kind of application meltdown might be imminent. I'll be happy to clarify further if needed. > What would it look like? > > > Here's what I'm thinking at the moment (having given the API very > little thought). In your application code: > > from django.core import log I'm not sure it's a good idea to have a wrapper, as I don't believe it's needed and may restrict the level of control you typically need to have over logging. You'll not be convinced by my just saying so - therefore, I'll be happy to work with you to understand what (in specific areas, including at the module level) you're trying to achieve, and explaining the best way to achieve it. > # Log to the default channel:
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 18, 6:21 pm, Eric Holscherwrote: > I have looked into Logging before for another project, and I found that > SQLAlchemy's support seemed to be a pretty good model to follow. They define > all of their loggers under the sqlalchemy namespace, and then you can > configure different handlers for different things[1]: > > I think that this would be necessary to have in Django, so that for > instance, I could listen to the django.orm logs, and not the django.http, or > listen to them with different handlers/levels. Yes, absolutely - this looks like exactly the right model. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
I have looked into Logging before for another project, and I found that SQLAlchemy's support seemed to be a pretty good model to follow. They define all of their loggers under the sqlalchemy namespace, and then you can configure different handlers for different things[1]: import logging logging.basicConfig() logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO) logging.getLogger('sqlalchemy.orm.unitofwork').setLevel(logging.DEBUG) I think that this would be necessary to have in Django, so that for instance, I could listen to the django.orm logs, and not the django.http, or listen to them with different handlers/levels. Their implementation[2] is a little confusing to me, but I think that having some prior art like this will allow us to better understand what we need, and how to accomplish it, so I thought I would throw it out there. 1: http://www.sqlalchemy.org/docs/05/dbengine.html#configuring-logging 2: http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/lib/sqlalchemy/log.py -- Eric Holscher Web Developer at The World Company in Lawrence, Ks http://ericholscher.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 6:25 pm, Simon Willisonwrote: > I think we should add logging to Django in version 1.2, implemented as > a light-weight wrapper around the Python logging module > (django.core.log maybe?) plus code to write errors to the Apache error > log under the mod_python handler and environ['wsgi.errors'] under WSGI > (meaningmod_wsgiwill write to the Apache error log as well). It isn't necessarily practical to use environ['wsgi.errors'] as that exists only for life of that request. Thus, anything done at time of module imports or in background threads wouldn't have access to it. You are better of just using sys.stderr. Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 4:04 pm, Russell Keith-Mageewrote: > I've seen several attempts to wrap Java loggers in a "nicer" > interface, and every one of them ended up hobbling some of the power > features of the logger. There is also the issue of our wrapper playing > nicely with the loggers already being used in the wild. I should clarify - by "lightweight wrapper" I basically mean a pre- configured log setup and a standard place to import the logger from - and maybe a tiny bit of syntactic sugar if it will make the common case more palatable. I'm mostly just interested in making the logging module an encouraged technique within the Django world. It should definitely play nicely with any already-existant logging code. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 10:25 am, Simon Willisonwrote: > That's 0.0006 of a second overhead for a page logging 100 SQL > statements. The performance overhead will go up if you attach a > handler, but that's fine - the whole point of a framework like > 'logging' is that you can log as much as you like but only act on > messages above a certain logging level. I woudln't worry about performance, lately I had a project doing massive logging, which resulted in a performance loss, but replacing my logging functions with simple lambda functions doing nothing (eg replace django.core.log.warn with lambda *args, *kwargs: pass if loglevel is error) gave me quite a performance boost. Aide from that: +1 on the proposal. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
+1 on the logging proposal. The stock python logging module is definitely a bit of a finicky and confusing creature, especially for people coming to Python for the first time with Django. -0 On the signals based approach. I would be wary of the potential performance overhead of replacing logging with signals and/or implement logging in certain high traffic areas with signals. It would definitely be interesting to see some proof of concept performance tests (i.e. what Simon did with the handlerless logging) before going too far down that path. -Sean O'Connor On Sep 17, 1:40 pm, Eric Florenzanowrote: > On Sep 17, 1:25 am, Simon Willison wrote: > > > 1. We'll be able to de-emphasise the current default "e-mail all > > errors to someone" behaviour, which doesn't scale at all well. > > I'm a big fan of this proposal, for exactly this reason. > > +1 > > Thanks, > Eric Florenzano --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Sep 17, 1:25 am, Simon Willisonwrote: > 1. We'll be able to de-emphasise the current default "e-mail all > errors to someone" behaviour, which doesn't scale at all well. I'm a big fan of this proposal, for exactly this reason. +1 Thanks, Eric Florenzano --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Thu, Sep 17, 2009 at 10:04 AM, Russell Keith-Mageewrote: > As for likely roadblocks: I've been led to believe that Adrian has > objections to framework-level logging. I have no idea as to the nature > of his objection, but ticket #5415 indicates that he is (or has been, > historically) in favor of adding signals that could be used for > logging or debugging purposes. I'm in favor of the signals approach as part of the core framework. In addition, django.contrib.logging, which could provide a generic solution, good for the 80 or 90% case would really rock. -- http://www.apgwoz.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
On Thu, Sep 17, 2009 at 4:25 PM, Simon Willisonwrote: > > I think we should add logging to Django in version 1.2, implemented as > a light-weight wrapper around the Python logging module > (django.core.log maybe?) plus code to write errors to the Apache error > log under the mod_python handler and environ['wsgi.errors'] under WSGI > (meaning mod_wsgi will write to the Apache error log as well). > > Benefits of logging as a core Django service > > > Adding logging to Django core would provide the following benefits: > > 1. We'll be able to de-emphasise the current default "e-mail all > errors to someone" behaviour, which doesn't scale at all well. > > 2. Having logging in the core framework will mean people start > actually using it, which will make it easier for people to debug their > Django apps. Right now adding "print" statements to a Django app is a > common debugging technique, but it's messy (you have to remember to > take them out again) and error prone - some production environments > throw errors if an app attempts to write to stdout. It's also not > obvious - many developers are surprised when I show them the > technique. > > 3. Logging in Django core rather than a 3rd party app will encourage > reusable applications to log things in a predictable way, standard > way. > > 4. 3rd party debugging tools such as the debug toolbar will be able to > hook in to Django's default logging behaviour. This could also lead to > plenty of additional 3rd party innovation - imagine a tool that looks > out for logged SQL that took longer than X seconds, or one that groups > together similar log messages, or streams log messages to IRC... > > 5. Built-in support for logging reflects a growing reality of modern > Web development: more and more sites have interfaces with external web > service APIs, meaning there are plenty of things that could go wrong > that are outside the control of the developer. Failing gracefully and > logging what happened is the best way to deal with 3rd party problems > - much better than throwing a 500 and leaving no record of what went > wrong. > > 6. Most importantly from my point of view, when a sysadmin asks where > Django logs errors in production we'll have a good answer for them! > > 7. As a general rule, I believe you can never have too much > information about what's going on with your web application. I've > never thought to myself "the problem with this bug is I've got too > much information about it". As for large log files, disk space is > cheap - and pluggable backends could ensure logs were sensibly > rotated. No disagreement here with any of these assertions. > Places logging would be useful > == > > - Unhandled exceptions that make it up to the top of the Django stack > (and would cause a 500 error to be returned in production) > - The development web server could use logging for showing processed > requests (where currently these are just printed to stdout). > - Failed attempts at signing in to the admin could be logged, making > security audits easier. > - We could replace (or complement) django.connection.queries with a > log of executed SQL. This would make the answer to the common question > "how do I see what SQL is being executed" much more obvious. > - Stuff that loads things from INSTALLED_APPS could log what is being > loaded, making it much easier to spot and debug errors caused by code > being incorrectly loaded. > - Likewise, the template engine could log which templates are being > loaded from where, making it easier to debug problems stemming from an > incorrectly configured TEMPLATE_DIRS setting. > - We could use logging to address the problems with the template > engine failing silently - maybe some template errors (the ones more > likely to be accidental than just people relying on the fail-silent > behaviour deliberately) should be logged as warnings. > > Most of the above would be set to a low log level which by default > would not be handled, displayed or stored anywhere (logging.info or > similar). Maybe "./manage.py runserver --loglevel=info" could cause > such logs to be printed to the terminal while the development server > is running. > > Problems and challenges > === > > 1. The Python logging module isn't very nicely designed - its Java > heritage shines through, and things like logging.basicConfig behave in > unintuitive ways (if you call basicConfig twice the second call fails > silently but has no effect). This is why I suggest wrapping it in our > own higher level interface. In the absence of specifics, this makes me a little bit nervous. The Python logging interface may be very Java-heavy and complex, but it is a thoroughly known quantity, and it houses a lot of features. I've seen several attempts to wrap Java loggers in a "nicer" interface, and every one of them ended up hobbling some of the power features of the
Re: Proposal for 1.2: built-in logging with django.core.log
On Thu, Sep 17, 2009 at 6:53 PM, Ivan Sagalaevwrote: > > Hi Simon, > > Simon Willison wrote: >> 1. We'll be able to de-emphasise the current default "e-mail all >> errors to someone" behaviour, which doesn't scale at all well. > > In a recent thread[1] on a similar topic Russel has also emphasized that > we should improve documentation about doing logging. To clarify - I think that documentation is the very least we should do. As your comments indicate, there are a lot of things you need to do in order to get logging right, so we should at the very least provide some documentation on how to do it right. Yours, Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
Hi Simon, Simon Willison wrote: > 1. We'll be able to de-emphasise the current default "e-mail all > errors to someone" behaviour, which doesn't scale at all well. In a recent thread[1] on a similar topic Russel has also emphasized that we should improve documentation about doing logging. > 3. Logging in Django core rather than a 3rd party app will encourage > reusable applications to log things in a predictable way, standard > way. Talking about predictable and standard way I want to be sure that we don't break existing logging. I.e. we at Yandex now have many reusable Django apps that don't setup loggers themselves but just log things into named loggers and expect them to be setup by a project that uses them. What I gather from your proposal is that you want the same model ("an app logs, a project setups") plus a nice declarative syntax in settings.py instead of boring creation of handlers, formatters and loggers. Right? > - We could replace (or complement) django.connection.queries with a > log of executed SQL. This would make the answer to the common question > "how do I see what SQL is being executed" much more obvious. In the thread that I was referring to[1] we kind of agreed on using a signal there. Then hooking a logger onto the signal is simple. > 5. People might leave logging on, then find their server disk has > filled up with log files and caused their site to crash. We had this problem with standard logging. Then we switched to a RotatingFileHandler which wasn't very good however because its behavior is simplistic and is not controllable by admins with an interface that they know, namely logrotate. Setting up logrotate also wasn't without problems. When it rotates a file it should let an app know about it and it uses SIG_HUP for that. However this instantly terminates Django's flup-based FastCGI server which we use. Now we've settled on a WatchedFileHandler ported from Python 2.6 logging module. It watches for file descriptor change and doesn't require SIG_HUP to pick up a new file. May be we should port it to Django and use it as a default handler for logging to file system. > 6. Logging levels are confusing - what exactly is the difference > between warn, info, error, debug, critical and fatal? We would need to > document this and make decisions on which ones get used for what > within the framework. May be we can just leave it for users to decide. It depends so much on how much an app actually wants from logging. The only standard thing I can think of is to have DEBUG = True imply level = logging.DEBUG (which includes everything more sever). DEBUG = False will imply logging.INFO then. What do you think? > # If you want custom log handlers - not sure how these would interact > with > # channels and log levels yet > LOG_HANDLERS = ( > 'django.core.log.handlers.LogToDatabase', > 'django.core.log.handlers.LogToEmail', > ) This is a hard problem really. Most handlers require different set of arguments. File names, email credentials, system idents for SysLog etc. Also there should be different formatters. For example there's no point to waste space in a syslog message on a timestamp since syslog tracks it itself... [1]: http://groups.google.com/group/django-developers/browse_frm/thread/9d0992e800cf7d68# --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal for 1.2: built-in logging with django.core.log
Definitely a +1 from me. -- Horst On Thu, Sep 17, 2009 at 10:25 AM, Simon Willisonwrote: > > I think we should add logging to Django in version 1.2, implemented as > a light-weight wrapper around the Python logging module > (django.core.log maybe?) plus code to write errors to the Apache error > log under the mod_python handler and environ['wsgi.errors'] under WSGI > (meaning mod_wsgi will write to the Apache error log as well). > > Benefits of logging as a core Django service > > > Adding logging to Django core would provide the following benefits: > > 1. We'll be able to de-emphasise the current default "e-mail all > errors to someone" behaviour, which doesn't scale at all well. > > 2. Having logging in the core framework will mean people start > actually using it, which will make it easier for people to debug their > Django apps. Right now adding "print" statements to a Django app is a > common debugging technique, but it's messy (you have to remember to > take them out again) and error prone - some production environments > throw errors if an app attempts to write to stdout. It's also not > obvious - many developers are surprised when I show them the > technique. > > 3. Logging in Django core rather than a 3rd party app will encourage > reusable applications to log things in a predictable way, standard > way. > > 4. 3rd party debugging tools such as the debug toolbar will be able to > hook in to Django's default logging behaviour. This could also lead to > plenty of additional 3rd party innovation - imagine a tool that looks > out for logged SQL that took longer than X seconds, or one that groups > together similar log messages, or streams log messages to IRC... > > 5. Built-in support for logging reflects a growing reality of modern > Web development: more and more sites have interfaces with external web > service APIs, meaning there are plenty of things that could go wrong > that are outside the control of the developer. Failing gracefully and > logging what happened is the best way to deal with 3rd party problems > - much better than throwing a 500 and leaving no record of what went > wrong. > > 6. Most importantly from my point of view, when a sysadmin asks where > Django logs errors in production we'll have a good answer for them! > > 7. As a general rule, I believe you can never have too much > information about what's going on with your web application. I've > never thought to myself "the problem with this bug is I've got too > much information about it". As for large log files, disk space is > cheap - and pluggable backends could ensure logs were sensibly > rotated. > > Places logging would be useful > == > > - Unhandled exceptions that make it up to the top of the Django stack > (and would cause a 500 error to be returned in production) > - The development web server could use logging for showing processed > requests (where currently these are just printed to stdout). > - Failed attempts at signing in to the admin could be logged, making > security audits easier. > - We could replace (or complement) django.connection.queries with a > log of executed SQL. This would make the answer to the common question > "how do I see what SQL is being executed" much more obvious. > - Stuff that loads things from INSTALLED_APPS could log what is being > loaded, making it much easier to spot and debug errors caused by code > being incorrectly loaded. > - Likewise, the template engine could log which templates are being > loaded from where, making it easier to debug problems stemming from an > incorrectly configured TEMPLATE_DIRS setting. > - We could use logging to address the problems with the template > engine failing silently - maybe some template errors (the ones more > likely to be accidental than just people relying on the fail-silent > behaviour deliberately) should be logged as warnings. > > Most of the above would be set to a low log level which by default > would not be handled, displayed or stored anywhere (logging.info or > similar). Maybe "./manage.py runserver --loglevel=info" could cause > such logs to be printed to the terminal while the development server > is running. > > Problems and challenges > === > > 1. The Python logging module isn't very nicely designed - its Java > heritage shines through, and things like logging.basicConfig behave in > unintuitive ways (if you call basicConfig twice the second call fails > silently but has no effect). This is why I suggest wrapping it in our > own higher level interface. > > 2. There may be some performance overhead, especially if we replace > mechanisms like django.connection.queries with logging. This should be > negligble: here's a simple benchmark: > > # ("hello " * 100) gives a 600 char string, long enough for a SQL > statement import timeit, logging t = timeit.Timer('logging.info("hello " * 100)', 'import
Re: Proposal for 1.2: built-in logging with django.core.log
+1 for this, another random thought which doesn't do your long post justice. But what are everyone's thoughts about log aggregation, taking logs from X app servers and combining them into a single location, something like Facebook's Scribe. I assume this could be built in as a separate log handler, but it would be nice for the guys who need this functionality to be able to achieve it easily, probably not right for core though. Mat On Thu, Sep 17, 2009 at 9:25 AM, Simon Willisonwrote: > > I think we should add logging to Django in version 1.2, implemented as > a light-weight wrapper around the Python logging module > (django.core.log maybe?) plus code to write errors to the Apache error > log under the mod_python handler and environ['wsgi.errors'] under WSGI > (meaning mod_wsgi will write to the Apache error log as well). > > Benefits of logging as a core Django service > > > Adding logging to Django core would provide the following benefits: > > 1. We'll be able to de-emphasise the current default "e-mail all > errors to someone" behaviour, which doesn't scale at all well. > > 2. Having logging in the core framework will mean people start > actually using it, which will make it easier for people to debug their > Django apps. Right now adding "print" statements to a Django app is a > common debugging technique, but it's messy (you have to remember to > take them out again) and error prone - some production environments > throw errors if an app attempts to write to stdout. It's also not > obvious - many developers are surprised when I show them the > technique. > > 3. Logging in Django core rather than a 3rd party app will encourage > reusable applications to log things in a predictable way, standard > way. > > 4. 3rd party debugging tools such as the debug toolbar will be able to > hook in to Django's default logging behaviour. This could also lead to > plenty of additional 3rd party innovation - imagine a tool that looks > out for logged SQL that took longer than X seconds, or one that groups > together similar log messages, or streams log messages to IRC... > > 5. Built-in support for logging reflects a growing reality of modern > Web development: more and more sites have interfaces with external web > service APIs, meaning there are plenty of things that could go wrong > that are outside the control of the developer. Failing gracefully and > logging what happened is the best way to deal with 3rd party problems > - much better than throwing a 500 and leaving no record of what went > wrong. > > 6. Most importantly from my point of view, when a sysadmin asks where > Django logs errors in production we'll have a good answer for them! > > 7. As a general rule, I believe you can never have too much > information about what's going on with your web application. I've > never thought to myself "the problem with this bug is I've got too > much information about it". As for large log files, disk space is > cheap - and pluggable backends could ensure logs were sensibly > rotated. > > Places logging would be useful > == > > - Unhandled exceptions that make it up to the top of the Django stack > (and would cause a 500 error to be returned in production) > - The development web server could use logging for showing processed > requests (where currently these are just printed to stdout). > - Failed attempts at signing in to the admin could be logged, making > security audits easier. > - We could replace (or complement) django.connection.queries with a > log of executed SQL. This would make the answer to the common question > "how do I see what SQL is being executed" much more obvious. > - Stuff that loads things from INSTALLED_APPS could log what is being > loaded, making it much easier to spot and debug errors caused by code > being incorrectly loaded. > - Likewise, the template engine could log which templates are being > loaded from where, making it easier to debug problems stemming from an > incorrectly configured TEMPLATE_DIRS setting. > - We could use logging to address the problems with the template > engine failing silently - maybe some template errors (the ones more > likely to be accidental than just people relying on the fail-silent > behaviour deliberately) should be logged as warnings. > > Most of the above would be set to a low log level which by default > would not be handled, displayed or stored anywhere (logging.info or > similar). Maybe "./manage.py runserver --loglevel=info" could cause > such logs to be printed to the terminal while the development server > is running. > > Problems and challenges > === > > 1. The Python logging module isn't very nicely designed - its Java > heritage shines through, and things like logging.basicConfig behave in > unintuitive ways (if you call basicConfig twice the second call fails > silently but has no effect). This is why I suggest wrapping it