Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
Hi Graham, I've spent 2 times 45 min recently for reading howtos on writing own middleware and after that have decided to postpone this grownup's magic :) I mean that Karen hit the right level for a middling person like I am. It worked out in about 2 minutes. Said that, I am actually very thankful for the link, it is indeed interesting, I am likely to return to that. best regards -- Valery A.Khamenya On Wed, May 27, 2009 at 11:42 PM, Graham Dumpleton < graham.dumple...@gmail.com> wrote: > > > > On May 28, 5:42 am, Valery Khamenya wrote: > > Hi Karen, > > > > I appreciate very much your reply, thank you. > > > > the line is injected, it does logging fine, let's wait for the next > > occurrence. (yep, things are not reproducible) > > Also see: > > > http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response > > This explains how you can wrap the WSGI application as a whole to > track details of request and response without having to go inside of > Django code and make changes. > > Graham > > > > > On Wed, May 27, 2009 at 5:51 PM, Karen Tracey > wrote: > > > On Wed, May 27, 2009 at 10:23 AM, Valery Khamenya >wrote: > > > > >> Hi Karen, > > > > >> thanks for reply. I am looking in log files that I believe are created > by > > >> apache, at least they are created via usual apache statement like: > > > > >> ... > > >> CustomLog /var/www/mysite/logs/access.log combined > > >> ... > > > > >> In this sense, yes, I do see only a single HTTP-request in logs, but > can > > >> track multiple calls. > > > > >> How can I register those multiple calls? Easily, I send only one > single > > >> email notification per veiw function call, but I receive multiple > emails > > >> after one HTTP-request. > > > > > So I'd probably debug by adding logging. Specifically something like > this: > > > > > print >> sys.stderr, "WSGIHandler called to handle request; > > > environ=%r" % environ > > > > > to django/core/handlers/wsgi.py as the first line in the __call__ > method > > > for WSGIHandler. Each resulting entry you see in the Apache error log > > > indicates Django was called by Apache to handle a request, and the > PATH_INFO > > > key in the environ will show what the request is for, exactly. > > > > > Similarly add logging to where you send the email. Issue your single > > > request and check the access and error logs for where the duplication > is > > > happening. > > > > > If you get multiple email send logs for a single WSGIHandler __call__ > then > > > it's resulting from within Django or your code, and you can head down > the > > > path of logging the call stack at the email send point, or adding more > > > logging along the execution path, to figure out what's going on. > > > > > If you get multiple WSGIHandler __call__ logs for a single entry in > access > > > log, then it's something outside of Django that is causing the problem. > I > > > have no idea how that would be happening. > > > > > If you get a single WSGIHandler __call__ log and a single email send > log, > > > but multiple emails, then something is duplicating the sent emails, or > there > > > is somewhere else in your code sending emails. > > > > > Karen > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
On May 28, 5:42 am, Valery Khamenya wrote: > Hi Karen, > > I appreciate very much your reply, thank you. > > the line is injected, it does logging fine, let's wait for the next > occurrence. (yep, things are not reproducible) Also see: http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response This explains how you can wrap the WSGI application as a whole to track details of request and response without having to go inside of Django code and make changes. Graham > > On Wed, May 27, 2009 at 5:51 PM, Karen Tracey wrote: > > On Wed, May 27, 2009 at 10:23 AM, Valery Khamenya wrote: > > >> Hi Karen, > > >> thanks for reply. I am looking in log files that I believe are created by > >> apache, at least they are created via usual apache statement like: > > >> ... > >> CustomLog /var/www/mysite/logs/access.log combined > >> ... > > >> In this sense, yes, I do see only a single HTTP-request in logs, but can > >> track multiple calls. > > >> How can I register those multiple calls? Easily, I send only one single > >> email notification per veiw function call, but I receive multiple emails > >> after one HTTP-request. > > > So I'd probably debug by adding logging. Specifically something like this: > > > print >> sys.stderr, "WSGIHandler called to handle request; > > environ=%r" % environ > > > to django/core/handlers/wsgi.py as the first line in the __call__ method > > for WSGIHandler. Each resulting entry you see in the Apache error log > > indicates Django was called by Apache to handle a request, and the PATH_INFO > > key in the environ will show what the request is for, exactly. > > > Similarly add logging to where you send the email. Issue your single > > request and check the access and error logs for where the duplication is > > happening. > > > If you get multiple email send logs for a single WSGIHandler __call__ then > > it's resulting from within Django or your code, and you can head down the > > path of logging the call stack at the email send point, or adding more > > logging along the execution path, to figure out what's going on. > > > If you get multiple WSGIHandler __call__ logs for a single entry in access > > log, then it's something outside of Django that is causing the problem. I > > have no idea how that would be happening. > > > If you get a single WSGIHandler __call__ log and a single email send log, > > but multiple emails, then something is duplicating the sent emails, or there > > is somewhere else in your code sending emails. > > > Karen --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
Hi Karen, I appreciate very much your reply, thank you. the line is injected, it does logging fine, let's wait for the next occurrence. (yep, things are not reproducible) best regards -- Valery A.Khamenya On Wed, May 27, 2009 at 5:51 PM, Karen Tracey wrote: > On Wed, May 27, 2009 at 10:23 AM, Valery Khamenya wrote: > >> Hi Karen, >> >> thanks for reply. I am looking in log files that I believe are created by >> apache, at least they are created via usual apache statement like: >> >> ... >> CustomLog /var/www/mysite/logs/access.log combined >> ... >> >> In this sense, yes, I do see only a single HTTP-request in logs, but can >> track multiple calls. >> >> How can I register those multiple calls? Easily, I send only one single >> email notification per veiw function call, but I receive multiple emails >> after one HTTP-request. >> > > So I'd probably debug by adding logging. Specifically something like this: > > print >> sys.stderr, "WSGIHandler called to handle request; > environ=%r" % environ > > to django/core/handlers/wsgi.py as the first line in the __call__ method > for WSGIHandler. Each resulting entry you see in the Apache error log > indicates Django was called by Apache to handle a request, and the PATH_INFO > key in the environ will show what the request is for, exactly. > > Similarly add logging to where you send the email. Issue your single > request and check the access and error logs for where the duplication is > happening. > > If you get multiple email send logs for a single WSGIHandler __call__ then > it's resulting from within Django or your code, and you can head down the > path of logging the call stack at the email send point, or adding more > logging along the execution path, to figure out what's going on. > > If you get multiple WSGIHandler __call__ logs for a single entry in access > log, then it's something outside of Django that is causing the problem. I > have no idea how that would be happening. > > If you get a single WSGIHandler __call__ log and a single email send log, > but multiple emails, then something is duplicating the sent emails, or there > is somewhere else in your code sending emails. > > Karen > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
On Wed, May 27, 2009 at 10:23 AM, Valery Khamenya wrote: > Hi Karen, > > thanks for reply. I am looking in log files that I believe are created by > apache, at least they are created via usual apache statement like: > > ... > CustomLog /var/www/mysite/logs/access.log combined > ... > > In this sense, yes, I do see only a single HTTP-request in logs, but can > track multiple calls. > > How can I register those multiple calls? Easily, I send only one single > email notification per veiw function call, but I receive multiple emails > after one HTTP-request. > So I'd probably debug by adding logging. Specifically something like this: print >> sys.stderr, "WSGIHandler called to handle request; environ=%r" % environ to django/core/handlers/wsgi.py as the first line in the __call__ method for WSGIHandler. Each resulting entry you see in the Apache error log indicates Django was called by Apache to handle a request, and the PATH_INFO key in the environ will show what the request is for, exactly. Similarly add logging to where you send the email. Issue your single request and check the access and error logs for where the duplication is happening. If you get multiple email send logs for a single WSGIHandler __call__ then it's resulting from within Django or your code, and you can head down the path of logging the call stack at the email send point, or adding more logging along the execution path, to figure out what's going on. If you get multiple WSGIHandler __call__ logs for a single entry in access log, then it's something outside of Django that is causing the problem. I have no idea how that would be happening. If you get a single WSGIHandler __call__ log and a single email send log, but multiple emails, then something is duplicating the sent emails, or there is somewhere else in your code sending emails. Karen --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
Hi Karen, thanks for reply. I am looking in log files that I believe are created by apache, at least they are created via usual apache statement like: ... CustomLog /var/www/mysite/logs/access.log combined ... In this sense, yes, I do see only a single HTTP-request in logs, but can track multiple calls. How can I register those multiple calls? Easily, I send only one single email notification per veiw function call, but I receive multiple emails after one HTTP-request. best regards -- Valery A.Khamenya On Wed, May 27, 2009 at 4:09 PM, Karen Tracey wrote: > On Wed, May 27, 2009 at 7:59 AM, Valery Khamenya wrote: > >> Hi Graham, >> >> I do not do touch anymore. However multiple view func calls per single >> HTTP-request still happen. >> > > It's far more likely that Django is being routed multiple HTTP requests > than that Django is calling your view function multiple times for a single > HTTP request. Every time I recall someone reporting this type of thing it > turns out that their templates have something specified that is causing > unexpected additional HTTP requests. If you turn on access logging in > Apache, do you really see only a single request resulting in multiple > executions of your view? > > Karen > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
On Wed, May 27, 2009 at 7:59 AM, Valery Khamenya wrote: > Hi Graham, > > I do not do touch anymore. However multiple view func calls per single > HTTP-request still happen. > It's far more likely that Django is being routed multiple HTTP requests than that Django is calling your view function multiple times for a single HTTP request. Every time I recall someone reporting this type of thing it turns out that their templates have something specified that is causing unexpected additional HTTP requests. If you turn on access logging in Apache, do you really see only a single request resulting in multiple executions of your view? Karen --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
Hi Graham, I do not do touch anymore. However multiple view func calls per single HTTP-request still happen. best regards -- Valery A.Khamenya On Wed, May 27, 2009 at 1:47 PM, Graham Dumpleton < graham.dumple...@gmail.com> wrote: > > > > On May 27, 9:11 pm, Valery wrote: > > Hi Graham > > > > the issue is still here even after mod_python is removed. I still > > register multiple calls of my view function for a single HTTP-request > > (proved via Django log file). For example, 25 calls per HTTP request > > today. > > > > And you can't believe, as always -- "I did nothing special!" (c) > > > > No one got the same? > > Does this happen even for the very first request against a newly > started Apache, or does the multiple calls only start occurring after > you have touched .wsgi file. > > BTW, touching the .wsgi file isn't doing what you think it does for > prefork/embedded mode. Read: > > http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode > > Graham > > > regards > > Valery > > > > On May 15, 1:55 am, Graham Dumpleton > > wrote: > > > > > On May 14, 11:08 pm, Valery wrote: > > > > > > By the way, regarding the 30 func calls per HTTP request you'd mean > > > > it is a bare conflict WSGI vs mod_python? > > > > > That is likely to be a Django issue, you will have to explain that one > > > better and maybe someone else can help. > > > > > Graham > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
On May 27, 9:11 pm, Valery wrote: > Hi Graham > > the issue is still here even after mod_python is removed. I still > register multiple calls of my view function for a single HTTP-request > (proved via Django log file). For example, 25 calls per HTTP request > today. > > And you can't believe, as always -- "I did nothing special!" (c) > > No one got the same? Does this happen even for the very first request against a newly started Apache, or does the multiple calls only start occurring after you have touched .wsgi file. BTW, touching the .wsgi file isn't doing what you think it does for prefork/embedded mode. Read: http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode Graham > regards > Valery > > On May 15, 1:55 am, Graham Dumpleton > wrote: > > > On May 14, 11:08 pm, Valery wrote: > > > > By the way, regarding the 30 func calls per HTTP request you'd mean > > > it is a bare conflict WSGI vs mod_python? > > > That is likely to be a Django issue, you will have to explain that one > > better and maybe someone else can help. > > > Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
Hi Graham the issue is still here even after mod_python is removed. I still register multiple calls of my view function for a single HTTP-request (proved via Django log file). For example, 25 calls per HTTP request today. And you can't believe, as always -- "I did nothing special!" (c) No one got the same? regards Valery On May 15, 1:55 am, Graham Dumpleton wrote: > On May 14, 11:08 pm, Valery wrote: > > > By the way, regarding the 30 func calls per HTTP request you'd mean > > it is a bare conflict WSGI vs mod_python? > > That is likely to be a Django issue, you will have to explain that one > better and maybe someone else can help. > > Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
On May 14, 11:08 pm, Valery wrote: > By the way, regarding the 30 func calls per HTTP request you'd mean > it is a bare conflict WSGI vs mod_python? That is likely to be a Django issue, you will have to explain that one better and maybe someone else can help. Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
Hi Graham, many thanks for your quick and very helpful reply! > You need to ascertain whether they were Apache parent process, Apache > server child worker processes, or mod_wsgi daemon mode processes. OK i'd check next time, i've seen your hint on how to do it, thanks! > > a. both issues appeared after I've switsched to WSGI dipatching model. > > Did you disable mod_python out of Apache when you switched. Loading > both at same time is not recommended as can cause some problems. mod_python is indeed *not* disabled because of other project running on the same server in mod_python. OK, great, I see, am going to change this. > > b. I use wsgi-based dispatching for apache prefork (unfortunatelly I am > > restricted for prefork currently) as descreibed here right in the beginning > > of the document:http://pinaxproject.com/docs/dev/deployment.html > > Be aware of the dangers of prefork. Read: > > http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa... Oh, interesting, I keep on reading it to the end after end of my answer. > Why are you restricted to using prefork? Also, that documentation says > to use daemon mode which makes the MPM irrelevant, because if you > configure it properly the application runs in separate process like in > fastcgi. I am restricted to prefork because of a small PHP-contact form currently used in other application running on the same server. If there would be a slim way to kick it away, it would be nice. However I see it as an overkill to create a Django project just to implement a short contact form within a static web-site... > What the documentation doesn't explain is to use the option: > [...] > http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDae... great, it helps a lot, thanks! > > c. Also, to restart my app I sometime use "touch my-dispatcher.wsgi" as it > > was mentioned somewhere in django docs. > > Perhaps also read mod_wsgi documentation on source code reloading to > find out what that is doing: > > http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode hm, I am curious what I am doing wrong, gotta read it. > You need to identify what role the process was doing. The display-name > option can help with that. Also set LogLevel to at least 'info' as > explained in: > http://code.google.com/p/modwsgi/wiki/DebuggingTechniques > Then you need to work out what the process is perhaps hanging on. OK, I'll do. By the way, regarding the 30 func calls per HTTP request you'd mean it is a bare conflict WSGI vs mod_python? Kind regards Valery --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Troubles after switching to WSGI: lost apache processes, multiple view func calls per HTTP request
On May 14, 7:14 pm, Valery Khamenya wrote:
> Hi,
>
> I have 2 issues that presumably could be related:
>
> 1. when I totally stop apache, I see that some apache processes remain
> running. When I repeat the stop operation, apache says that apache is not
> running, whereas the processes do exist. So, I have to kill them manually.
You need to ascertain whether they were Apache parent process, Apache
server child worker processes, or mod_wsgi daemon mode processes.
> 2. For a single HTTP request I see that multiple calls of the corresponding
> rendering function of a view.py are invoked. The overhead like 30 calls per
> single HTTP request is not acceptable neither from performance nor logic
> requirements.
>
> Details:
>
> a. both issues appeared after I've switsched to WSGI dipatching model.
Did you disable mod_python out of Apache when you switched. Loading
both at same time is not recommended as can cause some problems.
> b. I use wsgi-based dispatching for apache prefork (unfortunatelly I am
> restricted for prefork currently) as descreibed here right in the beginning
> of the document:http://pinaxproject.com/docs/dev/deployment.html
Be aware of the dangers of prefork. Read:
http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
Why are you restricted to using prefork? Also, that documentation says
to use daemon mode which makes the MPM irrelevant, because if you
configure it properly the application runs in separate process like in
fastcgi.
What the documentation doesn't explain is to use the option:
display-name=%{GLOBAL}
to WSGIDaemonProcess. This will allow you to use 'ps' to see whether
the Apache process is actually the Django process, ie., mod_wsgi
daemon process, or not. See the official mod_wsgi documentation for
more detail:
http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess
> c. Also, to restart my app I sometime use "touch my-dispatcher.wsgi" as it
> was mentioned somewhere in django docs.
Perhaps also read mod_wsgi documentation on source code reloading to
find out what that is doing:
http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode
> d. the file my-dispatcher.wsgi is basically like
> this:http://github.com/pinax/pinax/blob/1f6d2d1216c5e8539639ef561349e4277f...
>
> Your input is very welcome, thanks in advance.
>
> P.S. I doubt that the troubles are purely Pinax-related, but if so, I am
> sorry.
You need to identify what role the process was doing. The display-name
option can help with that. Also set LogLevel to at least 'info' as
explained in:
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques
Then you need to work out what the process is perhaps hanging on.
Graham
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
