Re: Django keeps growing, performance keeps dropping
Take a shot at enabling the profiling (Jeremy provided a link a few msgs back) - it'll give you a huge amount of detail. -joe On 3/12/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > > Well crud. My host won't turn on slow query logging. So, if you know > you've got a bunch of not-so-good queries (as shown above), how would > one try to spot them? > > > > On Mar 10, 4:58 pm, "gilhad" <[EMAIL PROTECTED]> wrote: > > On Mar 7, 4:26 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > > wrote: > > > > > My site just keeps getting bigger and bigger, and the performance has > > > gotten excruciatingly slow. > > > ... > > > Problem is, I really don't know what to do about it, or even what I > > > need to be looking at, really. I've gone through and looked at the > > > views and removed imports I didn't need (I had a lot of import * type > > > statements), but that only helped a little. I've tried figuring out > > > select_all, but don't know if that would really help me or not. > > > > (-; Very simple test: copy this code to another diretory, attach it to > > another nearly empty database and if it hugely speeds up thinks, then > > the problem is not in including libraries, but in having big database > > and/or bad algorithm for data manipulation. So there lies the way to > > get more speed ;-) > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
Well crud. My host won't turn on slow query logging. So, if you know you've got a bunch of not-so-good queries (as shown above), how would one try to spot them? On Mar 10, 4:58 pm, "gilhad" <[EMAIL PROTECTED]> wrote: > On Mar 7, 4:26 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > wrote: > > > My site just keeps getting bigger and bigger, and the performance has > > gotten excruciatingly slow. > > ... > > Problem is, I really don't know what to do about it, or even what I > > need to be looking at, really. I've gone through and looked at the > > views and removed imports I didn't need (I had a lot of import * type > > statements), but that only helped a little. I've tried figuring out > > select_all, but don't know if that would really help me or not. > > (-; Very simple test: copy this code to another diretory, attach it to > another nearly empty database and if it hugely speeds up thinks, then > the problem is not in including libraries, but in having big database > and/or bad algorithm for data manipulation. So there lies the way to > get more speed ;-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
On Mar 7, 4:26 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > My site just keeps getting bigger and bigger, and the performance has > gotten excruciatingly slow. > ... > Problem is, I really don't know what to do about it, or even what I > need to be looking at, really. I've gone through and looked at the > views and removed imports I didn't need (I had a lot of import * type > statements), but that only helped a little. I've tried figuring out > select_all, but don't know if that would really help me or not. (-; Very simple test: copy this code to another diretory, attach it to another nearly empty database and if it hugely speeds up thinks, then the problem is not in including libraries, but in having big database and/or bad algorithm for data manipulation. So there lies the way to get more speed ;-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
On Mar 9, 4:08 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > Both those numbers seem pretty excessive to me. My question to this > board is, what sort of query would lead a newbie like myself into this > sort of mess.. in other words, what would be an example of bad code > I've probably done and need to remedy? Slow query logging will tell you just that. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
I still don't have slow query logging set up, but I am looking at my database, and what jumped out was: Handler_read_rnd3,013 k The number of requests to read a row based on a fixed position. This is high if you are doing a lot of queries that require sorting of the result. You probably have a lot of queries that require MySQL to scan whole tables or you have joins that don't use keys properly. Handler_read_rnd_next 133 M The number of requests to read the next row in the data file. This is high if you are doing a lot of table scans. Generally this suggests that your tables are not properly indexed or that your queries are not written to take advantage of the indexes you have. Both those numbers seem pretty excessive to me. My question to this board is, what sort of query would lead a newbie like myself into this sort of mess.. in other words, what would be an example of bad code I've probably done and need to remedy? On Mar 7, 4:43 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > Bren, I hear ya, but I'm finding a lot of easy fixes while I really > pick through my (not-very-good) code. It's been pretty instructional. > > Thanks to everyone for your suggestions. Lots to chew on here. > > On Mar 7, 3:50 pm, "Bren" <[EMAIL PROTECTED]> wrote: > > > Personally, I would not get into those details until you profile your > > database. Like someone else said, if you're using MySQL turn on the > > slow query log. If you run Postgres I'm sure you can find plenty of > > documentation about how to monitor and tune that. I just did this and > > found two queries that were taking over a minute each to complete, so > > this was very helpful. > > > Also, do you do any caching? > > > I would start with the big stuff and see if that helps with > >performance. > > > On Mar 7, 3:18 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > > wrote: > > > > On a more specific note, I'm finding lots of places I've done > > > something like: > > > if request.user.is_authenticated() and request.user.is_staff): > > > > It seems to me there's a redundant check being made there. Wouldn't > > > one have to be authenticated for is_staff to evaluate to true, meaning > > > I can safely just say > > > > if request.user.is_staff? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
Bren, I hear ya, but I'm finding a lot of easy fixes while I really pick through my (not-very-good) code. It's been pretty instructional. Thanks to everyone for your suggestions. Lots to chew on here. On Mar 7, 3:50 pm, "Bren" <[EMAIL PROTECTED]> wrote: > Personally, I would not get into those details until you profile your > database. Like someone else said, if you're using MySQL turn on the > slow query log. If you run Postgres I'm sure you can find plenty of > documentation about how to monitor and tune that. I just did this and > found two queries that were taking over a minute each to complete, so > this was very helpful. > > Also, do you do any caching? > > I would start with the big stuff and see if that helps with > performance. > > On Mar 7, 3:18 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > wrote: > > > On a more specific note, I'm finding lots of places I've done > > something like: > > if request.user.is_authenticated() and request.user.is_staff): > > > It seems to me there's a redundant check being made there. Wouldn't > > one have to be authenticated for is_staff to evaluate to true, meaning > > I can safely just say > > > if request.user.is_staff? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
Personally, I would not get into those details until you profile your database. Like someone else said, if you're using MySQL turn on the slow query log. If you run Postgres I'm sure you can find plenty of documentation about how to monitor and tune that. I just did this and found two queries that were taking over a minute each to complete, so this was very helpful. Also, do you do any caching? I would start with the big stuff and see if that helps with performance. On Mar 7, 3:18 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > On a more specific note, I'm finding lots of places I've done > something like: > if request.user.is_authenticated() and request.user.is_staff): > > It seems to me there's a redundant check being made there. Wouldn't > one have to be authenticated for is_staff to evaluate to true, meaning > I can safely just say > > if request.user.is_staff? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
i do not know the answer to your question, but it reminded me that
there is a decorator like @login_required or something that you could
use in stead of checking your self. (if it applies to your case).
konstantin
On Mar 7, 4:18 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
wrote:
> On a more specific note, I'm finding lots of places I've done
> something like:
> if request.user.is_authenticated() and request.user.is_staff):
>
> It seems to me there's a redundant check being made there. Wouldn't
> one have to be authenticated for is_staff to evaluate to true, meaning
> I can safely just say
>
> if request.user.is_staff?
>
> On Mar 7, 2:45 pm, "Karsu" <[EMAIL PROTECTED]> wrote:
>
>
>
> > Hello,
>
> > First, If you use many relations it is very important to use JOIN
> > like: Example.objects.filter(first_object__second__third=something)
>
> > It is important to use select_related too (also in template!!)
>
> > example:
>
> > {% for item in example.objects.all %}
> >{{ item.related_object.value }}
> > {% endfor %}
>
> > TO
>
> > {% for item in example.objects.select_related %}
> >{{ item.related_object.value }}
> > {% endfor %}- Hide quoted text -
>
> - Show quoted text -
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
On a more specific note, I'm finding lots of places I've done
something like:
if request.user.is_authenticated() and request.user.is_staff):
It seems to me there's a redundant check being made there. Wouldn't
one have to be authenticated for is_staff to evaluate to true, meaning
I can safely just say
if request.user.is_staff?
On Mar 7, 2:45 pm, "Karsu" <[EMAIL PROTECTED]> wrote:
> Hello,
>
> First, If you use many relations it is very important to use JOIN
> like: Example.objects.filter(first_object__second__third=something)
>
> It is important to use select_related too (also in template!!)
>
> example:
>
> {% for item in example.objects.all %}
>{{ item.related_object.value }}
> {% endfor %}
>
> TO
>
> {% for item in example.objects.select_related %}
>{{ item.related_object.value }}
> {% endfor %}
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
Hello,
First, If you use many relations it is very important to use JOIN
like: Example.objects.filter(first_object__second__third=something)
It is important to use select_related too (also in template!!)
example:
{% for item in example.objects.all %}
{{ item.related_object.value }}
{% endfor %}
TO
{% for item in example.objects.select_related %}
{{ item.related_object.value }}
{% endfor %}
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
On 3/7/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > Sounds good. How does one go about profiling? Is there a tutorial or > something you can point me to? If you're on Linux and Apache, you'll like this: http://www.rkblog.rk.edu.pl/w/p/django-profiling-hotshot-and-kcachegrind/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
First, what database platform do you use: MySQL or Postgres. MySQL has ways of logging slow queries (see /etc/mysql/my.cnf), this will give you an understanding of which queries are slow and you may be able to add indices where necessary. Secondly, if you use PHPmyAdmin it can tell you what tuning parameters need to be changed. You may want to do this on your development environment, not on production. I have no experience with Postgres. But learning how to tune a database is something that you can learn thru the MySQL or Postgres channels. Sometimes an increase in table or buffer size can make a serious difference in database responsiveness. good luck. On Mar 7, 9:47 am, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > Sounds good. How does one go about profiling? Is there a tutorial or > something you can point me to? > > Also, I appreciate everyone's patience with these dumb and vague > questions. It's frustrating being a stupid newbie. > > For what it's worth, I have found some things, most notably that I was > - for whatever reason - cycling through every post for every topic > when I built the topic list. That might be slowing things down a bit. > > On Mar 7, 10:26 am, "James Bennett" <[EMAIL PROTECTED]> wrote: > > > On 3/7/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > > > I think the problem lies in the forum (a modified Myghtyboard install) > > > and in it's relationship to users. Since both forum posts and users > > > keeps growing, I wonder if I've created some sort of exponential > > > growth situation for myself. > > > Probably the first thing to do is start profiling to figure out where > > the time is actually being spent; without knowing where your > > bottleneck is, it's hard to suggest possible solutions. > > > -- > > "Bureaucrat Conrad, you are technically correct -- the best kind of > > correct." --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
"[EMAIL PROTECTED]" <[EMAIL PROTECTED]> writes: > For what it's worth, I have found some things, most notably that I was > - for whatever reason - cycling through every post for every topic > when I built the topic list. That might be slowing things down a bit. Such things are the reason for bad performance in many applications. Selections should be performed inside the database. That's one thing that databases are for. That means you should have a topic table or an attribute in your model and use filters then. Regards, Frank --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
Sounds good. How does one go about profiling? Is there a tutorial or something you can point me to? Also, I appreciate everyone's patience with these dumb and vague questions. It's frustrating being a stupid newbie. For what it's worth, I have found some things, most notably that I was - for whatever reason - cycling through every post for every topic when I built the topic list. That might be slowing things down a bit. On Mar 7, 10:26 am, "James Bennett" <[EMAIL PROTECTED]> wrote: > On 3/7/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > > I think the problem lies in the forum (a modified Myghtyboard install) > > and in it's relationship to users. Since both forum posts and users > > keeps growing, I wonder if I've created some sort of exponential > > growth situation for myself. > > Probably the first thing to do is start profiling to figure out where > the time is actually being spent; without knowing where your > bottleneck is, it's hard to suggest possible solutions. > > -- > "Bureaucrat Conrad, you are technically correct -- the best kind of correct." --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
On 3/7/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > I think the problem lies in the forum (a modified Myghtyboard install) > and in it's relationship to users. Since both forum posts and users > keeps growing, I wonder if I've created some sort of exponential > growth situation for myself. Probably the first thing to do is start profiling to figure out where the time is actually being spent; without knowing where your bottleneck is, it's hard to suggest possible solutions. -- "Bureaucrat Conrad, you are technically correct -- the best kind of correct." --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
It's entirely possible my data relations aren't optimal. The database may need indexes, I may be making too many queries. All of these are very good possibilities. Problem is, I don't really know where to begin checking this stuff. Obviously, I'm in over my head, but I'm trying to learn to swim here. On Mar 7, 9:52 am, Michael Radziej <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED]: > > > My site just keeps getting bigger and bigger, and the performance has > > gotten excruciatingly slow. > > - database needs indexes? > - check how many queries your views generate. Perhaps you need > reorganization or select_related. > > -- > noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg - > Tel +49-911-9352-0 - Fax +49-911-9352-100http://www.noris.de- The > IT-Outsourcing Company > > Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk - > Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
[EMAIL PROTECTED]: > My site just keeps getting bigger and bigger, and the performance has > gotten excruciatingly slow. - database needs indexes? - check how many queries your views generate. Perhaps you need reorganization or select_related. -- noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg - Tel +49-911-9352-0 - Fax +49-911-9352-100 http://www.noris.de - The IT-Outsourcing Company Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk - Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Django keeps growing, performance keeps dropping
Hello, may be your data model is not optimal? too many relations? relations are too complex? konstantin On Mar 7, 10:26 am, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > My site just keeps getting bigger and bigger, and the performance has > gotten excruciatingly slow. > > I think the problem lies in the forum (a modified Myghtyboard install) > and in it's relationship to users. Since both forum posts and users > keeps growing, I wonder if I've created some sort of exponential > growth situation for myself. > > Problem is, I really don't know what to do about it, or even what I > need to be looking at, really. I've gone through and looked at the > views and removed imports I didn't need (I had a lot of import * type > statements), but that only helped a little. I've tried figuring out > select_all, but don't know if that would really help me or not. > > Any suggestions, anyone willing to look at the code, anything... it's > all welcome. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Django keeps growing, performance keeps dropping
My site just keeps getting bigger and bigger, and the performance has gotten excruciatingly slow. I think the problem lies in the forum (a modified Myghtyboard install) and in it's relationship to users. Since both forum posts and users keeps growing, I wonder if I've created some sort of exponential growth situation for myself. Problem is, I really don't know what to do about it, or even what I need to be looking at, really. I've gone through and looked at the views and removed imports I didn't need (I had a lot of import * type statements), but that only helped a little. I've tried figuring out select_all, but don't know if that would really help me or not. Any suggestions, anyone willing to look at the code, anything... it's all welcome. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
