Re: Django model object allocation performance

[oyiptong]

That was exactly my point. I got around my issue by just using the
dictionaries all over my application.

This did bypass the creation of the objects. My cpu usage and load
average went to drastically.

On Feb 2, 4:33 am, Thomas Weholt  wrote:
> I might be missing something here, but in the first example you do:
>
> len(list(m))
>
> and the next example you do:
>
> len(list(m.values()))
>
> As far as I know, calling .values() will bypass the creation of Django
> ORM objects and just return a dictionary for each record instead of a
> model instance. This is much more efficient if you just need the
> values anyway.
>
> Regards,
> Thomas
>
>
>
>
>
>
>
>
>
> On Wed, Feb 2, 2011 at 10:28 AM, Daniel Roseman  wrote:
> > On Tuesday, February 1, 2011 11:52:49 PM UTC, oyiptong wrote:
>
> >> Hello,
>
> >> I have been seeing a big performance degradation with my application
> >> in production.
> >> The traffic is roughly 2.5K pageviews per day. I can expect each page
> >> to load 100 model objects in memory.
>
> >> Some might overlap, but I have a large inventory of objects to show.
>
> >> I have noticed that pages have been taking longer and longer to load,
> >> at a point where its unacceptable.
> >> Looking at the wsgi processes, i found that they a request seems to be
> >> taking up a large amount of CPU usage.
>
> >> I've been poking around to see how I could improve things, and I've
> >> noticed this behavior:
>
> >> from project.app.models import Model
> >> m = Model.objects.all()[200:300]
> >> len(list(m))
>
> >> This takes several seconds.
>
> >> from project.app.models import Model
> >> m = Model.objects.all()[400:500]
> >> len(list(m.values()))
>
> >> This is much faster. If you're gonna try it, make sure you choose a
> >> range of objects that are not already in memory.
>
> >> Is the only difference between the two object allocation?
> >> If object allocation is what is costing me so much CPU power, how can
> >> I get around it?
> >> Is using the values method the only option?
>
> > Is `len` just an example? If that's really all you need, you should be using
> > m.count() instead.
> > --
> > DR.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django users" group.
> > To post to this group, send email to django-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > django-users+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/django-users?hl=en.
>
> --
> Mvh/Best regards,
> Thomas Weholthttp://www.weholt.org

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django model object allocation performance

[oyiptong]

Yeah, it was just an example

On Feb 2, 4:28 am, Daniel Roseman  wrote:
> On Tuesday, February 1, 2011 11:52:49 PM UTC, oyiptong wrote:
>
> > Hello,
>
> > I have been seeing a big performance degradation with my application
> > in production.
> > The traffic is roughly 2.5K pageviews per day. I can expect each page
> > to load 100 model objects in memory.
>
> > Some might overlap, but I have a large inventory of objects to show.
>
> > I have noticed that pages have been taking longer and longer to load,
> > at a point where its unacceptable.
> > Looking at the wsgi processes, i found that they a request seems to be
> > taking up a large amount of CPU usage.
>
> > I've been poking around to see how I could improve things, and I've
> > noticed this behavior:
>
> > from project.app.models import Model
> > m = Model.objects.all()[200:300]
> > len(list(m))
>
> > This takes several seconds.
>
> > from project.app.models import Model
> > m = Model.objects.all()[400:500]
> > len(list(m.values()))
>
> > This is much faster. If you're gonna try it, make sure you choose a
> > range of objects that are not already in memory.
>
> > Is the only difference between the two object allocation?
> > If object allocation is what is costing me so much CPU power, how can
> > I get around it?
> > Is using the values method the only option?
>
> Is `len` just an example? If that's really all you need, you should be using
> m.count() instead.
> --
> DR.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django model object allocation performance

[Thomas Weholt]

I might be missing something here, but in the first example you do:

len(list(m))

and the next example you do:

len(list(m.values()))

As far as I know, calling .values() will bypass the creation of Django
ORM objects and just return a dictionary for each record instead of a
model instance. This is much more efficient if you just need the
values anyway.

Regards,
Thomas

On Wed, Feb 2, 2011 at 10:28 AM, Daniel Roseman  wrote:
> On Tuesday, February 1, 2011 11:52:49 PM UTC, oyiptong wrote:
>>
>> Hello,
>>
>> I have been seeing a big performance degradation with my application
>> in production.
>> The traffic is roughly 2.5K pageviews per day. I can expect each page
>> to load 100 model objects in memory.
>>
>> Some might overlap, but I have a large inventory of objects to show.
>>
>> I have noticed that pages have been taking longer and longer to load,
>> at a point where its unacceptable.
>> Looking at the wsgi processes, i found that they a request seems to be
>> taking up a large amount of CPU usage.
>>
>> I've been poking around to see how I could improve things, and I've
>> noticed this behavior:
>>
>> from project.app.models import Model
>> m = Model.objects.all()[200:300]
>> len(list(m))
>>
>> This takes several seconds.
>>
>> from project.app.models import Model
>> m = Model.objects.all()[400:500]
>> len(list(m.values()))
>>
>> This is much faster. If you're gonna try it, make sure you choose a
>> range of objects that are not already in memory.
>>
>> Is the only difference between the two object allocation?
>> If object allocation is what is costing me so much CPU power, how can
>> I get around it?
>> Is using the values method the only option?
>
> Is `len` just an example? If that's really all you need, you should be using
> m.count() instead.
> --
> DR.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>



-- 
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django model object allocation performance

[Daniel Roseman]

On Tuesday, February 1, 2011 11:52:49 PM UTC, oyiptong wrote:
>
> Hello, 
>
> I have been seeing a big performance degradation with my application 
> in production. 
> The traffic is roughly 2.5K pageviews per day. I can expect each page 
> to load 100 model objects in memory. 
>
> Some might overlap, but I have a large inventory of objects to show. 
>
> I have noticed that pages have been taking longer and longer to load, 
> at a point where its unacceptable. 
> Looking at the wsgi processes, i found that they a request seems to be 
> taking up a large amount of CPU usage. 
>
> I've been poking around to see how I could improve things, and I've 
> noticed this behavior: 
>
> from project.app.models import Model 
> m = Model.objects.all()[200:300] 
> len(list(m)) 
>
> This takes several seconds. 
>
> from project.app.models import Model 
> m = Model.objects.all()[400:500] 
> len(list(m.values())) 
>
> This is much faster. If you're gonna try it, make sure you choose a 
> range of objects that are not already in memory. 
>
> Is the only difference between the two object allocation? 
> If object allocation is what is costing me so much CPU power, how can 
> I get around it? 
> Is using the values method the only option?


Is `len` just an example? If that's really all you need, you should be using 
m.count() instead.
--
DR. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Django model object allocation performance

[oyiptong]

Hello,

I have been seeing a big performance degradation with my application
in production.
The traffic is roughly 2.5K pageviews per day. I can expect each page
to load 100 model objects in memory.

Some might overlap, but I have a large inventory of objects to show.

I have noticed that pages have been taking longer and longer to load,
at a point where its unacceptable.
Looking at the wsgi processes, i found that they a request seems to be
taking up a large amount of CPU usage.

I've been poking around to see how I could improve things, and I've
noticed this behavior:

from project.app.models import Model
m = Model.objects.all()[200:300]
len(list(m))

This takes several seconds.

from project.app.models import Model
m = Model.objects.all()[400:500]
len(list(m.values()))

This is much faster. If you're gonna try it, make sure you choose a
range of objects that are not already in memory.

Is the only difference between the two object allocation?
If object allocation is what is costing me so much CPU power, how can
I get around it?
Is using the values method the only option?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.