Re: django 1.1 performance versus django 1.2 performance

[Steve Holden]

Don't know whether this is relevant or not but the Python Software
Foundation has just decided to establish a facility that allows speed
comparison of different Python releases. The thinking behind this is (at
least) twofold:

1) People will more easily be able to determine which is the best Python
implementation for them to use on their specific problems, and

2) Developers will be able to see relatively easily whether their changes
make a positive or negative difference to performance, which /might/ help
eliminate the performance sawtoooth (gets worse with a major release and
then minor releases incre,entally improve it).

I don't know whether we can share our shiny new hardware with anyone else,
though ;-)

regards
 Stefe

On Sat, Apr 23, 2011 at 1:19 AM, Russell Keith-Magee <
russ...@keith-magee.com> wrote:

> > On Fri, Apr 22, 2011 at 9:22 PM, Shawn Milochik 
> wrote:
> >>
> >> This was mentioned in Eric Florenzano's talk at DjangoCon 2010. Each
> >> version has gotten slower.
> >>
> >> I haven't heard anything about the cause or plans to fix this, though.
> >> If you've got a good test suite you can always use tools like Python's
> >> profile module to track down slowdowns in your Django code and
> >> contribute that knowledge to the developers' list.
> >>
>
> On Sat, Apr 23, 2011 at 11:26 AM, Peter Portante
>  wrote:
> > Is there a plan in place to address this?
>
> No - there isn't a plan to address this, because it isn't clear what "this"
> is.
>
> While it is known that there has been a slowdown between versions,
> that slowdown has been accompanied by a massive increase in
> functionality -- for example, the 1.1->1.2 transition introduced
> support for multiple databases. To the best of my knowledge, the
> performance slowdown highlighted by Eric at Djangocon was relatively
> small - 5-10%, not on the order of 30-50% slowdown. This matches with
> my personal experience of upgrading.
>
> There are two performance issues in database operations that I'm aware
> of that people commonly point to.
>
> Firstly -- deep cloning of query sets. Cloning a queryset is an
> expensive operation, and if you are making extensive use of operations
> that clone (e.g., very long chains of complex filters) you may see
> performance issues. This isn't something that can be easily avoided --
> in order to preserve the API guarantees of a query set, deep cloning
> is required.
>
> Secondly, the backwards compatibility infrastructure introduced in
> order to support multiple databases. In order to guarantee backwards
> compatibility, the value preparation methods on fields needed to use
> some moderately expensive introspection methods. However because these
> methods were introduced for 1.2, and we're currently preparing 1.4,
> these introspection methods have just been removed from trunk.
>
> Beyond these two issues -- we're always interested in improving
> performance, so if you can identify a specific performance issue,
> we're happy to look at it. The more evidence you can provide (profile
> runs, sample queries that are observably slower, and so on), the
> easier it will be to address these problems.
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>


-- 
Steve Holden+1 571 484 6266  +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.1 performance versus django 1.2 performance

[Peter Portante]

But what changed in 1.2 vs. 1.1 that caused cloning to become slower? Is it
fixes for correctness that were missing?

And if 1.3 is slower than 1.2, why the slow down there?

If the slowness is for a good reason (correctness is about the only one I
can think of that is really valid), then fine. But if the slowdown is just
because it is slower and no one knows why, let's add a performance test for
cloning and make sure Django does not slow down (yeah, this is a hand wave
as performance is a tough thing to maintain, but well worth the effort).

Also, just simply creating a DB object is slower with get_or_create. See
attached script that I used to harp on 1.1 vs. 1.2.5 (slightly modified to
change the names, etc. and requires a proper django view to work). I ran
this against both sqlite and mysql.

-peter


On Mon, Apr 25, 2011 at 9:36 PM, Alexander Schepanovski
wrote:

> > Is there a way to identify which queries use cloning?
>
> Every filter, exclude, order_by, select_related and some other calls
> clones it. Slicing too.
>
> > Is there a way rewrite those queries to avoid cloning overhead?
>
> You can rewrite
> qs.filter(a=1).exclude(b=2) as qs.filter(Q(a=1) & !Q(b=2))
> or
> qs = qs.filter(a=1); ...; qs = qs.filter(b=2) as q = {}; q['a'] =
> 1; ...; q['b'] = 2; qs = qs.filter(**q)
>
> But you can't avoid several cloning procs for queryset not modifying
> django or monkey patching.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.



pound.py
Description: Binary data

Re: django 1.1 performance versus django 1.2 performance

[Alexander Schepanovski]

> Is there a way to identify which queries use cloning?

Every filter, exclude, order_by, select_related and some other calls
clones it. Slicing too.

> Is there a way rewrite those queries to avoid cloning overhead?

You can rewrite
qs.filter(a=1).exclude(b=2) as qs.filter(Q(a=1) & !Q(b=2))
or
qs = qs.filter(a=1); ...; qs = qs.filter(b=2) as q = {}; q['a'] =
1; ...; q['b'] = 2; qs = qs.filter(**q)

But you can't avoid several cloning procs for queryset not modifying
django or monkey patching.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.1 performance versus django 1.2 performance

[ydjango]

Is there a way to identify which queries use cloning?

Is there a way rewrite those queries to avoid cloning overhead?

(If many of the queries use cloning then  migrating one of my django
apps to 1.2.x may not be right for me.)

On Apr 25, 9:39 am, djeff  wrote:
> Russ,
>
> Thanks for the response.  You have confirmed what I'm seeing in the profile
> dump.  The cloning of a queryset is expensive and it is more expensive in
> 1.2 than with 1.1.  I really appreciate the feedback.  
>
> Jeff Fischer

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.1 performance versus django 1.2 performance

[djeff]

Russ,

Thanks for the response.  You have confirmed what I'm seeing in the profile 
dump.  The cloning of a queryset is expensive and it is more expensive in 
1.2 than with 1.1.  I really appreciate the feedback.  

Jeff Fischer

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.1 performance versus django 1.2 performance

[Alexander Schepanovski]

I am experiencing considerable performance problems with ORM requests,
too. In django 1.2.3.

And I found two slow procedures: first is queryset cloning (sql.Query
objects deepcopying is actually slow) and second - sql generation
(especially, generating list of fields). Both could be a result of
adding new features that made sql.Query objects very complex.

I opened a topic about expensive cloning on django-developers:
http://groups.google.com/group/django-developers/browse_thread/thread/5943bc1bab0711b0/95ab0714ef746469?lnk=gst=queryset+cloning#95ab0714ef746469

and found some people with same problem.

I also use a monkey patch to avoid this problem by making querysets
mutating, not cloning:
https://gist.github.com/872145

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.1 performance versus django 1.2 performance

[akaariai]

On Apr 23, 11:54 am, Torsten Bronger 
wrote:
> Does anybody have estimates or figures concerning the impact of
> middleware?  I love the middleware framework and I make extensive
> use of it but sometimes I'm afraid it slows down every request
> significantly, especially if you have a project with cheap view
> functions.
>
> And then, the number of "canonical" middleware has increased between
> Django versions as far as I can see.

The performance impact of a middleware depends almost completely on
what a given middleware does. If I am not mistaken adding a middleware
that does nothing gives you just one function call per the
middleware's functions (process_request, process_view, ...) when
processing a request. If you do not have insane amounts of middlewares
defined then there should be no performance problems.

However, if you do something complex in the middleware then
performance will of course be affected.

In general I have a feeling that Django is still fast enough, and the
added features outweigh the performance penalty.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.1 performance versus django 1.2 performance

[Torsten Bronger]

Hallöchen!

Russell Keith-Magee writes:

> [...]
>
> No - there isn't a plan to address this, because it isn't clear
> what "this" is.
>
> While it is known that there has been a slowdown between versions,
> that slowdown has been accompanied by a massive increase in
> functionality -- for example, the 1.1->1.2 transition introduced
> support for multiple databases. To the best of my knowledge, the
> performance slowdown highlighted by Eric at Djangocon was
> relatively small - 5-10%, not on the order of 30-50%
> slowdown. This matches with my personal experience of upgrading.

Does anybody have estimates or figures concerning the impact of
middleware?  I love the middleware framework and I make extensive
use of it but sometimes I'm afraid it slows down every request
significantly, especially if you have a project with cheap view
functions.

And then, the number of "canonical" middleware has increased between
Django versions as far as I can see.

Tschö,
Torsten.

-- 
Torsten BrongerJabber ID: torsten.bron...@jabber.rwth-aachen.de
  or http://bronger-jmp.appspot.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.1 performance versus django 1.2 performance

[Russell Keith-Magee]

> On Fri, Apr 22, 2011 at 9:22 PM, Shawn Milochik  wrote:
>>
>> This was mentioned in Eric Florenzano's talk at DjangoCon 2010. Each
>> version has gotten slower.
>>
>> I haven't heard anything about the cause or plans to fix this, though.
>> If you've got a good test suite you can always use tools like Python's
>> profile module to track down slowdowns in your Django code and
>> contribute that knowledge to the developers' list.
>>

On Sat, Apr 23, 2011 at 11:26 AM, Peter Portante
 wrote:
> Is there a plan in place to address this?

No - there isn't a plan to address this, because it isn't clear what "this" is.

While it is known that there has been a slowdown between versions,
that slowdown has been accompanied by a massive increase in
functionality -- for example, the 1.1->1.2 transition introduced
support for multiple databases. To the best of my knowledge, the
performance slowdown highlighted by Eric at Djangocon was relatively
small - 5-10%, not on the order of 30-50% slowdown. This matches with
my personal experience of upgrading.

There are two performance issues in database operations that I'm aware
of that people commonly point to.

Firstly -- deep cloning of query sets. Cloning a queryset is an
expensive operation, and if you are making extensive use of operations
that clone (e.g., very long chains of complex filters) you may see
performance issues. This isn't something that can be easily avoided --
in order to preserve the API guarantees of a query set, deep cloning
is required.

Secondly, the backwards compatibility infrastructure introduced in
order to support multiple databases. In order to guarantee backwards
compatibility, the value preparation methods on fields needed to use
some moderately expensive introspection methods. However because these
methods were introduced for 1.2, and we're currently preparing 1.4,
these introspection methods have just been removed from trunk.

Beyond these two issues -- we're always interested in improving
performance, so if you can identify a specific performance issue,
we're happy to look at it. The more evidence you can provide (profile
runs, sample queries that are observably slower, and so on), the
easier it will be to address these problems.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.1 performance versus django 1.2 performance

[Peter Portante]

Is there a plan in place to address this?

On Fri, Apr 22, 2011 at 9:22 PM, Shawn Milochik  wrote:

> This was mentioned in Eric Florenzano's talk at DjangoCon 2010. Each
> version has gotten slower.
>
> I haven't heard anything about the cause or plans to fix this, though.
> If you've got a good test suite you can always use tools like Python's
> profile module to track down slowdowns in your Django code and
> contribute that knowledge to the developers' list.
>
> Shawn
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django 1.1 performance versus django 1.2 performance

[Shawn Milochik]

This was mentioned in Eric Florenzano's talk at DjangoCon 2010. Each
version has gotten slower.

I haven't heard anything about the cause or plans to fix this, though.
If you've got a good test suite you can always use tools like Python's
profile module to track down slowdowns in your Django code and
contribute that knowledge to the developers' list.

Shawn

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.