Re: Benchmarking 1.5 vs 1.6

[Anssi Kääriäinen]

I applied the patch with comment modification to 1.6.x and master.

It seems the change did a lot of good to all of query benchmarks. Most 
1.6.x ORM results are now 1.1 to 1.3x faster than in 1.5.x. Full results in 
the end of this post.

There is still some problems, template rendering seems to be 10% slower. 
model_creation is still 1.15x slower than in 1.5.x. url_reverse is 1.2x 
slower. I don't have time to investigate these more, and in any case these 
aren't critical regressions in any way.

For those who want to run the suite - it seems that if you have new enough 
processor the speed throttling is done in silicon. In that case it will be 
really, really hard to get stable results from djangobench. The best you 
can do is use big amount of trials (-t) and look at the minimum runtime 
instead of the average. There might be some tools for your operating system 
to switch CPU throttling to minimum, so google for those.

 - Anssi

The full results again for latest stable/1.6.x vs stable/1.5.x:

Running 'default_middleware' benchmark ...
Min: 0.000664 -> 0.000336: 1.9766x faster
Avg: 0.000794 -> 0.000398: 1.9932x faster
Significant (t=6.607238)
Stddev: 0.00031 -> 0.00029: 1.0583x smaller (N = 50)

Running 'form_clean' benchmark ...
Min: 0.21 -> 0.22: 1.0455x slower
Avg: 0.26 -> 0.23: 1.1314x faster
Not significant
Stddev: 0.2 -> 0.0: 6.9991x smaller (N = 50)

Running 'form_create' benchmark ...
Min: 0.46 -> 0.55: 1.1979x slower
Avg: 0.49 -> 0.000737: 14.9229x slower # Outlier, as mentioned earlier
Not significant
Stddev: 0.1 -> 0.00481: 415.8844x larger (N = 50)

Running 'l10n_render' benchmark ...
Min: 0.005955 -> 0.006949: 1.1669x slower
Avg: 0.006347 -> 0.007402: 1.1661x slower
Significant (t=-3.120463)
Stddev: 0.00150 -> 0.00186: 1.2448x larger (N = 50)

Running 'model_creation' benchmark ...
Min: 0.000170 -> 0.000199: 1.1697x slower
Avg: 0.000186 -> 0.000218: 1.1725x slower
Significant (t=-2.009837)
Stddev: 0.8 -> 0.8: 1.0107x smaller (N = 50)

Running 'model_delete' benchmark ...
Min: 0.000225 -> 0.000240: 1.0668x slower
Avg: 0.000232 -> 0.000256: 1.1057x slower
Significant (t=-4.861734)
Stddev: 0.2 -> 0.3: 1.7163x larger (N = 50)

Running 'model_save_existing' benchmark ...
Min: 0.033559 -> 0.011296: 2.9709x faster
Avg: 0.034041 -> 0.011363: 2.9959x faster
Significant (t=169.025764)
Stddev: 0.00094 -> 0.00010: 9.0688x smaller (N = 50)

Running 'model_save_new' benchmark ...
Min: 0.022818 -> 0.011384: 2.0044x faster
Avg: 0.033812 -> 0.011772: 2.8721x faster
Significant (t=82.502660)
Stddev: 0.00177 -> 0.00065: 2.7339x smaller (N = 50)

Running 'multi_value_dict' benchmark ...
Min: 0.43 -> 0.44: 1.0335x slower
Avg: 0.79 -> 0.81: 1.0363x slower
Not significant
Stddev: 0.2 -> 0.2: 1.0079x larger (N = 50)

Running 'qs_filter_chaining' benchmark ...
Min: 0.002281 -> 0.000742: 3.0742x faster
Avg: 0.002331 -> 0.000765: 3.0468x faster
Significant (t=133.899536)
Stddev: 0.7 -> 0.4: 1.5650x smaller (N = 50)

Running 'query_aggregate' benchmark ...
Min: 0.000254 -> 0.000209: 1.2144x faster
Avg: 0.000316 -> 0.000215: 1.4680x faster
Significant (t=3.067214)
Stddev: 0.00023 -> 0.1: 16.9474x smaller (N = 50)

Running 'query_all' benchmark ...
Min: 0.024302 -> 0.021535: 1.1285x faster
Avg: 0.025855 -> 0.023171: 1.1158x faster
Significant (t=5.678997)
Stddev: 0.00243 -> 0.00230: 1.0566x smaller (N = 50)

Running 'query_all_multifield' benchmark ...
Min: 0.050120 -> 0.048104: 1.0419x faster
Avg: 0.052968 -> 0.050711: 1.0445x faster
Significant (t=3.563558)
Stddev: 0.00353 -> 0.00276: 1.2772x smaller (N = 50)

Running 'query_annotate' benchmark ...
Min: 0.000521 -> 0.000438: 1.1894x faster
Avg: 0.000533 -> 0.000453: 1.1781x faster
Significant (t=11.172131)
Stddev: 0.3 -> 0.4: 1.2886x larger (N = 50)

Running 'query_complex_filter' benchmark ...
Min: 0.000166 -> 0.000119: 1.3948x faster
Avg: 0.000172 -> 0.000123: 1.3995x faster
Significant (t=12.315270)
Stddev: 0.3 -> 0.1: 2.4050x smaller (N = 50)

Running 'query_count' benchmark ...
Min: 0.000211 -> 0.000170: 1.2412x faster
Avg: 0.000215 -> 0.000177: 1.2131x faster
Significant (t=9.985689)
Stddev: 0.1 -> 0.3: 4.9672x larger (N = 50)

Running 'query_dates' benchmark ...
Min: 0.000540 -> 0.000433: 1.2472x faster
Avg: 0.000550 -> 0.000446: 1.2348x faster
Significant (t=21.074766)
Stddev: 0.2 -> 0.3: 1.4789x larger (N = 50)

Running 'query_delete' benchmark ...
Min: 0.000260 -> 0.000215: 1.2095x faster
Avg: 0.000284 -> 0.000228: 1.2420x faster
Significant (t=7.900090)
Stddev: 0.4 -> 0.3: 1.1254x smaller (N = 50)

Running 'query_delete_related' benchmark ...
Min: 0.000308 -> 0.000275: 1.1206x faster
Avg: 0.000337 -> 0.000300: 1.1247x faster
Not significant
Stddev: 0.00017 -> 0.00013: 1.3042x smaller (N = 50)

Running 'query_distinct' benchmark ...
Min: 0.000339 -> 0.000281: 1.2071x faster
Avg: 0.000352 -> 0.000285: 1.2336x faster
Significant

Re: Benchmarking 1.5 vs 1.6

[Jeremy Dunck]

It may be useful to place a comment where @wraps was removed, lest some later 
do-gooder "fix" it.

On Sep 17, 2013, at 12:06 AM, Marc Tamlyn  wrote:

> Can't say I'm hugely worried about perfect tracebacks and introspection with 
> internal kind of functions here if it affects performance. The patch looks 
> good. I'd add a comment to say why we're not using @wraps in that context 
> though so someone doesn't go back and add it.
> 
> On 17 Sep 2013 07:57, "Anssi Kääriäinen"  wrote:
>> I did some investigation to query_iterator, and most of the slowdown is 
>> caused by functools.wraps. When used in nested context @wraps seems to be 
>> really slow. For example, try this with and without @wraps: 
>> http://pastebin.com/5ejAKuKd
>> 
>> Should we care about this slowdown? Can we do without @wraps? My 
>> understanding is that @wraps gives you __name__, __module__ and __doc__ 
>> which are easy to do manually. In addition it gives introspection the right 
>> args and kwargs for the wrapped function (so that IDEs see it correctly). 
>> This will be a lot harder to do manually, and doing so will likely result in 
>> similar slowdown.
>> 
>> The biggest hit case is if somebody is looping over a raw SQL queryset with 
>> .fetchone(). This use case will be severely punished. Even .execute(); 
>> .fetchmany() has a 1.5x slowdown (the benchmark is this one: 
>> https://github.com/django/djangobench/commit/94fc28d99f95c65d26d0fad8af44e46c49282220
>>  - simple SELECT + fetchall() for the ten returned rows.
>> 
>> I have created a patch that removes the usage of @wraps and makes 
>> connection.wrap_database_errors a cached_property. The cached_property 
>> should be safe as the context manager created by wrap_database_errors is 
>> stateless. This patch gets rid of the slowdown (1.00-1.05x slower in 
>> repeated runs for raw_sql benchmark). With just removed @wraps the slowdown 
>> is 1.1x. The patch is here: 
>> https://github.com/akaariai/django/commit/eafc373c16350abe51c565c8aefbe36cabf5219e
>> 
>> Should I apply the patch, and should I apply it to stable/1.6.x, too? 
>> Removal of @wraps is really low risk, usage of @cached_property is also low 
>> risk, but not as low as removal of @wraps.
>> 
>>  - Anssi
>> 
>> On Saturday, September 14, 2013 9:21:43 PM UTC+3, Anssi Kääriäinen wrote:
>>> 
>>> I just ran djangobench comparing 1.5 to 1.6. The biggest thing seems to be 
>>> form_create bechmark, the average is 15x slower. But for some reason the 
>>> minimum runtime is just 1.16x slower. This seems a bit strange. This might 
>>> be worth more investigation.
>>> 
>>> Otherwise there doens't seem to be anything alarming going on. Most ORM 
>>> operations are slightly faster (likely due to __deepcopy__ removal).
>>> 
>>> There are a couple of individual benchmarks worth mentioning... url_reverse 
>>> and query_iterator are slower (1.15x and 1.12x respectively). The 
>>> default_middleware and qs_filter_chaining benchmarks have gained a lot 
>>> (2.0x and 3.1x respectively). model_save_existing/model_save_new are faster 
>>> (2x+), but model_creation is slightly slower. I suspect there is something 
>>> going on with connection creation or transactions for model_creation, after 
>>> all model_creation is a convenience method for model.__init__ + 
>>> model_save_new which both aren't slower.
>>> 
>>> What else? url_reverse and template compliation under i18n slightly slower. 
>>> For the rest of benchmarks, see below.
>>> 
>>> Running 'default_middleware' benchmark ...
>>> Min: 0.001260 -> 0.000554: 2.2755x faster
>>> Avg: 0.001389 -> 0.000681: 2.0394x faster
>>> Significant (t=8.222155)
>>> Stddev: 0.00037 -> 0.00048: 1.2843x larger (N = 50)
>>> 
>>> Running 'form_clean' benchmark ...
>>> Min: 0.42 -> 0.42: no change
>>> Avg: 0.49 -> 0.43: 1.1341x faster
>>> Not significant
>>> Stddev: 0.4 -> 0.1: 6.6718x smaller (N = 50)
>>> 
>>> Running 'form_create' benchmark ...
>>> Min: 0.85 -> 0.99: 1.1657x slower
>>> Avg: 0.88 -> 0.001396: 15.8410x slower
>>> Not significant
>>> Stddev: 0.1 -> 0.00916: 783.3314x larger (N = 50)
>>> 
>>> Running 'l10n_render' benchmark ...
>>> Min: 0.011702 -> 0.014151: 1.2093x slower
>>> Avg: 0.012200 -> 0.014846: 1.2169x slower
>>> Significant (t=-4.824893)
>>> Stddev: 0.00233 -> 0.00310: 1.3311x larger (N = 50)
>>> 
>>> Running 'model_creation' benchmark ...
>>> Min: 0.000311 -> 0.000400: 1.2860x slower
>>> Avg: 0.000336 -> 0.000432: 1.2840x slower
>>> Significant (t=-2.799691)
>>> Stddev: 0.00015 -> 0.00019: 1.2751x larger (N = 50)
>>> 
>>> Running 'model_delete' benchmark ...
>>> Min: 0.000407 -> 0.000468: 1.1500x slower
>>> Avg: 0.000423 -> 0.000484: 1.1440x slower
>>> Significant (t=-7.131460)
>>> Stddev: 0.5 -> 0.4: 1.1807x smaller (N = 50)
>>> 
>>> Running 'model_save_existing' benchmark ...
>>> Min: 0.062923 -> 0.021829: 2.8826x faster
>>> Avg: 0.063206 -> 0.022001: 2.8729x faster
>>> Significant (t=793.026302)
>>> Stddev: 0.00034 -> 0.0001

Re: Benchmarking 1.5 vs 1.6

[Marc Tamlyn]

Can't say I'm hugely worried about perfect tracebacks and introspection
with internal kind of functions here if it affects performance. The patch
looks good. I'd add a comment to say why we're not using @wraps in that
context though so someone doesn't go back and add it.
On 17 Sep 2013 07:57, "Anssi Kääriäinen"  wrote:

> I did some investigation to query_iterator, and most of the slowdown is
> caused by functools.wraps. When used in nested context @wraps seems to be
> really slow. For example, try this with and without @wraps:
> http://pastebin.com/5ejAKuKd
>
> Should we care about this slowdown? Can we do without @wraps? My
> understanding is that @wraps gives you __name__, __module__ and __doc__
> which are easy to do manually. In addition it gives introspection the right
> args and kwargs for the wrapped function (so that IDEs see it correctly).
> This will be a lot harder to do manually, and doing so will likely result
> in similar slowdown.
>
> The biggest hit case is if somebody is looping over a raw SQL queryset
> with .fetchone(). This use case will be severely punished. Even .execute();
> .fetchmany() has a 1.5x slowdown (the benchmark is this one:
> https://github.com/django/djangobench/commit/94fc28d99f95c65d26d0fad8af44e46c49282220-
>  simple SELECT + fetchall() for the ten returned rows.
>
> I have created a patch that removes the usage of @wraps and makes
> connection.wrap_database_errors a cached_property. The cached_property
> should be safe as the context manager created by wrap_database_errors is
> stateless. This patch gets rid of the slowdown (1.00-1.05x slower in
> repeated runs for raw_sql benchmark). With just removed @wraps the slowdown
> is 1.1x. The patch is here:
> https://github.com/akaariai/django/commit/eafc373c16350abe51c565c8aefbe36cabf5219e
>
> Should I apply the patch, and should I apply it to stable/1.6.x, too?
> Removal of @wraps is really low risk, usage of @cached_property is also low
> risk, but not as low as removal of @wraps.
>
>  - Anssi
>
> On Saturday, September 14, 2013 9:21:43 PM UTC+3, Anssi Kääriäinen wrote:
>>
>> I just ran djangobench comparing 1.5 to 1.6. The biggest thing seems to
>> be form_create bechmark, the average is 15x slower. But for some reason the
>> minimum runtime is just 1.16x slower. This seems a bit strange. This might
>> be worth more investigation.
>>
>> Otherwise there doens't seem to be anything alarming going on. Most ORM
>> operations are slightly faster (likely due to __deepcopy__ removal).
>>
>> There are a couple of individual benchmarks worth mentioning...
>> url_reverse and query_iterator are slower (1.15x and 1.12x respectively).
>> The default_middleware and qs_filter_chaining benchmarks have gained a lot
>> (2.0x and 3.1x respectively). model_save_existing/model_**save_new are
>> faster (2x+), but model_creation is slightly slower. I suspect there is
>> something going on with connection creation or transactions for
>> model_creation, after all model_creation is a convenience method for
>> model.__init__ + model_save_new which both aren't slower.
>>
>> What else? url_reverse and template compliation under i18n slightly
>> slower. For the rest of benchmarks, see below.
>>
>> Running 'default_middleware' benchmark ...
>> Min: 0.001260 -> 0.000554: 2.2755x faster
>> Avg: 0.001389 -> 0.000681: 2.0394x faster
>> Significant (t=8.222155)
>> Stddev: 0.00037 -> 0.00048: 1.2843x larger (N = 50)
>>
>> Running 'form_clean' benchmark ...
>> Min: 0.42 -> 0.42: no change
>> Avg: 0.49 -> 0.43: 1.1341x faster
>> Not significant
>> Stddev: 0.4 -> 0.1: 6.6718x smaller (N = 50)
>>
>> Running 'form_create' benchmark ...
>> Min: 0.85 -> 0.99: 1.1657x slower
>> Avg: 0.88 -> 0.001396: 15.8410x slower
>> Not significant
>> Stddev: 0.1 -> 0.00916: 783.3314x larger (N = 50)
>>
>> Running 'l10n_render' benchmark ...
>> Min: 0.011702 -> 0.014151: 1.2093x slower
>> Avg: 0.012200 -> 0.014846: 1.2169x slower
>> Significant (t=-4.824893)
>> Stddev: 0.00233 -> 0.00310: 1.3311x larger (N = 50)
>>
>> Running 'model_creation' benchmark ...
>> Min: 0.000311 -> 0.000400: 1.2860x slower
>> Avg: 0.000336 -> 0.000432: 1.2840x slower
>> Significant (t=-2.799691)
>> Stddev: 0.00015 -> 0.00019: 1.2751x larger (N = 50)
>>
>> Running 'model_delete' benchmark ...
>> Min: 0.000407 -> 0.000468: 1.1500x slower
>> Avg: 0.000423 -> 0.000484: 1.1440x slower
>> Significant (t=-7.131460)
>> Stddev: 0.5 -> 0.4: 1.1807x smaller (N = 50)
>>
>> Running 'model_save_existing' benchmark ...
>> Min: 0.062923 -> 0.021829: 2.8826x faster
>> Avg: 0.063206 -> 0.022001: 2.8729x faster
>> Significant (t=793.026302)
>> Stddev: 0.00034 -> 0.00014: 2.3570x smaller (N = 50)
>>
>> Running 'model_save_new' benchmark ...
>> Min: 0.041341 -> 0.021819: 1.8947x faster
>> Avg: 0.063127 -> 0.022379: 2.8208x faster
>> Significant (t=76.086085)
>> Stddev: 0.00344 -> 0.00158: 2.1770x smaller (N = 50)
>>
>> Running 'multi_value_dict' benchmark ...
>> Min: 0.000

Re: Benchmarking 1.5 vs 1.6

[Anssi Kääriäinen]

I did some investigation to query_iterator, and most of the slowdown is 
caused by functools.wraps. When used in nested context @wraps seems to be 
really slow. For example, try this with and without @wraps: 
http://pastebin.com/5ejAKuKd

Should we care about this slowdown? Can we do without @wraps? My 
understanding is that @wraps gives you __name__, __module__ and __doc__ 
which are easy to do manually. In addition it gives introspection the right 
args and kwargs for the wrapped function (so that IDEs see it correctly). 
This will be a lot harder to do manually, and doing so will likely result 
in similar slowdown.

The biggest hit case is if somebody is looping over a raw SQL queryset with 
.fetchone(). This use case will be severely punished. Even .execute(); 
.fetchmany() has a 1.5x slowdown (the benchmark is this one: 
https://github.com/django/djangobench/commit/94fc28d99f95c65d26d0fad8af44e46c49282220
 
- simple SELECT + fetchall() for the ten returned rows.

I have created a patch that removes the usage of @wraps and makes 
connection.wrap_database_errors a cached_property. The cached_property 
should be safe as the context manager created by wrap_database_errors is 
stateless. This patch gets rid of the slowdown (1.00-1.05x slower in 
repeated runs for raw_sql benchmark). With just removed @wraps the slowdown 
is 1.1x. The patch is here: 
https://github.com/akaariai/django/commit/eafc373c16350abe51c565c8aefbe36cabf5219e

Should I apply the patch, and should I apply it to stable/1.6.x, too? 
Removal of @wraps is really low risk, usage of @cached_property is also low 
risk, but not as low as removal of @wraps.

 - Anssi

On Saturday, September 14, 2013 9:21:43 PM UTC+3, Anssi Kääriäinen wrote:
>
> I just ran djangobench comparing 1.5 to 1.6. The biggest thing seems to be 
> form_create bechmark, the average is 15x slower. But for some reason the 
> minimum runtime is just 1.16x slower. This seems a bit strange. This might 
> be worth more investigation.
>
> Otherwise there doens't seem to be anything alarming going on. Most ORM 
> operations are slightly faster (likely due to __deepcopy__ removal).
>
> There are a couple of individual benchmarks worth mentioning... 
> url_reverse and query_iterator are slower (1.15x and 1.12x respectively). 
> The default_middleware and qs_filter_chaining benchmarks have gained a lot 
> (2.0x and 3.1x respectively). model_save_existing/model_save_new are faster 
> (2x+), but model_creation is slightly slower. I suspect there is something 
> going on with connection creation or transactions for model_creation, after 
> all model_creation is a convenience method for model.__init__ + 
> model_save_new which both aren't slower.
>
> What else? url_reverse and template compliation under i18n slightly 
> slower. For the rest of benchmarks, see below.
>
> Running 'default_middleware' benchmark ...
> Min: 0.001260 -> 0.000554: 2.2755x faster
> Avg: 0.001389 -> 0.000681: 2.0394x faster
> Significant (t=8.222155)
> Stddev: 0.00037 -> 0.00048: 1.2843x larger (N = 50)
>
> Running 'form_clean' benchmark ...
> Min: 0.42 -> 0.42: no change
> Avg: 0.49 -> 0.43: 1.1341x faster
> Not significant
> Stddev: 0.4 -> 0.1: 6.6718x smaller (N = 50)
>
> Running 'form_create' benchmark ...
> Min: 0.85 -> 0.99: 1.1657x slower
> Avg: 0.88 -> 0.001396: 15.8410x slower
> Not significant
> Stddev: 0.1 -> 0.00916: 783.3314x larger (N = 50)
>
> Running 'l10n_render' benchmark ...
> Min: 0.011702 -> 0.014151: 1.2093x slower
> Avg: 0.012200 -> 0.014846: 1.2169x slower
> Significant (t=-4.824893)
> Stddev: 0.00233 -> 0.00310: 1.3311x larger (N = 50)
>
> Running 'model_creation' benchmark ...
> Min: 0.000311 -> 0.000400: 1.2860x slower
> Avg: 0.000336 -> 0.000432: 1.2840x slower
> Significant (t=-2.799691)
> Stddev: 0.00015 -> 0.00019: 1.2751x larger (N = 50)
>
> Running 'model_delete' benchmark ...
> Min: 0.000407 -> 0.000468: 1.1500x slower
> Avg: 0.000423 -> 0.000484: 1.1440x slower
> Significant (t=-7.131460)
> Stddev: 0.5 -> 0.4: 1.1807x smaller (N = 50)
>
> Running 'model_save_existing' benchmark ...
> Min: 0.062923 -> 0.021829: 2.8826x faster
> Avg: 0.063206 -> 0.022001: 2.8729x faster
> Significant (t=793.026302)
> Stddev: 0.00034 -> 0.00014: 2.3570x smaller (N = 50)
>
> Running 'model_save_new' benchmark ...
> Min: 0.041341 -> 0.021819: 1.8947x faster
> Avg: 0.063127 -> 0.022379: 2.8208x faster
> Significant (t=76.086085)
> Stddev: 0.00344 -> 0.00158: 2.1770x smaller (N = 50)
>
> Running 'multi_value_dict' benchmark ...
> Min: 0.83 -> 0.85: 1.0230x slower
> Avg: 0.000165 -> 0.000168: 1.0164x slower
> Not significant
> Stddev: 0.5 -> 0.5: 1.0339x larger (N = 50)
>
> Running 'qs_filter_chaining' benchmark ...
> Min: 0.004264 -> 0.001362: 3.1311x faster
> Avg: 0.004311 -> 0.001391: 3.0996x faster
> Significant (t=170.435257)
> Stddev: 0.00010 -> 0.7: 1.2953x smaller (N = 50)
>
> Running 'query_aggregate' benchmark ...
> Min: 0.00

Re: Benchmarking 1.5 vs 1.6

[Anssi Kääriäinen]

This was a case where there was a tradeoff between correctness and 
speed. Previously EmptyQuerySet (the class used by .none() internally) 
wasn't a real QuerySet class. This caused a couple of problems 
(subclassing and no error checking at least). Now .none() just generates 
a WhereNode for the query that ensures no query is ran on execution and 
no objects are returned when the queryset is iterated.


 - Anssi

On 09/16/2013 04:54 AM, charettes wrote:

I guess it's related to a2396a4c8f[1] and #19184[2].

[1] 
https://github.com/django/django/commit/a2396a4c8f2ccd7f91adee6d8c2e9c31f13f0e3f

[2] https://code.djangoproject.com/ticket/19184

Le dimanche 15 septembre 2013 20:05:43 UTC-4, Curtis Maloney a écrit :

So what's going on here:

Running 'query_none' benchmark ...
Min: 0.44 -> 0.000262: 5.9674x slower
Avg: 0.47 -> 0.000290: 6.1906x slower
Significant (t=-12.805744)
Stddev: 0.1 -> 0.00013: 14.5148x larger (N = 50)

--
Curtis



On 15 September 2013 16:31, Anssi Kääriäinen > wrote:

On Sunday, September 15, 2013 3:13:09 AM UTC+3, Curtis Maloney
wrote:

Hey, thanks for that!

It would be nice to have something that would chart this
over time... something like some people have set up for GCC.

I've never been able to get djangobench to give meaningful
results, otherwise I'd do it.

Mmm... perhaps I've just found a use for my odroid :)


I have a simple setup for generating graphs over time. Results
are something like this:
http://users.tkk.fi/~akaariai/djbench/model_save/
.
Unfortunately I haven't found the time to automate benchmarking.

When using djangobench it is extremely important to turn all
sorts of cpu speed throttling off. Power saving features will
cause random results. It seems the older your hardware is, the
better results you will get. Even then, if you forget to
disable cpu throttling you will get results like this:
http://users.tkk.fi/~akaariai/djbench/model_creation/


I managed to spot why form_create is 15x slower in average -
it is all about translation startup speed. A string that
wasn't translatable before now is, and I guess the first time
that string is translated will cause a huge spike in runtime.
Luckily the rest of iterations are about same as always. In
short, form_create was a false alarm. Or, see
http://users.tkk.fi/~akaariai/djbench/form_create/


 - Anssi
-- 
You received this message because you are subscribed to the

Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to django-develop...@googlegroups.com
.
To post to this group, send email to
django-d...@googlegroups.com .
Visit this group at
http://groups.google.com/group/django-developers
.
For more options, visit
https://groups.google.com/groups/opt_out
.


--
You received this message because you are subscribed to the Google 
Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-developers+unsubscr...@googlegroups.com.

To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Benchmarking 1.5 vs 1.6

[charettes]

I guess it's related to a2396a4c8f[1] and #19184[2].

[1] 
https://github.com/django/django/commit/a2396a4c8f2ccd7f91adee6d8c2e9c31f13f0e3f
[2] https://code.djangoproject.com/ticket/19184

Le dimanche 15 septembre 2013 20:05:43 UTC-4, Curtis Maloney a écrit :
>
> So what's going on here:
>
> Running 'query_none' benchmark ...
> Min: 0.44 -> 0.000262: 5.9674x slower
> Avg: 0.47 -> 0.000290: 6.1906x slower
> Significant (t=-12.805744)
> Stddev: 0.1 -> 0.00013: 14.5148x larger (N = 50)
>
> --
> Curtis
>
>
>
> On 15 September 2013 16:31, Anssi Kääriäinen 
> > wrote:
>
>> On Sunday, September 15, 2013 3:13:09 AM UTC+3, Curtis Maloney wrote:
>>>
>>> Hey, thanks for that!
>>>
>>> It would be nice to have something that would chart this over time... 
>>> something like some people have set up for GCC.
>>>
>>> I've never been able to get djangobench to give meaningful results, 
>>> otherwise I'd do it.
>>>
>>> Mmm... perhaps I've just found a use for my odroid :) 
>>>
>>
>> I have a simple setup for generating graphs over time. Results are 
>> something like this: http://users.tkk.fi/~akaariai/djbench/model_save/.  
>> Unfortunately I haven't found the time to automate benchmarking.
>>
>> When using djangobench it is extremely important to turn all sorts of cpu 
>> speed throttling off. Power saving features will cause random results. It 
>> seems the older your hardware is, the better results you will get. Even 
>> then, if you forget to disable cpu throttling you will get results like 
>> this: http://users.tkk.fi/~akaariai/djbench/model_creation/
>>
>> I managed to spot why form_create is 15x slower in average - it is all 
>> about translation startup speed. A string that wasn't translatable before 
>> now is, and I guess the first time that string is translated will cause a 
>> huge spike in runtime. Luckily the rest of iterations are about same as 
>> always. In short, form_create was a false alarm. Or, see 
>> http://users.tkk.fi/~akaariai/djbench/form_create/
>>
>>  - Anssi
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to 
>> django-d...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Benchmarking 1.5 vs 1.6

[Curtis Maloney]

So what's going on here:

Running 'query_none' benchmark ...
Min: 0.44 -> 0.000262: 5.9674x slower
Avg: 0.47 -> 0.000290: 6.1906x slower
Significant (t=-12.805744)
Stddev: 0.1 -> 0.00013: 14.5148x larger (N = 50)

--
Curtis



On 15 September 2013 16:31, Anssi Kääriäinen wrote:

> On Sunday, September 15, 2013 3:13:09 AM UTC+3, Curtis Maloney wrote:
>>
>> Hey, thanks for that!
>>
>> It would be nice to have something that would chart this over time...
>> something like some people have set up for GCC.
>>
>> I've never been able to get djangobench to give meaningful results,
>> otherwise I'd do it.
>>
>> Mmm... perhaps I've just found a use for my odroid :)
>>
>
> I have a simple setup for generating graphs over time. Results are
> something like this: http://users.tkk.fi/~akaariai/djbench/model_save/.
> Unfortunately I haven't found the time to automate benchmarking.
>
> When using djangobench it is extremely important to turn all sorts of cpu
> speed throttling off. Power saving features will cause random results. It
> seems the older your hardware is, the better results you will get. Even
> then, if you forget to disable cpu throttling you will get results like
> this: http://users.tkk.fi/~akaariai/djbench/model_creation/
>
> I managed to spot why form_create is 15x slower in average - it is all
> about translation startup speed. A string that wasn't translatable before
> now is, and I guess the first time that string is translated will cause a
> huge spike in runtime. Luckily the rest of iterations are about same as
> always. In short, form_create was a false alarm. Or, see
> http://users.tkk.fi/~akaariai/djbench/form_create/
>
>  - Anssi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Benchmarking 1.5 vs 1.6

[Anssi Kääriäinen]

On Sunday, September 15, 2013 3:13:09 AM UTC+3, Curtis Maloney wrote:
>
> Hey, thanks for that!
>
> It would be nice to have something that would chart this over time... 
> something like some people have set up for GCC.
>
> I've never been able to get djangobench to give meaningful results, 
> otherwise I'd do it.
>
> Mmm... perhaps I've just found a use for my odroid :) 
>

I have a simple setup for generating graphs over time. Results are 
something like this: http://users.tkk.fi/~akaariai/djbench/model_save/.  
Unfortunately I haven't found the time to automate benchmarking.

When using djangobench it is extremely important to turn all sorts of cpu 
speed throttling off. Power saving features will cause random results. It 
seems the older your hardware is, the better results you will get. Even 
then, if you forget to disable cpu throttling you will get results like 
this: http://users.tkk.fi/~akaariai/djbench/model_creation/

I managed to spot why form_create is 15x slower in average - it is all 
about translation startup speed. A string that wasn't translatable before 
now is, and I guess the first time that string is translated will cause a 
huge spike in runtime. Luckily the rest of iterations are about same as 
always. In short, form_create was a false alarm. Or, see 
http://users.tkk.fi/~akaariai/djbench/form_create/

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Benchmarking 1.5 vs 1.6

[Curtis Maloney]

Hey, thanks for that!

It would be nice to have something that would chart this over time...
something like some people have set up for GCC.

I've never been able to get djangobench to give meaningful results,
otherwise I'd do it.

Mmm... perhaps I've just found a use for my odroid :)

--
Curtis


On 15 September 2013 04:21, Anssi Kääriäinen wrote:

> I just ran djangobench comparing 1.5 to 1.6. The biggest thing seems to be
> form_create bechmark, the average is 15x slower. But for some reason the
> minimum runtime is just 1.16x slower. This seems a bit strange. This might
> be worth more investigation.
>
> Otherwise there doens't seem to be anything alarming going on. Most ORM
> operations are slightly faster (likely due to __deepcopy__ removal).
>
> There are a couple of individual benchmarks worth mentioning...
> url_reverse and query_iterator are slower (1.15x and 1.12x respectively).
> The default_middleware and qs_filter_chaining benchmarks have gained a lot
> (2.0x and 3.1x respectively). model_save_existing/model_save_new are faster
> (2x+), but model_creation is slightly slower. I suspect there is something
> going on with connection creation or transactions for model_creation, after
> all model_creation is a convenience method for model.__init__ +
> model_save_new which both aren't slower.
>
> What else? url_reverse and template compliation under i18n slightly
> slower. For the rest of benchmarks, see below.
>
> Running 'default_middleware' benchmark ...
> Min: 0.001260 -> 0.000554: 2.2755x faster
> Avg: 0.001389 -> 0.000681: 2.0394x faster
> Significant (t=8.222155)
> Stddev: 0.00037 -> 0.00048: 1.2843x larger (N = 50)
>
> Running 'form_clean' benchmark ...
> Min: 0.42 -> 0.42: no change
> Avg: 0.49 -> 0.43: 1.1341x faster
> Not significant
> Stddev: 0.4 -> 0.1: 6.6718x smaller (N = 50)
>
> Running 'form_create' benchmark ...
> Min: 0.85 -> 0.99: 1.1657x slower
> Avg: 0.88 -> 0.001396: 15.8410x slower
> Not significant
> Stddev: 0.1 -> 0.00916: 783.3314x larger (N = 50)
>
> Running 'l10n_render' benchmark ...
> Min: 0.011702 -> 0.014151: 1.2093x slower
> Avg: 0.012200 -> 0.014846: 1.2169x slower
> Significant (t=-4.824893)
> Stddev: 0.00233 -> 0.00310: 1.3311x larger (N = 50)
>
> Running 'model_creation' benchmark ...
> Min: 0.000311 -> 0.000400: 1.2860x slower
> Avg: 0.000336 -> 0.000432: 1.2840x slower
> Significant (t=-2.799691)
> Stddev: 0.00015 -> 0.00019: 1.2751x larger (N = 50)
>
> Running 'model_delete' benchmark ...
> Min: 0.000407 -> 0.000468: 1.1500x slower
> Avg: 0.000423 -> 0.000484: 1.1440x slower
> Significant (t=-7.131460)
> Stddev: 0.5 -> 0.4: 1.1807x smaller (N = 50)
>
> Running 'model_save_existing' benchmark ...
> Min: 0.062923 -> 0.021829: 2.8826x faster
> Avg: 0.063206 -> 0.022001: 2.8729x faster
> Significant (t=793.026302)
> Stddev: 0.00034 -> 0.00014: 2.3570x smaller (N = 50)
>
> Running 'model_save_new' benchmark ...
> Min: 0.041341 -> 0.021819: 1.8947x faster
> Avg: 0.063127 -> 0.022379: 2.8208x faster
> Significant (t=76.086085)
> Stddev: 0.00344 -> 0.00158: 2.1770x smaller (N = 50)
>
> Running 'multi_value_dict' benchmark ...
> Min: 0.83 -> 0.85: 1.0230x slower
> Avg: 0.000165 -> 0.000168: 1.0164x slower
> Not significant
> Stddev: 0.5 -> 0.5: 1.0339x larger (N = 50)
>
> Running 'qs_filter_chaining' benchmark ...
> Min: 0.004264 -> 0.001362: 3.1311x faster
> Avg: 0.004311 -> 0.001391: 3.0996x faster
> Significant (t=170.435257)
> Stddev: 0.00010 -> 0.7: 1.2953x smaller (N = 50)
>
> Running 'query_aggregate' benchmark ...
> Min: 0.000467 -> 0.000412: 1.1331x faster
> Avg: 0.000479 -> 0.000423: 1.1322x faster
> Significant (t=11.625112)
> Stddev: 0.2 -> 0.2: 1.0048x smaller (N = 50)
>
> Running 'query_all' benchmark ...
> Min: 0.047256 -> 0.043987: 1.0743x faster
> Avg: 0.049713 -> 0.046489: 1.0693x faster
> Significant (t=3.604195)
> Stddev: 0.00423 -> 0.00470: 1.1104x larger (N = 50)
>
> Running 'query_all_multifield' benchmark ...
> Min: 0.103770 -> 0.099138: 1.0467x faster
> Avg: 0.107616 -> 0.103382: 1.0409x faster
> Significant (t=4.033836)
> Stddev: 0.00495 -> 0.00553: 1.1182x larger (N = 50)
>
> Running 'query_annotate' benchmark ...
> Min: 0.000987 -> 0.000865: 1.1408x faster
> Avg: 0.001006 -> 0.000881: 1.1417x faster
> Significant (t=17.891371)
> Stddev: 0.4 -> 0.3: 1.2235x smaller (N = 50)
>
> Running 'query_complex_filter' benchmark ...
> Min: 0.000308 -> 0.000218: 1.4136x faster
> Avg: 0.000317 -> 0.000223: 1.4175x faster
> Significant (t=27.463980)
> Stddev: 0.2 -> 0.1: 1.3219x smaller (N = 50)
>
> Running 'query_count' benchmark ...
> Min: 0.000392 -> 0.000339: 1.1561x faster
> Avg: 0.000401 -> 0.000347: 1.1571x faster
> Significant (t=13.345028)
> Stddev: 0.2 -> 0.2: 1.0469x smaller (N = 50)
>
> Running 'query_dates' benchmark ...
> Min: 0.001032 -> 0.000884: 1.1678x faster
> Avg: 0.001055 -> 0.000900: 1.1730x faster
> Significant (t=21.848869)
> Stddev

Benchmarking 1.5 vs 1.6

[Anssi Kääriäinen]

I just ran djangobench comparing 1.5 to 1.6. The biggest thing seems to be 
form_create bechmark, the average is 15x slower. But for some reason the 
minimum runtime is just 1.16x slower. This seems a bit strange. This might 
be worth more investigation.

Otherwise there doens't seem to be anything alarming going on. Most ORM 
operations are slightly faster (likely due to __deepcopy__ removal).

There are a couple of individual benchmarks worth mentioning... url_reverse 
and query_iterator are slower (1.15x and 1.12x respectively). The 
default_middleware and qs_filter_chaining benchmarks have gained a lot 
(2.0x and 3.1x respectively). model_save_existing/model_save_new are faster 
(2x+), but model_creation is slightly slower. I suspect there is something 
going on with connection creation or transactions for model_creation, after 
all model_creation is a convenience method for model.__init__ + 
model_save_new which both aren't slower.

What else? url_reverse and template compliation under i18n slightly slower. 
For the rest of benchmarks, see below.

Running 'default_middleware' benchmark ...
Min: 0.001260 -> 0.000554: 2.2755x faster
Avg: 0.001389 -> 0.000681: 2.0394x faster
Significant (t=8.222155)
Stddev: 0.00037 -> 0.00048: 1.2843x larger (N = 50)

Running 'form_clean' benchmark ...
Min: 0.42 -> 0.42: no change
Avg: 0.49 -> 0.43: 1.1341x faster
Not significant
Stddev: 0.4 -> 0.1: 6.6718x smaller (N = 50)

Running 'form_create' benchmark ...
Min: 0.85 -> 0.99: 1.1657x slower
Avg: 0.88 -> 0.001396: 15.8410x slower
Not significant
Stddev: 0.1 -> 0.00916: 783.3314x larger (N = 50)

Running 'l10n_render' benchmark ...
Min: 0.011702 -> 0.014151: 1.2093x slower
Avg: 0.012200 -> 0.014846: 1.2169x slower
Significant (t=-4.824893)
Stddev: 0.00233 -> 0.00310: 1.3311x larger (N = 50)

Running 'model_creation' benchmark ...
Min: 0.000311 -> 0.000400: 1.2860x slower
Avg: 0.000336 -> 0.000432: 1.2840x slower
Significant (t=-2.799691)
Stddev: 0.00015 -> 0.00019: 1.2751x larger (N = 50)

Running 'model_delete' benchmark ...
Min: 0.000407 -> 0.000468: 1.1500x slower
Avg: 0.000423 -> 0.000484: 1.1440x slower
Significant (t=-7.131460)
Stddev: 0.5 -> 0.4: 1.1807x smaller (N = 50)

Running 'model_save_existing' benchmark ...
Min: 0.062923 -> 0.021829: 2.8826x faster
Avg: 0.063206 -> 0.022001: 2.8729x faster
Significant (t=793.026302)
Stddev: 0.00034 -> 0.00014: 2.3570x smaller (N = 50)

Running 'model_save_new' benchmark ...
Min: 0.041341 -> 0.021819: 1.8947x faster
Avg: 0.063127 -> 0.022379: 2.8208x faster
Significant (t=76.086085)
Stddev: 0.00344 -> 0.00158: 2.1770x smaller (N = 50)

Running 'multi_value_dict' benchmark ...
Min: 0.83 -> 0.85: 1.0230x slower
Avg: 0.000165 -> 0.000168: 1.0164x slower
Not significant
Stddev: 0.5 -> 0.5: 1.0339x larger (N = 50)

Running 'qs_filter_chaining' benchmark ...
Min: 0.004264 -> 0.001362: 3.1311x faster
Avg: 0.004311 -> 0.001391: 3.0996x faster
Significant (t=170.435257)
Stddev: 0.00010 -> 0.7: 1.2953x smaller (N = 50)

Running 'query_aggregate' benchmark ...
Min: 0.000467 -> 0.000412: 1.1331x faster
Avg: 0.000479 -> 0.000423: 1.1322x faster
Significant (t=11.625112)
Stddev: 0.2 -> 0.2: 1.0048x smaller (N = 50)

Running 'query_all' benchmark ...
Min: 0.047256 -> 0.043987: 1.0743x faster
Avg: 0.049713 -> 0.046489: 1.0693x faster
Significant (t=3.604195)
Stddev: 0.00423 -> 0.00470: 1.1104x larger (N = 50)

Running 'query_all_multifield' benchmark ...
Min: 0.103770 -> 0.099138: 1.0467x faster
Avg: 0.107616 -> 0.103382: 1.0409x faster
Significant (t=4.033836)
Stddev: 0.00495 -> 0.00553: 1.1182x larger (N = 50)

Running 'query_annotate' benchmark ...
Min: 0.000987 -> 0.000865: 1.1408x faster
Avg: 0.001006 -> 0.000881: 1.1417x faster
Significant (t=17.891371)
Stddev: 0.4 -> 0.3: 1.2235x smaller (N = 50)

Running 'query_complex_filter' benchmark ...
Min: 0.000308 -> 0.000218: 1.4136x faster
Avg: 0.000317 -> 0.000223: 1.4175x faster
Significant (t=27.463980)
Stddev: 0.2 -> 0.1: 1.3219x smaller (N = 50)

Running 'query_count' benchmark ...
Min: 0.000392 -> 0.000339: 1.1561x faster
Avg: 0.000401 -> 0.000347: 1.1571x faster
Significant (t=13.345028)
Stddev: 0.2 -> 0.2: 1.0469x smaller (N = 50)

Running 'query_dates' benchmark ...
Min: 0.001032 -> 0.000884: 1.1678x faster
Avg: 0.001055 -> 0.000900: 1.1730x faster
Significant (t=21.848869)
Stddev: 0.4 -> 0.3: 1.1197x smaller (N = 50)

Running 'query_delete' benchmark ...
Min: 0.000475 -> 0.000425: 1.1178x faster
Avg: 0.000530 -> 0.000436: 1.2153x faster
Significant (t=2.191750)
Stddev: 0.00030 -> 0.4: 8.3495x smaller (N = 50)

Running 'query_delete_related' benchmark ...
Min: 0.000565 -> 0.000550: 1.0278x faster
Avg: 0.000623 -> 0.000605: 1.0289x faster
Not significant
Stddev: 0.00034 -> 0.00031: 1.0908x smaller (N = 50)

Running 'query_distinct' benchmark ...
Min: 0.000634 -> 0.000570: 1.1121x faster
Avg: 0.000643 -> 0.000584: 1.1016x