Django 3.0 -> 3.1 performance degradation

[Jonatan Heyman]

Hi!

I've recently upgraded an old Django app from Django 1.11 all the way up to 
Django 4.0. Even though the app is pretty large, the migration went smooth 
(or so I thought). However, when all tests passed and I deployed the app to 
production, I saw that the response time of the app took a significant hit 
(the response time for the 95th percentile went from about 300 ms to 900 
ms, and the median from about 100 ms to 250 ms). I also saw that the server 
CPU load had increased significantly.

I wrote some load tests and started downgrading Django. Now I've narrowed 
down the problem to appear when I upgrade from Django 3.0.x to Django 
3.1.x, with no other significant changes to my app code.

Is this something someone else has experienced, or does anyone know what 
might be going on?

Are there any known changes that might increase CPU usage? In that case, I 
could throw more hardware at the problem.

Any ideas, pointers or insights would be greatly appreciated!

Best,
Jonatan

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5f05c06c-bd22-417b-9688-20aca9632bcbn%40googlegroups.com.

Re: Django 3.0 -> 3.1 performance degradation

[Antonis Christofides]

Hi! This is a very interesting problem, and I'm afraid I'm not qualified to 
answer it. What I'd like to point out is that, if I had done all the work that 
you seem to have done, I would be tempted to start bisecting in between versions 
3 and 3.1 in order to find the exact commit that causes the problem.


(However, in order to avoid that, I might also take a second look at the release 
notes, in case there was some information in there. And I might also ask in the 
developers mailing list—many of those who might know the answer don't read 
django-users.)


It's possible it's a Django issue, but it's also possible it isn't. For example, 
it could be some error in your caches or templates or other configuration that 
happens to only manifest itself in Django>=3.1.


In any case, I'm eager to learn how this plays out.

Regards,

Antonis


On 04/05/2022 12.12, Jonatan Heyman wrote:


Hi!

I've recently upgraded an old Django app from Django 1.11 all the way up to 
Django 4.0. Even though the app is pretty large, the migration went smooth (or 
so I thought). However, when all tests passed and I deployed the app to 
production, I saw that the response time of the app took a significant hit 
(the response time for the 95th percentile went from about 300 ms to 900 ms, 
and the median from about 100 ms to 250 ms). I also saw that the server CPU 
load had increased significantly.


I wrote some load tests and started downgrading Django. Now I've narrowed down 
the problem to appear when I upgrade from Django 3.0.x to Django 3.1.x, with 
no other significant changes to my app code.


Is this something someone else has experienced, or does anyone know what might 
be going on?


Are there any known changes that might increase CPU usage? In that case, I 
could throw more hardware at the problem.


Any ideas, pointers or insights would be greatly appreciated!

Best,
Jonatan
--
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/5f05c06c-bd22-417b-9688-20aca9632bcbn%40googlegroups.com 
.


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/160b9b6a-fc89-cb54-7dec-39ef1ffafaba%40antonischristofides.com.

Re: Django 3.0 -> 3.1 performance degradation

[Jonatan Heyman]

Hi! And thanks for the reply!

I've read the release notes but couldn't see anything that popped out as 
the possible cause. But I'm going to give them another read. Currently, the 
app is running fine on 3.0, and I haven't started debugging what might be 
the cause. I started with a message to this list, hoping that this would 
sound familiar to someone else :).

Bisecting between 3.0 and 3.1 is a possibility, though it's time-consuming 
since I have to rebuild the Docker image, deploy it to the staging server 
and run the load test for a little while between each test.

I'll post an update when I make any progress!

Best,
Jonatan

On Thursday, May 5, 2022 at 2:37:06 PM UTC+2 Antonis Christofides wrote:

> Hi! This is a very interesting problem, and I'm afraid I'm not qualified 
> to answer it. What I'd like to point out is that, if I had done all the 
> work that you seem to have done, I would be tempted to start bisecting in 
> between versions 3 and 3.1 in order to find the exact commit that causes 
> the problem.
>
> (However, in order to avoid that, I might also take a second look at the 
> release notes, in case there was some information in there. And I might 
> also ask in the developers mailing list—many of those who might know the 
> answer don't read django-users.)
>
> It's possible it's a Django issue, but it's also possible it isn't. For 
> example, it could be some error in your caches or templates or other 
> configuration that happens to only manifest itself in Django>=3.1.
>
> In any case, I'm eager to learn how this plays out.
>
> Regards,
>
> Antonis
>
> On 04/05/2022 12.12, Jonatan Heyman wrote:
>
> Hi!
>
> I've recently upgraded an old Django app from Django 1.11 all the way up 
> to Django 4.0. Even though the app is pretty large, the migration went 
> smooth (or so I thought). However, when all tests passed and I deployed the 
> app to production, I saw that the response time of the app took a 
> significant hit (the response time for the 95th percentile went from about 
> 300 ms to 900 ms, and the median from about 100 ms to 250 ms). I also saw 
> that the server CPU load had increased significantly.
>
> I wrote some load tests and started downgrading Django. Now I've narrowed 
> down the problem to appear when I upgrade from Django 3.0.x to Django 
> 3.1.x, with no other significant changes to my app code.
>
> Is this something someone else has experienced, or does anyone know what 
> might be going on?
>
> Are there any known changes that might increase CPU usage? In that case, I 
> could throw more hardware at the problem.
>
> Any ideas, pointers or insights would be greatly appreciated!
>
> Best,
> Jonatan
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/5f05c06c-bd22-417b-9688-20aca9632bcbn%40googlegroups.com
>  
> 
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/542bbe8d-c067-423e-9676-6e4b3bab2146n%40googlegroups.com.

Re: Django 3.0 -> 3.1 performance degradation

[Jason]

also would be helpful if you could give some examples.  All that is stated 
is "there's a significant slowdown", but nothing where or if there's been 
any tracing in place.  

Is this happening with db queries?  
serializing? 
rendering? 
are you using old packages which might have open/closed issues about 
performance?

On Friday, May 6, 2022 at 5:03:33 PM UTC-4 Jonatan Heyman wrote:

> Hi! And thanks for the reply!
>
> I've read the release notes but couldn't see anything that popped out as 
> the possible cause. But I'm going to give them another read. Currently, the 
> app is running fine on 3.0, and I haven't started debugging what might be 
> the cause. I started with a message to this list, hoping that this would 
> sound familiar to someone else :).
>
> Bisecting between 3.0 and 3.1 is a possibility, though it's time-consuming 
> since I have to rebuild the Docker image, deploy it to the staging server 
> and run the load test for a little while between each test.
>
> I'll post an update when I make any progress!
>
> Best,
> Jonatan
>
> On Thursday, May 5, 2022 at 2:37:06 PM UTC+2 Antonis Christofides wrote:
>
>> Hi! This is a very interesting problem, and I'm afraid I'm not qualified 
>> to answer it. What I'd like to point out is that, if I had done all the 
>> work that you seem to have done, I would be tempted to start bisecting in 
>> between versions 3 and 3.1 in order to find the exact commit that causes 
>> the problem.
>>
>> (However, in order to avoid that, I might also take a second look at the 
>> release notes, in case there was some information in there. And I might 
>> also ask in the developers mailing list—many of those who might know the 
>> answer don't read django-users.)
>>
>> It's possible it's a Django issue, but it's also possible it isn't. For 
>> example, it could be some error in your caches or templates or other 
>> configuration that happens to only manifest itself in Django>=3.1.
>>
>> In any case, I'm eager to learn how this plays out.
>>
>> Regards,
>>
>> Antonis
>>
>> On 04/05/2022 12.12, Jonatan Heyman wrote:
>>
>> Hi!
>>
>> I've recently upgraded an old Django app from Django 1.11 all the way up 
>> to Django 4.0. Even though the app is pretty large, the migration went 
>> smooth (or so I thought). However, when all tests passed and I deployed the 
>> app to production, I saw that the response time of the app took a 
>> significant hit (the response time for the 95th percentile went from about 
>> 300 ms to 900 ms, and the median from about 100 ms to 250 ms). I also saw 
>> that the server CPU load had increased significantly.
>>
>> I wrote some load tests and started downgrading Django. Now I've narrowed 
>> down the problem to appear when I upgrade from Django 3.0.x to Django 
>> 3.1.x, with no other significant changes to my app code.
>>
>> Is this something someone else has experienced, or does anyone know what 
>> might be going on?
>>
>> Are there any known changes that might increase CPU usage? In that case, 
>> I could throw more hardware at the problem.
>>
>> Any ideas, pointers or insights would be greatly appreciated!
>>
>> Best,
>> Jonatan
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-users...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-users/5f05c06c-bd22-417b-9688-20aca9632bcbn%40googlegroups.com
>>  
>> 
>> .
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c0712c2b-a796-4bbb-a20a-df31efa5ae8fn%40googlegroups.com.