Re: Daphne SIGKILLS on production server

[Vaisagh Viswanathan]

I forgot to mention supervisor is my process manager.

On Monday, December 16, 2019 at 12:08:55 PM UTC+8, Vaisagh Viswanathan 
wrote:
>
> Hi,
>
> We recently changed to django 2.x and channels. 2.2 from a system that has 
> been running relatively stable on channels 1.x for almost 3 years now. Our 
> tests did not show any issues when we were running django channels in our 
> test servers. And even when we tried load testing with frequent requests, 
> the errors were not unexpected i.e. there would be conenctions dropped and 
> a few 502s.  ANd as soon as the load was cut off things would work fine. 
>
> This changed drastically when we deployed it in our prod environment. Our 
> daphne process SIGKILLS every few minutes and as a result our websockets 
> connections are extremely unstable and requires frequent reconnects. Our 
> setup is such that we have an AWS ALB sitting in front of NGINX that 
> directs all non-websockets traffic to a gunicorn process (which is pretty 
> stable) and all websockets traffic to a daphne process. 
>
> We tried Uvicorn in the middle as well, howeer Uvicorn wouldn't sigkill 
> itself but it was just as unresponsive. 
>
> Our ELB logs show a steady stream of 502 errors. 
>
> We have been dealing with these issues for around 2 days now and are 
> completely lost. If you coud give some advice on where we should be looking 
> at. What could be causing SIGKILLS, that would be seriously awesome. 
>
> Regards,
> Vaisagh
>
>
> *This message contains confidential information and is intended only for 
> the individual named. If you are not the named addressee you should not 
> disseminate, distribute or copy this e-mail. Please notify the sender 
> immediately by e-mail if you have received this e-mail by mistake and 
> delete this e-mail from your system. E-mail transmission cannot be 
> guaranteed to be secure or error-free as information could be intercepted, 
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
> The sender therefore does not accept liability for any errors or omissions 
> in the contents of this message, which arise as a result of e-mail 
> transmission. "impress.ai <http://impress.ai>" and "Impress" are brands of 
> Ideatory Pte. Ltd., #08-01, 80 Robinson Road, Singapore - 068898, 
> https://impress.ai <https://impress.ai>.*
>

-- 
_This message contains confidential information and is intended only for 
the individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
The sender therefore does not accept liability for any errors or omissions 
in the contents of this message, which arise as a result of e-mail 
transmission. "impress.ai <http://impress.ai>" and "Impress" are brands of 
Ideatory Pte. Ltd., #08-01, 80 Robinson Road, Singapore - 068898, 
https://impress.ai <https://impress.ai>._

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8b5653b4-c756-4e25-a3f0-fa5caeaca856%40googlegroups.com.

Daphne SIGKILLS on production server

[Vaisagh Viswanathan]

Hi,

We recently changed to django 2.x and channels. 2.2 from a system that has
been running relatively stable on channels 1.x for almost 3 years now. Our
tests did not show any issues when we were running django channels in our
test servers. And even when we tried load testing with frequent requests,
the errors were not unexpected i.e. there would be conenctions dropped and
a few 502s.  ANd as soon as the load was cut off things would work fine.

This changed drastically when we deployed it in our prod environment. Our
daphne process SIGKILLS every few minutes and as a result our websockets
connections are extremely unstable and requires frequent reconnects. Our
setup is such that we have an AWS ALB sitting in front of NGINX that
directs all non-websockets traffic to a gunicorn process (which is pretty
stable) and all websockets traffic to a daphne process.

We tried Uvicorn in the middle as well, howeer Uvicorn wouldn't sigkill
itself but it was just as unresponsive.

Our ELB logs show a steady stream of 502 errors.

We have been dealing with these issues for around 2 days now and are
completely lost. If you coud give some advice on where we should be looking
at. What could be causing SIGKILLS, that would be seriously awesome.

Regards,
Vaisagh

-- 
_This message contains confidential information and is intended only for 
the individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
The sender therefore does not accept liability for any errors or omissions 
in the contents of this message, which arise as a result of e-mail 
transmission. "impress.ai " and "Impress" are brands of 
Ideatory Pte. Ltd., #08-01, 80 Robinson Road, Singapore - 068898, 
https://impress.ai ._

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CADjVUwkjG856hP6XXuj4_fmKdoY05ZVY88qVoUW-1SRR%2Bd16EA%40mail.gmail.com.

Django channels 1.1.3 not found on pip

[Vaisagh Viswanathan]

I am using django-channels for my project and I needed the specific bug fix 
that is part of 1.1.3 version. However, even though the release notes: 
https://channels.readthedocs.io/en/stable/releases/index.html say that 
1.1.3 is out, I can't find it on pip.

Is there generally a delay in this appearing? When can I expect it to be 
updated?

Regards,
Vaisagh


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/ccbaf392-8039-4fca-a070-9cbfe37e1959%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Structure for deploying django channels in producting in AWS

[Vaisagh Viswanathan]

Hi,

I'm trying to setup up my django project to use django channels in a 
dockerized setup. The whole setup seems to work fine on a local server and 
when I use runserver instead of daphne.  When I run the setup using daphne, 
the web page gives no response 

The  page isn’t working

* didn’t send any data.

ERR_EMPTY_RESPONSE


I am not sure at what point this is really failing. I tried running daphne 
with verbosity 2 and it seems the connections are not reaching it. 

The setup is like this:

I am running the whole set up on AWS. I have a Redis cluster that I am 
connecting to and using as a channel layer. I assume this is working fine, 
since the runserver seems to be working.

Following this I changed the setup to have two docker containers. The first 
one runs this script and has port 8000 of container mapped to port 80 of 
host machine:

cd myproject
python manage.py collectstatic --no-input
python manage.py migrate
python manage.py createsu
daphne myproject.asgi:channel_layer -b 0.0.0.0 -p 8000 --verbosity 2

The second container simply has this:

cd myproject  
python manage.py runworker


The logs from both containers indicate that they are both waiting for some 
requests to reach them. But nothing seems to be reaching them.

I have no idea what I'm supposed to be fixing here. DO i need to put 
nginx/apache in front of it for daphne to work??

Regards,
Vaisagh

-- 
*This message contains confidential information and is intended only for 
the individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
The sender therefore does not accept liability for any errors or omissions 
in the contents of this message, which arise as a result of e-mail 
transmission. If verification is required please request a hard-copy 
version. Ideatory Pte. Ltd., #02-18, 71 Ayer Rajah Crescent, Singapore - 
139951, www.ideatory.co .*

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/b18d4163-1ccf-45d5-9dac-04b44c77245c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.