Hi all,
I have this weird problem: REST API response is not returned to callers
correctly. It can only be recreated under some situation. Maybe it is a
caching problem, or incorrect configurations, or bug somewhere (???)
The configuration is Nginx<->uWsgi<->Django Rest Framework (DRF) under
Ubuntu.
I will summarize the business logic and the problem:
=========================================
All REST APIs are protected by Basic Authentication (username/password info
is persisted in Postgres)
DRF gets the authentication header and does the authentication itself.
There is a POST REST API "login" ; it returns a response with the user
information (like username).
There are other REST APIs, and say one of them is POST API called "XXX".
To recreate the problem:
====================
Client(the tool Postman) calls "login" API with user "A" authentication.
Response is good with user "A" info; response is 200 (as expected)
Client calls "XXX" API with user "B". It returns 500 (as expected).
Client calls "login" again repeatedly with user "A" authentication; now we
see intermittent problem; sometimes the returned response is about user
"A", but sometimes it is about user "B". Correct answer should be always
about user "A". All responses are 200.
My investigation:
==============
When the problem happens, if I look at the Nginx logs, I can see the new
entries with response status 200 but when I look at the Django logs, I
don't see the new entries. Based on this observation, I think Django is
being skipped somehow. Maybe uWsgi is being skipped as well. Maybe Nginx is
caching some response for certain situation. But it seems all is triggered
by the 500 response.
Any debugging tips is appreciated. I am just looking for new idea on how to
approach the problem.
Thanks in advance.
Steve
Below is configuration details:
========================
Ubuntu 16.04 LTS
uWSGI (2.0.14)
Django (1.9.12)
djangorestframework (3.3.3)
nginx version: nginx/1.10.0 (Ubuntu)
===========nginx.conf====================
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
log_format auth
'http_authorization= "$http_authorization" '
'request= "$request" '
'time_local= "$time_local" '
'status= "$status" '
;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
============nginx default file===================
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
#try_files $uri $uri/ =404;
try_files $uri $uri/ /index.html;
}
}
============nginx config file for django================
# the upstream component nginx needs to connect to
upstream django {
# we will be using socket connection
server unix:///tmp/zlims_uwsgi.sock;
}
# configuration of the server
server {
# use this version to receive all requests
listen 8000 default_server;
# the domain name it will serve for
# server_name _;
charset utf-8;
# server root, any location block path will be relative to this root
root /opt/zlims;
# max upload size
client_max_body_size 75M; # adjust to taste
# Django media
location /media {
#alias /opt/zlims/media; # your Django project's media files -
amend as required
}
location /static {
alias /opt/zlims/be/static/; # your Django project's static files -
amend as required
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass django;
#include /opt/zlims/uwsgi_params; # the uwsgi_params file you
installed
include uwsgi_params; # the uwsgi_params file you installed
proxy_cache_bypass $cookie_nocache $arg_nocache$arg_comment;
proxy_cache_bypass $http_pragma $http_authorization;
proxy_no_cache $cookie_nocache $arg_nocache$arg_comment;
proxy_no_cache $http_pragma $http_authorization;
access_log /var/log/nginx/auth.log auth;
}
}
========= uwsgi_params file======================
uwsgi_param QUERY_STRING $query_string;
uwsgi_param REQUEST_METHOD $request_method;
uwsgi_param CONTENT_TYPE $content_type;
uwsgi_param CONTENT_LENGTH $content_length;
uwsgi_param REQUEST_URI $request_uri;
uwsgi_param PATH_INFO $document_uri;
uwsgi_param DOCUMENT_ROOT $document_root;
uwsgi_param SERVER_PROTOCOL $server_protocol;
uwsgi_param REQUEST_SCHEME $scheme;
uwsgi_param HTTPS $https if_not_empty;
uwsgi_param REMOTE_ADDR $remote_addr;
uwsgi_param REMOTE_PORT $remote_port;
uwsgi_param SERVER_PORT $server_port;
uwsgi_param SERVER_NAME $server_name;
=============uwsgi vassel .ini file=====================
[uwsgi]
set-placeholder = base_dir = /opt/zlims/be
chdir =/opt/zlims/be
module =zlims.wsgi
home =/opt/zlims/venv
master =true
processes =10
socket =/tmp/%n.sock
chmod-socket =666
vacuum =true
daemonize =/var/log/zlims/uwsgi_zlims.log
logdate =%%d/%%m/%%Y %%T
log-maxsize =10000000
disable-logging =true # disable request logging
lazy-apps =true
single-interpreter =true
enable-threads =true
========uwsgi service script (for systemd) ===========================
# uwsgi service script
[Unit]
Description=uWSGI Emperor service
After=syslog.target
[Service]
ExecStart=/usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals/
Restart=always
KillSignal=SIGQUIT
Type=notify
StandardError=syslog
NotifyAccess=all
[Install]
WantedBy=multi-user.target
_______________________________________________
uWSGI mailing list
uWSGI@lists.unbit.it
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
