NGINX + tomcat 8.0.35 (110: Connection timed out)

[Ayub Khan]

During high load of 16k requests per minute, we notice below error in log.

 [error] 2437#2437: *13335389 upstream timed out (110: Connection timed
out) while reading response header from upstream,  server: jahez.net,
request: "GET /serviceContext/ServiceName?callback= HTTP/1.1", upstream: "
http://127.0.0.1:8080/serviceContext/ServiceName

Below is the flow of requests:

cloudflare-->AWS ALB--> NGINX--> Tomcat-->Elastic-search

In NGINX we have the below config

location /serviceContext/ServiceName{

    proxy_pass          http://localhost:8080/serviceContext/ServiceName;
   proxy_http_version  1.1;
    proxy_set_header    Connection          $connection_upgrade;
    proxy_set_header    Upgrade             $http_upgrade;
    proxy_set_header    Host                      $host;
    proxy_set_header    X-Real-IP              $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;


        proxy_buffers 16 16k;
        proxy_buffer_size 32k;
}

below is tomcat connector config

<Connector port="8080"
               protocol="org.apache.coyote.http11.Http11NioProtocol"
               connectionTimeout="200" maxThreads="50000"
               URIEncoding="UTF-8"
               redirectPort="8443" />


We monitor the open file using *watch "sudo ls /proc/`cat
/var/run/tomcat8.pid`/fd/ | wc -l" *the number of tomcat open files keeps
increasing slowing the responses. the only option to recover from this is
to restart tomcat.