Hi, We encountered a big problem this afternoon, which crashed for a while one of our websites, a java (tomcat+lift) application. We are using Haproxy 1.5.
For our backend, we're doing something like this, using tcp-check because we need to check status AND a string, which is not possible with http-check : backend backend-mywebsite balance roundrobin option redispatch option tcp-check tcp-check send GET\ /check \ HTTP/1.1\r\nHost:\ www.mywebsite.fr\r\nConnection:\ close\r\n tcp-check send \r\n tcp-check expect string HTTP/1.1\ 200\ OK tcp-check expect rstring "healthStatus":"(Healthy|DegradedMode)" cookie JSESSIONID prefix nocache server s1 s1:11503 weight 1 check inter 10s fall 3 rise 2 ssl cookie s1 server s2 s2:11503 weight 1 check inter 10s fall 3 rise 2 ssl cookie s2 server s3 s3:11503 weight 1 check inter 10s fall 3 rise 2 ssl cookie s3 server s4 s4:11503 weight 1 check inter 10s fall 3 rise 2 ssl cookie s4 For some reasons, the /check page didn't returned the correct application status and our / returned a 500 even if /check was OK, so we decided to check /. After 20 minutes, our application crashed. In fact, our 4 fronts crashed at the same time, and if we restarted them, 20 minutes after, they crashed again. We lost some time because we were really thinking on a software bug, before we realize the root cause. * Each tcp-check send opens a session on the application * Each session, on the / page, consumes 500 KB * session duration : 30 minutes * We have 4 Haproxy, doing 2 checks (the app provides 2 websites, so one check for each Host: ), 6 times per minute = 48 checks, each minute. On each front. * After 20 minutes : more than 450 MB used in the app for sessions * Full GC, crash So, my question is : Is it possible to get and store the JSESSIONID cookie returned by the tcp-check expect (or something like this), and send it with the tcp-check send, to reuse the same session ? Is there a way for a health check to use persistent cookie session (always the same, one per server), returned by the check ? Thank you very much, Sebastien Rohaut
