On Thu, Sep 17, 2020 at 10:56:39AM +0200, Maciej Zdeb wrote: > Hi, > > Our config is quite complex and I'm trying to narrow it down. It is > occurring only on one production haproxy cluster (which consists of 6 > servers in each of two data centers) with significant load - crashes occurs > on random servers so I would exclude memory corruption.
When I'm saying "memory corruption" I don't necessarily mean hardware error, most likely a software error. For example a write after free to a memory location, or any such thing, which typically happens outside of the visible code path. > I'm suspecting SPOE or/and LUA script both are used to send metadata about > each request to an external endpoint. Yesterday I disabled this feature in > one datacenter to verify. Great! > Our build is done in docker (Ubuntu bionic) with kernel 4.9.184-linuxkit, > crash is on Ubuntu bionic 4.15.0-55-generic, using: > haproxy 2.0.17 > openssl 1.1.1f > pcre 8.44 > lua 5.3.5 > lrandom (PRNG for lua, we're using it for 2 or 3 years without any > problems, and soon we will drop it from our build) Never heard of this last one, not that it would make it suspicious at all, just that it might indicate you're having a slightly different workload than most common ones and can help spotting directions where to look for the problem. > compiled in following way: (...) OK, nothing unusual here, thanks for the details. Let's wait for your new tests to narrow down the issue a little bit more, then. Thanks, Willy
