Hi Maciej,
On Wed, Nov 18, 2020 at 12:33:23PM +0100, Maciej Zdeb wrote:
> Hi again,
>
> So "# some headers manipulation, nothing different then on other clusters"
> was the important factor in config. Under this comment I've hidden from you
> one of our LUA scripts that is doing header manipulation like deleting all
> headers from request when its name begins with "abc*". We're doing it on
> all HAProxy servers, but only here it has such a big impact on the CPU,
> because of huge RPS.
>
> If I understand correctly:
> with nbproc = 20, lua interpreter worked on every process
> with nbproc=1, nbthread=20, lua interpreter works on single process/thread
>
> I suspect that running lua on multiple threads is not a trivial task...
You've indeed nailed it down to the exact *last* showstopper against
killing nbproc (which is why I didn't kill it in 2.3). Lua is inherenty
single-threaded. We've started to work with Thierry on porting it to
threads. Based on limited support of threading in Lua (requires a
recompilation of the library to pass lock callbacks), and the very low
number of people who require high Lua performance, we currently suspect
that we'll adopt the following approach:
- the default will not change
- lock callback registration will be supported when a multi-threaded
Lua library will be detected so that, by default, you just have to
rebuild liblua (possibly after applying a patch) to get full threading
support. I initially thought it would be a problem, but I was wrong,
we used to explain everyone how to build lua 5.3 when 5.2 was the
only one shipped in distros, and it never was a problem, so here it
should be the same. This would allow to instantly boost the Lua
performance without touching the Lua code.
- add a specific global option to use thread-local, lockless, Lua
stacks that are all independent on each other (just like you currently
have with processes). However there will likely be some constraints
at init time. This would be the fastest and cleanest, essentially for
those who have control over their Lua code (the vast majority of users).
And it would not require to rebuild the library.
I *absolutely* want to see threaded Lua support in 2.4, otherwise we'll
be stuck with this nasty nbproc that is causing more and more trouble to
the internal architecture.
So I guess we'll use you as a beta tester once we're starting to see
promising solutions ;-)
Cheers,
Willy
