Re: HAProxy bottleneck/tuning
Hi, On Sat, Sep 28, 2019 at 07:07:02PM +0200, Emmanuel BILLOT wrote: > Hi, > > We use HAProxy as a LB for many usage, including LB for Squid and user > acces on Internet. > Users frequently grumble about "Internet speed" (classical...) and we want > to be sure that it is not a LB issue (msconfig, or system bottleneck ?). > > How could we find is haproxy is "undersized" in config or if the hosting > system (CentOS) is not correctly configured (file desc ? memory ?) Well, this sounds more like a sysadmin training than an haproxy-specific question, but a few things you should have a look at : - is your system swapping ? (it must never) - do you see in your system logs "conntrack table full" ? If so you're hitting some misconfigured conntrack limits - do you see in your system logs "TCP: sending SYN cookies" ? If so you're likely running haproxy with too low a maxconn setting, resulting in connections not being accepted - do you see in your haproxy logs flags "sC" or "SC" while you know your servers are working fine ? If so that could indicate a failure to allocate a source port or a file descriptor for an outgoing connection - do you see your CPU steadily running above, say, 50% ? If so you cannot exclude frequent 100% peaks possibly causing some sub-second delays not reported by the system tools. - and if you're running in a VM, you can redo all this above inside the hypervisor, and in the guest you should also look at the "st" field in vmstat to make sure your CPU is not stolen by other VMs (or goodbye low latency), and you can also run "ping" to your VM from an external host on the same LAN and make sure the latency never jumps above 1 millisecond on a mostly idle network or 2-3 ms on a loaded network or it can indicate that you're have performance issues caused by noisy neighbors on your machine. - connection setup timers exhibiting multiple of 3s in haproxy logs usually indicate packet drops between haproxy and the servers - client timeout errors during request like "cR" often indicate drops or MTU issues between the client and haproxy (sometimes caused by bogus virtualized network drivers). Hoping this helps, Willy
Re: [PATCH 2/2] BUG/MINOR: lua: Make the arrays in the list of headers 1-indexed
Hi guys,
On Thu, Sep 26, 2019 at 04:35:31PM +0200, Tim Düsterhus wrote:
> Adis,
>
> Am 26.09.19 um 16:01 schrieb Adis Nezirovic:
> > While I agree that using zero based array indexing is a mistake (wearing
> > my Lua hat), I don't think your proposal will make situation any better.
> > Actually ot might hurt existing Lua code in unforeseen ways.
>
> Yes, I understand that it breaks backwards compatibility. That's why I
> added the cover letter with the rationale of duplicating the first value
> instead of making the 0-index a nil.
To be honest I'm totally ignorant of this code area and it's even not
very clear to me (even after reading the commit messages) what is the
extent of this change nor what this headers array corresponds to.
However what I can say is that when implementing a transition towards
a final solution by having an intermediate solution (like you're
proposing by keeping index 0), it's important to have a plan to get rid
of the intermediate solution at some point. Here, by duplicating entries
I'm not seeing any way to properly cover this final transition, nor to
safely avoid breakage of existing applications. Thus it will turn an
existing problem into two other ones.
I could instead propose two solutions :
- either we knowingly break existing code for 2.1+ by changing the
array numbering to match Lua standards ; those migrating from 2.0
to 2.1 or 2.2 will have to modify their code. We can explain them
how to make portable code by checking the haproxy version from
within Lua if required;
- or we decide to rename this "headers" field on the long term (if
it's the only one, which is really the part I'm ignoring) and create
a new one ("hdrs" ?) numbered from 1. We can thus keep the two in
parallel till 2.2 included so that at least one LTS version covers
both the old and new API, and in 2.3 we remove headers. If we can
emit an API warning when "headers" is first met, it's even better.
I just don't know how we can make one structure field alias another
one, I can understand that it's not just a matter of setting +1 on
a pointer, but I guess that we have hooks behind the curtains which
intercept this "headers" thing.
There may also be other solutions, but definitely an API change needs
to pass through something visible so that affected code at some point
has to be fixed if we want to get rid of a past mistake.
Willy
Re: [PATCH 1/3] BUG/MINOR: lua: Properly initialize the buffer's fields for string samples in hlua_lua2(smp|arg)
On Sun, Sep 29, 2019 at 11:03:07PM +0200, Tim Duesterhus wrote: > `size` is used in conditional jumps and valgrind complains: (...) Looks pretty good, whole series applied, thank you Tim! Willy
[PATCH 4/4] CLEANUP: proxy: Remove `proxy_tbl_by_name`
Willy,
and a last one cleaning up the old proxy_tbl_by_name function so that the
bug will be gone for good.
Best regards
Tim Duesterhus
Apply with `git am --scissors` to automatically cut the commit message.
-- >8 --
Subject: [PATCH 4/4] CLEANUP: proxy: Remove `proxy_tbl_by_name`
It is no longer required as of 1b8e68e89aadb2740d79034c217257a78e9747b9
and is no longer used when #306 is fixed.
---
include/proto/proxy.h | 8
1 file changed, 8 deletions(-)
diff --git a/include/proto/proxy.h b/include/proto/proxy.h
index 558718ba2..7f108b507 100644
--- a/include/proto/proxy.h
+++ b/include/proto/proxy.h
@@ -97,14 +97,6 @@ static inline struct proxy *proxy_be_by_name(const char
*name)
return proxy_find_by_name(name, PR_CAP_BE, 0);
}
-/* Find the table having name . The name may also start with a '#' to
- * reference a numeric id. NULL is returned if not found.
- */
-static inline struct proxy *proxy_tbl_by_name(const char *name)
-{
- return proxy_find_by_name(name, 0, 1);
-}
-
/* this function initializes all timeouts for proxy p */
static inline void proxy_reset_timeouts(struct proxy *proxy)
{
--
2.23.0
[PATCH 2/3] CLEANUP: lua: Get rid of obsolete (size_t *) cast in hlua_lua2(smp|arg)
This was required for the `chunk` API (`data` was an int), but is not required with the `buffer` API. --- src/hlua.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/hlua.c b/src/hlua.c index be959d847..bc6c7f767 100644 --- a/src/hlua.c +++ b/src/hlua.c @@ -419,7 +419,7 @@ static int hlua_lua2arg(lua_State *L, int ud, struct arg *arg) case LUA_TSTRING: arg->type = ARGT_STR; - arg->data.str.area = (char *)lua_tolstring(L, ud, (size_t *)&arg->data.str.data); + arg->data.str.area = (char *)lua_tolstring(L, ud, &arg->data.str.data); /* We don't know the actual size of the underlying allocation, so be conservative. */ arg->data.str.size = arg->data.str.data; arg->data.str.head = 0; @@ -562,7 +562,7 @@ static int hlua_lua2smp(lua_State *L, int ud, struct sample *smp) case LUA_TSTRING: smp->data.type = SMP_T_STR; smp->flags |= SMP_F_CONST; - smp->data.u.str.area = (char *)lua_tolstring(L, ud, (size_t *)&smp->data.u.str.data); + smp->data.u.str.area = (char *)lua_tolstring(L, ud, &smp->data.u.str.data); /* We don't know the actual size of the underlying allocation, so be conservative. */ smp->data.u.str.size = smp->data.u.str.data; smp->data.u.str.head = 0; -- 2.23.0
[PATCH 3/3] BUG/MEDIUM: lua: Store stick tables into the sample's `t` field
This patch fixes issue #306. This bug was introduced in the stick table refactoring in 1b8e68e89aadb2740d79034c217257a78e9747b9. This fix must be backported to 2.0. --- src/hlua.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/hlua.c b/src/hlua.c index bc6c7f767..cefaf3801 100644 --- a/src/hlua.c +++ b/src/hlua.c @@ -713,8 +713,8 @@ __LJMP int hlua_lua2arg_check(lua_State *L, int first, struct arg *argp, memcpy(trash.area, argp[idx].data.str.area, argp[idx].data.str.data); trash.area[argp[idx].data.str.data] = 0; - argp[idx].data.prx = proxy_tbl_by_name(trash.area); - if (!argp[idx].data.prx) + argp[idx].data.t = stktable_find_by_name(trash.area); + if (!argp[idx].data.t) WILL_LJMP(luaL_argerror(L, first + idx, "table doesn't exist")); argp[idx].type = ARGT_TAB; break; -- 2.23.0
[PATCH 1/3] BUG/MINOR: lua: Properly initialize the buffer's fields for string samples in hlua_lua2(smp|arg)
`size` is used in conditional jumps and valgrind complains: ==24145== Conditional jump or move depends on uninitialised value(s) ==24145==at 0x4B3028: smp_is_safe (sample.h:98) ==24145==by 0x4B3028: smp_make_safe (sample.h:125) ==24145==by 0x4B3028: smp_to_stkey (stick_table.c:936) ==24145==by 0x4B3F2A: sample_conv_in_table (stick_table.c:1113) ==24145==by 0x420AD4: hlua_run_sample_conv (hlua.c:3418) ==24145==by 0x54A308F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==24145==by 0x54AFEFC: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==24145==by 0x54A29F1: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==24145==by 0x54A3523: lua_resume (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0) ==24145==by 0x426433: hlua_ctx_resume (hlua.c:1097) ==24145==by 0x42D7F6: hlua_action (hlua.c:6218) ==24145==by 0x43A414: http_req_get_intercept_rule (http_ana.c:3044) ==24145==by 0x43D946: http_process_req_common (http_ana.c:500) ==24145==by 0x457892: process_stream (stream.c:2084) Found while investigating issue #306. A variant of this issue exists since 55da165301b4de213dacf57f1902c2142e867775, which was using the old `chunk` API instead of the `buffer` API thus this patch must be backported to HAProxy 1.6 and higher. --- src/hlua.c | 6 ++ 1 file changed, 6 insertions(+) diff --git a/src/hlua.c b/src/hlua.c index 1c87daae3..be959d847 100644 --- a/src/hlua.c +++ b/src/hlua.c @@ -420,6 +420,9 @@ static int hlua_lua2arg(lua_State *L, int ud, struct arg *arg) case LUA_TSTRING: arg->type = ARGT_STR; arg->data.str.area = (char *)lua_tolstring(L, ud, (size_t *)&arg->data.str.data); + /* We don't know the actual size of the underlying allocation, so be conservative. */ + arg->data.str.size = arg->data.str.data; + arg->data.str.head = 0; break; case LUA_TUSERDATA: @@ -560,6 +563,9 @@ static int hlua_lua2smp(lua_State *L, int ud, struct sample *smp) smp->data.type = SMP_T_STR; smp->flags |= SMP_F_CONST; smp->data.u.str.area = (char *)lua_tolstring(L, ud, (size_t *)&smp->data.u.str.data); + /* We don't know the actual size of the underlying allocation, so be conservative. */ + smp->data.u.str.size = smp->data.u.str.data; + smp->data.u.str.head = 0; break; case LUA_TUSERDATA: -- 2.23.0
Re: using hashicorp vault for storing SSL certs
Hey, Sure, I'm using Vault PKI from years. If you want to store certs you probably want to use secret backend, with PKI (Public Key Infrastructure) you can maintain your own certificate infrastructure and issue certificates from Vault. I often use HAProxy with consul-template to automate process of cert rotation. Best On Sun, Sep 29, 2019 at 10:53 AM Илья Шипицин wrote: > hello, > > is anybody using https://www.vaultproject.io/docs/secrets/pki/index.html > for storing certs ? (I want to avoid reinventing the wheel here) > > thanks, > Ilya Shipitcin > -- Łukasz Jagiełło lukaszjagielloorg
Re: using hashicorp vault for storing SSL certs
Hi. Sun Sep 29 19:52:57 GMT+02:00 2019 Илья Шипицин : > hello, > is anybody using https://www.vaultproject.io/docs/secrets/pki/index.html for > storing certs ? (I want to avoid reinventing the wheel here) I not yet. I planned to go dig deeper into that topic but haven't the time to do it. It would be really great to get some information an experience about this topic. > thanks, Ilya Shipitcin Regards Aleks
using hashicorp vault for storing SSL certs
hello, is anybody using https://www.vaultproject.io/docs/secrets/pki/index.html for storing certs ? (I want to avoid reinventing the wheel here) thanks, Ilya Shipitcin
