Re: testing and validating complex haproxy.conf rules
Hi Dave
On 01.04.20 00:36, Dave Cottlehuber wrote:
On Tue, 31 Mar 2020, at 07:53, Aleksandar Lazic wrote:
Hi Dave.
On 31.03.20 09:24, Dave Cottlehuber wrote:
hi all,
Our main haproxy.conf has practically become sentient... it's reached the
point where the number of url redirects and similar incantations is very
hard to reason about, and certainly not test or validate, until it's
shipped. In fact I deploy to a "B" cluster node, and verify most changes
on a spare production node. This is not always possible to ensure that
existing acls and url redirects aren't broken by the changes.
For example:
https://%[hdr(host)]%[url,regsub(/$,)] ...
didn't do what the person who deployed it thinks it does - easy enough to
fix. How could we have tested this locally before committing it?
Is there any easy-ish way to try out these rules, almost like you
could in a REPL?
Once we've written them, and committed them to our ansible repos, is there
any way to unit test the whole config, to avoid regressions?
90% of these commits relate to remapping and redirecting urls from patterns.
Please can you tell us which version of HAProxy and some more details
from the config.
Maybe you can split the redirects, for example can you use a map for
the host part.
thanks Aleks,
In this case it's haproxy 2.1, and the config is complex.
This is a generic problem, not one for a single rule -- I need to find a way
to enable other people "unit test" their changes, before committing, and,
once committed, to avoid breaking production, be able to validate that the
most recent change doesn't break existing functions (more unit tests but
over the whole config). I can spin up a full staging environment if
necessary but I'm hoping somebody has a clever hack to avoid this.
Our newer stuff looks a bit like this with a map file:
http-requestredirect code 301 location
%[capture.req.uri,map(/usr/local/etc/haproxy/redirects.map)] if {
capture.req.uri,map(/usr/local/etc/haproxy/redirects.map) -m found }
but there are hundreds of acls that can overlap, or even override the
straightforward logic of the map. That's what I need to find a way to deal with.
Well I think that you reach the limit of a static config file.
How about to use some filters like SPOE or Fcgi-app
http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#9
I assume from the commits in 2.2 that there will be a filter possibility with
LUA but I don't know how usable or complete it will be.
Maybe a staging environment isn't such a bad idea ;-)
A+
Dave
Regards
Aleks
Re: testing and validating complex haproxy.conf rules
On Tue, 31 Mar 2020, at 07:53, Aleksandar Lazic wrote:
> Hi Dave.
>
> On 31.03.20 09:24, Dave Cottlehuber wrote:
> > hi all,
> >
> > Our main haproxy.conf has practically become sentient... it's reached the
> > point where the number of url redirects and similar incantations is very
> > hard to reason about, and certainly not test or validate, until it's
> > shipped. In fact I deploy to a "B" cluster node, and verify most changes
> > on a spare production node. This is not always possible to ensure that
> > existing acls and url redirects aren't broken by the changes.
> >
> > For example:
> >
> > https://%[hdr(host)]%[url,regsub(/$,)] ...
> >
> > didn't do what the person who deployed it thinks it does - easy enough to
> > fix. How could we have tested this locally before committing it?
> >
> > Is there any easy-ish way to try out these rules, almost like you
> > could in a REPL?
> >
> > Once we've written them, and committed them to our ansible repos, is there
> > any way to unit test the whole config, to avoid regressions?
> >
> > 90% of these commits relate to remapping and redirecting urls from patterns.
>
> Please can you tell us which version of HAProxy and some more details
> from the config.
> Maybe you can split the redirects, for example can you use a map for
> the host part.
thanks Aleks,
In this case it's haproxy 2.1, and the config is complex.
This is a generic problem, not one for a single rule -- I need to find a way
to enable other people "unit test" their changes, before committing, and,
once committed, to avoid breaking production, be able to validate that the
most recent change doesn't break existing functions (more unit tests but
over the whole config). I can spin up a full staging environment if
necessary but I'm hoping somebody has a clever hack to avoid this.
Our newer stuff looks a bit like this with a map file:
http-requestredirect code 301 location
%[capture.req.uri,map(/usr/local/etc/haproxy/redirects.map)] if {
capture.req.uri,map(/usr/local/etc/haproxy/redirects.map) -m found }
but there are hundreds of acls that can overlap, or even override the
straightforward logic of the map. That's what I need to find a way to deal with.
A+
Dave
Re: testing and validating complex haproxy.conf rules
Hi Dave. On 31.03.20 09:24, Dave Cottlehuber wrote: hi all, Our main haproxy.conf has practically become sentient... it's reached the point where the number of url redirects and similar incantations is very hard to reason about, and certainly not test or validate, until it's shipped. In fact I deploy to a "B" cluster node, and verify most changes on a spare production node. This is not always possible to ensure that existing acls and url redirects aren't broken by the changes. For example: https://%[hdr(host)]%[url,regsub(/$,)] ... didn't do what the person who deployed it thinks it does - easy enough to fix. How could we have tested this locally before committing it? Is there any easy-ish way to try out these rules, almost like you could in a REPL? Once we've written them, and committed them to our ansible repos, is there any way to unit test the whole config, to avoid regressions? 90% of these commits relate to remapping and redirecting urls from patterns. Please can you tell us which version of HAProxy and some more details from the config. Maybe you can split the redirects, for example can you use a map for the host part. A+ Dave Regards Aleks
testing and validating complex haproxy.conf rules
hi all, Our main haproxy.conf has practically become sentient... it's reached the point where the number of url redirects and similar incantations is very hard to reason about, and certainly not test or validate, until it's shipped. In fact I deploy to a "B" cluster node, and verify most changes on a spare production node. This is not always possible to ensure that existing acls and url redirects aren't broken by the changes. For example: https://%[hdr(host)]%[url,regsub(/$,)] ... didn't do what the person who deployed it thinks it does - easy enough to fix. How could we have tested this locally before committing it? Is there any easy-ish way to try out these rules, almost like you could in a REPL? Once we've written them, and committed them to our ansible repos, is there any way to unit test the whole config, to avoid regressions? 90% of these commits relate to remapping and redirecting urls from patterns. A+ Dave
