Server transparency

[haproxy]

Ha proxiers,

Wondering if it might be possible to have return traffic of transparent L4 
proxies seem to come from individual real server or at least leave from same IP 
stack as request was received from.

Problem is I’ve got a HA proxy binding to multiple ‘public’ IP stacks with 
client transparency, each load balancing for different smtp pools, and every 
real server of every pool has the same default GW on the ‘private’ side of my 
HA proxy (though they could be configured to pool specific ‘private' side 
default GWs).

I would like every real server, whenever they establish an outbound connection, 
to been seen as either their individually IP or at least as their individually 
HA proxy ‘public’ side IP, right now every 'real' outbound connection just 
seems to come from the same IP stack, ie. the one with the HA proxy’s default 
GW.

Would it be possible to change this somehow, or do I need to split into 
separate HA proxies each on separate VMs and thus get different ‘public’ side 
default GW per pool?

TIA

/Steffen

Re: Use cases for haproxy

[Etienne Champetier]

port 443 sharing, ie nginx + ssh + openvpn (tcp) + JitsiVideoBridge on one
ip/server/port 443
with 'usesrc clientip' the server even see the real client ip

2016-03-20 11:41 GMT+01:00 Unknown User :

> Hi all,
>
> I am interested in knowing some of the use cases people have used haproxy
> for.
> This is in addition to standard load balancing. Are there any quirky
> problems solved? Any out of the box thinking?
>
> Thanks
>
>

servers multiple sources

[Beluc]

Hi,

We're trying to find a way to have multiple sources per server and
thus bypass 64k connections per server.

We already tried with SNAT iptables :
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to 10.0.0.1-10.0.10

without success because kernel is hashing real source ip and real
destination ip, so only one source ip nated is used (aka same as using
one different source per server).

Any idea on achieving this ? maybe in lua ?

Regards,

Re: servers multiple sources

[James Brown]

Why not just add each server multiple times with a different src parameter
and a different name.

Something like

backend my_be
mode tcp
server server1_src1 10.1.0.1 source 10.0.0.1
server server1_src2 10.1.0.1 source 10.0.0.2
server server2_src1 10.1.0.2 source 10.0.0.1
server server2_src2 10.1.0.2 source 10.0.0.2

On Mon, Mar 21, 2016 at 8:20 AM, Beluc  wrote:

> Hi,
>
> We're trying to find a way to have multiple sources per server and
> thus bypass 64k connections per server.
>
> We already tried with SNAT iptables :
> iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to 10.0.0.1-10.0.10
>
> without success because kernel is hashing real source ip and real
> destination ip, so only one source ip nated is used (aka same as using
> one different source per server).
>
> Any idea on achieving this ? maybe in lua ?
>
> Regards,
>
>


-- 
James Brown
Engineer

DOC Patch: tune.vars.xxx-max-size

[Daniel Schneller]

From 29bddd461c30bc850633350ac81e3c9fd7b56cb8 Mon Sep 17 00:00:00 2001
From: Daniel Schneller 
Date: Mon, 21 Mar 2016 20:46:57 +0100
Subject: [PATCH] DOC: Clarify tunes.vars.xxx-max-size settings

Adds a little more clarity to the description of the maximum sizes of
the different variable scopes and adds a note about what happens when
the space allocated for variables is too small.

Also fixes some typos and grammar/spelling issues re/ variables and
their naming conventions, copied throughout the document.
---
 doc/configuration.txt | 227 +-
 1 file changed, 114 insertions(+), 113 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index c9cca4f..5147626 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -1400,16 +1400,22 @@ tune.vars.global-max-size 
 tune.vars.reqres-max-size 
 tune.vars.sess-max-size 
 tune.vars.txn-max-size 
-  These four tunes helps to manage the allowed amount of memory used by the
-  variables system. "global" limits the memory for all the systems. "sess" 
limit
-  the memory by session, "txn" limits the memory by transaction and "reqres"
-  limits the memory for each request or response processing. during the
-  accounting, "sess" embbed "txn" and "txn" embed "reqres".
-
-  By example, we considers that "tune.vars.sess-max-size" is fixed to 100,
-  "tune.vars.txn-max-size" is fixed to 100, "tune.vars.reqres-max-size" is
-  also fixed to 100. If we create a variable "txn.var" that contains 100 bytes,
-  we cannot create any more variable in the other contexts.
+  These four tunes help to manage the maximum amount of memory used by the
+  variables system. "global" limits the overall amount of memory available
+  for all scopes. "sess" limits the memory for the session scope, "txn" for
+  the transaction scope, and "reqres" limits the memory for each request or
+  response processing.
+  Memory accounting is hierarchical, meaning more coarse grained limits
+  include the finer grained ones: "sess" includes "txn", and "txn" includes
+  "reqres".
+
+  For example, when "tune.vars.sess-max-size" is limited to 100,
+  "tune.vars.txn-max-size" and "tune.vars.reqres-max-size" cannot exceed
+  100 either. If we create a variable "txn.var" that contains 100 bytes,
+  all available space is consumed.
+  Notice that exceeding the limits at runtime will not result in an error
+  message, but values might be cut off or corrupted. So make sure to accurately
+  plan for the amount of space needed to store all your variables.
 
 tune.zlib.memlevel 
   Sets the memLevel parameter in zlib initialization for each session. It
@@ -3765,17 +3771,17 @@ http-request { allow | deny | tarpit | auth [realm 
] | redirect  |
   Is used to set the contents of a variable. The variable is declared
   inline.
 
- The name of the variable starts by an indication about its
-   scope. The allowed scopes are:
- "sess" : the variable is shared with all the session,
- "txn"  : the variable is shared with all the transaction
+ The name of the variable starts with an indication about
+   its scope. The scopes allowed are:
+ "sess" : the variable is shared with the whole session
+ "txn"  : the variable is shared with the transaction
   (request and response)
- "req"  : the variable is shared only during the request
+ "req"  : the variable is shared only during request
+  processing
+ "res"  : the variable is shared only during response
   processing
- "res"  : the variable is shared only during the response
-  processing.
This prefix is followed by a name. The separator is a '.'.
-   The name may only contain characters 'a-z', 'A-Z', '0-9',
+   The name may only contain characters 'a-z', 'A-Z', '0-9'
and '_'.
 
  Is a standard HAProxy expression formed by a sample-fetch
@@ -4077,17 +4083,17 @@ http-response { allow | deny | add-header   
| set-nice  |
   Is used to set the contents of a variable. The variable is declared
   inline.
 
- The name of the variable starts by an indication about its
-   scope. The allowed scopes are:
- "sess" : the variable is shared with all the session,
- "txn"  : the variable is shared with all the transaction
+ The name of the variable starts with an indication about
+   its scope. The scopes allowed are:
+ "sess" : the variable is shared with the whole session
+ "txn"  : the variable is shared with the transaction
   (request and response)
-   

Back pressure strategy for decentralized HAProxy setup

[Sergii Mikhtoniuk]

Hi,

My question is more about HTTP/REST in general, but I can't think of a
better place to find experts than this mailing list.

Can you share your approaches to providing back pressure with HTTP? I'm
talking about handling cases when upstream service is overloaded due to
increased request rate, or due to some problem that reduces its normal
capacity, e.g. hardware issues or database backup.

To give you more context - we are running a set of RESTful microservices
that are very latency sensitive. We use HAProxy in a fully decentralized
fashion to route the requests (aka SmartStack
). Suppose we
have N instances of service A that need to speak to M instances of service
B - every instance of A runs local HAProxy, which is automatically
reconfigured whenever we scale service B up or down.

This  model worked really well for us, and it's also becoming very common
in microservice environments (e.g. container schedulers such as Marathon
and Kubernetes), but somehow noone ever mentions how to implement back
pressure in it.

The decentralized nature prevents us from being able to use the only back
pressure mechanism HAProxy has - maxconn/maxqueue/timeout queue. Even if we
assume that request distribution is uniform and set A's HAProxy maxconn to
`Capacity(B) / N` this model breaks when we have another service C that
also makes requests to B, and we do have a complex service topology.

The way we are currently solving this:
- we assume that upstream service (B) is the only place where we know the
actual capacity and current load
- so it's the upstream service that makes a decision whether to accept
request for processing or to decline it
- if request is declined we want HAProxy to try the next server in the list
(`option redispatch`)
- the only way to achieve this (due to use of `splice`) is to prevent TCP
connection from ever being established
- so we use IPTables to set a hard limit on the number of active TCP
connection per service port
- when all `retries` instances of upstream service are busy - we fail
request very fast, allowing caller to perform any load shedding strategy
(asking client to try again, returning cached result etc.)

This solution worked well for us, but has a number of downsides:
- relies on iptables and conntrack, and a lot of kernel variable tweaking
- does not play well with Keep-Alive connections
- hard to use in containers, especially with network isolation

So we are looking at replacing it with something that works on application
level protocol. One idea would be:
- have yet another HAProxy, but now on the upstream service side
- it's only purpose will be to limit `maxconn` and maintain a queue of
connections
- ideally implement an Active Queue Management (AQM) strategy such as CoDel
 to allow this queue to absorb
short-term request bursts, but at the same time prevent bufferbloat-like
"standing queues"

To summarize:
- Can anyone recommend a better solution for back pressure with
decentralized HAProxy setup?
- Do you think AQM for connection queues will be a good addition to HAProxy?


Thanks,
Sergii