Re: Haproxy custom log - apache replacement

[Benoit GEORGELIN - Association Web4all]

Hello ! 

thanks for your answer, I will have a closer look about this configuration. I 
have seen it in the documentation before but never tried it 

Regards, 
Benoit 

FOB HK FOB sz

[FOB HK FOB sz]

大陆至深圳港口运输，报关，中港线     联系 邮 箱：nanfeng...@163.com
一、大陆至香港散货  免费上门提货
0.68元/公斤；    泡货120元/立方；
近期经常外出，有订单当天收货的请给我打电话。  祝你工作顺利、开心快乐！  
东莞市南丰物流（香港）有限公司
联   系 人：13922925659/ {Mobile}13662814456  咨询接单QQ：654340993 联系 邮 箱：nanfeng...@163.com运 输 地址：东莞东城牛山景观路1-3栋（市委党校对面）>服务只有起点，满意满意终点<*为了保证你能继续收到我们的优质信息，请将我们加入你的联系人列表。如本类邮件打扰您的工作，请备注邮箱账号回复到本邮箱，我司将不再发送。谢

Re: Haproxy custom log - apache replacement

[Aleksandar Lazic]

Hi.

Am 08-09-2016 21:16, schrieb Benoit GEORGELIN - Association Web4all:

Hello members of the list,

I was wondering if Haproxy custom log could log the same things as 
apache log format does ?


Well due to the fact that we don't know your apache log format I assume 
you mean CLF (common log format) ;-).

http://httpd.apache.org/docs/2.4/mod/mod_log_config.html

I also not know which version of haproxy you use so I refer to the 
latest stable one.


There is a predefined crf format in haproxy which is documented here

http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#8.2
http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#8.2.4

Haproxy is on the top for HTTP servers farms and I would like to stop 
logging on the HTTP servers side (apache or ngnix) and consume the log 
made by Haproxy with everything I get from a typical Apache log


I think, at that time, Haproxy does not log everything like apache will 
do, but maybe it's something than can be changed ?


HAProxy is able to log much more the apache ;-).

You will need to add some lines like these to get the additional 
information from the request.


##
 # log the beginning of the referrer
 capture request  header Referer len 20
 capture request  header User-agent len 20
##

As described here.
http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#8.8


That would be nice and will save a lot of time/ressources
In my case, this is what i'm doing :

- Haproxy
Loadblance to 5 HTTP servers
Apache2 custom log with X-Forwarder-for

- Apache2
Each HTTP serveur runs apache2 and log http requests locally to 
access_domain_log


Every night, merging of the log and then process the full log with Http 
log analyser like Awstats


Before you can feed Awstats with the new logs you will need to 
pre-process it.


Maybe the halog tool ( 
http://git.haproxy.org/?p=haproxy-1.6.git;a=tree;f=contrib/halog;h=189001c583ddcf3e35060171a58c39fd5115b9c2;hb=HEAD 
) can help.


There is also a doc for this tool.
https://www.haproxy.com/static/media/uploads/eng/resources/appnotes_0054_analyze_haproxy_logs_with_halog_en.pdf


Store the logs to a dedicated storage for archiving

If I could get the same HTTP information directly from Haproxy log , 
this will same us a lot of time an ressource usage.


Haproxy logs -> log shipper -> elasticsearch /hadoop  -> kibana or 
something else to display log information


Thanks for your help

Cordialement,

Benoît G


Best regards
Aleks

[PATCH] New DNS parser

[Baptiste]

Hi all,

Please find in attachment 10 patches to cover the following new topic in
HAProxy:

1. a new DNS parser, which stores the DNS response into a DNS structure,
instead of manipulating a buffer.
=> it doesn't add any feature by itself, but it will make DNS consumer
tasks much easier when using DNS responses

2. when the DNS response finishes with a CNAME, now HAProxy sends a new
query, changing the query type (from  to A or A to )

I heavily tested the code, but I'd like more people to test it in their own
environment.

We can now move forward on the next big development: filling servers in a
backend based on records read in a DNS responses.

Conrad: I have a quick and dirty and not finished patch to read and store
SRV records. If you want to use it for your own dev, please let me know.

Baptiste
From 2d196c70952be351508e3ee154d6c57d5cefee2e Mon Sep 17 00:00:00 2001
From: Baptiste Assmann 
Date: Mon, 18 Apr 2016 19:42:57 +0200
Subject: [PATCH 01/11] CLEANUP/MINOR dns: comment do not follow up code update

The loop comment is not appropriate anymore and needed to be updated
according to the code.

backport: no
---
 src/dns.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/dns.c b/src/dns.c
index c76637f..b9dce6b 100644
--- a/src/dns.c
+++ b/src/dns.c
@@ -725,7 +725,7 @@ int dns_get_ip_from_response(unsigned char *resp, unsigned char *resp_end,
 /* move forward data_len for analyzing next record in the response */
 reader += data_len;
 		} /* switch (record type) */
-	} /* for i 0 to ancount */
+	} /* list for each record entries */
 
 	/* Select an IP regarding configuration preference.
 	 * Top priority is the prefered network ip version,
-- 
1.9.1

From 83e6c3f60ade30a175b40b17f312fbf1e5b5aae2 Mon Sep 17 00:00:00 2001
From: Baptiste Assmann 
Date: Sat, 26 Mar 2016 14:12:50 +0100
Subject: [PATCH 02/11] MINOR: chunk: new strncat function

Purpose of this function is to append data to the end of a chunk when
we know only the pointer to the beginning of the string and the string
length.
---
 include/common/chunk.h | 13 +
 1 file changed, 13 insertions(+)

diff --git a/include/common/chunk.h b/include/common/chunk.h
index aac5282..205523c 100644
--- a/include/common/chunk.h
+++ b/include/common/chunk.h
@@ -120,6 +120,19 @@ static inline int chunk_strcat(struct chunk *chk, const char *str)
 	return 1;
 }
 
+/* appends  characters from str after .
+ * Returns 0 in case of failure.
+ */
+static inline int chunk_strncat(struct chunk *chk, const char *str, int nb)
+{
+	if (unlikely(chk->len < 0 || chk->len + nb >= chk->size))
+		return 0;
+
+	memcpy(chk->str + chk->len, str, nb);
+	chk->len += nb;
+	return 1;
+}
+
 /* Adds a trailing zero to the current chunk and returns the pointer to the
  * following part. The purpose is to be able to use a chunk as a series of
  * short independant strings with chunk_* functions, which do not need to be
-- 
1.9.1

From 1a8a094fcdc40bc10200ed4d036a5e553904b4f3 Mon Sep 17 00:00:00 2001
From: Baptiste Assmann 
Date: Sat, 26 Mar 2016 15:09:48 +0100
Subject: [PATCH 03/11] MINOR: dns: wrong DNS_MAX_UDP_MESSAGE value

Current implementation of HAProxy's DNS resolution expect only 512 bytes
of data in the response.
Update DNS_MAX_UDP_MESSAGE to match this.

Backport: can be backported to 1.6
---
 include/types/dns.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/types/dns.h b/include/types/dns.h
index 1b240fa..19c6edc 100644
--- a/include/types/dns.h
+++ b/include/types/dns.h
@@ -30,7 +30,7 @@
  */
 #define DNS_MAX_LABEL_SIZE	63
 #define DNS_MAX_NAME_SIZE	255
-#define DNS_MAX_UDP_MESSAGE	4096
+#define DNS_MAX_UDP_MESSAGE	512
 
 /* DNS error messages */
 #define DNS_TOO_LONG_FQDN	"hostname too long"
-- 
1.9.1

From adac8d19d888cf63315290f71ccfe0dcfd05b80d Mon Sep 17 00:00:00 2001
From: Baptiste Assmann 
Date: Wed, 9 Dec 2015 14:02:01 +0100
Subject: [PATCH 04/11] MINOR: dns: new MAX values

DNS_MIN_RECORD_SIZE: minimal size of a DNS record

DNS_MAX_QUERY_RECORDS: maximum number of query records we allow.
  For now, we send one DNS query per request.

DNS_MAX_ANSWER_RECORDS: maximum number of records we may found in a
  response

WIP dns: new MAX values
---
 include/types/dns.h | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/include/types/dns.h b/include/types/dns.h
index 19c6edc..01d42f9 100644
--- a/include/types/dns.h
+++ b/include/types/dns.h
@@ -32,6 +32,16 @@
 #define DNS_MAX_NAME_SIZE	255
 #define DNS_MAX_UDP_MESSAGE	512
 
+/* DNS minimun record size: 1 char + 1 NULL + type + class */
+#define DNS_MIN_RECORD_SIZE	( 1 + 1 + 2 + 2 )
+
+/* maximum number of query records in a DNS response
+ * For now, we allow only one */
+#define DNS_MAX_QUERY_RECORDS 1
+
+/* maximum number of answer record in a DNS response */
+#define DNS_MAX_ANSWER_RECORDS ((DNS_MAX_UDP_MESSAGE - DNS_HEADER_SIZE) / DNS_MIN_RECORD_SIZE)
+
 /* 

Haproxy custom log - apache replacement

[Benoit GEORGELIN - Association Web4all]

Hello members of the list, 

I was wondering if Haproxy custom log could log the same things as apache log 
format does ? 

Haproxy is on the top for HTTP servers farms and I would like to stop logging 
on the HTTP servers side (apache or ngnix) and consume the log made by Haproxy 
with everything I get from a typical Apache log 

I think, at that time, Haproxy does not log everything like apache will do, but 
maybe it's something than can be changed ? 

That would be nice and will save a lot of time/ressources 
In my case, this is what i'm doing : 

- Haproxy 
Loadblance to 5 HTTP servers 
Apache2 custom log with X-Forwarder-for 

- Apache2 
Each HTTP serveur runs apache2 and log http requests locally to 
access_domain_log 

Every night, merging of the log and then process the full log with Http log 
analyser like Awstats 
Store the logs to a dedicated storage for archiving 


If I could get the same HTTP information directly from Haproxy log , this will 
same us a lot of time an ressource usage. 

Haproxy logs -> log shipper -> elasticsearch /hadoop -> kibana or something 
else to display log information 

Thanks for your help 

Cordialement, 

Benoît G 

Re: Bytes in / out counters for TCP Keepalive Sessions

[Daniel Schneller]

Adding the list back. Sorry for dropping it earlier. 


> On 8 Sep 2016, at 19:56, PiBa-NL  wrote:
> 
> Hi,
> Op 8-9-2016 om 15:43 schreef Daniel Schneller:
>>> http://cbonte.github.io/haproxy-dconv/1.7/snapshot/configuration.html#4.2-option%20contstats
>> Indeed, that sounds like it. So, 1.6 would not have helped me here ;)
>> But good to know that this is the expected behavior.
> Just for clarity.. despite my link pointing to a 1.7 manual page, 1.4 already 
> have that same contstat option available for you to use. 
> http://cbonte.github.io/haproxy-dconv/1.4/configuration.html#option contstats
> Regards,
> PiBa-NL

Damn. Thanks for pointing that out again, I did not even think to search for it 
in older doc releases! Very cool. :)

Daniel

Re: Haproxy 1.6.9 failed to compile regex

[Lukas Tribus]

Hello Veiko,


Am 08.09.2016 um 13:11 schrieb Veiko Kukk:
Yes, turned out, build box had newer pcre installed (7 vs 8 major 
version). Compiling pcre statically solved that error/problem.


Good to hear. If you link against a shared library, you will have to 
make sure they match in major versions (same OS), otherwise you will 
have major problems with it (OpenSSL major release differences also lead 
to haproxy crashes). If you link statically to a library, you have to 
remember to upgrade the library and recompile haproxy, if the library 
has a vulnerability.




Means that haproxy -vv reported "Built with PCRE version" version 
wrong previously. That confused me.


This returns the output of pcre_version() and yes, the text should be 
renamed to "Running with PCRE version", because the runtime library 
returns the version in this case.





BTW: openssl version is also reported wrong by haproxy -vv.


OpenSSL informations on the other hand should be correct. "Built with" 
returns the build time OPENSSL_VERSION_TEXT and "Running on" makes the 
library return the version ("SSLeay_version(SSLEAY_VERSION)"). There is 
really no interpretation going on at haproxy level.


Not sure why you would see different "Running on" informations on the 
same box, pointing to the same libssl and libcrypto library in the ldd 
output. That doesn't make sense to me.




Lukas

Re: Incorrect counters in stats interface

[Dmitry Sivachenko]

> On 07 Sep 2016, at 23:12, David Birdsong  wrote:
> 
> Queue Cur is a gauge and so not representative of historical values.
> 
> Queue Max of zero is telling though.
> 
> In addition to queue timeout, there are other ways haproxy can synthesize an 
> http response on behalf of the backend server. Check for connection errors.
> 


Also, awk '{if ($11 >= 500) print};' gives me no matches from haproxy.log, but 
counter for backend's Total Sessions 5xx respones does increase.