Re: Re: CPU 100% when waiting for the client timeout

[baiyang]

Hi Willy,

I've been upgraded our server to Ubuntu 14.04 LTS:

root@WD-G0-SRP1:~# uname -a
Linux WD-G0-SRP1 3.13.0-70-generic #113-Ubuntu SMP Mon Nov 16 18:34:13 UTC 2015 
x86_64 x86_64 x86_64 GNU/Linux

And this problem appeared again. HAProxy -vv:

HA-Proxy version 1.6.2 2015/11/03
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat 
-Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3.4
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1 14 Mar 2012
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

I caught both strace log and show sess all output this time: 
http://baiy.cn/tmp/log-1124.rar 

Thanks :-)
 
--

   Best Regards
  BaiYang
  baiy...@gmail.com
  http://baiy.cn
 < END OF EMAIL >  
 
 
From: Willy Tarreau
Date: 2015-11-22 01:55
To: baiyang
CC: Lukas Tribus; haproxy
Subject: Re: Re: CPU 100% when waiting for the client timeout
Hi Baiyang,
 
On Sun, Nov 22, 2015 at 01:22:42AM +0800, baiyang wrote:
> Hi Willy,
> 
> It reproduced again:
 
Ah bad :-/
 
(...)
> I've captured the strace call log successfully this time. You are right,
> there are full of epoll_wait and gettimeofday calls.
 
Excellent, thanks for doing this.
 
> Here is the broken request:
> Nov 22 01:06:32 WD-G0-SRP1 haproxy[1259]: 2.145.41.3:56014 
> [22/Nov/2015:00:50:01.683] https-in~ g0n2/n2 1263/0/0/3/991286 200 84517 - - 
> CD-- 8/8/1/1/0 0/0 "GET /lib/ext/ext-all.js HTTP/1.1"
> 
> I have been uploaded the total 40 seconds strace log and full haproxy.log to:
> http://baiy.cn/tmp/log-1122.rar 
 
OK I just downloaded it, thank you. That's interesting, it's not a
closed fd that is reported, it's an old timer that is not processed
(for a reason I have no idea about) or a task which remains active
but not processed, which causes epoll_wait's timeout to remain zero,
hence the busy loop. This clearly is an haproxy bug.
 
Did you by chance have the opportunity to retrieve a "show sess all"
on the CLI ? I can easily understand that you didn't think about it
facing the situation.
 
At least that gives us a few insights about what to look for :-/
 
Thanks!
willy

Re: Could we reduce spam in the list?

[Thrawn]

@PiBa-NL If you're interested, I've developed a NoScript surrogate script for 
cleaning up the online mail archives.


If you don't use NoScript, you could probably port it to Greasemonkey or 
similar.

https://forums.informaction.com/viewtopic.php?f=26&t=20652

Re: simply copy mapped value into acl

[Willy Tarreau]

On Tue, Nov 24, 2015 at 01:44:07AM +0100, Dennis Jacobfeuerborn wrote:
> On 23.11.2015 22:38, Willy Tarreau wrote:
> > Hi Andrew,
> > 
> > On Mon, Nov 23, 2015 at 12:19:22PM -0600, Andrew Hayworth wrote:
> >> That said, the difference between one and two map lookups is
> >> negligible, so I don't think you're saving much this way and it'll
> >> make configuration harder in some ways if you keep adding IPs. I can
> >> vouch from personal experience in production that we run 2+ map
> >> lookups on every request at Braintree (hundreds of req/s) and it adds
> >> no noticeable latency. Food for thought. :)
> > 
> > I agree, depending on the map size, you'll see something between 1 and 10
> > million lookups per second, it's quite cheap. I do abuse them as well and
> > I don't bother about the number of lookups. Sure if I have to match a
> > country among a few tens, I'll certainly use a variable first, but otherwise
> > I don't care.
> 
> I'm going to use multiple lookups until I can upgrade to 1.6 because I
> agree that it's not going to cause any major issues in our current setup.
> It's just that when I'm looking the same value up multiple times in a
> map of about 800.000 values my coding sensibilities immediately go for
> storing the result in a variable instead.

Just FWIW in 1.5 there's a tricky alternative to variables. You can perform
a sample capture in TCP rules, and reuse this capture later :

tcp-request content capture src,map(ip-to-country) len 2
...
http-request add-header x-country %[capture.req.hdr(0)]
...
use_backend country_XXX if { capture.req.hdr(0) XXX }

It will effectively work as a variable, though this variable will be dumped
into the logs. Since it's the source address you're interested in, you can
even do that in "tcp-request connection" rules, and it will be performed
only once per connection, which is the most optimal situation. But as you
can see it's a trick and it is limited because you can't do much using HTTP
inputs for example.

Regards,
Willy

Bike Bicycle

[Sandra]

Dear Sir/Madam,

Happy to introduce my company and myself to you.

This is Symi from China, our company Dongguan Worthy Hardware Co,is an 
experienced manufacturer which produces and sells bike all over the world, we 
have strong production and sales team, our products are produced with 
customer’s requirements and our price is very fair to our customer.

Our main products are mountain bikes, fat bikes, fold bike and road bike, we 
also have bicycle parts, such as rims, tires, hubs and so on.

If you are interested in my products please contact me freely.

Look forward to hearing from you.

Best Regards,

Symi
Dongguan Worthy Hardware Co.,Ltd 
No 11 of Lane 1 ,Road 1 ,Jiumenzhai Village ,Humen Town, Dongguan City 
,Guangdong Province ,523898,China.
Tel:+86-769-89919645 0769-8991 8145
Fax:+86-769-81838010
Skype:whys03
Http://www.wyhardare.en.alibaba.com

Re: simply copy mapped value into acl

[Dennis Jacobfeuerborn]

On 23.11.2015 22:38, Willy Tarreau wrote:
> Hi Andrew,
> 
> On Mon, Nov 23, 2015 at 12:19:22PM -0600, Andrew Hayworth wrote:
>> That said, the difference between one and two map lookups is
>> negligible, so I don't think you're saving much this way and it'll
>> make configuration harder in some ways if you keep adding IPs. I can
>> vouch from personal experience in production that we run 2+ map
>> lookups on every request at Braintree (hundreds of req/s) and it adds
>> no noticeable latency. Food for thought. :)
> 
> I agree, depending on the map size, you'll see something between 1 and 10
> million lookups per second, it's quite cheap. I do abuse them as well and
> I don't bother about the number of lookups. Sure if I have to match a
> country among a few tens, I'll certainly use a variable first, but otherwise
> I don't care.

I'm going to use multiple lookups until I can upgrade to 1.6 because I
agree that it's not going to cause any major issues in our current setup.
It's just that when I'm looking the same value up multiple times in a
map of about 800.000 values my coding sensibilities immediately go for
storing the result in a variable instead.

Regards,
  Dennis

Re: simply copy mapped value into acl

[Dennis Jacobfeuerborn]

On 23.11.2015 19:19, Andrew Hayworth wrote:
> On Mon, Nov 23, 2015 at 10:52 AM, Dennis Jacobfeuerborn
>  wrote:
>> Hm, I wasn't aware of the -M flag, thanks!
>>
>> However in you example you again you have to do multiple lookups even
>> though that shouldn't be necessary. I can make decisions based on the
>> fact that the IP is present in the map but what I really want to do is
>> make a decision based on what the actual value for that IP in the map is
>> i.e. if the value is "de" then I want to do one thing and if it is "at"
>> then I want to do something else.
>>
>> Regards,
>>   Dennis
>>
> 
> I see. I believe you could accomplish what you want with something like
> 
> http-request set-var(req.ip_lookup) src,map_ip()
> http-request set-header X-Test wasxx if { var(req.ip_lookup) -m str xx }
> http-request set-header X-Test wasyy if { var(req.ip_lookup) -m str yy }
> 
> http-request set-var is new in HAProxy 1.6. Also note the implicit ACL
> definition with '{}' - that's syntactic sugar so you don't have to
> declare them before (which would be very verbose).
> 
> That said, the difference between one and two map lookups is
> negligible, so I don't think you're saving much this way and it'll
> make configuration harder in some ways if you keep adding IPs. I can
> vouch from personal experience in production that we run 2+ map
> lookups on every request at Braintree (hundreds of req/s) and it adds
> no noticeable latency. Food for thought. :)
> 
> http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#http-request
> (especially note the section about variable scope).

set-var is precisely the thing I was looking for, thanks!

Since I need to update to 1.6 for this but am really busy over the next
two week at least I'll probably use your first suggestion for now and
then look into the set-var version in a few weeks.
In general the ability to set a variable from a map and then later use
that for further processing is a nice feature to have,

Thanks for pointing me in the right direction!

Regards,
  Dennis

RE: Owncloud through Haproxy makes upload not possible

[Lukas Tribus]

Hi,



>> Still seeing poll() in this trace. Are you sure nokqueue was removed
>> in the configuration and haproxy was restarted?
> Yes, I definitely did that.
> [...]
> Total: 3 (3 usable), will use kqueue.

I don't get it. The trace doesn't match the configuration.

When you start haproxy with the debug enabled (-d), what do you see
in the sixth line (Using *blabla()* as the polling mechanism)?

So this affects only the SSL enabled frontend, the frontend "http-in" works
just fine, is that correct?


Lukas




  

Re: simply copy mapped value into acl

[Willy Tarreau]

Hi Andrew,

On Mon, Nov 23, 2015 at 12:19:22PM -0600, Andrew Hayworth wrote:
> That said, the difference between one and two map lookups is
> negligible, so I don't think you're saving much this way and it'll
> make configuration harder in some ways if you keep adding IPs. I can
> vouch from personal experience in production that we run 2+ map
> lookups on every request at Braintree (hundreds of req/s) and it adds
> no noticeable latency. Food for thought. :)

I agree, depending on the map size, you'll see something between 1 and 10
million lookups per second, it's quite cheap. I do abuse them as well and
I don't bother about the number of lookups. Sure if I have to match a
country among a few tens, I'll certainly use a variable first, but otherwise
I don't care.

Willy

RE: ssl parameters ignored

[Lukas Tribus]

Hi,


> When testing this config I get:
>
> [ALERT] 326/202736 (24201) : SSLv3 support requested but unavailable.
> Configuration file is valid
>
> After testing with ssllabs I also noticed tlsv10 and tlsv11 were still
> enabled. Downgrading to haproxy 1.5.14 removes the error when testing
> the config and shows the tls protocols as disabled when using ssllabs.

I don't see whats wrong here.

First of all, even if your ssllib was build without SSL3 support, the behavior
would be different:

- you should only see this with the force-sslv3 keyword, not with no-sslv3
- if you see it, it should be a fatal error, aborting, instead your output
  suggests its just a warning (Configuration file is valid)

  
Are you sure that the executable was cleanly build (first "make clean",
only then "make ...")?

Can you elaborate what kind of OS we are talking about, and where the
openssl lib comes from (is it just a openssl-dev package from the
repository, or a custom build? static or shared?)



Thanks,

Lukas

  

Re: Connect() failed using unix sockets

[Willy Tarreau]

On Fri, Nov 20, 2015 at 08:37:58PM +0100, Lukas Tribus wrote:
> >>
> >> So anybody know what resource "free ports" relates to in the unix
> >> domain socket case? Are there any other debug options to find out
> >> more about what is happening.
> >
> > I suspect the connect() call returns EAGAIN
> 
> Digging some more, it looks like the kernel returns EAGAIN when the
> backlog on the other side is full (the backend doesn't accept() fast
> enough).

Yes this reminds me something as well. Greg, did you change the
backlog size on your server ? It's frequent to see values as low
as 128 or even 5 in some programs, maybe you're in a similar
situation ? It could be interesting to see if increasing the
net.core.somaxconn sysctl helps here. I have an old memory of it
being used by Unix sockets but I don't remember exactly how.

Cheers,
Willy

Re: Could we reduce spam in the list?

[Willy Tarreau]

On Mon, Nov 23, 2015 at 03:20:06PM +0100, Vincent Gallissot wrote:
> This is obviously a spam and Barracuda let it go.
> Don't you use other headers to help yourself detect spams ?
> Cause this is really annoying on this mailing list.

What is even more annoying is to find valid e-mails being tagged
as spam. And since it already happened a few times, I even thought
about removing the anti-spam. If those boring "stop the spam"
subjects come up again, this will make me think even more about it.

It's amazing how much the people who complain the most about spams
are the ones contributing the least on the list. People who seek
help do not complain and those who help don't either. People who
subscribe here do it on purpose, it's not mandatory to participate,
so I don't see why people complain.

Willy

Re: ssl parameters ignored

[Sander Klein]

Hey Lukas,

On 2015-11-23 21:27, Lukas Tribus wrote:
1.5.15 is probably affected as well (the error above comes from a build 
fix

for libssl that has been backported to 1.5).


Heh, didn't notice that release, else I would have tested with that 
one...


Can you provide "haproxy -vv" output of both 1.5.14 and 1.6.2 
executables?


Yes!

[ALERT] 326/214402 (27635) : SSLv3 support requested but unavailable.
HA-Proxy version 1.6.2 2015/11/03
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fstack-protector-strong -Wformat 
-Werror=format-security -D_FORTIFY_SOURCE=2

  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 
200


Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Built with OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015
Running on OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND


Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

---

HA-Proxy version 1.5.14 2015/07/02
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fstack-protector-strong -Wformat 
-Werror=format-security -D_FORTIFY_SOURCE=2

  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 
200


Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015
Running on OpenSSL version : OpenSSL 1.0.1k 8 Jan 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND


Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Greets,

Sander

Re: ssl parameters ignored

[Lukas Tribus]

Hi Sander,


> When testing this config I get:
>
> [ALERT] 326/202736 (24201) : SSLv3 support requested but unavailable.
> Configuration file is valid
>
> After testing with ssllabs I also noticed tlsv10 and tlsv11 were still
> enabled. Downgrading to haproxy 1.5.14 removes the error when testing
> the config and shows the tls protocols as disabled when using ssllabs.

1.5.15 is probably affected as well (the error above comes from a build fix
for libssl that has been backported to 1.5).

Not quite sure why we would see this behavior though.


Can you provide "haproxy -vv" output of both 1.5.14 and 1.6.2 executables?



Thanks,

Lukas

  

ssl parameters ignored

[Sander Klein]

Hi All,

I'm running haproxy 1.6.2 and it seems it ignores the values given with 
ssl-default-bind-options and/or ssl-default-server-options.


I have the following in my global conf:

ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
ssl-default-bind-ciphers 
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11
ssl-default-server-ciphers 
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS



When testing this config I get:

[ALERT] 326/202736 (24201) : SSLv3 support requested but unavailable.
Configuration file is valid

After testing with ssllabs I also noticed tlsv10 and tlsv11 were still 
enabled. Downgrading to haproxy 1.5.14 removes the error when testing 
the config and shows the tls protocols as disabled when using ssllabs.


Did something change betweern 1.5 and 1.6 so my config doesn't work 
anymore?


Greets,

Sander

Re: Could we reduce spam in the list?

[PiBa-NL]

You might get no or little spam in your mailbox, but have you ever tried 
finding something in list archives? Half of the topics aren't related to 
haproxy.. Which does make the archives less usable.. And yes gmail does 
a nice job at filtering, but also passes the occasional spam mail, not 
that much to really worry about but just saying spamfilters cant catch 
everything.. I dont think i can match the amount of time gmail spend to 
configure theirs spam filters to do a better job in-house..


Take a look at the archives, they are messy.., should the archives 
filter mails they think are spam?:

http://marc.info/?l=haproxy&r=1&b=201511&w=2
http://blog.gmane.org/gmane.comp.web.haproxy?set_blog_all=yes

Yes i have seen & read several of the the previous spam discussions. And 
contributions by non-list-members are of-course welcome.  Not sure if 
they should perhaps get some 'confirm your contribution' link by email 
they should click before the mail gets processed on the list? Though 
that would require quite a bit of effort to handle a problem most people 
on the list dont have a big problem with. And who is going to maintain 
the infrastructure to make such a thing possible.?.


I agree a note about the spam policy would be nice to have near where 
the subscription address is written. It might reduce the 'stop spam' 
discussions..


Regards,
PiBa-NL

Op 23-11-2015 om 16:06 schreef Kobus Bensch:
Again, I get no SPAM on this list. As a user of HAProxy, this list is 
about HAProxy. The few people that do complain about SPAM, need to 
sort it in-house. As mentioned, this subject has been covered in depth 
and please no more. Read previous posts.


On 23/11/2015 14:20, Vincent Gallissot wrote:

Barracuda headers are not really usefull :

Spam received on this mailing list on Fri, 20 Nov 2015 15:41:02 +0100
Subject "Comparer les offres d'assurances pour votre animal de compagnie"

Here are Barracuda headers :
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of
TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=1000.0
tests=DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE

This is obviously a spam and Barracuda let it go.
Don't you use other headers to help yourself detect spams ?
Cause this is really annoying on this mailing list.

Regards,
Vincent

On 11/23/2015 11:00 AM, William Lewis wrote:

Perhaps something about the spam philosophy and reasoning on this list
could be included on the website and/or initial subscription email. It
is a subject that comes up again and again.

Sent from my iPhone


On 23 Nov 2015, at 09:53, Jarno Huuskonen  wrote:

Hi,


On Mon, Nov 23, Alexandros Afentoulis wrote:
Hello there,

dear list administrators could you possibly do something about spam in
the list? Perhaps allow only subscribers to send or something similar?

I'm having hard time coping with the amount of spam between the
legitimate messages of the list. I think reducing spam would profit us all.

This has been covered before. Just search list archives.

What you can do is filter messages based on
X-Barracuda-Spam-Score:
X-Barracuda-Spam-Status:
headers. This should keep most of the spam out of your inbox.

-Jarno

--
Jarno Huuskonen



--
Kobus Bensch Trustpay Global LTD email signature Kobus Bensch
Senior Systems Administrator
Address:  22 & 24 | Frederick Sanger Road | Guildford | Surrey | GU2 7YD
DDI:  0207 871 3958
Tel:  0207 871 3890
Email: kobus.ben...@trustpayglobal.com 



Trustpay Global Limited is an authorised Electronic Money Institution 
regulated by the Financial Conduct Authority registration number 
900043. Company No 07427913 Registered in England and Wales with 
registered address 130 Wood Street, London, EC2V 6DL, United Kingdom.


For further details please visit our website at www.trustpayglobal.com 
.


The information in this email and any attachments are confidential and 
remain the property of Trustpay Global Ltd unless agreed by contract. 
It is intended solely for the person to whom or the entity to which it 
is addressed. If you are not the intended recipient you may not use, 
disclose, copy, distribute, print or rely on the content of this email 
or its attachments. If this email has been received by you in error 
please advise the sender and delete the email from your system. 
Trustpay Global Ltd does not accept any liability for any personal 
view expressed in this message.

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

On 11/23/15 17:29, Lukas Tribus wrote:
>>> Ok, could you redo this trace with the "-d" option and
>>> without the nokqueue configuration.
>> Attached.
> 
> Still seeing poll() in this trace. Are you sure nokqueue was removed
> in the configuration and haproxy was restarted?
> 
> Please also provide the output of "haproxy -vv".
> 
> 
> Thanks,
> 
> Lukas   
> 
Yes, I definitely did that.
haproxy -vv:
HA-Proxy version 1.6.2 2015/11/03
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1
USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1p-freebsd 9 Jul 2015
Running on OpenSSL version : OpenSSL 1.0.1p-freebsd 9 Jul 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.37 2015-04-28
PCRE library supports JIT : yes
Built without Lua support
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY

Available polling systems :
 kqueue : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.




signature.asc
Description: OpenPGP digital signature

Re: simply copy mapped value into acl

[Andrew Hayworth]

On Mon, Nov 23, 2015 at 10:52 AM, Dennis Jacobfeuerborn
 wrote:
> Hm, I wasn't aware of the -M flag, thanks!
>
> However in you example you again you have to do multiple lookups even
> though that shouldn't be necessary. I can make decisions based on the
> fact that the IP is present in the map but what I really want to do is
> make a decision based on what the actual value for that IP in the map is
> i.e. if the value is "de" then I want to do one thing and if it is "at"
> then I want to do something else.
>
> Regards,
>   Dennis
>

I see. I believe you could accomplish what you want with something like

http-request set-var(req.ip_lookup) src,map_ip()
http-request set-header X-Test wasxx if { var(req.ip_lookup) -m str xx }
http-request set-header X-Test wasyy if { var(req.ip_lookup) -m str yy }

http-request set-var is new in HAProxy 1.6. Also note the implicit ACL
definition with '{}' - that's syntactic sugar so you don't have to
declare them before (which would be very verbose).

That said, the difference between one and two map lookups is
negligible, so I don't think you're saving much this way and it'll
make configuration harder in some ways if you keep adding IPs. I can
vouch from personal experience in production that we run 2+ map
lookups on every request at Braintree (hundreds of req/s) and it adds
no noticeable latency. Food for thought. :)

http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#http-request
(especially note the section about variable scope).

-- 
- Andrew Hayworth

Re: simply copy mapped value into acl

[Dennis Jacobfeuerborn]

On 23.11.2015 17:04, Andrew Hayworth wrote:
> On Mon, Nov 23, 2015 at 6:26 AM, Dennis Jacobfeuerborn
>  wrote:
>>
>> On 16.11.2015 15:25, Dennis Jacobfeuerborn wrote:
>>> Hi,
>>> I'm trying to figure out the best way to match a source ip against an ip
>>> mapping file and make decisions based on that. What I'm now doing is this:
>>>
>>> acl acl_is_xx src,map_ip() -m str xx
>>> acl acl_is_yy src,map_ip() -m str yy
>>>
>>> acl acl_value src,map_ip() -m copy
>>> http-request set-header X-Test wasxx if acl_value==xx
>>> http-request set-header X-Test wasyy if acl_value==yy
>>>
> 
>>> Is there an alternative way to express something like this?
>>
>> Does nobody have any idea how to accomplish this?
>> This is happening in a GeoIP context and I'm now planning to make
>> distinct lookups for four different countries which seems pretty wasteful.
> 
> Hi Dennis -
> 
> I think this configuration would do what you want:
> 
> acl has_ip_map src -M -f 
> http-request set-header X-Test %[src,map_ip()] if has_ip_map
> 
> The idea is that map-file is a two-column file like so:
> 
> # ip_addr name
> 1.2.3.4 wasxx
> 5.6.7.8 wasyy
> 
> This works by treating  as both a map AND and acl (the '-M'
> flag does that). When you treat it as an ACL, it only evaluates the
> first column. This lets you test the ip address, to see if it's one
> you care about.
> 
> Then, you use the  as a map, and you look up the value for
> your header based on the ip address - but only if the prior acl
> evaluated to true.
> 
> An added benefit of this is that you can scale out easily to many
> values in the map/acl file without polluting your configuration.
> Additionally, you can use the socket commands to dynamically add
> things to the map/acl without reloading haproxy, if you wanted
> (something like 'add map   ').

Hm, I wasn't aware of the -M flag, thanks!

However in you example you again you have to do multiple lookups even
though that shouldn't be necessary. I can make decisions based on the
fact that the IP is present in the map but what I really want to do is
make a decision based on what the actual value for that IP in the map is
i.e. if the value is "de" then I want to do one thing and if it is "at"
then I want to do something else.

Regards,
  Dennis

RE: Owncloud through Haproxy makes upload not possible

[Lukas Tribus]

>> Ok, could you redo this trace with the "-d" option and
>> without the nokqueue configuration.
> Attached.

Still seeing poll() in this trace. Are you sure nokqueue was removed
in the configuration and haproxy was restarted?

Please also provide the output of "haproxy -vv".


Thanks,

Lukas 

Re: simply copy mapped value into acl

[Andrew Hayworth]

On Mon, Nov 23, 2015 at 6:26 AM, Dennis Jacobfeuerborn
 wrote:
>
> On 16.11.2015 15:25, Dennis Jacobfeuerborn wrote:
> > Hi,
> > I'm trying to figure out the best way to match a source ip against an ip
> > mapping file and make decisions based on that. What I'm now doing is this:
> >
> > acl acl_is_xx src,map_ip() -m str xx
> > acl acl_is_yy src,map_ip() -m str yy
> >
> > acl acl_value src,map_ip() -m copy
> > http-request set-header X-Test wasxx if acl_value==xx
> > http-request set-header X-Test wasyy if acl_value==yy
> >

> > Is there an alternative way to express something like this?
>
> Does nobody have any idea how to accomplish this?
> This is happening in a GeoIP context and I'm now planning to make
> distinct lookups for four different countries which seems pretty wasteful.

Hi Dennis -

I think this configuration would do what you want:

acl has_ip_map src -M -f 
http-request set-header X-Test %[src,map_ip()] if has_ip_map

The idea is that map-file is a two-column file like so:

# ip_addr name
1.2.3.4 wasxx
5.6.7.8 wasyy

This works by treating  as both a map AND and acl (the '-M'
flag does that). When you treat it as an ACL, it only evaluates the
first column. This lets you test the ip address, to see if it's one
you care about.

Then, you use the  as a map, and you look up the value for
your header based on the ip address - but only if the prior acl
evaluated to true.

An added benefit of this is that you can scale out easily to many
values in the map/acl file without polluting your configuration.
Additionally, you can use the socket commands to dynamically add
things to the map/acl without reloading haproxy, if you wanted
(something like 'add map   ').

Hope that helps!

- Andrew Hayworth

Re: Could we reduce spam in the list?

[Kobus Bensch]

Again, I get no SPAM on this list. As a user of HAProxy, this list is 
about HAProxy. The few people that do complain about SPAM, need to sort 
it in-house. As mentioned, this subject has been covered in depth and 
please no more. Read previous posts.


On 23/11/2015 14:20, Vincent Gallissot wrote:

Barracuda headers are not really usefull :

Spam received on this mailing list on Fri, 20 Nov 2015 15:41:02 +0100
Subject "Comparer les offres d'assurances pour votre animal de compagnie"

Here are Barracuda headers :
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of
TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=1000.0
tests=DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE

This is obviously a spam and Barracuda let it go.
Don't you use other headers to help yourself detect spams ?
Cause this is really annoying on this mailing list.

Regards,
Vincent

On 11/23/2015 11:00 AM, William Lewis wrote:

Perhaps something about the spam philosophy and reasoning on this list
could be included on the website and/or initial subscription email. It
is a subject that comes up again and again.

Sent from my iPhone


On 23 Nov 2015, at 09:53, Jarno Huuskonen  wrote:

Hi,


On Mon, Nov 23, Alexandros Afentoulis wrote:
Hello there,

dear list administrators could you possibly do something about spam in
the list? Perhaps allow only subscribers to send or something similar?

I'm having hard time coping with the amount of spam between the
legitimate messages of the list. I think reducing spam would profit us all.

This has been covered before. Just search list archives.

What you can do is filter messages based on
X-Barracuda-Spam-Score:
X-Barracuda-Spam-Status:
headers. This should keep most of the spam out of your inbox.

-Jarno

--
Jarno Huuskonen



--
Kobus Bensch Trustpay Global LTD email signature Kobus Bensch
Senior Systems Administrator
Address:  22 & 24 | Frederick Sanger Road | Guildford | Surrey | GU2 7YD
DDI:  0207 871 3958
Tel:  0207 871 3890
Email: kobus.ben...@trustpayglobal.com 



--


Trustpay Global Limited is an authorised Electronic Money Institution 
regulated by the Financial Conduct Authority registration number 900043. 
Company No 07427913 Registered in England and Wales with registered address 
130 Wood Street, London, EC2V 6DL, United Kingdom.


For further details please visit our website at www.trustpayglobal.com.

The information in this email and any attachments are confidential and 
remain the property of Trustpay Global Ltd unless agreed by contract. It is 
intended solely for the person to whom or the entity to which it is 
addressed. If you are not the intended recipient you may not use, disclose, 
copy, distribute, print or rely on the content of this email or its 
attachments. If this email has been received by you in error please advise 
the sender and delete the email from your system. Trustpay Global Ltd does 
not accept any liability for any personal view expressed in this message.

Re: Could we reduce spam in the list?

[Kobus Bensch]

Have you looked at the previous discussions?

On 23/11/2015 10:00, William Lewis wrote:

Perhaps something about the spam philosophy and reasoning on this list
could be included on the website and/or initial subscription email. It
is a subject that comes up again and again.

Sent from my iPhone


On 23 Nov 2015, at 09:53, Jarno Huuskonen  wrote:

Hi,


On Mon, Nov 23, Alexandros Afentoulis wrote:
Hello there,

dear list administrators could you possibly do something about spam in
the list? Perhaps allow only subscribers to send or something similar?

I'm having hard time coping with the amount of spam between the
legitimate messages of the list. I think reducing spam would profit us all.

This has been covered before. Just search list archives.

What you can do is filter messages based on
X-Barracuda-Spam-Score:
X-Barracuda-Spam-Status:
headers. This should keep most of the spam out of your inbox.

-Jarno

--
Jarno Huuskonen



--
Kobus Bensch Trustpay Global LTD email signature Kobus Bensch
Senior Systems Administrator
Address:  22 & 24 | Frederick Sanger Road | Guildford | Surrey | GU2 7YD
DDI:  0207 871 3958
Tel:  0207 871 3890
Email: kobus.ben...@trustpayglobal.com 



--


Trustpay Global Limited is an authorised Electronic Money Institution 
regulated by the Financial Conduct Authority registration number 900043. 
Company No 07427913 Registered in England and Wales with registered address 
130 Wood Street, London, EC2V 6DL, United Kingdom.


For further details please visit our website at www.trustpayglobal.com.

The information in this email and any attachments are confidential and 
remain the property of Trustpay Global Ltd unless agreed by contract. It is 
intended solely for the person to whom or the entity to which it is 
addressed. If you are not the intended recipient you may not use, disclose, 
copy, distribute, print or rely on the content of this email or its 
attachments. If this email has been received by you in error please advise 
the sender and delete the email from your system. Trustpay Global Ltd does 
not accept any liability for any personal view expressed in this message.

Re: Could we reduce spam in the list?

[Vincent Gallissot]

Barracuda headers are not really usefull :

Spam received on this mailing list on Fri, 20 Nov 2015 15:41:02 +0100
Subject "Comparer les offres d'assurances pour votre animal de compagnie"

Here are Barracuda headers :
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of
TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=1000.0
tests=DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE

This is obviously a spam and Barracuda let it go.
Don't you use other headers to help yourself detect spams ?
Cause this is really annoying on this mailing list.

Regards,
Vincent

On 11/23/2015 11:00 AM, William Lewis wrote:
> Perhaps something about the spam philosophy and reasoning on this list
> could be included on the website and/or initial subscription email. It
> is a subject that comes up again and again.
> 
> Sent from my iPhone
> 
>> On 23 Nov 2015, at 09:53, Jarno Huuskonen  wrote:
>>
>> Hi,
>>
>>> On Mon, Nov 23, Alexandros Afentoulis wrote:
>>> Hello there,
>>>
>>> dear list administrators could you possibly do something about spam in
>>> the list? Perhaps allow only subscribers to send or something similar?
>>>
>>> I'm having hard time coping with the amount of spam between the
>>> legitimate messages of the list. I think reducing spam would profit us all.
>>
>> This has been covered before. Just search list archives.
>>
>> What you can do is filter messages based on
>> X-Barracuda-Spam-Score:
>> X-Barracuda-Spam-Status:
>> headers. This should keep most of the spam out of your inbox.
>>
>> -Jarno
>>
>> --
>> Jarno Huuskonen
>>
> 

-- 
Vincent Gallissot
System & network administrator
M6 Web - Groupe M6 
49 quai Rambaud 69002 LYON - FRANCE
Tel : +33(0)4 26 83 70 87

Re: simply copy mapped value into acl

[Dennis Jacobfeuerborn]

On 16.11.2015 15:25, Dennis Jacobfeuerborn wrote:
> Hi,
> I'm trying to figure out the best way to match a source ip against an ip
> mapping file and make decisions based on that. What I'm now doing is this:
> 
> acl acl_is_xx src,map_ip() -m str xx
> acl acl_is_yy src,map_ip() -m str yy
> 
> http-request set-header X-Test wasxx if acl_is_xx ...
> http-request set-header X-Test wasyy if acl_is_yy ...
> 
> While this works my problem is that this requires two map look-ups. What
> i would really like to do is this (pseudo code):
> 
> acl acl_value src,map_ip() -m copy
> http-request set-header X-Test wasxx if acl_value==xx
> http-request set-header X-Test wasyy if acl_value==yy
> 
> That way you only would have to do one look-up in the map and then
> determine the the different cases based on simple string matches.
> 
> As far as I can tell though ACLs only allow for matching and not for a
> straight forward copy like I tried to express with the "-m copy" above.
> 
> Is there an alternative way to express something like this?

Does nobody have any idea how to accomplish this?
This is happening in a GeoIP context and I'm now planning to make
distinct lookups for four different countries which seems pretty wasteful.

Regards,
  Dennis

Re: Selecting back end from host header

[SL]

> use_backend %[req.hdr(host),lower]

Is there anyway to extract the host subdomain to use like this (rather than
the full hostname)?  I can't see any obvious way to do it from the manual,
but perhaps I'm missing something.

On 14 November 2015 at 22:14, SL  wrote:

> @Igor - thanks, I didn't know about that map feature, I think it could be
> useful in a number of situations.
>
> @Baptiste - Ah, it is possible dynamically, that's great - thank you!
>
> On 14 November 2015 at 22:05, Baptiste  wrote:
>
>> On Sat, Nov 14, 2015 at 3:21 PM, SL  wrote:
>> > Hi,
>> >
>> > We have quite a large number of backends, and are selecting which back
>> end
>> > to use based on the host specified in the request.  (Note these are not
>> > loadbalanced, we have to target them individually).
>> >
>> > Currently we are doing this with ACLs, e.g. for each:
>> >
>> > acl svr1_request hdr_beg(host) -i svr1
>> >
>> > then:
>> >
>> > use_backend svr1 if svr1_request
>> >
>> > (An example request host in this case would be svr1.example.com)
>> >
>> > Using ACLs like this means that we have a large number of repeated ACLs
>> and
>> > use_backends.  It's a bit cumbersome, difficult to maintain, and I
>> suspect
>> > not very efficient.
>> >
>> > Is there a better way to do this?  What would be ideal, is some way to
>> take
>> > the subdomain of the request host, and simply select a backend whose
>> name
>> > matched, but I don't know of any way to do that.  Is such a thing
>> possible?
>> >
>> > Thank you
>> >
>> > S
>> >
>>
>> Hi,
>>
>> As Igor stated, you can use content from this blog article:
>>
>> http://blog.haproxy.com/2015/01/26/web-application-name-to-backend-mapping-in-haproxy/
>>
>> very usefull if you want to map many host header to the same backend.
>> If you want a single host header per backend, then, simply use
>>
>>   use_backend %[req.hdr(host),lower
>>
>> Then each backend must use the host header as the backend name and
>> you're done, dynamic routing with a single use_backend rule, whatever
>> the number of backends you have.
>>
>> Baptiste
>>
>
>

Re: Could we reduce spam in the list?

[William Lewis]

Perhaps something about the spam philosophy and reasoning on this list
could be included on the website and/or initial subscription email. It
is a subject that comes up again and again.

Sent from my iPhone

> On 23 Nov 2015, at 09:53, Jarno Huuskonen  wrote:
>
> Hi,
>
>> On Mon, Nov 23, Alexandros Afentoulis wrote:
>> Hello there,
>>
>> dear list administrators could you possibly do something about spam in
>> the list? Perhaps allow only subscribers to send or something similar?
>>
>> I'm having hard time coping with the amount of spam between the
>> legitimate messages of the list. I think reducing spam would profit us all.
>
> This has been covered before. Just search list archives.
>
> What you can do is filter messages based on
> X-Barracuda-Spam-Score:
> X-Barracuda-Spam-Status:
> headers. This should keep most of the spam out of your inbox.
>
> -Jarno
>
> --
> Jarno Huuskonen
>

Re: Could we reduce spam in the list?

[Jarno Huuskonen]

Hi,

On Mon, Nov 23, Alexandros Afentoulis wrote:
> Hello there,
> 
> dear list administrators could you possibly do something about spam in
> the list? Perhaps allow only subscribers to send or something similar?
> 
> I'm having hard time coping with the amount of spam between the
> legitimate messages of the list. I think reducing spam would profit us all.

This has been covered before. Just search list archives.

What you can do is filter messages based on
X-Barracuda-Spam-Score:
X-Barracuda-Spam-Status:
headers. This should keep most of the spam out of your inbox.

-Jarno

-- 
Jarno Huuskonen

Re: Could we reduce spam in the list?

[Kobus Bensch]

I dont get any spam. Any I do get is dealt with swiftly by my own 
anti-spam measures.


This subject has been discussed at length. I dont think any action is 
required.


On 23/11/2015 09:41, Alexandros Afentoulis wrote:

Hello there,

dear list administrators could you possibly do something about spam in
the list? Perhaps allow only subscribers to send or something similar?

I'm having hard time coping with the amount of spam between the
legitimate messages of the list. I think reducing spam would profit us all.

Greetings,
Alex



--
Kobus Bensch Trustpay Global LTD email signature Kobus Bensch
Senior Systems Administrator
Address:  22 & 24 | Frederick Sanger Road | Guildford | Surrey | GU2 7YD
DDI:  0207 871 3958
Tel:  0207 871 3890
Email: kobus.ben...@trustpayglobal.com 



--


Trustpay Global Limited is an authorised Electronic Money Institution 
regulated by the Financial Conduct Authority registration number 900043. 
Company No 07427913 Registered in England and Wales with registered address 
130 Wood Street, London, EC2V 6DL, United Kingdom.


For further details please visit our website at www.trustpayglobal.com.

The information in this email and any attachments are confidential and 
remain the property of Trustpay Global Ltd unless agreed by contract. It is 
intended solely for the person to whom or the entity to which it is 
addressed. If you are not the intended recipient you may not use, disclose, 
copy, distribute, print or rely on the content of this email or its 
attachments. If this email has been received by you in error please advise 
the sender and delete the email from your system. Trustpay Global Ltd does 
not accept any liability for any personal view expressed in this message.

Could we reduce spam in the list?

[Alexandros Afentoulis]

Hello there,

dear list administrators could you possibly do something about spam in
the list? Perhaps allow only subscribers to send or something similar?

I'm having hard time coping with the amount of spam between the
legitimate messages of the list. I think reducing spam would profit us all.

Greetings,
Alex