Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

I'm not sure now, but it seems that packets routed in Haproxy are
somehow protected from dumping:
root@owncloud:~ # tcpdump -i em1 -vv port 80
tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size
10 bytes
^C
0 packets captured
3774 packets received by filter
0 packets dropped by kernel

I do it directly in the jail and have /dev/bpf enabled.


On 11/25/15 18:03, Lukas Tribus wrote:
>> I'm not sure why, but after doing haproxy -vv I now get kevent()
>> in truss output. I'm attaching another truss output.
> 
> Ok, so thats not the problem, good.
> 
> 
>> While browsing the logs I've notices that besides the usual 200 
>> output, when uploading was finished (unsuccessfully) I got the
>> following 504 code: Nov 24 11:03:13 anongoth haproxy[76436]:
>> 46.248.161.165:23648 [24/Nov/2015:11:02:23.521] https-in~
>> owncloud/node1 54/0/3/-1/50265 504 194 - - sH-- 0/0/0/0/0 0/0
>> "POST /index.php/apps/files/ajax/upload.php HTTP/1.1"
> 
> I think we will have to take a look at the actual backend traffic
> that generates this timeout (according to haproxy the server didn't
> respond in time).
> 
> Can you reproduce this and tcpdump the unencrypted backend traffic
> so that we can take a look (make sure the content can be
> disclosed)?
> 
> In that case please provide the logs again, so we can compare the
> haproxy log and the backend tcp traffic.
> 
> 
> 
> Regards,
> 
> Lukas
> 
> 
> 
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJWVvQ7AAoJEC9nKukRsfY+OrwQAIGMAlSZtCHhTLu3gElzA1sG
Q7CMXydW+XSh9Y2mF2F7zS1LKTT2vow8FsXYvVDl2IKEaxgUbh1abVbbOcoSpPsy
fOx/x7OGzBDgvz6JdfPwpcBioFtqXFxbvtYgSdjYSUjDCnJAxyB4EGkx+aiM3Ysu
TWLWmIbwguHExgsj9AsczBEzBwbX1NuvsTH3wffeYFYkDZZeCzlBL0ewLJdL58JQ
pTgYWTFy00oz7Vjv42BeTuYiba37sndNV1tS+xOwNjY78aMCK1hvgsGPyCO3nKjz
YJ+UEW7NPH6NRI5Z3guTi5yoJ/6Gx36ZM+pA/dZ3LC7SLWlStGMJt7MbPKZ9EZGV
cx6W2ZK+Sz/Arj2cvNpCgIrcUDWvoteJ8ym//Z7ag2KkA7iXV+idGThyHV6UNrFZ
IsRJrcVhNaN/iYB1m4IteuTAOMVb8AxtdhYBZlI78wgQSwBYdT6x5QiP6RsMjvzb
CYPbd2aeZLfvjLKNLCa3Qsp2zkMZlbPFBAuh0RzQEAwazOmMV+V59xW/ikDsX7Be
w5oIy0n0aZW352XBDGCNM79MrY6nJs9JhJOoySbl391mJcYU+xJdF0CNy+DVRLyH
RPKAnXyg/t92DdZ1DetHWGH/V+BVdTU4I9Ejv+ObKiuzbVLksv7tQz7Kvji10Xzo
JtMeRfEiVWfIb9YEos+2
=JAEC
-END PGP SIGNATURE-

RE: Owncloud through Haproxy makes upload not possible

[Lukas Tribus]

> I'm not sure now, but it seems that packets routed in Haproxy are
> somehow protected from dumping:
> root@owncloud:~ # tcpdump -i em1 -vv port 80
> tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size
> 10 bytes
> ^C
> 0 packets captured
> 3774 packets received by filter
> 0 packets dropped by kernel
>
> I do it directly in the jail and have /dev/bpf enabled.

Is em1 the correct interface? Maybe it traverses loopback (even
though the ip addresses are from em0 or em1)?


Lukas

  

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

No, I have checked both lo0 and lo1 (I have lo1 set up for jails).

On 11/26/15 15:21, Lukas Tribus wrote:
>> I'm not sure now, but it seems that packets routed in Haproxy are
>> somehow protected from dumping:
>> root@owncloud:~ # tcpdump -i em1 -vv port 80
>> tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size
>> 10 bytes
>> ^C
>> 0 packets captured
>> 3774 packets received by filter
>> 0 packets dropped by kernel
>>
>> I do it directly in the jail and have /dev/bpf enabled.
> 
> Is em1 the correct interface? Maybe it traverses loopback (even
> though the ip addresses are from em0 or em1)?
> 
> 
> Lukas
> 
> 
> 

RE: Owncloud through Haproxy makes upload not possible

[Lukas Tribus]

> No, I have checked both lo0 and lo1 (I have lo1 set up for jails).

Can you try outside of the jail? Somehow it must be possible
to tcpdump this traffic.

Lukas

  

RE: Owncloud through Haproxy makes upload not possible

[Lukas Tribus]

> I'm not sure why, but after doing haproxy -vv I now get kevent() in
> truss output. I'm attaching another truss output.

Ok, so thats not the problem, good.


> While browsing the logs I've notices that besides the usual 200
> output, when uploading was
> finished (unsuccessfully) I got the following 504 code:
> Nov 24 11:03:13 anongoth haproxy[76436]: 46.248.161.165:23648
> [24/Nov/2015:11:02:23.521] https-in~ owncloud/node1 54/0/3/-1/50265 504
> 194 - - sH-- 0/0/0/0/0 0/0 "POST /index.php/apps/files/ajax/upload.php
> HTTP/1.1"

I think we will have to take a look at the actual backend traffic that generates
this timeout (according to haproxy the server didn't respond in time).

Can you reproduce this and tcpdump the unencrypted backend traffic so
that we can take a look (make sure the content can be disclosed)?

In that case please provide the logs again, so we can compare the haproxy
log and the backend tcp traffic.



Regards,

Lukas

  

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

I'm not sure why, but after doing haproxy -vv I now get kevent() in
truss output. I'm attaching another truss output. While browsing the
logs I've notices that besides the usual 200 output, when uploading was
finished (unsuccessfully) I got the following 504 code:
Nov 24 11:03:13 anongoth haproxy[76436]: 46.248.161.165:23648
[24/Nov/2015:11:02:23.521] https-in~ owncloud/node1 54/0/3/-1/50265 504
194 - - sH-- 0/0/0/0/0 0/0 "POST /index.php/apps/files/ajax/upload.php
HTTP/1.1"

And yes, HTTP-only connections work just fine. The problem is only with
HTTPS.

On 11/23/15 17:29, Lukas Tribus wrote:
>>> Ok, could you redo this trace with the "-d" option and
>>> without the nokqueue configuration.
>> Attached.
> 
> Still seeing poll() in this trace. Are you sure nokqueue was removed
> in the configuration and haproxy was restarted?
> 
> Please also provide the output of "haproxy -vv".
> 
> 
> Thanks,
> 
> Lukas   
> 
0.000129552 gettimeofday({1448359288.348990 },0x0) = 0 (0x0)
0.000275237 gettimeofday({1448359288.349133 },0x0) = 0 (0x0)
1.063129893 kevent(0,0x0,0,{},11,{1.0 }) = 0 (0x0)
1.063276067 gettimeofday({1448359289.412129 },0x0) = 0 (0x0)
1.063388508 gettimeofday({1448359289.412254 },0x0) = 0 (0x0)
2.126672885 kevent(0,0x0,0,{},11,{1.0 }) = 0 (0x0)
2.126818082 gettimeofday({1448359290.475671 },0x0) = 0 (0x0)
2.126929545 gettimeofday({1448359290.475794 },0x0) = 0 (0x0)
3.190221256 kevent(0,0x0,0,{},11,{1.0 }) = 0 (0x0)
3.190371829 gettimeofday({1448359291.539241 },0x0) = 0 (0x0)
3.190486715 gettimeofday({1448359291.539341 },0x0) = 0 (0x0)
4.254129927 kevent(0,0x0,0,{},11,{1.0 }) = 0 (0x0)
4.254290767 gettimeofday({1448359292.603139 },0x0) = 0 (0x0)
4.254413964 gettimeofday({1448359292.603282 },0x0) = 0 (0x0)
4.867281122 kevent(0,0x0,0,{0x7,EVFILT_READ,0x0,0,0xab0,0x0},11,{1.0 }) = 1 (0x1)
4.867456139 gettimeofday({1448359293.216325 },0x0) = 0 (0x0)
4.867594002 read(7,"\^W\^C\^C@0",5)		 = 5 (0x5)
4.867793463 read(7,"\n\v\^N\M-l\M-t\M-@<\M-,\^R\M-]"...,16432) = 4099 (0x1003)
4.868166964 read(7,0x801e2900b,12333)		 ERR#35 'Resource temporarily unavailable'
4.868429490 gettimeofday({1448359293.217276 },0x0) = 0 (0x0)
4.896999380 kevent(0,0x0,0,{0x7,EVFILT_READ,0x0,0,0x558,0x0},11,{1.0 }) = 1 (0x1)
4.897145554 gettimeofday({1448359293.246014 },0x0) = 0 (0x0)
4.897272173 read(7,"];0p\M-k\M-g-\M-ly\M-=\M-y\M^X"...,12333) = 1368 (0x558)
4.897645185 read(7,"\M^[V\^B\M-Ro\M-"*\M-`\M-9\M^P"...,10965) = 4104 (0x1008)
4.897970776 read(7,0x801e2a56b,6861)		 ERR#35 'Resource temporarily unavailable'
4.898126727 gettimeofday({1448359293.246979 },0x0) = 0 (0x0)
4.919487169 kevent(0,0x0,0,{0x7,EVFILT_READ,0x0,0,0x558,0x0},11,{1.0 }) = 1 (0x1)
4.919659742 gettimeofday({1448359293.268509 },0x0) = 0 (0x0)
4.919802005 read(7,"\M-I\M^D\M-d\M-#\^Q\M-[\M^J\M^R"...,6861) = 1368 (0x558)
4.920178439 read(7,"\M-9\M-P6\M-2\M-x\M^J\M-+\M-S\^?"...,5493) = 2736 (0xab0)
4.920513319 read(7,"\M^[\M-Q\M^V\M^Z\M-@\M-M\M-ZV^"...,2757) = 1368 (0x558)
4.920835976 read(7,0x801e2bacb,1389)		 ERR#35 'Resource temporarily unavailable'
4.920986061 gettimeofday({1448359293.269841 },0x0) = 0 (0x0)
4.925489094 kevent(0,0x0,0,{0x7,EVFILT_READ,0x0,0,0x558,0x0},11,{1.0 }) = 1 (0x1)
4.925639179 gettimeofday({1448359293.274509 },0x0) = 0 (0x0)
4.925798064 read(7,"\M-l}\M-I_\M-c<\^D\M^P\M^_\M-p$"...,1389) = 1389 (0x56d)
4.926543599 socket(PF_INET,SOCK_STREAM,6)	 = 2 (0x2)
4.926686351 fcntl(2,F_SETFL,O_NONBLOCK)		 = 0 (0x0)
4.926824214 setsockopt(0x2,0x6,0x1,0x4cc2b4,0x4,0x4000) = 0 (0x0)
4.927016342 connect(2,{ AF_INET 192.168.11.14:80 },16) ERR#36 'Operation now in progress'
4.927144427 gettimeofday({1448359293.276014 },0x0) = 0 (0x0)
4.927261757 kevent(0,0x0,0,{0x7,EVFILT_READ,0x0,0,0xff3,0x0},11,{0.0 }) = 1 (0x1)
4.927395220 gettimeofday({1448359293.276263 },0x0) = 0 (0x0)
4.927608370 sendto(2,"POST /index.php/apps/files/ajax/"...,8220,0x20080,NULL,0x0) = 8220 (0x201c)
4.927756010 gettimeofday({1448359293.276611 },0x0) = 0 (0x0)
4.927853786 kevent(0,0x0,0,{0x7,EVFILT_READ,0x0,0,0xff3,0x0},11,{0.0 }) = 1 (0x1)
4.927947161 gettimeofday({1448359293.276819 },0x0) = 0 (0x0)
4.928033203 read(7,"\^W\^C\^C@0",5)		 = 5 (0x5)
4.928122178 read(7,"\M-^dtN\M-t\M-wQ\M^K\M-<\M-j\rY"...,16432) = 4078 (0xfee)
4.928382260 read(7,0x801e28ff6,12354)		 ERR#35 'Resource temporarily unavailable'
4.928731806 sendto(2,"?\M-u\M-=\M-d\M--\^[\M-d\M-q!\^Z"...,8192,0x20080,NULL,0x0) = 8192 (0x2000)
4.928816870 recvfrom(2,0x801e55014,8192,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable'
4.928915135 kevent(0,{0x2,EVFILT_READ,EV_ADD,0,0x0,0x0},1,0x0,0,0x0) = 0 (0x0)
4.928996777 gettimeofday({1448359293.277869 },0x0) = 0 (0x0)
4.943972503 kevent(0,0x0,0,{0x7,EVFILT_READ,0x0,0,0x558,0x0},11,{1.0 }) = 1 (0x1)
4.944135298 gettimeofday({1448359293.292987 },0x0) = 0 (0x0)
4.944272672 read(7,"\M-{\M-3\M-wx\M^Mxq\M^Z\M-%\M-U"...,12354) = 1368 (0x558)
4.944598752 

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

On 11/23/15 17:29, Lukas Tribus wrote:
>>> Ok, could you redo this trace with the "-d" option and
>>> without the nokqueue configuration.
>> Attached.
> 
> Still seeing poll() in this trace. Are you sure nokqueue was removed
> in the configuration and haproxy was restarted?
> 
> Please also provide the output of "haproxy -vv".
> 
> 
> Thanks,
> 
> Lukas   
> 
Yes, I definitely did that.
haproxy -vv:
HA-Proxy version 1.6.2 2015/11/03
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1
USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1p-freebsd 9 Jul 2015
Running on OpenSSL version : OpenSSL 1.0.1p-freebsd 9 Jul 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.37 2015-04-28
PCRE library supports JIT : yes
Built without Lua support
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY

Available polling systems :
 kqueue : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.




signature.asc
Description: OpenPGP digital signature

RE: Owncloud through Haproxy makes upload not possible

[Lukas Tribus]

Hi,



>> Still seeing poll() in this trace. Are you sure nokqueue was removed
>> in the configuration and haproxy was restarted?
> Yes, I definitely did that.
> [...]
> Total: 3 (3 usable), will use kqueue.

I don't get it. The trace doesn't match the configuration.

When you start haproxy with the debug enabled (-d), what do you see
in the sixth line (Using *blabla()* as the polling mechanism)?

So this affects only the SSL enabled frontend, the frontend "http-in" works
just fine, is that correct?


Lukas




  

RE: Owncloud through Haproxy makes upload not possible

[Lukas Tribus]

>> Ok, could you redo this trace with the "-d" option and
>> without the nokqueue configuration.
> Attached.

Still seeing poll() in this trace. Are you sure nokqueue was removed
in the configuration and haproxy was restarted?

Please also provide the output of "haproxy -vv".


Thanks,

Lukas 

RE: Owncloud through Haproxy makes upload not possible

[Lukas Tribus]

Hi Piotr,


>> - try nokqueue mode [1]
> Didn't change anything.
>> - try option http-no-delay [2]
> Didn't change anything.

Ok, please remove both options again.



>> - check cpu usage (system and haproxy) while uploading
> Load average is about 0.2-0.4

What we have to find out is if haproxy or the system are
in some kind of call loop. The average CPU load is therefor
not that important, its the CPU load in the very second the
upload happens thats important.



>> - truss ([3]) haproxy while uploading
> Attached.

Ok, could you redo this trace with the "-d" option and
without the nokqueue configuration.



> Done, both LibreSSL 2.2.4 and OpenSSL 1.0.1p are affected.
>
> I'll get the tcpdump later on, can you deduce anything from those
> "Resource temporarily unavailable" from my log?

Error 35 is EAGAIN, afaik this is expected (and should be handled
without problems).




Regards,

Lukas

  

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 11/19/15 22:43, Igor Cicimov wrote:
> 
> It might be something specific to BSD os causing issues for you 
> since I haven't heard of anyone complaining about ssl till now.
> You can also try latest stable 1.5.15 since I can't see any 1.6 
> specific feature in your config.
> 
Unfortunately, using 1.5.15 didn't change anything. The logs are:
Nov 20 10:23:51 anongoth haproxy[86788]: 46.248.161.165:57472
[20/Nov/2015:10:23:46.069] https-in~ owncloud/node1 4958/0/0/31/4989
200 319 - -  12/6/1/2/0 0/0 "GET
/core/css/images/ui-bg_flat_35_1d2d44_40x100.png HTTP/1.1"
Nov 20 10:23:51 anongoth haproxy[86788]: 46.248.161.165:13241
[20/Nov/2015:10:23:46.075] https-in~ owncloud/node1 4958/0/1/24/4983
200 323 - -  12/6/0/1/0 0/0 "GET
/core/css/images/ui-bg_highlight-soft_100_ee_1x100.png HTTP/1.1"
Nov 20 10:23:58 anongoth haproxy[86788]: 46.248.161.165:57472
[20/Nov/2015:10:23:51.058] https-in~ owncloud/node1 6752/0/0/448/7221
200 862 - -  12/6/1/2/0 0/0 "GET
/index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1"
Nov 20 10:24:23 anongoth haproxy[86788]: 46.248.161.165:13241
[20/Nov/2015:10:23:51.058] https-in~ owncloud/node1
31819/0/1/520/32362 200 862 - -  12/6/2/3/0 0/0 "GET
/index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1"
Nov 20 10:24:43 anongoth haproxy[86788]: 46.248.161.165:13481
[20/Nov/2015:10:23:45.309] https-in~ owncloud/node1 5791/0/0/-1/58637
- -1 0 - - CD-- 2/2/2/2/0 0/0 "POST
/index.php/apps/files/ajax/upload.php HTTP/1.1"
Nov 20 10:25:11 anongoth haproxy[86788]: 46.248.161.165:57472
[20/Nov/2015:10:23:58.280] https-in~ owncloud/node1 14900/0/1/-1/73036
- -1 0 - - CD-- 1/1/1/1/0 0/0 "POST
/index.php/apps/files/ajax/upload.php HTTP/1.1"
Nov 20 10:28:21 anongoth haproxy[86788]: 46.248.161.165:45063
[20/Nov/2015:10:26:54.272] https-in~ owncloud/node1 58/0/1/-1/87092
504 194 - - sH-- 0/0/0/0/0 0/0 "POST
/index.php/apps/files/ajax/upload.php HTTP/1.1"
Nov 20 10:28:22 anongoth haproxy[86788]: 46.248.161.165:17696
[20/Nov/2015:10:28:21.697] https-in~ owncloud/node1 372/0/0/408/801
200 862 - -  1/1/0/1/0 0/0 "GET
/index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1"
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJWTuo/AAoJEC9nKukRsfY+L58P/jNTo/tTh0MiwEXuULvKIwnp
abVg4bJ5ObF3MJjQSwoB5oqsAOM14sjCg/x/zPb8B+SCkYf5rciR1tdZQEyY+1Ml
DkzN6WRg5zztv/5FwAH2A3ZmaVeMShmhWtdWlgFJszCVKIHNEziBRp5pQI50zWD7
RadW6EFXe9Sw6Di0NhzoZoF9sIeYymI1fq45OKlRdxc5Usdy5jtpfHsvFP9EUT+Q
NLOye1tXMJq3VT/HdrErOp3C7kNxffYmcNZR+uSFWyaRInpV+ME0gseyOnImkO+c
og+03j6tFTUZeSndPOJSdo/Po5ldf5XoBZp2ToyC8Fz16O627ZsIzXFbaeNh9SUt
VpZTByYR91LRm473rYoJjCik+uI8RsbYWa2tNYx7xUMBAkufN7EHeQ4CnBeLhy4h
26gbhIvpyAs2wyB/0nlm5rEEZim8fLbDMgs03HaBifqcp8Wjsy/6kLlF/j6TjOX3
H1fV9P/5KbfctvsBaeCPkx5fndLDBmaN1ryWPWeCIojluu+6IMDiUQgko3v/b5Yy
jq6m7ay1Kwq8FLiVkAnUjxDmxoe5bFvi1G2tcKaaFwNAtFsmOLlxRKVgOF3RpQy7
LoJyczPWR8ucO/WF5z9oAxapx5EolH6aYQoWe0QGRkDJwkVroLBOq6Gi0JDITw9Y
VO7rSZNnxdbWYPcVQcld
=EShE
-END PGP SIGNATURE-

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 11/19/15 22:43, Igor Cicimov wrote:
> 
> On 20/11/2015 7:23 AM, "Piotr Kubaj"  > wrote:
>> 
>> On 11/19/2015 17:01, Janusz Dziemidowicz wrote:
>>> 2015-11-19 15:45 GMT+01:00 Piotr Kubaj  >:
 Now, about RSA vs ECDSA. I simply don't trust ECDSA. There
 are quite a lot of questions about constants used by ECDSA,
 which seem to be chosen quite arbitrarily by its creator,
 which happens to be NSA. These questions of course remain
 unanswered. Even respected scientists like Schneier say that
 RSA should be used instead (see
 
> https://www.schneier.com/blog/archives/2013/09/the_nsa_is_brea.html#c1
67
>>
> 
>> 5929
>>> 
>>> But ECDSA itself does not contain any constants (see
>>> 
> https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorit
hm).
>>
>  Yes, you have to choose domain parameters and most commonly used
> are
>>> NIST ones. But you can also use brainpool curves, which
>>> specifically avoid using any arbitrary constants (see 
>>> http://www.ecc-brainpool.org/download/Domain-parameters.pdf)
>>> and they are even defined for TLS
>>> (https://tools.ietf.org/html/rfc7027) and apparently supported
>>> by latest OpenSSL. Unfortunately not by anything else. OK,
>>> anyway that's your preference, I'm not going to argue about
> ECDSA or not;)
>>> 
 ). When I'm done setting my HTTP(S) services, I'll simply
 limit incoming connections connections on my firewall so
 DDOS'ing won't be possible, unless you DDOS my firewall :)
>>> 
>>> I've never said anything about DDoS. In such setup there is no
>>> need for distributed DoS. The CPU usage of RSA 8192 is so high
>>> that a single shell script running on a single attack machine
>>> can kill any server. If you are willing to limit your
>>> connection rate on a firewall to a few per second, then fine;)
>>> 
>>> As for your problem. Now that it seems like SSL problem, can
>>> you just try with RSA 4096 or 2048? RSA 8192 is really not much
>>> tested in most code, so maybe the problem is in fact related.
>>> 
>> Unfortunately, accessing my HTTPS services by only OpenSSL is out
>> of the question. Besides, I use LibreSSL and am not sure it
>> supports it, since OpenBSD people got rid of quite a lot of
>> unnecessary code.
>> 
>> So I can only choose ECDSA or RSA.
>> 
>> I don't think limiting my connections is a bad idea vs choosing
>> weaker RSA. As I said before, I actually expect only a few
>> connections at once.
>> 
>> I've generated RSA 2048 cert with: openssl req -x509 -newkey
>> rsa:2048 -keyout haproxy.pem -out haproxy.pem -days 3650 -nodes
>> 
>> That is, I didn't use any non-default options, such as SHA512. 
>> Unfortunately, it doesn't yield any result. I'm now considering 
>> switching to SSL Pass-through, and configuring HTTPS in each of
>> my WWW servers, it may be much quicker considering how long I've
>> been getting Haproxy to work.
>> 
> It might be something specific to BSD os causing issues for you
> since I haven't heard of anyone complaining about ssl till now. You
> can also try latest stable 1.5.15 since I can't see any 1.6
> specific feature in your config.
> 
Unfortunately, using 1.5.15 didn't change anything. The logs are:
Nov 20 10:23:51 anongoth haproxy[86788]: 46.248.161.165:57472
[20/Nov/2015:10:23:46.069] https-in~ owncloud/node1 4958/0/0/31/4989
200 319 - -  12/6/1/2/0 0/0 "GET
/core/css/images/ui-bg_flat_35_1d2d44_40x100.png HTTP/1.1"
Nov 20 10:23:51 anongoth haproxy[86788]: 46.248.161.165:13241
[20/Nov/2015:10:23:46.075] https-in~ owncloud/node1 4958/0/1/24/4983
200 323 - -  12/6/0/1/0 0/0 "GET
/core/css/images/ui-bg_highlight-soft_100_ee_1x100.png HTTP/1.1"
Nov 20 10:23:58 anongoth haproxy[86788]: 46.248.161.165:57472
[20/Nov/2015:10:23:51.058] https-in~ owncloud/node1 6752/0/0/448/7221
200 862 - -  12/6/1/2/0 0/0 "GET
/index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1"
Nov 20 10:24:23 anongoth haproxy[86788]: 46.248.161.165:13241
[20/Nov/2015:10:23:51.058] https-in~ owncloud/node1
31819/0/1/520/32362 200 862 - -  12/6/2/3/0 0/0 "GET
/index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1"
Nov 20 10:24:43 anongoth haproxy[86788]: 46.248.161.165:13481
[20/Nov/2015:10:23:45.309] https-in~ owncloud/node1 5791/0/0/-1/58637
- -1 0 - - CD-- 2/2/2/2/0 0/0 "POST
/index.php/apps/files/ajax/upload.php HTTP/1.1"
Nov 20 10:25:11 anongoth haproxy[86788]: 46.248.161.165:57472
[20/Nov/2015:10:23:58.280] https-in~ owncloud/node1 14900/0/1/-1/73036
- -1 0 - - CD-- 1/1/1/1/0 0/0 "POST
/index.php/apps/files/ajax/upload.php HTTP/1.1"
Nov 20 10:28:21 anongoth haproxy[86788]: 46.248.161.165:45063
[20/Nov/2015:10:26:54.272] https-in~ owncloud/node1 58/0/1/-1/87092
504 194 - - sH-- 0/0/0/0/0 0/0 "POST
/index.php/apps/files/ajax/upload.php HTTP/1.1"
Nov 20 10:28:22 anongoth haproxy[86788]: 46.248.161.165:17696

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

On 11/19/15 22:43, Igor Cicimov wrote:
>
> It might be something specific to BSD os causing issues for you
> since I haven't heard of anyone complaining about ssl till now.
> You can also try latest stable 1.5.15 since I can't see any 1.6
> specific feature in your config.
>
Unfortunately, using 1.5.15 didn't change anything. The logs are:
Nov 20 10:23:51 anongoth haproxy[86788]: 46.248.161.165:57472
[20/Nov/2015:10:23:46.069] https-in~ owncloud/node1 4958/0/0/31/4989
200 319 - -  12/6/1/2/0 0/0 "GET
/core/css/images/ui-bg_flat_35_1d2d44_40x100.png HTTP/1.1"
Nov 20 10:23:51 anongoth haproxy[86788]: 46.248.161.165:13241
[20/Nov/2015:10:23:46.075] https-in~ owncloud/node1 4958/0/1/24/4983
200 323 - -  12/6/0/1/0 0/0 "GET
/core/css/images/ui-bg_highlight-soft_100_ee_1x100.png HTTP/1.1"
Nov 20 10:23:58 anongoth haproxy[86788]: 46.248.161.165:57472
[20/Nov/2015:10:23:51.058] https-in~ owncloud/node1 6752/0/0/448/7221
200 862 - -  12/6/1/2/0 0/0 "GET
/index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1"
Nov 20 10:24:23 anongoth haproxy[86788]: 46.248.161.165:13241
[20/Nov/2015:10:23:51.058] https-in~ owncloud/node1
31819/0/1/520/32362 200 862 - -  12/6/2/3/0 0/0 "GET
/index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1"
Nov 20 10:24:43 anongoth haproxy[86788]: 46.248.161.165:13481
[20/Nov/2015:10:23:45.309] https-in~ owncloud/node1 5791/0/0/-1/58637
-1 0 - - CD-- 2/2/2/2/0 0/0 "POST
/index.php/apps/files/ajax/upload.php HTTP/1.1"
Nov 20 10:25:11 anongoth haproxy[86788]: 46.248.161.165:57472
[20/Nov/2015:10:23:58.280] https-in~ owncloud/node1 14900/0/1/-1/73036
-1 0 - - CD-- 1/1/1/1/0 0/0 "POST
/index.php/apps/files/ajax/upload.php HTTP/1.1"
Nov 20 10:28:21 anongoth haproxy[86788]: 46.248.161.165:45063
[20/Nov/2015:10:26:54.272] https-in~ owncloud/node1 58/0/1/-1/87092
504 194 - - sH-- 0/0/0/0/0 0/0 "POST
/index.php/apps/files/ajax/upload.php HTTP/1.1"
Nov 20 10:28:22 anongoth haproxy[86788]: 46.248.161.165:17696
[20/Nov/2015:10:28:21.697] https-in~ owncloud/node1 372/0/0/408/801
200 862 - -  1/1/0/1/0 0/0 "GET
/index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1"

Re: Owncloud through Haproxy makes upload not possible

[Bryan Talbot]

On Fri, Nov 20, 2015 at 1:39 AM, Piotr Kubaj  wrote:

>
>
> Unfortunately, using 1.5.15 didn't change anything. The logs are:
>

We can see from the logs below that the connection is aborting with CD or
sH codes.

The docs say:

 CD   The client unexpectedly aborted during data transfer. This can be
  caused by a browser crash, by an intermediate equipment between the
  client and haproxy which decided to actively break the connection,
  by network routing issues between the client and haproxy, or by a
  keep-alive session between the server and the client terminated first
  by the client.


 sH   The "timeout server" stroke before the server could return its
  response headers. This is the most common anomaly, indicating too
  long transactions, probably caused by server or database saturation.
  The immediate workaround consists in increasing the "timeout server"
  setting, but it is important to keep in mind that the user experience
  will suffer from these long response times. The only long term
  solution is to fix the application.


The logs show that haproxy is able to accept the connection from your
client and make the connection to a backend; however, the client is then
being disconnected before the server responds or the server is taking too
long to respond and haproxy returns an error.

Maybe you have a firewall that's causing troubles?



Nov 20 10:24:43 anongoth haproxy[86788]: 46.248.161.165:13481
> [20/Nov/2015:10:23:45.309] https-in~ owncloud/node1 5791/0/0/-1/58637
> -1 0 - - CD-- 2/2/2/2/0 0/0 "POST
> /index.php/apps/files/ajax/upload.php HTTP/1.1"
> Nov 20 10:25:11 anongoth haproxy[86788]: 46.248.161.165:57472
> [20/Nov/2015:10:23:58.280] https-in~ owncloud/node1 14900/0/1/-1/73036
> -1 0 - - CD-- 1/1/1/1/0 0/0 "POST
> /index.php/apps/files/ajax/upload.php HTTP/1.1"
> Nov 20 10:28:21 anongoth haproxy[86788]: 46.248.161.165:45063
> [20/Nov/2015:10:26:54.272] https-in~ owncloud/node1 58/0/1/-1/87092
> 504 194 - - sH-- 0/0/0/0/0 0/0 "POST
> /index.php/apps/files/ajax/upload.php HTTP/1.1"
> Nov 20 10:28:22 anongoth haproxy[86788]: 46.248.161.165:17696
>

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

On 11/20/2015 21:05, Bryan Talbot wrote:
> On Fri, Nov 20, 2015 at 1:39 AM, Piotr Kubaj  > wrote:
> 
> 
> 
> Unfortunately, using 1.5.15 didn't change anything. The logs are:
> 
> 
> We can see from the logs below that the connection is aborting with CD
> or sH codes.
> 
> The docs say:
> 
>  CD   The client unexpectedly aborted during data transfer. This can be
>   caused by a browser crash, by an intermediate equipment between the
>   client and haproxy which decided to actively break the connection,
>   by network routing issues between the client and haproxy, or by a
>   keep-alive session between the server and the client terminated 
> first
>   by the client.
> 
> 
>  sH   The "timeout server" stroke before the server could return its
>   response headers. This is the most common anomaly, indicating too
>   long transactions, probably caused by server or database saturation.
>   The immediate workaround consists in increasing the "timeout server"
>   setting, but it is important to keep in mind that the user 
> experience
>   will suffer from these long response times. The only long term
>   solution is to fix the application.
> 
> 
> The logs show that haproxy is able to accept the connection from your
> client and make the connection to a backend; however, the client is then
> being disconnected before the server responds or the server is taking
> too long to respond and haproxy returns an error.
> 
> Maybe you have a firewall that's causing troubles?
> 
> 
> 
> Nov 20 10:24:43 anongoth haproxy[86788]: 46.248.161.165:13481
> 
> [20/Nov/2015:10:23:45.309] https-in~ owncloud/node1 5791/0/0/-1/58637
> -1 0 - - CD-- 2/2/2/2/0 0/0 "POST
> /index.php/apps/files/ajax/upload.php HTTP/1.1"
> Nov 20 10:25:11 anongoth haproxy[86788]: 46.248.161.165:57472
> 
> [20/Nov/2015:10:23:58.280] https-in~ owncloud/node1 14900/0/1/-1/73036
> -1 0 - - CD-- 1/1/1/1/0 0/0 "POST
> /index.php/apps/files/ajax/upload.php HTTP/1.1"
> Nov 20 10:28:21 anongoth haproxy[86788]: 46.248.161.165:45063
> 
> [20/Nov/2015:10:26:54.272] https-in~ owncloud/node1 58/0/1/-1/87092
> 504 194 - - sH-- 0/0/0/0/0 0/0 "POST
> /index.php/apps/files/ajax/upload.php HTTP/1.1"
> Nov 20 10:28:22 anongoth haproxy[86788]: 46.248.161.165:17696
> 
> 
Nope, it's not the firewall. I'm able to connect between other jails.
I've also successfully set up SSL Pass-through and It Just Works(tm).



signature.asc
Description: OpenPGP digital signature

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

On 11/20/2015 21:05, Bryan Talbot wrote:
> On Fri, Nov 20, 2015 at 1:39 AM, Piotr Kubaj  > wrote:
>
>
>
> Unfortunately, using 1.5.15 didn't change anything. The logs are:
>
>
> We can see from the logs below that the connection is aborting with CD
> or sH codes.
>
> The docs say:
>
>  CD   The client unexpectedly aborted during data transfer. This
can be
>   caused by a browser crash, by an intermediate equipment
between the
>   client and haproxy which decided to actively break the
connection,
>   by network routing issues between the client and haproxy, or
by a
>   keep-alive session between the server and the client
terminated first
>   by the client.
>
>
>  sH   The "timeout server" stroke before the server could return its
>   response headers. This is the most common anomaly,
indicating too
>   long transactions, probably caused by server or database
saturation.
>   The immediate workaround consists in increasing the "timeout
server"
>   setting, but it is important to keep in mind that the user
experience
>   will suffer from these long response times. The only long term
>   solution is to fix the application.
>
>
> The logs show that haproxy is able to accept the connection from your
> client and make the connection to a backend; however, the client is then
> being disconnected before the server responds or the server is taking
> too long to respond and haproxy returns an error.
>
> Maybe you have a firewall that's causing troubles?
>
>
>
> Nov 20 10:24:43 anongoth haproxy[86788]: 46.248.161.165:13481
> 
> [20/Nov/2015:10:23:45.309] https-in~ owncloud/node1 5791/0/0/-1/58637
> -1 0 - - CD-- 2/2/2/2/0 0/0 "POST
> /index.php/apps/files/ajax/upload.php HTTP/1.1"
> Nov 20 10:25:11 anongoth haproxy[86788]: 46.248.161.165:57472
> 
> [20/Nov/2015:10:23:58.280] https-in~ owncloud/node1 14900/0/1/-1/73036
> -1 0 - - CD-- 1/1/1/1/0 0/0 "POST
> /index.php/apps/files/ajax/upload.php HTTP/1.1"
> Nov 20 10:28:21 anongoth haproxy[86788]: 46.248.161.165:45063
> 
> [20/Nov/2015:10:26:54.272] https-in~ owncloud/node1 58/0/1/-1/87092
> 504 194 - - sH-- 0/0/0/0/0 0/0 "POST
> /index.php/apps/files/ajax/upload.php HTTP/1.1"
> Nov 20 10:28:22 anongoth haproxy[86788]: 46.248.161.165:17696
> 
>
Nope, it's not the firewall. I'm able to connect between other jails.
I've also successfully set up SSL Pass-through and It Just Works(tm).

RE: Owncloud through Haproxy makes upload not possible

[Lukas Tribus]

Hi Piotr,


> Unfortunately, using 1.5.15 didn't change anything.

a few things I would suggest to try/troubleshoot:

- try nokqueue mode [1]
- try option http-no-delay [2]
- check cpu usage (system and haproxy) while uploading
- truss ([3]) haproxy while uploading
- tcpdump the frontend connection
- statically link against openssl, to confirm that the issue happens
  with both openssl and libressl (README has some suggestions howto
  compile openssl locally and linking haproxy against it without
  affecting your system's ssllib)


Also can you provide the output of "haproxy -vv"?


Regards,

Lukas

[1] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#nokqueue
[2] 
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-option%20http-no-delay
[3] https://www.freebsd.org/cgi/man.cgi?truss


  

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 11/18/15 19:45, Bryan Talbot wrote:
> Hard to guess what the issue is but haproxy logging would probably
> help. However, the logging configuration is a bit of a mess as it's
> configured to be "all on" and "all off" at the same time. There are
> also other configuration oddities.

Well, I'm actually quite new to Haproxy, so I just followed some
random tutorial for domain proxy. It worked (I've got a few websites
on it), so I've just assumed it's ok.

> 4096 bit DH params will be pretty slow to handshake. Maybe that's
> okay in your circumstance though since you seem to be using this
> for a personal use and not expecting a high connection rate. You
> also have a 8 kbit RSA self signed certificate and using 256 bit
> ciphers which increase TLS overhead.
I want it to be secure, and I don't want to touch my settings for
quite a while so I just took the strongest algorithms there are, and
2x recommended values for things like private key, or DH params. The
hardware is pretty powerful and I've already checked that I don't have
a huge load.


> Is that where the logging socket is on FreeBSD now? I haven't used 
> FreeBSD in quite a while.
Yes, the default is /var/run/log.

> maxconn 4096 user daemon group daemon daemon
> 
> defaults modehttp option  httplog option  dontlognull option
> forwardfor option  http-server-close option  httpclose option
> tcplog option  dontlog-normal
> 
> 
> You have both tcp logging and http logging enabled at the same
> time. In addition, you also have all logging disabled with
> "dontlognull" and "dontlog-normal". If all your proxies are HTTP
> like you've shown, just enable httplog and remove the tcplog
> option. When troubleshooting, enable logging at least normal
> connections.
> 
> You also do not want to use both httpclose and http-server-close
> since they conflict. Remove option httpclose.
> 
> Timeouts are also missing and you should be getting warnings about
> that too.
> 
> [WARNING] 321/103152 (87887) : config : missing timeouts for proxy 
> 'https-in'. | While not properly invalid, you will certainly
> encounter various problems | with such a configuration. To fix
> this, please ensure that all following | timeouts are set to a
> non-zero value: 'client', 'connect', 'server'.
Well, it seems that the person that wrote the tutorial I used doesn't
know much about Haproxy either...

> frontend http-in bind 192.168.11.3:80  
> reqadd X-Forwarded-Proto:\ http redirect scheme https code 301 if
> !{ ssl_fc }
> 
> 

> 
> 
> 
> 
> 
> frontend https-in option httplog option forwardfor option
> http-server-close option httpclose
> 
> 
> Don't need to repeat all of the above since it should be set in
> defaults above (if set properly). With no "log global" you won't
> get any logs anyway and are probably seeing a warning when haproxy
> checks the config or starts.
> 
> [WARNING] 321/103109 (87884) : config : log format ignored for
> proxy 'https-in' since it has no log address.
> 
> 
> 
> 
> 
> 
> rspadd Public-Key-Pins:\ 
> pin-sha256="1Pw5h93NOsPw6j/vaTYl5VvW9cmtuZXtNP3cVz10hKo=";\ 
> max-age=15768000;\ includeSubDomains
> 
> 
> AFAIK, HPKP is only somewhat supported by only the most recent
> browser releases. I believe that it's also ignored by them for
> certificates which are self-signed or signed by a CA that is not in
> the browsers system-defined CA set. Probably doesn't cause your
> issue but who knows -- it is still experimental.
> 
> The "http-response set-header" supported in haproxy 1.5 and later
> is more powerful and easier to read than the old reqadd and rspadd
> features.
> 
> 
> 
> 
> bind 192.168.11.3:443  ssl crt 
> /usr/local/etc/haproxy.pem ciphers AES256+EECDH:AES256+EDH
> force-tlsv12 no-sslv3
> 
> 
> Don't need to repeat these options that are already set globally.
> 
> 
> 
> 
> 
> backend 10amd64 server node1 192.168.11.3:81
>  cookie A check
> 
> 
> 
> Setting sticky cookies and not using them is probably harmless but 
> what's the point?
> 
> -Bryan
> 


Debug logs from uploading a file:
00c1:https-in.clicls[0007:]
00c1:https-in.closed[0007:]
00c2:https-in.accept(0006)=0007 from [xxx:59266]
00c2:https-in.clireq[0007:]: POST
/index.php/apps/files/ajax/upload.php HTTP/1.1
00c2:https-in.clihdr[0007:]: Host: owncloud.anongoth.pl
00c2:https-in.clihdr[0007:]: User-Agent: Mozilla/5.0
(Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
00c2:https-in.clihdr[0007:]: Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
00c2:https-in.clihdr[0007:]: Accept-Language: en-US, en
00c2:https-in.clihdr[0007:]: Accept-Encoding: gzip, deflate
00c2:https-in.clihdr[0007:]: DNT: 1
00c2:https-in.clihdr[0007:]: requesttoken:
e6185c74050ccda163a7f6d9299933a12b9fae66e46b97d1eed750c9ecd71dd2|C4wwTFE

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

On 11/19/2015 17:01, Janusz Dziemidowicz wrote:
> 2015-11-19 15:45 GMT+01:00 Piotr Kubaj :
>> Now, about RSA vs ECDSA. I simply don't trust ECDSA. There are quite a
>> lot of questions about constants used by ECDSA, which seem to be
>> chosen quite arbitrarily by its creator, which happens to be NSA.
>> These questions of course remain unanswered. Even respected scientists
>> like Schneier say that RSA should be used instead (see
>> https://www.schneier.com/blog/archives/2013/09/the_nsa_is_brea.html#c167
>> 5929
> 
> But ECDSA itself does not contain any constants (see
> https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm).
> Yes, you have to choose domain parameters and most commonly used are
> NIST ones. But you can also use brainpool curves, which specifically
> avoid using any arbitrary constants (see
> http://www.ecc-brainpool.org/download/Domain-parameters.pdf) and they
> are even defined for TLS (https://tools.ietf.org/html/rfc7027) and
> apparently supported by latest OpenSSL. Unfortunately not by anything
> else.
> OK, anyway that's your preference, I'm not going to argue about ECDSA or not;)
> 
>> ). When I'm done setting my HTTP(S) services, I'll simply limit
>> incoming connections connections on my firewall so DDOS'ing won't be
>> possible, unless you DDOS my firewall :)
> 
> I've never said anything about DDoS. In such setup there is no need
> for distributed DoS. The CPU usage of RSA 8192 is so high that a
> single shell script running on a single attack machine can kill any
> server.
> If you are willing to limit your connection rate on a firewall to a
> few per second, then fine;)
> 
> As for your problem. Now that it seems like SSL problem, can you just
> try with RSA 4096 or 2048? RSA 8192 is really not much tested in most
> code, so maybe the problem is in fact related.
> 
Unfortunately, accessing my HTTPS services by only OpenSSL is out of the
question. Besides, I use LibreSSL and am not sure it supports it, since
OpenBSD people got rid of quite a lot of unnecessary code.

So I can only choose ECDSA or RSA.

I don't think limiting my connections is a bad idea vs choosing weaker
RSA. As I said before, I actually expect only a few connections at once.

I've generated RSA 2048 cert with:
openssl req -x509 -newkey rsa:2048 -keyout haproxy.pem -out haproxy.pem
-days 3650 -nodes

That is, I didn't use any non-default options, such as SHA512.
Unfortunately, it doesn't yield any result. I'm now considering
switching to SSL Pass-through, and configuring HTTPS in each of my WWW
servers, it may be much quicker considering how long I've been getting
Haproxy to work.



signature.asc
Description: OpenPGP digital signature

Re: Owncloud through Haproxy makes upload not possible

[Igor Cicimov]

On 20/11/2015 7:23 AM, "Piotr Kubaj"  wrote:
>
> On 11/19/2015 17:01, Janusz Dziemidowicz wrote:
> > 2015-11-19 15:45 GMT+01:00 Piotr Kubaj :
> >> Now, about RSA vs ECDSA. I simply don't trust ECDSA. There are quite a
> >> lot of questions about constants used by ECDSA, which seem to be
> >> chosen quite arbitrarily by its creator, which happens to be NSA.
> >> These questions of course remain unanswered. Even respected scientists
> >> like Schneier say that RSA should be used instead (see
> >>
https://www.schneier.com/blog/archives/2013/09/the_nsa_is_brea.html#c167
> >> 5929
> >
> > But ECDSA itself does not contain any constants (see
> > https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm
).
> > Yes, you have to choose domain parameters and most commonly used are
> > NIST ones. But you can also use brainpool curves, which specifically
> > avoid using any arbitrary constants (see
> > http://www.ecc-brainpool.org/download/Domain-parameters.pdf) and they
> > are even defined for TLS (https://tools.ietf.org/html/rfc7027) and
> > apparently supported by latest OpenSSL. Unfortunately not by anything
> > else.
> > OK, anyway that's your preference, I'm not going to argue about ECDSA
or not;)
> >
> >> ). When I'm done setting my HTTP(S) services, I'll simply limit
> >> incoming connections connections on my firewall so DDOS'ing won't be
> >> possible, unless you DDOS my firewall :)
> >
> > I've never said anything about DDoS. In such setup there is no need
> > for distributed DoS. The CPU usage of RSA 8192 is so high that a
> > single shell script running on a single attack machine can kill any
> > server.
> > If you are willing to limit your connection rate on a firewall to a
> > few per second, then fine;)
> >
> > As for your problem. Now that it seems like SSL problem, can you just
> > try with RSA 4096 or 2048? RSA 8192 is really not much tested in most
> > code, so maybe the problem is in fact related.
> >
> Unfortunately, accessing my HTTPS services by only OpenSSL is out of the
> question. Besides, I use LibreSSL and am not sure it supports it, since
> OpenBSD people got rid of quite a lot of unnecessary code.
>
> So I can only choose ECDSA or RSA.
>
> I don't think limiting my connections is a bad idea vs choosing weaker
> RSA. As I said before, I actually expect only a few connections at once.
>
> I've generated RSA 2048 cert with:
> openssl req -x509 -newkey rsa:2048 -keyout haproxy.pem -out haproxy.pem
> -days 3650 -nodes
>
> That is, I didn't use any non-default options, such as SHA512.
> Unfortunately, it doesn't yield any result. I'm now considering
> switching to SSL Pass-through, and configuring HTTPS in each of my WWW
> servers, it may be much quicker considering how long I've been getting
> Haproxy to work.
>
It might be something specific to BSD os causing issues for you since I
haven't heard of anyone complaining about ssl till now. You can also try
latest stable 1.5.15 since I can't see any 1.6 specific feature in your
config.

Re: Owncloud through Haproxy makes upload not possible

[Janusz Dziemidowicz]

2015-11-19 15:45 GMT+01:00 Piotr Kubaj :
> Now, about RSA vs ECDSA. I simply don't trust ECDSA. There are quite a
> lot of questions about constants used by ECDSA, which seem to be
> chosen quite arbitrarily by its creator, which happens to be NSA.
> These questions of course remain unanswered. Even respected scientists
> like Schneier say that RSA should be used instead (see
> https://www.schneier.com/blog/archives/2013/09/the_nsa_is_brea.html#c167
> 5929

But ECDSA itself does not contain any constants (see
https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm).
Yes, you have to choose domain parameters and most commonly used are
NIST ones. But you can also use brainpool curves, which specifically
avoid using any arbitrary constants (see
http://www.ecc-brainpool.org/download/Domain-parameters.pdf) and they
are even defined for TLS (https://tools.ietf.org/html/rfc7027) and
apparently supported by latest OpenSSL. Unfortunately not by anything
else.
OK, anyway that's your preference, I'm not going to argue about ECDSA or not;)

> ). When I'm done setting my HTTP(S) services, I'll simply limit
> incoming connections connections on my firewall so DDOS'ing won't be
> possible, unless you DDOS my firewall :)

I've never said anything about DDoS. In such setup there is no need
for distributed DoS. The CPU usage of RSA 8192 is so high that a
single shell script running on a single attack machine can kill any
server.
If you are willing to limit your connection rate on a firewall to a
few per second, then fine;)

As for your problem. Now that it seems like SSL problem, can you just
try with RSA 4096 or 2048? RSA 8192 is really not much tested in most
code, so maybe the problem is in fact related.

-- 
Janusz Dziemidowicz

Re: Owncloud through Haproxy makes upload not possible

[Piotr Kubaj]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 11/19/15 14:41, Janusz Dziemidowicz wrote:
> 2015-11-19 11:13 GMT+01:00 Piotr Kubaj :
>>> 4096 bit DH params will be pretty slow to handshake. Maybe
>>> that's okay in your circumstance though since you seem to be
>>> using this for a personal use and not expecting a high
>>> connection rate. You also have a 8 kbit RSA self signed
>>> certificate and using 256 bit ciphers which increase TLS
>>> overhead.
>> I want it to be secure, and I don't want to touch my settings
>> for quite a while so I just took the strongest algorithms there
>> are, and 2x recommended values for things like private key, or DH
>> params. The hardware is pretty powerful and I've already checked
>> that I don't have a huge load.
> 
> Take note, that increasing RSA size twice reduces number of 
> connections you can accept 10 times or more. For example, my quite
> powerful desktop with recent CPU can accept 973 connections per
> second per core for RSA 2048, 136 connections per second for RSA
> 4096. OpenSSL does not have tests for RSA 8192, but that would be
> in the order of a few connections per second. RSA 8192 is really an
> overkill, it would be possible to DoS your server with a simple
> shell script;) If you want a state of the art cryptography you 
> should probably use ECDSA certificate, it will be both faster and
> more secure.
> 
I've tweaked my haproxy.conf a little, now it looks like this:
global
#ssl-default-bind-options no-sslv3 no-tls-tickets force-tlsv12
#ssl-default-bind-ciphers AES256+EECDH:AES256+EDH
#tune.ssl.default-dh-param 4096
log /var/run/log local0
maxconn 4096
user daemon
group daemon
daemon

defaults
log global
modehttp
option  httplog
option forwardfor
option http-server-close
timeout connect 5000ms
timeout client 5ms
timeout server 5ms


frontend http-in
bind 192.168.11.3:80
acl host_phpsysinfo hdr_dom(host) -i phpsysinfo.anongoth.pl
acl host_owncloud hdr_dom(host) -i owncloud.anongoth.pl
acl host_roundcube hdr_dom(host) -i roundcube.anongoth.pl
acl host_10amd64 hdr_dom(host) -i 10amd64.anongoth.pl
acl host_10armv6 hdr_dom(host) -i 10armv6.anongoth.pl

## figure out which one to use
use_backend phpsysinfo if host_phpsysinfo
use_backend owncloud if host_owncloud
use_backend roundcube if host_roundcube
use_backend 10amd64 if host_10amd64
use_backend 10armv6 if host_10armv6

#   redirect scheme https code 301 if !{ ssl_fc }

frontend https-in
#option httplog
#option forwardfor
#option http-server-close
#option httpclose
#http-response set-header Strict-Transport-Security
"max-age=1600; includeSubDomains; preload;"
#http-response set-header Public-Key-Pins:
"pin-sha256="1Pw5h93NOsPw6j/vaTYl5VvW9cmtuZXtNP3cVz10hKo=";
max-age=15768000; includeSubDomains"
#bind 192.168.11.3:443 ssl crt /usr/local/etc/haproxy.pem ciphers
AES256+EECDH:AES256+EDH force-tlsv12 no-sslv3
bind 192.168.11.3:443 ssl crt /usr/local/etc/haproxy.pem
acl host_phpsysinfo hdr_dom(host) -i phpsysinfo.anongoth.pl
acl host_owncloud hdr_dom(host) -i owncloud.anongoth.pl
acl host_roundcube hdr_dom(host) -i roundcube.anongoth.pl
acl host_10amd64 hdr_dom(host) -i 10amd64.anongoth.pl
acl host_10armv6 hdr_dom(host) -i 10armv6.anongoth.pl

## figure out which one to use
use_backend phpsysinfo if host_phpsysinfo
use_backend owncloud if host_owncloud
use_backend roundcube if host_roundcube
use_backend 10amd64 if host_10amd64
use_backend 10armv6 if host_10armv6

backend 10amd64
server node1 192.168.11.3:81

backend 10armv6
server node1 192.168.11.3:81

backend phpsysinfo
server node1 192.168.11.3:81

backend owncloud
server node1 192.168.11.14:80

backend roundcube
server node1 192.168.1.101:80






With that, I can tell that the issue is about HTTPS section. HTTP
connection through Haproxy goes fine, but HTTPS still has a problem.
I've also got regular Haproxy logs, I'm not sure if they can help
diagnose the problem:
Nov 19 15:04:58 anongoth haproxy[11792]: 46.248.161.165:54997
[19/Nov/2015:15:04:52.559] https-in owncloud/node1 5900/0/0/0/5901 200
319 - -  2/2/0/1/0 0/0 "GET
/core/css/images/ui-bg_flat_35_1d2d44_40x100.png HTTP/1.1"
Nov 19 15:04:58 anongoth haproxy[11792]: 46.248.161.165:54996
[19/Nov/2015:15:04:52.555] https-in owncloud/node1 5911/0/0/1/5912 200
323 - -  2/2/0/1/0 0/0 "GET
/core/css/images/ui-bg_highlight-soft_100_ee_1x100.png HTTP/1.1"
Nov 19 15:05:00 anongoth haproxy[11792]: 46.248.161.165:54997
[19/Nov/2015:15:04:58.459] https-in owncloud/node1 27/0/0/1876/1931
200 1076 - -  3/3/0/1/0 0/0 "POST

Re: Owncloud through Haproxy makes upload not possible

[Janusz Dziemidowicz]

2015-11-19 11:13 GMT+01:00 Piotr Kubaj :
>> 4096 bit DH params will be pretty slow to handshake. Maybe that's
>> okay in your circumstance though since you seem to be using this
>> for a personal use and not expecting a high connection rate. You
>> also have a 8 kbit RSA self signed certificate and using 256 bit
>> ciphers which increase TLS overhead.
> I want it to be secure, and I don't want to touch my settings for
> quite a while so I just took the strongest algorithms there are, and
> 2x recommended values for things like private key, or DH params. The
> hardware is pretty powerful and I've already checked that I don't have
> a huge load.

Take note, that increasing RSA size twice reduces number of
connections you can accept 10 times or more.
For example, my quite powerful desktop with recent CPU can accept 973
connections per second per core for RSA 2048, 136 connections per
second for RSA 4096. OpenSSL does not have tests for RSA 8192, but
that would be in the order of a few connections per second. RSA 8192
is really an overkill, it would be possible to DoS your server with a
simple shell script;) If you want a state of the art cryptography you
should probably use ECDSA certificate, it will be both faster and more
secure.

-- 
Janusz Dziemidowicz

Re: Owncloud through Haproxy makes upload not possible

[Bryan Talbot]

On Wed, Nov 18, 2015 at 3:45 AM, Piotr Kubaj  wrote:

> Hi,
>
> I've got a home server with 1 public IP, on which I host a couple of my
> websites. Each of them is in a separate jail. Haproxy listens on the
> outgoing IP and directs the traffic to the appropriate jail. Each of my
> websites works fast. However, if uploading files in Owncloud goes VERY
> slow and in the end I get a Bad Gateway error if the file is larger than
> ~100KB. Smaller files go through, but slowly. If I make the Owncloud
> jail listen on the external IP and connect directly to it, there's no
> problem, so it must be something about Haproxy configuration.
>


Hard to guess what the issue is but haproxy logging would probably help.
However, the logging configuration is a bit of a mess as it's configured to
be "all on" and "all off" at the same time. There are also other
configuration oddities.



>
> My operating system is FreeBSD 10.2-RELEASE-p7/amd64. Each jail is at
> the same version. Haproxy is at 1.6.2 version. I'm not sure if that
> matters, but I use Lighttpd 1.4.37 as a WWW server.
>
> Below is my haproxy.conf:
> global
> ssl-default-bind-options no-sslv3 no-tls-tickets force-tlsv12
> ssl-default-bind-ciphers AES256+EECDH:AES256+EDH
> tune.ssl.default-dh-param 4096
>

4096 bit DH params will be pretty slow to handshake. Maybe that's okay in
your circumstance though since you seem to be using this for a personal use
and not expecting a high connection rate. You also have a 8 kbit RSA self
signed certificate and using 256 bit ciphers which increase TLS overhead.




> log /var/run/log local0 notice
>

Is that where the logging socket is on FreeBSD now? I haven't used FreeBSD
in quite a while.



> maxconn 4096
> user daemon
> group daemon
> daemon
>
> defaults
> modehttp
> option  httplog
> option  dontlognull
> option  forwardfor
> option  http-server-close
> option  httpclose
> option  tcplog
> option  dontlog-normal
>

You have both tcp logging and http logging enabled at the same time. In
addition, you also have all logging disabled with "dontlognull" and
"dontlog-normal". If all your proxies are HTTP like you've shown, just
enable httplog and remove the tcplog option. When troubleshooting, enable
logging at least normal connections.

You also do not want to use both httpclose and http-server-close since they
conflict. Remove option httpclose.

Timeouts are also missing and you should be getting warnings about that too.

[WARNING] 321/103152 (87887) : config : missing timeouts for proxy
'https-in'.
   | While not properly invalid, you will certainly encounter various
problems
   | with such a configuration. To fix this, please ensure that all
following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.




>
>
> frontend http-in
> bind 192.168.11.3:80
>  reqadd X-Forwarded-Proto:\ http
> redirect scheme https code 301 if !{ ssl_fc }
>

Why bother to add a request header which will never be used?




>
> frontend https-in
> option httplog
> option forwardfor
> option http-server-close
> option httpclose
>

Don't need to repeat all of the above since it should be set in defaults
above (if set properly).
With no "log global" you won't get any logs anyway and are probably seeing
a warning when haproxy checks the config or starts.

[WARNING] 321/103109 (87884) : config : log format ignored for proxy
'https-in' since it has no log address.






> rspadd Public-Key-Pins:\
> pin-sha256="1Pw5h93NOsPw6j/vaTYl5VvW9cmtuZXtNP3cVz10hKo=";\
> max-age=15768000;\ includeSubDomains
>

AFAIK, HPKP is only somewhat supported by only the most recent browser
releases. I believe that it's also ignored by them for certificates which
are self-signed or signed by a CA that is not in the browsers
system-defined CA set. Probably doesn't cause your issue but who knows --
it is still experimental.

The "http-response set-header" supported in haproxy 1.5 and later is more
powerful and easier to read than the old reqadd and rspadd features.




> bind 192.168.11.3:443 ssl crt /usr/local/etc/haproxy.pem ciphers
> AES256+EECDH:AES256+EDH force-tlsv12 no-sslv3
>

Don't need to repeat these options that are already set globally.




>
> backend 10amd64
> server node1 192.168.11.3:81 cookie A check
>
>

Setting sticky cookies and not using them is probably harmless but what's
the point?

-Bryan

Re: Owncloud through Haproxy makes upload not possible

[Janusz Dziemidowicz]

2015-11-18 19:45 GMT+01:00 Bryan Talbot :
> AFAIK, HPKP is only somewhat supported by only the most recent browser
> releases. I believe that it's also ignored by them for certificates which
> are self-signed or signed by a CA that is not in the browsers system-defined
> CA set. Probably doesn't cause your issue but who knows -- it is still
> experimental.

There is also one more detail people often miss about HPKP. In order
for HPKP to work, you MUST have a backup pin, that is a pin for a
certificate that is offline. That means at least two pins, otherwise
this whole header is ignored. See RFC7469 section 2.5. Also use tools
in browsers, like Chrome net internals, to verify that it is correctly
noted by the browser.

-- 
Janusz Dziemidowicz