RE: mode tcp with mode http/shoutcast radio issue
This worked. Thanks for the help Willy. -Original Message- From: Willy Tarreau [] Sent: Friday 11, March 03, 2011 10:39 To: Florescu, Dan Alexandru Cc: haproxy@formilux.org Subject: Re: mode tcp with mode http/shoutcast radio issue On Wed, Mar 09, 2011 at 04:41:49PM +0200, Florescu, Dan Alexandru wrote: > This works. Problem is that I cannot use acl's like the following: > hdr(User-Agent) -i WinampMPEG/5.60 > hdr(Host) -i 80.86.106.35 > > I guess this is because it receives the tcp stream and doen't analyze headers > anymore. If the requests are enough HTTP compliant, then you might be able to use the above in TCP mode provided you wait for a complete HTTP request. It works like this : acl winamp hdr(User-Agent) -i WinampMPEG/5.60 tcp-request inspect-delay 5s tcp-request content accept if HTTP use-backend XXX if winamp ... The idea is that the "tcp-request accept" ACL will wait until the contents parse as a valid HTTP request (at most 5s). Once this is OK, the ACLs relying on headers will be able to match. Regards, Willy The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.com/disclaimer.html
RE: How to deal with proxying shoutcast / ICY protocol
I do not think you can see this with mode tcp, because of the lack of
X-Forwarded-For.
Your request could have been possible with mode http with the following
settings:
in haproxy -> option forwardfor (I see you already have this enabled)
in squid -> (in this order)
logformat squid [%{%d/%m/%Y-%H:%M:%S}tl] %{X-Forwarded-For}>h through %>a
%Ss/%03Hs %rm/%Hs %ru %Sh/%h
cache_access_log /var/log/squid3/access.log squid
and disable cache_access_log if it is enabled.
From the logformat above, the essential parts are:
%{X-Forwarded-For}>h = read the X-Forwarded-For header in the http request
%>a = display originating server or peer IP address
for more reference consult http://devel.squid-cache.org/customlog/logformat.html
Do try this with mode tcp as I am currently unable to.
-Original Message-
From: David Young
Sent: Friday 11, March 03, 2011 2:05
To: haproxy@formilux.org
Subject: Re: How to deal with proxying shoutcast / ICY protocol
Works as advertised, thank you :)
One thing is now missing from my setup, and that's the ability to
identify WHICH source IPs are accessing WHICH sites.
Without haproxy, I'm able to do this with squid's access.log, and with
"mode http", I'm able to see this in the haproxy logs.
When I enable "mode tcp" though, to work around this silly shoutcast
issue, I get source IPs _only_ in the haproxy.log, and my squid access
log reports every request has having a source IP of my haproxy host.
Any ideas?
Thanks,
D
On 10/03/11 10:22 PM, David Young wrote:
> Hi Willy, Dan,
>
> Thanks for pointing out that I was still in http mode - that was an
> oversight, and should solve my problem for now. Long term, I'd rather
> use http mode over tcp for flexibility, but unfortunately I can't get
> away from having to support Shoutcast (and any other dodgy protocol you
> can think of!) :(
>
> Dan - I guess we're stuck making backend decisions on destination IP
> only - any reason you're using Squid 3.0 over 3.1?
>
> I'll test in the lab tomorrow and report back results :)
>
> D
>
>
>
> On 10/03/11 9:43 PM, Florescu, Dan Alexandru wrote:
>> Hi David,
>>
>> I'm kind of amazed that we have both the same problem (
>> http://www.formilux.org/archives/haproxy/1103/4397.html ) in almost the same
>> circumstances, and we both asked for help at the same time :)
>>
>> I too am performing LB between proxies.
>> The issue with ICY is also in Squid. You need the upgrade_http0.9 (
>> http://www.squid-cache.org/Doc/config/upgrade_http0.9/ ) option. But this
>> one works only with Squid 2.7 and 3.1, not with 3.0. I myself have a 3.0 for
>> the main Squid, so I thought I would balance it with Haproxy (elegant
>> solution) to a Squid 3.1.
>>
>> If I try streaming directly on the 3.1 Squid, it will work, but through
>> Haproxy it will fail.
>> It will work through Haproxy if you enable mode tcp. Now, I see you have
>> enabled mode tcp in frontend, but you didn't do this for the backend aswell.
>> You also have a mode http in the defaults section, so if you don't
>> explicitly specify it in the backend, it will get the default-specified mode
>> (http in your case).
>> so: mode tcp in backend and you're ok.
>>
>> I myself have another issue, I cannot send the requests for ICY/radio
>> streaming through my second squid because I don't have any relevant options
>> to use in ACLs, so all requests are sent to the default_backend.
>>
>>
>> -Original Message-
>> From: David Young [
>> Sent: Thursday 10, March 03, 2011 4:36
>> To: haproxy@formilux.org
>> Subject: How to deal with proxying shoutcast / ICY protocol
>>
>> Hi folks,
>>
>> First-time poster here - we've been working on implementing haproxy to
>> perform load balancing between our backend squid proxies.
>>
>> I stumbled across an issue today when I realized that I can't listen to
>> streaming radio via my haproxy instance, but that it works fine if I
>> direct my request to one of by backend proxies directly.
>>
>> A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the
>> probable cause
>> (http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10).
>>
>> I thought that if I configured haproxy in "mode tcp" instead of "mode
>> http", that it'd behave more like a simple loadbalancer, and just pass
>> the request onto my squid backend verbatim, but that doesn't seem to
>> have worked either.
>>
>> An example URL I'm having
RE: How to deal with proxying shoutcast / ICY protocol
>> I guess we're stuck making backend decisions on destination IP Well, true enough, BUT: when you use a proxy in, let's say Winamp, and add a stream server and play it, the Destination IP seen by Haproxy will not be the stream server's one (external IP), it will be the Proxy server's one (internal IP) - because you are in fact establishing a connection to Squid and it connects afterwards to the intended destination. This is why I can't use this method. >> any reason you're using Squid 3.0 over 3.1? It's a production environment with several hundred people using it at the same time, so I can't upgrade just yet. -Original Message- From: David Young Sent: Thursday 10, March 03, 2011 11:22 To: haproxy@formilux.org Subject: Re: How to deal with proxying shoutcast / ICY protocol Hi Willy, Dan, Thanks for pointing out that I was still in http mode - that was an oversight, and should solve my problem for now. Long term, I'd rather use http mode over tcp for flexibility, but unfortunately I can't get away from having to support Shoutcast (and any other dodgy protocol you can think of!) :( Dan - I guess we're stuck making backend decisions on destination IP only - any reason you're using Squid 3.0 over 3.1? I'll test in the lab tomorrow and report back results :) D On 10/03/11 9:43 PM, Florescu, Dan Alexandru wrote: > Hi David, > > I'm kind of amazed that we have both the same problem ( > http://www.formilux.org/archives/haproxy/1103/4397.html ) in almost the same > circumstances, and we both asked for help at the same time :) > > I too am performing LB between proxies. > The issue with ICY is also in Squid. You need the upgrade_http0.9 ( > http://www.squid-cache.org/Doc/config/upgrade_http0.9/ ) option. But this one > works only with Squid 2.7 and 3.1, not with 3.0. I myself have a 3.0 for the > main Squid, so I thought I would balance it with Haproxy (elegant solution) > to a Squid 3.1. > > If I try streaming directly on the 3.1 Squid, it will work, but through > Haproxy it will fail. > It will work through Haproxy if you enable mode tcp. Now, I see you have > enabled mode tcp in frontend, but you didn't do this for the backend aswell. > You also have a mode http in the defaults section, so if you don't explicitly > specify it in the backend, it will get the default-specified mode (http in > your case). > so: mode tcp in backend and you're ok. > > I myself have another issue, I cannot send the requests for ICY/radio > streaming through my second squid because I don't have any relevant options > to use in ACLs, so all requests are sent to the default_backend. > > > -Original Message- > From: David Young [ > Sent: Thursday 10, March 03, 2011 4:36 > To: haproxy@formilux.org > Subject: How to deal with proxying shoutcast / ICY protocol > > Hi folks, > > First-time poster here - we've been working on implementing haproxy to > perform load balancing between our backend squid proxies. > > I stumbled across an issue today when I realized that I can't listen to > streaming radio via my haproxy instance, but that it works fine if I > direct my request to one of by backend proxies directly. > > A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the > probable cause > (http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10). > > I thought that if I configured haproxy in "mode tcp" instead of "mode > http", that it'd behave more like a simple loadbalancer, and just pass > the request onto my squid backend verbatim, but that doesn't seem to > have worked either. > > An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3 > > Even in "mode tcp", if I request that URL through by browser from > haproxy, I get a 502 error, whereas if I requested it directly from the > backend squid instance which services my request anyway, I get the > expected headers and stream. > > I thought I'd try telnetting to the haproxy / squid ports respectively, > to test, and I received differing output (below). > > Can anybody enlighten me? > > Many thanks, > David > > -- Relevant haproxy.cfg -- > > #- > # Global settings > #- > global > log 127.0.0.1 local2 > log /dev/log local2 > chroot /var/lib/haproxy > pidfile /var/run/haproxy.pid > maxconn 4000 > userhaproxy > group haproxy > daemon > > #--
RE: How to deal with proxying shoutcast / ICY protocol
Hi David, I'm kind of amazed that we have both the same problem ( http://www.formilux.org/archives/haproxy/1103/4397.html ) in almost the same circumstances, and we both asked for help at the same time :) I too am performing LB between proxies. The issue with ICY is also in Squid. You need the upgrade_http0.9 ( http://www.squid-cache.org/Doc/config/upgrade_http0.9/ ) option. But this one works only with Squid 2.7 and 3.1, not with 3.0. I myself have a 3.0 for the main Squid, so I thought I would balance it with Haproxy (elegant solution) to a Squid 3.1. If I try streaming directly on the 3.1 Squid, it will work, but through Haproxy it will fail. It will work through Haproxy if you enable mode tcp. Now, I see you have enabled mode tcp in frontend, but you didn't do this for the backend aswell. You also have a mode http in the defaults section, so if you don't explicitly specify it in the backend, it will get the default-specified mode (http in your case). so: mode tcp in backend and you're ok. I myself have another issue, I cannot send the requests for ICY/radio streaming through my second squid because I don't have any relevant options to use in ACLs, so all requests are sent to the default_backend. -Original Message- From: David Young [ Sent: Thursday 10, March 03, 2011 4:36 To: haproxy@formilux.org Subject: How to deal with proxying shoutcast / ICY protocol Hi folks, First-time poster here - we've been working on implementing haproxy to perform load balancing between our backend squid proxies. I stumbled across an issue today when I realized that I can't listen to streaming radio via my haproxy instance, but that it works fine if I direct my request to one of by backend proxies directly. A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the probable cause (http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10). I thought that if I configured haproxy in "mode tcp" instead of "mode http", that it'd behave more like a simple loadbalancer, and just pass the request onto my squid backend verbatim, but that doesn't seem to have worked either. An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3 Even in "mode tcp", if I request that URL through by browser from haproxy, I get a 502 error, whereas if I requested it directly from the backend squid instance which services my request anyway, I get the expected headers and stream. I thought I'd try telnetting to the haproxy / squid ports respectively, to test, and I received differing output (below). Can anybody enlighten me? Many thanks, David -- Relevant haproxy.cfg -- #- # Global settings #- global log 127.0.0.1 local2 log /dev/log local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 userhaproxy group haproxy daemon #- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #- defaults modehttp log global option dontlognull option http-server-close option httplog # Log on start of connection, not completion option logasap # log errors separately option log-separate-errors option forwardfor option redispatch timeout connect 1 # default 10 second time out if a backend is not found timeout client 30 timeout server 30 maxconn 6 retries 3 #- # main frontend which proxys to the backends #- frontend test_frontend *:5001 mode tcp option tcpka default_backend test_backend backend test_backend balance roundrobin server localhost localhost:3128 -- Telnet to haproxy infront of single squid tcp backend -- DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 5001 Trying x.x.x.x... Connected to webscan-dev.blahblah.net. Escape character is '^]'. GET http://66.225.205.47/;stream.mp3 HTTP/1.0 502 Bad Gateway Cache-Control: no-cache Connection: close Content-Type: text/html 502 Bad Gateway The server returned an invalid or incomplete response. Connection closed by foreign host. DavidBook:~ davidy$ -- Telnet to squid directly -- DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 3128 Trying x.x.x.x... Connected to webscan-dev.blahblah.net. Escape character is '^]'. GET http://66.225.205.47/;stream.mp3 ICY 200 OK Date: Thu, 10 Mar 2011 02:25:55 GMT icy-notice1: This stream requires http://www.winamp.com/";>Winamp icy-notice2: SHOUTcast Distributed Network Audio Server/Linux v1.9.8
mode tcp with mode http/shoutcast radio issue
Hello, I'm trying to use: frontend front mode http and backend web mode http backend radio mode tcp But will error with Unable to use proxy with wrong mode, required: http, has: tcp. Ok, this is understandable, so I changed to: frontend front mode tcp and backend web mode http backend radio mode tcp This works. Problem is that I cannot use acl's like the following: hdr(User-Agent) -i WinampMPEG/5.60 hdr(Host) -i 80.86.106.35 I guess this is because it receives the tcp stream and doen't analyze headers anymore. So I tried to match at destination ip using dst 80.86.106.35. This also fails because I am using a proxy, and the connection is in fact being addressed to the proxy ip, not the external host that I am trying to contact. If I use dst $proxy_ip, it matches. But it matchess _ALL_ connections, and I want to separate http ones from radio streaming. Also, trying to use mode http for the backend radio will give me 502 Bad Gateway errors when trying to connect to Shoutcast radio servers. mode tcp is the only way I could make it work. Also found a patch for haproxy+icey but I'm not very keen on recompiling. Any ideas? The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.com/disclaimer.html
RE: HAproxy and instant messagers (Ym)
Hi Willy, Thank you for the explanations. In fact I had assumed that CD/cD was something related to timing out and connection closing, and so I increased contimeout from 5000 (5ms) to 5 (5s) (BTW, I think that was a typo in my config). This has solved my problem. Although, looking at the timers from those three lines, nothing is below 5 seconds (6, 50003, 57935). Maybe I missed the specific lines in the log. Anyway, it works now. Cheers. -Original Message- From: Willy Tarreau Sent: Monday 14, February 02, 2011 12:41 Subject: Re: HAproxy and instant messagers (Ym) Hi, On Mon, Feb 14, 2011 at 10:35:15AM +0200, Florescu, Dan Alexandru wrote: > Anyone willing to help? It's possible that people are not much skilled in this application :-/ > Also, if I can get some help interpreting the following lines extracted from > haproxy.log: It's a good point you got the logs. Here are the explanations : > balance roundrobin/ proxy 0/0/1/26004/6 200 2054 - - CD-- > 470/470/456/456/0 0/0 "POST http://98.139.60.242/ HTTP/1.1" > balance roundrobin/ proxy 0/0/1/-1/50003 504 194 - - sH-- 490/490/489/489/0 > 0/0 "POST http://98.139.60.242/ HTTP/1.1" > balance roundrobin/ proxy 0/0/0/2/57935 407 47897 - - cD-- 531/531/528/528/0 > 0/0 "GET http://i1.ytimg.com/i/ HTTP/1.1" > specifically, what do those flags mean? e.g. CD, sH, cD CD means the Client closed the connection during the Data phase. I don't know how Ym works but it's possible that both sides exchange small chunks, and the client has actively stopped doing so (eg: click Stop, close the browser window, ...). sH means the server has failed to respond to a request within the configured timeout. In your case, the server did not say anything within 50 seconds. Since you're connecting through Squid, I suspect that in fact squid failed to establish a connection to the server (server may be overloaded) and that haproxy timed out first. You can increase your haproxy timeouts if you want, so that you get the Squid's error in the response. cD means the client timed out during the data phase. Most likely there were data to read in the buffer but the client did not care to read them, and haproxy timed out after 58 seconds. Since haproxy is between a browser and a squid, it needs to be configured to look for the Proxy-Connection header instead of the Connection header. I think it is what is causing these issues, because the client and squid can't agree on what scheme to use (close/keep-alive). You have to enable "option http-use-proxy-header" for that. Also, you'd better use "option http-server-close" than "httpclose", as it will maintain the connection alive on the client side. Regards, Willy The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.com/disclaimer.html
RE: HAproxy and instant messagers (Ym)
Anyone willing to help? Also, if I can get some help interpreting the following lines extracted from haproxy.log: balance roundrobin/ proxy 0/0/1/26004/6 200 2054 - - CD-- 470/470/456/456/0 0/0 "POST http://98.139.60.242/ HTTP/1.1" balance roundrobin/ proxy 0/0/1/-1/50003 504 194 - - sH-- 490/490/489/489/0 0/0 "POST http://98.139.60.242/ HTTP/1.1" balance roundrobin/ proxy 0/0/0/2/57935 407 47897 - - cD-- 531/531/528/528/0 0/0 "GET http://i1.ytimg.com/i/ HTTP/1.1" specifically, what do those flags mean? e.g. CD, sH, cD Thank you. Sent: Friday 11, February 02, 2011 13:09 To: 'haproxy@formilux.org' Subject: HAproxy and instant messagers (Ym) Hello, I'm using Haproxy to balance HTTP requests to Squid servers for web access. Http requests seem to work fine but when I connect to Yahoo messenger, I am randomly kicked out and logged in again. If I connect directly through Squid, I do not encounter these disconnects, so I am assuming it has something to do with my Haproxy config. Maybe it timeouts the connection.. I can't figure it out. Here is my config: defaults log global modehttp option httplog option dontlognull option redispatch option abortonclose retries 3 maxconn 9000 contimeout 5000 clitimeout 5 srvtimeout 5 frontend front bind 0.0.0.0: mode http option forwardfor default_backend back backend roundrobin mode http option httpclose balance roundrobin server proxy 192.168.100.5:3128 check inter 2000 fall 3 server proxy_bkp 192.168.100.6:3128 check inter 2000 fall 3 backup I also tried using server proxy 192.168.100.5:3128 cookie 2 check. Any suggestions (adding / removing options from the config)? Scope: being used by several hundred people for http browsing, voip and IM. The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.com/disclaimer.html
HAproxy and instant messagers (Ym)
Hello, I'm using Haproxy to balance HTTP requests to Squid servers for web access. Http requests seem to work fine but when I connect to Yahoo messenger, I am randomly kicked out and logged in again. If I connect directly through Squid, I do not encounter these disconnects, so I am assuming it has something to do with my Haproxy config. Maybe it timeouts the connection.. I can't figure it out. Here is my config: defaults log global modehttp option httplog option dontlognull option redispatch option abortonclose retries 3 maxconn 9000 contimeout 5000 clitimeout 5 srvtimeout 5 frontend front bind 0.0.0.0: mode http option forwardfor default_backend back backend roundrobin mode http option httpclose balance roundrobin server proxy 192.168.100.5:3128 check inter 2000 fall 3 server proxy_bkp 192.168.100.6:3128 check inter 2000 fall 3 backup I also tried using server proxy 192.168.100.5:3128 cookie 2 check. Any suggestions (adding / removing options from the config)? Scope: being used by several hundred people for http browsing, voip and IM. The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.com/disclaimer.html
