Re: [PATCH] chunked requests
mån 2009-07-13 klockan 18:51 -0600 skrev Alex Rousskov: does not allow is not the same as does not specify. Those who think that what is not forbidden is allowed, say that GET requests may have a body... It would be nice for 2616bis to clarify that (one way or another), but they are probably focusing on more important stuff. Has already been clarified to explicitly state the relaxed view on this since quite some time back (year or more) For current text see: http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#message.body Regards Henrik
Re: Hello from Mozilla
tor 2009-07-02 klockan 17:59 -0700 skrev Jason Duell: 1) I have a quick question about the limitations that you put by default on the CONNECT method. squid.conf contains # Deny CONNECT to other than SSL ports http_access deny CONNECT !SSL_ports So squid by default only allows CONNECT to port 443. I assume this is a rule-of-thumb security measure, since you figured that port 443 is the only common legitimate case for CONNECT? How long has this been the default behavior? Do you know if other proxy servers out there also do this? As you already know it's a security measure. I would expect it to be implemented by quite many as it's mentioned in the original Netscape document where CONNECT was first specified. I am not sure if Netscape still has that document, but it was later published as an internet-draft http://tools.ietf.org/html/draft-luotonen-ssl-tunneling-03 Due to this fact, the proxy cannot verify that the protocol being spoken is really SSL, and so the proxy configuration should explicitly limit allowed connections to well-known SSL ports (such as 443 for HTTPS, 563 for SNEWS, as assigned by the Internet Assigned Numbers Authority). When CONNECT later entered HTTP standards-track via the TLS working group it was for a slightly different model however (HTTP/1.1 Upgrade mechanism), and that text ONLY talks about port 80 and the security risks of allowing other ports. http://tools.ietf.org/html/rfc2817#section-8.2 A generic TCP tunnel is fraught with security risks. First, such authorization should be limited to a small number of known ports. The Upgrade: mechanism defined here only requires onward tunneling at port 80. Second, since tunneled data is opaque to the proxy, there are additional risks to tunneling to other well-known or reserved ports. A putative HTTP client CONNECTing to port 25 could relay spam via SMTP, for example. And the HTTP Upgrade mechanism generally haven't taken off yet with most of the net stuck in the https model, so there haven't been much (any before you) demands for CONNECT to port 80. I noticed that in section 3.1.3 the spec relies implicitly on CONNECT being allowed to arbitrary ports. But this is not the case for default installs of squid, and thus I fear that the general approach may be flawed. As you can see above all specifications available for the CONNECT method recommends very strict rules regarding which ports is allowed. I suppose we could ask that you allow arbitrary CONNECT access (or at least to the well-known websockets ports: 80/81/815). But I'm guessing that wouldn't help much, as it would probably take many years for that change to roll out across the net. Indeed. And as discussed we would never ship a default config which allows arbitrary ports, just a carefully selected list of ports. For what it's worth neither 81 or 815 is registered with IANA, and the web-sockets draft you referenced has expired. Regards Henrik
Re: Hello from Mozilla
tis 2009-07-14 klockan 23:35 +0200 skrev Henrik Nordstrom: For what it's worth neither 81 or 815 is registered with IANA, and the web-sockets draft you referenced has expired. Correction, the draft is not expired as it was recently resubmitted to IETF, but very much a work in progress. The correct link for reading up on websockets is http://dev.w3.org/html5/websockets/ and/or http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol There was a very brief discussion regarding websockets in the HTTPbis IETF WG in January. But it didn't spur very much interest among the HTTP people and discussions quickly died, not too surprisingly as it's not HTTP even if the initial connection setup is wrapped in an HTTP style message to be able to piggy-back on port 80. http://lists.w3.org/Archives/Public/ietf-http-wg/2009JanMar/0008.html The actual discussions regarding this family of wire protocols now take place on the HYBI IETF mailing list https://www.ietf.org/mailman/listinfo/hybi and from a quick scan of the discussions there apparently is several (two to four) proposals being discussed. and it's still very early to say where things will end up. There is not yet an official IETF working group dealing with this, just an informal gathering to discuss the problem area to see in what direction an eventual standardization effort should go. Regards Henrik
Re: Hello from Mozilla
ons 2009-07-08 klockan 07:00 + skrev Ian Hickson: Well, the client is a WebSocket client, so it can always generate the exact byte sequence specified in the spec almost by definition. The server side is restrictive, but should be implementable without too much trouble so long as there is a way to register a protocol handler that can prevent the HTTP side of the server from sending any bytes at all as part of the response. I agree that this is suboptimal, but we're somewhat forced into this by one of our security design requirements (that the handshake be very specific to avoid enabling request smuggling or other such behaviour -- basically we really want to be sure we're talking to a WebSocket server at the end of the handshake). If you use HTTP Upgrade as the initial handshake mechanism and expect this to be layered ontop of HTTP servers then you SHOULD use HTTP for this sequence. What takes place after the HTTP/1.1 response to the Upgrade is entirely yours, but until then it's HTTP. Hardcoding this sequence is just silly, and will cause a lot of trouble later on. You already touch some aspects of HTTP such as authentication. Now throw things like NTLM or Digest authentication into the mix and you will quickly find that the current specification is a bit too limiting if the intention is that the upgrade requests should be processed by an HTTP server. Also regarding HTTP Upgrade semantics. If you want to follow that model then there is not really any relation between the initial Upgrade request and what takes place after the upgrade has completed. It's a complete switch of protocols, with their own semantics. Regards Henrik
Re: [PATCH] Bug 2680: ** helper errors after -k rotate
lör 2009-07-11 klockan 20:08 +1200 skrev Amos Jeffries: It's most visible in rotate because Squid is intended to keep running with a hot-swap of its logs. Previously the sequence was causing two full sets of helpers to be started, and a period of overlap before the async closure of the old set happened. Yes, this is what Squid-2 does. It schedules the existing set to be closed (no more new requests assigned to them), and starts a new set. The old set dies as soon as it's completed doing what it's doing at the moment.. (including statful associations). But now that we are enforcing the number of helpers started to prevent memory bloat the overlap prevents the new set from starting at all. Hmm.. why? haven't been much of a problem in all the years we have been doing this.. except possibly for some poorly configured SquidGuard setups with large text lists without db backends... The safety checks in helper itself which prevent live restart attempts are tuned to only do so on incremental helper deaths, not to wholesale 100% loss of helpers. Yes. Our options are: 1 don't shutdown helpers on rotate (this ha been another open bug) 2 make helpers code extremely aggressive to keep helpers running (at cost of being able to detect failures) 3 make mainRotate async Or 4, go back to don't strictly enforce the number of helpers? Is there something I've missed that means some helpers MUST be restarted on rotate? Helpers are restarted because their stderr is connected to cache.log in append mode. Regards Henrik
Re: [PATCH] Bug 2680: ** helper errors after -k rotate
On Wed, 2009-07-15 at 00:52 +0200, Henrik Nordstrom wrote: Or 4, go back to don't strictly enforce the number of helpers? +1 I don't know what this strictly-enforce thing is, but it sounds unneeded as we used to fire up the right number of helpers anyway. -Rob signature.asc Description: This is a digitally signed message part
Re: Importing recent trunk changes to 3.1
Hello, Thank you to all who responded! It sounds like there is consensus that the changes should be imported. When I committed the changes to trunk, I merged them from the 3p1-plus branch on lp, using the cherrypicking method suggested by Robert: trunk% bzr merge -r branch:../v3.1..branch:3p1-plus ../3p1-plus The 3p1-plus branch is based on the official 3.1 branch and contains all of the approved changes. Thus, the above command means, roughly, merge the accumulated changes on 3p1-plus branch into trunk. [ Unfortunately, bzr does not copy commit history when cherrypicking (yet). So my plan to preserve that in trunk failed. ] I am not sure how we should import the changes to the official 3.1 branch now: 1) Merge from 3p1-plus (will preserve commit history, I think!) 2) Cherrypick from trunk (may help with future cherrupicking?) Robert, what is the best strategy here? Amos, will you do the import? The 3p1-plus branch, if you need it, is at https://code.launchpad.net/~rousskov/squid/3p1-plus Thank you, Alex. On 07/13/2009 05:04 AM, Henrik Nordstrom wrote: mån 2009-07-13 klockan 14:53 +1200 skrev Amos Jeffries: Lets kick the vote off now then +1 on both I suppose, assuming the code has been reasonably tested already and can easily be backported to 3.1. But ultimately it's your call. On the down side 3.1 is already quite delayed as it is... Another alternative is to plan for a short-term 3.2 feature release between 3.1 and 3.2 with reasonable feature backports where possible.. possibly even dropping some code from current 3.1 if that would help 3.1 getting out sooner. But this probably costs even more developer time. Regards Henrik
Re: Hello from Mozilla
On 07/14/2009 04:38 PM, Henrik Nordstrom wrote: ons 2009-07-08 klockan 07:00 + skrev Ian Hickson: Well, the client is a WebSocket client, so it can always generate the exact byte sequence specified in the spec almost by definition. The server side is restrictive, but should be implementable without too much trouble so long as there is a way to register a protocol handler that can prevent the HTTP side of the server from sending any bytes at all as part of the response. I agree that this is suboptimal, but we're somewhat forced into this by one of our security design requirements (that the handshake be very specific to avoid enabling request smuggling or other such behaviour -- basically we really want to be sure we're talking to a WebSocket server at the end of the handshake). If you use HTTP Upgrade as the initial handshake mechanism and expect this to be layered ontop of HTTP servers then you SHOULD use HTTP for this sequence. What takes place after the HTTP/1.1 response to the Upgrade is entirely yours, but until then it's HTTP. Hardcoding this sequence is just silly, and will cause a lot of trouble later on. I strongly agree with this. Frankly, I suspect (hope?) that IETF gurus would prevent publishing an RFC that kind of reuses the HTTP Upgrade mechanism but then declares many compliant Upgrade exchanges invalid. If you think your approach is the right one, I would suggest openly discussing it with the right IETF folks as early as possible, to avoid wasting your time on an idea they will be blocked later. HTTP hard-coding seems to be a small, albeit critical, part of WebSocket so changing it to avoid conflicts with HTTP may be possible without significant negative effects on the rest of the draft. Good luck, Alex. You already touch some aspects of HTTP such as authentication. Now throw things like NTLM or Digest authentication into the mix and you will quickly find that the current specification is a bit too limiting if the intention is that the upgrade requests should be processed by an HTTP server. Also regarding HTTP Upgrade semantics. If you want to follow that model then there is not really any relation between the initial Upgrade request and what takes place after the upgrade has completed. It's a complete switch of protocols, with their own semantics. Regards Henrik
Re: Hello from Mozilla
On Tue, 14 Jul 2009, Alex Rousskov wrote: If you think your approach is the right one, I would suggest openly discussing it with the right IETF folks as early as possible, to avoid wasting your time on an idea they will be blocked later. WebSocket is being discussed in the hybi IETF list. HTTP hard-coding seems to be a small, albeit critical, part of WebSocket so changing it to avoid conflicts with HTTP may be possible without significant negative effects on the rest of the draft. The handshake is a pretty critical part of the security model of the WebSocket protocol. I don't really see how we can continue to have the safe handshake while allowing either the client or the server to send any arbitrary string. WebSocket isn't an HTTP-upgraded protocol; it's just that its handshake happens to be such that it can trick HTTP servers into thinking that it is. In other words, HTTP Upgrade is not the initial handshake mechanism, it just looks like it is if you don't examine it closely. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
On 07/14/2009 06:01 PM, Ian Hickson wrote: On Tue, 14 Jul 2009, Alex Rousskov wrote: HTTP hard-coding seems to be a small, albeit critical, part of WebSocket so changing it to avoid conflicts with HTTP may be possible without significant negative effects on the rest of the draft. The handshake is a pretty critical part of the security model of the WebSocket protocol. I don't really see how we can continue to have the safe handshake while allowing either the client or the server to send any arbitrary string. Perhaps the WebSocket secure handshake should start _after_ the successful upgrade? Is not that how HTTP Upgrade was intended to be used? WebSocket isn't an HTTP-upgraded protocol; it's just that its handshake happens to be such that it can trick HTTP servers into thinking that it is. In other words, HTTP Upgrade is not the initial handshake mechanism, it just looks like it is if you don't examine it closely. I think I understand the intent behind the trick, but if a message looks like an HTTP message to the HTTP server or intermediary, the HTTP server or intermediary may start doing HTTP-valid things to it, and those things will not be valid from WebSocket point of view. Henrik provided a few examples of that already. I want to avoid the following Squid bug report 5 years from now: Title: Squid breaks FooBar Comment1: FooBar, a WebSocket application, works fine unless there is a transparent Squid proxy in the way. I have attached a packet trace. Comment2: Closed as invalid. Squid seems to be handling the HTTP messages correctly. Squid is not responsible to what happens after Upgrade, inside the tunnel. Comment3: Reopened as critical. I need to make this work. Please do something! I have read somewhere that if Squid would not add a Via or XFF header and also that extra space character after a column, everything would just work! Comment4: Changed to enhancement: Rewrite Squid to support WebSocket-compatible Upgrade exchange. Meanwhile, consider writing a post-Squid eCAP module that will rewrite Squid-altered HTTP messages to conform to WebSocket. Cheers, Alex.
Re: Hello from Mozilla
On Tue, 14 Jul 2009, Alex Rousskov wrote: On 07/14/2009 06:01 PM, Ian Hickson wrote: On Tue, 14 Jul 2009, Alex Rousskov wrote: HTTP hard-coding seems to be a small, albeit critical, part of WebSocket so changing it to avoid conflicts with HTTP may be possible without significant negative effects on the rest of the draft. The handshake is a pretty critical part of the security model of the WebSocket protocol. I don't really see how we can continue to have the safe handshake while allowing either the client or the server to send any arbitrary string. Perhaps the WebSocket secure handshake should start _after_ the successful upgrade? Is not that how HTTP Upgrade was intended to be used? If there are any bytes allowed from the client or the server before the handshake starts, then it is no longer secure. The idea is to make sure you can't smuggle though payloads from other protocols, since otherwise you could use WebSocket to connect to services that aren't expecting it. WebSocket isn't an HTTP-upgraded protocol; it's just that its handshake happens to be such that it can trick HTTP servers into thinking that it is. In other words, HTTP Upgrade is not the initial handshake mechanism, it just looks like it is if you don't examine it closely. I think I understand the intent behind the trick, but if a message looks like an HTTP message to the HTTP server or intermediary, the HTTP server or intermediary may start doing HTTP-valid things to it, and those things will not be valid from WebSocket point of view. Right -- and those things will prevent the connection, exactly as intended. The idea is to make sure that if there is anything in between that _isn't_ WebSocket-aware, the connection be dropped before the author has any chance of sending data down the pipe. If we didn't do this, then it's possible that, e.g., Squid could be tricked into sending data from third-party servers in a way that the WebSocket client could not detect, thus breaching the same-origin security model. I want to avoid the following Squid bug report 5 years from now: Title: Squid breaks FooBar Comment1: FooBar, a WebSocket application, works fine unless there is a transparent Squid proxy in the way. I have attached a packet trace. Comment2: Closed as invalid. Squid seems to be handling the HTTP messages correctly. Squid is not responsible to what happens after Upgrade, inside the tunnel. Comment3: Reopened as critical. I need to make this work. Please do something! I have read somewhere that if Squid would not add a Via or XFF header and also that extra space character after a column, everything would just work! Comment4: Changed to enhancement: Rewrite Squid to support WebSocket-compatible Upgrade exchange. Meanwhile, consider writing a post-Squid eCAP module that will rewrite Squid-altered HTTP messages to conform to WebSocket. If Squid changes the bytes that are sent from the client or by the server, then this bug will likely exist, yes. So don't do that. :-) If a man-in- the-middle proxy server modifies the bytes of connections it doesn't fully understand, then WebSocket isn't the only thing that will break. (Note that a min-in-the-middle proxy is not the same as a transparent proxy as defined by the HTTP spec. Per spec, a transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification.) -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: Hello from Mozilla
ons 2009-07-15 klockan 00:51 + skrev Ian Hickson: Right -- and those things will prevent the connection, exactly as intended. The idea is to make sure that if there is anything in between that _isn't_ WebSocket-aware, the connection be dropped before the author has any chance of sending data down the pipe. If we didn't do this, then it's possible that, e.g., Squid could be tricked into sending data from third-party servers in a way that the WebSocket client could not detect, thus breaching the same-origin security model. Which is a clear sign that you MUST NOT use a message that even resembles a HTTP message for the WebSocket handshake if you don't think that the HTTP model can accomplish the protocol you need. As all processing MUST bypass the HTTP stack of the server anyway the message may in fact look any way you like as long as it can be identified.. But I honestly don't get what you are talking about here. How would using HTTP Upgrade as intended possibly break the WebSocket security model? The WebSocket client has full control over the handshake, and is responsible for sending it as a well formed HTTP message and also parsing the HTTP response and should guarantee that it DOES NOT deliver any non-websocket payload (i.e. 407 responses, 5xx responses etc) to the WebSocket API user... If Squid changes the bytes that are sent from the client or by the server, then this bug will likely exist, yes. So don't do that. :-) If a man-in- the-middle proxy server modifies the bytes of connections it doesn't fully understand, then WebSocket isn't the only thing that will break. Squid operates on HTTP messages. While doing so HTTP REQUIRES it to modify the message in certain ways, and allows it to modify it in several other ways. - Addition of Via header (MUST) - Various other informative headers added by Squid. (fully allowed) But it's not my point. What I said is not even related to Squid but general protocol design and your design goals with the websocket wire protocol handshake resembling HTTP. (Note that a min-in-the-middle proxy is not the same as a transparent proxy as defined by the HTTP spec. Per spec, a transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification.) Err, a semantically transparent proxy still modifies the request in many ways. Via headers is the most obvious, but there is many many other small things going on. It's just not allowed to change the request or response in such way that the request/response means something else. But yes, a transparently intercepting proxy is a different thing and what Alex meant. Regards Henrik
Re: Hello from Mozilla
On 07/14/2009 06:51 PM, Ian Hickson wrote: I want to avoid the following Squid bug report 5 years from now: Title: Squid breaks FooBar Comment1: FooBar, a WebSocket application, works fine unless there is a transparent Squid proxy in the way. I have attached a packet trace. Comment2: Closed as invalid. Squid seems to be handling the HTTP messages correctly. Squid is not responsible to what happens after Upgrade, inside the tunnel. Comment3: Reopened as critical. I need to make this work. Please do something! I have read somewhere that if Squid would not add a Via or XFF header and also that extra space character after a column, everything would just work! Comment4: Changed to enhancement: Rewrite Squid to support WebSocket-compatible Upgrade exchange. Meanwhile, consider writing a post-Squid eCAP module that will rewrite Squid-altered HTTP messages to conform to WebSocket. If Squid changes the bytes that are sent from the client or by the server, then this bug will likely exist, yes. So don't do that. :-) From HTTP point of view, some byte changes are not really changes and some are actually required. Squid changes HTTP headers today and will probably continue to do so 5 years from now, even when deployed as a man in the middle. If a man-in- the-middle proxy server modifies the bytes of connections it doesn't fully understand, then WebSocket isn't the only thing that will break. WebSocket made the handshake bytes look like something Squid thinks it understands. That is the whole point of the argument. You are sending an HTTP-looking message that is not really an HTTP message. I think this is a recipe for trouble, even though it might solve some problem in some environments. (Note that a min-in-the-middle proxy is not the same as a transparent proxy as defined by the HTTP spec. Per spec, a transparent proxy is a proxy that does not modify the request or response beyond what is required for proxy authentication and identification.) That is why I quoted transparent. There is little transparent about most transparent hijacking proxies. It is the reality that is not covered by the HTTP specs but WebSocket will have to deal with it. Moreover, I suspect similar arguments can be applied to surrogates and even forward proxies that are covered by the specs and the specs allow them to modify HTTP messages. Cheers, Alex.
Re: Hello from Mozilla
On Tue, 14 Jul 2009, Alex Rousskov wrote: WebSocket made the handshake bytes look like something Squid thinks it understands. That is the whole point of the argument. You are sending an HTTP-looking message that is not really an HTTP message. I think this is a recipe for trouble, even though it might solve some problem in some environments. Could you elaborate on what bytes Squid thinks it should change in the WebSocket handshake? -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
