Re: WebSockets negotiation over HTTP
(cc'ing hybi by request, since this e-mail involves changes to the WebSocket protocol) On Thu, 22 Oct 2009, Mark Nottingham wrote: On 22/10/2009, at 10:52 AM, Ian Hickson wrote: Until the upgrade is complete, you're speaking HTTP and working with HTTP implementations. How so? A WebSocket client is always talking Web Socket, even if it might also sound like HTTP. Yes, but to someone examining the traffic -- whether in a debugger or an intermediary -- it looks, smells and quacks like HTTP. A person looking the traffic in a debugger will clearly see it's WebSocket, since it says WebSocket at least twice in the request (once on the second line). An intermediary is supposed to treat it as HTTP; that is part of how we get intermediaries to drop the connection (they should fail to recognise the Upgrade and drop the connection). It declares itself to be HTTP/1.1 by using the HTTP/1.1 protocol identifier in the request-line. What's the point of doing that -- i.e., why not use WebSockets/1.0? Wouldn't that be an inappropriate use of port 80? Have you verified that implementations (e.g., Apache module API) will give you byte-level access to what's on the wire in the request, and byte-level control over what goes out in the response? On the server side, you don't need wire-level control over what's coming in, only over what's going out. Yes, you do, because section 5.2 specifies headers as whitespace-sensitive and header-names as case-sensitive. You can completely ignore the incoming handshake, actually. Servers aren't required to do anything with the incoming handshake. There's no big interop issue with them handling whitespace differently in practice, so long as the client-side UA follows the spec to the letter. This looks an awful lot like a redirect. There's no redirection involved here. It's just confirming the opened URL, as part of the handshake. The TCP connection is not closed (unless the handshake fails, and then it's not reopened). I see now that you have the client-side fail a connection where the URL doesn't match, but that's really not obvious in 5.1. Please put some context in there and reinforce that the URL has to be the URL of the current script, not just any script. Ok, I've added a note at the end of that section explaining that the user agent will fail the connection if the strings don't match what the UA sent. Please let me know if you'd like anything else clarified; I don't really know exactly what should be made clearer. Did this get into -50? Don't see anything in the diff... Loos like there was some update issue. Try now: http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol#section-5.1 You can always find the most up-to-date text on the WHATWG site, if the IETF update process fails (or when there's an IETF meeting in progress, since apparently we're not allowed to make progress during IETF meetings): http://www.whatwg.org/specs/web-apps/current-work/complete.html#websocket-protocol The most effective way of doing this would be to actually define the new headers' semantics in your draft; Websocket-Location, for example, is only defined as client-side and server-side behaviours. I know this is probably intentional, but people will read all sorts of things into this header (especially since its name is so similar to other HTTP headers) unless you give some indication of what it means. I've tried adding more information to the sections that define headers, to describe what they do. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: WebSockets negotiation over HTTP
On Sat, 17 Oct 2009, Mark Nottingham wrote: On 17/10/2009, at 9:09 AM, Ian Hickson wrote: On Wed, 14 Oct 2009, Mark Nottingham wrote: Section 5.2 does constrain the bytes the server accepts from the client, thereby conflicting with HTTP, but only in some small details. In particular, it makes HTTP header field-names case-sensitive, and requires certain arrangements of whitespace in them. Ian, if you can address these small things in section 5.2 it would help. If a WebSocket client is connecting to a WebSocket server, then this isn't HTTP, it's just the WebSocket protocol. So whether the fields are parsed like HTTP is presumably not a problem. If an HTTP client is connecting to a WebSocket server, then the server's response is going to be garbage (from the HTTP client's perspective) anyway, much like if an HTTP client were to connect to an SMTP server. So how the server parses the fields doesn't really matter. If a WebSocket client is connecting to a WebSocket server, then the requirements in this section don't apply to the server. If an HTTP client is connecting to an HTTP server, then the whole spec doesn't apply. Which case is the one you are concerned about? Are my conclusions above incorrect? Until the upgrade is complete, you're speaking HTTP and working with HTTP implementations. How so? A WebSocket client is always talking Web Socket, even if it might also sound like HTTP. Have you verified that implementations (e.g., Apache module API) will give you byte-level access to what's on the wire in the request, and byte-level control over what goes out in the response? On the server side, you don't need wire-level control over what's coming in, only over what's going out. There's already a WebSocket module for Apache, by the way: http://code.google.com/p/pywebsocket/ Overall, I guess I'm just not seeing how running WebSockets on port 80 (i.e., co-existant with a HTTP server) is ever a good idea. I wouldn't recommend co-existing with a port 80 HTTP server. The co-existing support is really for port 443. Since a sizeable portion of the Internet is accessed through proxies (e.g., hotels, universities, corporations, mobile phones, and some ISPs), and none of the deployed infrastructure will support WebSockets, deploying in this fashion alone won't be workable on the open Internet; people using this technique will have to also deploy a fallback server on a different port. So, why bother, and why force people to write the code for fallback? What value is there in doing it this way? By and large, you can connect over port 443 without the proxy getting in the way. That's the model that I would expect most Web Socket deployments to use. Despite all of this, you say: The simplest method is to use port 80 to get a direct connection to a Web Socket server. Port 80 traffic, however, will often be intercepted by HTTP proxies, which can lead to the connection failing to be established. which I think is misleading; this is far from the simplest way to use WebSockets, from a deployment perspective. True. I've tried to reword this to avoid this possible ambiguity. The other aspect here is that you're really not using Upgrade in an appropriate fashion; as mentioned before, its intended use is to upgrade *this* TCP connection, not redirect to another one. There's only one TCP connection established. As far as I can tell, WebSocket never does a redirect of any kind. -48 5.1 says: Send the string WebSocket-Location followed by a U+003A COLON (:) and a U+0020 SPACE, followed by the URL of the Web Socket script, followed by a CRLF pair (0x0D 0x0A). For instance: WebSocket-Location: ws://example.com/demo NOTE: Do not include the port if it is the default port for Web Socket protocol connections of the type in question (80 for unencrypted connections and 443 for encrypted connections). This looks an awful lot like a redirect. There's no redirection involved here. It's just confirming the opened URL, as part of the handshake. The TCP connection is not closed (unless the handshake fails, and then it's not reopened). I see now that you have the client-side fail a connection where the URL doesn't match, but that's really not obvious in 5.1. Please put some context in there and reinforce that the URL has to be the URL of the current script, not just any script. Ok, I've added a note at the end of that section explaining that the user agent will fail the connection if the strings don't match what the UA sent. Please let me know if you'd like anything else clarified; I don't really know exactly what should be made clearer. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,.
Re: WebSockets negotiation over HTTP
On Mon, 19 Oct 2009, Amos Jeffries wrote: Ian Hickson wrote: On Wed, 14 Oct 2009, Amos Jeffries wrote: 4.1.13 still has a fragility issue in that it assumes the Upgrade: and Connection: headers will retain both their specific sending order and be the very first headers in the reply. It will work in most situations, but proxies which 'correct' the headers order to have Date: first will kill WebSockets. That's intentional; such proxies don't know about Web Sockets (if they did, they wouldn't be modifying the headers!) and thus clearly can't really be trusted to route the traffic unmodified. At this point of the handshake the client is the only software which knows it's using WebSockets. The server may validate-parse the headers mime syntax before sub-parsing the request line. At this point all its seen is the GET and HTTP/1.1. So... the server and any middleware will be in a state right now thinking that HTTP/1.1 is in use and will do appropriate HTTP/1.1 header alterations. It is not until the server reply accepting the Upgrade: request is received by middleware that WebSockets protocol actions can start happening. Agreed. I don't see how that affects my point though. 4.1 14 thru 4.1.23 appear to be a very conflated description of parsing the headers. It seems to me that referencing rfc2616 section 4.2 should be sufficient for the parse Unfortunately, HTTP doesn't define how to parse headers. It defines the semantics of valid headers, but doesn't say, e.g., what headers are present in the following: HTTP/1.1 200 OK : Bar Foo Quux Section 4.2 is clear: Each header field consists of a name followed by a colon (:) and the field value. Field names are case-insensitive. So what are the headers in the (invalid) HTTP response above? NP: WebSockets as of draft-49 requires (1.2) The first three lines in each case are hard-coded (the exact case and order matters) which is a breach of the final statement above. That final statement permits middeleware to uppercase or CamelCase the headers on a whim without altering their meaning. The entire point of the handshake is to detect such middleware and fail the connection when it is detected. References RFC822 section 3.1 for the BNF. Which states: B.1. SYNTAX message = *field *(CRLF *text) field =field-name : [field-body] CRLF field-name = 1*any CHAR, excluding CTLs, SPACE, and : field-body = *text [CRLF LWSP-char field-body] ... C.1.1. FIELD NAMES These now must be a sequence of printable characters. They may not contain any LWSP-chars. ... which requires a minimum of one ASCII byte header names which may not include ':' or whitespace or non-printables. NP: WebSockets draft-49 changes the bytes to UNICODE format and permits non-printables which are not LF or CR. Right. In your above demo request is HTTP/1.1 invalid: * first header line has no token in the field-name portion, * second line has CRLF in the name portion, * third line has zero-byte name portion. I am aware that it is invalid. My point is that HTTP doesn't define how it is to be parsed (it leaves it undefined), which is IMHO unacceptable for a protocol specification, and that is why WebSocket doesn't defer to HTTP. Since you have spec'd that only valid HTTP/1.1 is acceptable this will be dropped by any WebSockets aware software even if its accepted by WebSockets. Sure, but the following: HTTP/1.1 101 Web Socket Protocol Handshake Upgrade: WebSocket Connection: Upgrade WebSocket-Origin: http://example.com WebSocket-Location: ws://example.com/demo WebSocket-Protocol: sample : ...has defined processing on the client-side when it is sent back as a WebSocket handshake, while if I deferred to HTTP, it's handling would be undefined (and indeed a range of behaviours from allowing the connection to failing the connection altogether would be allowed, which is far too vague to lead to good interoperability!). For completeness the rest of rfc822sect3.1 used by rfc2616 specs: B.2. SEMANTICS Headers occur before the message body and are terminated by a null line (i.e., two contiguous CRLFs). A line which continues a header field begins with a SPACE or HTAB character, while a line beginning a field starts with a printable character which is not a colon. A field-name consists of one or more printable characters (excluding colon, space, and control-characters). A field-name MUST be contained on one line. Upper and lower case are not dis- tinguished when comparing field-names. .. the third clause there prohibits headers like your example Foo: FooCRLF : header textCRLF
Re: WebSockets negotiation over HTTP
On 22/10/2009, at 10:52 AM, Ian Hickson wrote: Until the upgrade is complete, you're speaking HTTP and working with HTTP implementations. How so? A WebSocket client is always talking Web Socket, even if it might also sound like HTTP. Yes, but to someone examining the traffic -- whether in a debugger or an intermediary -- it looks, smells and quacks like HTTP. It declares itself to be HTTP/1.1 by using the HTTP/1.1 protocol identifier in the request-line. What's the point of doing that -- i.e., why not use WebSockets/1.0? Have you verified that implementations (e.g., Apache module API) will give you byte-level access to what's on the wire in the request, and byte-level control over what goes out in the response? On the server side, you don't need wire-level control over what's coming in, only over what's going out. Yes, you do, because section 5.2 specifies headers as whitespace- sensitive and header-names as case-sensitive. There's already a WebSocket module for Apache, by the way: http://code.google.com/p/pywebsocket/ Cool. Despite all of this, you say: The simplest method is to use port 80 to get a direct connection to a Web Socket server. Port 80 traffic, however, will often be intercepted by HTTP proxies, which can lead to the connection failing to be established. which I think is misleading; this is far from the simplest way to use WebSockets, from a deployment perspective. True. I've tried to reword this to avoid this possible ambiguity. I see those changes in -50; looks good (and a very elegant change). Thanks. This looks an awful lot like a redirect. There's no redirection involved here. It's just confirming the opened URL, as part of the handshake. The TCP connection is not closed (unless the handshake fails, and then it's not reopened). I see now that you have the client-side fail a connection where the URL doesn't match, but that's really not obvious in 5.1. Please put some context in there and reinforce that the URL has to be the URL of the current script, not just any script. Ok, I've added a note at the end of that section explaining that the user agent will fail the connection if the strings don't match what the UA sent. Please let me know if you'd like anything else clarified; I don't really know exactly what should be made clearer. Did this get into -50? Don't see anything in the diff... The most effective way of doing this would be to actually define the new headers' semantics in your draft; Websocket-Location, for example, is only defined as client-side and server-side behaviours. I know this is probably intentional, but people will read all sorts of things into this header (especially since its name is so similar to other HTTP headers) unless you give some indication of what it means. -- Mark Nottingham m...@yahoo-inc.com
Re: WebSockets negotiation over HTTP
Ian Hickson wrote: On Wed, 14 Oct 2009, Amos Jeffries wrote: 4.1.12 prescribes semi-implicitly that HTTP/1.0 and HTTP/1.2 etc are not compatible. Maybe thats what you want. *very* minor enhancement would be to make that explicitly stated. I've added a note to this effect. 4.1.13 still has a fragility issue in that it assumes the Upgrade: and Connection: headers will retain both their specific sending order and be the very first headers in the reply. It will work in most situations, but proxies which 'correct' the headers order to have Date: first will kill WebSockets. That's intentional; such proxies don't know about Web Sockets (if they did, they wouldn't be modifying the headers!) and thus clearly can't really be trusted to route the traffic unmodified. At this point of the handshake the client is the only software which knows its using WebSockets. The server may validate-parse the headers mime syntax before sub-parsing the request line. At this point all its seen is the GET and HTTP/1.1. So... the server and any middleware will be in a state right now thinking that HTTP/1.1 is in use and will do appropriate HTTP/1.1 header alterations. It is not until the server reply accepting the Upgrade: request is received by middleware that WebSockets protocol actions can start happening. 4.1 14 thru 4.1.23 appear to be a very conflated description of parsing the headers. It seems to me that referencing rfc2616 section 4.2 should be sufficient for the parse Unfortunately, HTTP doesn't define how to parse headers. It defines the semantics of valid headers, but doesn't say, e.g., what headers are present in the following: HTTP/1.1 200 OK : Bar Foo Quux Section 4.2 is clear: Each header field consists of a name followed by a colon (:) and the field value. Field names are case-insensitive. NP: WebSockets as of draft-49 requires (1.2) The first three lines in each case are hard-coded (the exact case and order matters) which is a breach of the final statement above. That final statement permits middeleware to uppercase or CamelCase the headers on a whim without altering their meaning. References RFC822 section 3.1 for the BNF. Which states: B.1. SYNTAX message = *field *(CRLF *text) field =field-name : [field-body] CRLF field-name = 1*any CHAR, excluding CTLs, SPACE, and : field-body = *text [CRLF LWSP-char field-body] ... C.1.1. FIELD NAMES These now must be a sequence of printable characters. They may not contain any LWSP-chars. ... which requires a minimum of one ASCII byte header names which may not include ':' or whitespace or non-printables. NP: WebSockets draft-49 changes the bytes to UNICODE format and permits non-printables which are not LF or CR. In your above demo request is HTTP/1.1 invalid: * first header line has no token in the field-name portion, * second line has CRLF in the name portion, * third line has zero-byte name portion. Any one of which will be either dropped by existing middleware or handled as HTTP/0.9 with body content: : BarCRLF FooCRLF QuuxCRLF The first handling method is good the second may be a major headache. Since you have spec'd that only valid HTTP/1.1 is acceptable this will be dropped by any WebSockets aware software even if its accepted by WebSockets. For completeness the rest of rfc822sect3.1 used by rfc2616 specs: B.2. SEMANTICS Headers occur before the message body and are terminated by a null line (i.e., two contiguous CRLFs). A line which continues a header field begins with a SPACE or HTAB character, while a line beginning a field starts with a printable character which is not a colon. A field-name consists of one or more printable characters (excluding colon, space, and control-characters). A field-name MUST be contained on one line. Upper and lower case are not dis- tinguished when comparing field-names. .. the third clause there prohibits headers like your example Foo: FooCRLF : header textCRLF Supporting the second clause (LWS) will not affect the client sent data. But will help WebSockets cope with headers using very long Cookie data and long auth credentials. For Web Sockets I would like to have well-defined processing in the face of any input, even invalid input. I'd also like to not require that the processing for headers be as complicated as HTTP's (with continuation lines, multiple headers being merged, etc). Understood. I'm hoping the above spec 2616 + 822 segments are sufficiently clear for you on what is and is not permitted on the headers. Things which are not valid HTTP/1.1 as above are of course badly broken WebSockets as well. You can spec as a broad cover that non-valid HTTP/1.1 is a fail connection. and do away with 4.1.15 through 4.1.21. Similar to
Re: WebSockets negotiation over HTTP
On Wed, 14 Oct 2009, Mark Nottingham wrote: Section 5.2 does constrain the bytes the server accepts from the client, thereby conflicting with HTTP, but only in some small details. In particular, it makes HTTP header field-names case-sensitive, and requires certain arrangements of whitespace in them. Ian, if you can address these small things in section 5.2 it would help. If a WebSocket client is connecting to a WebSocket server, then this isn't HTTP, it's just the WebSocket protocol. So whether the fields are parsed like HTTP is presumably not a problem. If an HTTP client is connecting to a WebSocket server, then the server's response is going to be garbage (from the HTTP client's perspective) anyway, much like if an HTTP client were to connect to an SMTP server. So how the server parses the fields doesn't really matter. If a WebSocket client is connecting to a WebSocket server, then the requirements in this section don't apply to the server. If an HTTP client is connecting to an HTTP server, then the whole spec doesn't apply. Which case is the one you are concerned about? Are my conclusions above incorrect? The other aspect here is that you're really not using Upgrade in an appropriate fashion; as mentioned before, its intended use is to upgrade *this* TCP connection, not redirect to another one. There's only one TCP connection established. As far as I can tell, WebSocket never does a redirect of any kind. * Ian, are you just trying to exceed 100 drafts, thereby crashing the IETF? :) No, just following the release early release often policy. 48 is nothing; HTML5 itself is currently at release 4153. I've had more than 48 releases of HTML5 in the last four days. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: WebSockets negotiation over HTTP
On Fri, 4 Sep 2009, Robert Collins wrote: On Fri, 2009-09-04 at 01:44 +, Ian Hickson wrote: One very real example of this would be the web server or an fully WebSocket capable intermediary sending back bytes ... example #1 suppose there was an intermediary translating websockets-over-http to websockets-port-81 which used HTTP to format said headers of confirmation. Here is the problem I have with this: Why would we suppose the existence of such an intermediary? Why would anyone ever implementa WebSocket-specific proxy? Welcome to the internet :). Seriously, Why would we suppose the existence of an intermediary that intercepts and MITM's SSL connections? I wouldn't. That sounds like a security disaster. Or HTTP - its even got a defined proxy facility, there is no need to take over and insert different behaviour, is there? For HTTP it makes sense since you get edge caching gains. We *should* assume that firewall vendors and ISP's will do arguably insane things, because they have time and again in the past. Ok, but that doesn't mean we should go out of our way to make such an insane thing easier, especially at the cost of security. example #2 is where the traffic is processed by an HTTP-only intermediary which sees the 'Upgrade:' header and flags the connection for transparent pass-thru (This by the way is the desirable method of making Squid support WebSockets). Being a good HTTP relay it accepts these bytes: HTTP/1.1 101 Web Socket Protocol Handshake Upgrade: WebSocket Connection: Upgrade It violates HTTP by omitting the Via and other headers your spec omits to handle. And passes these on: HTTP/1.1 101 Web Socket Protocol Handshake Connection: Upgrade Upgrade: WebSocket then moves to tunnel mode for you. Why would it violate HTTP in all the ways you mention? If it goes to such extreme lengths to have transparent pass-through to the point of violating HTTP, why would it then go out of its way to reorder header lines? Because sysadmins do this! Not for long, since if they do, it simply won't work. :-) Don't ask us to justify the weird and wonderful things we run into. I'm not asking you to justify the weird things people do; I'm asking you to justify the request to reduce the security of the handshake to make it easier for people to do those weird things. From my perspective, such a proxy would raise all kinds of alarm bells to me, and I would be _glad_ that the connection failed. If it didn't, I wouldn't be sure we could trust the rest of the data. Again, welcome to the internet :P. Seriously, if its not digitally signed, you *can't* trust the rest of the data. Sure. I don't mean to say that I _would_ trust it otherwise, but that this would just be a warning sign to me. The MITM isn't the WebSocket client. In this situation, it's a (non-compliant, since it forwarded by-hop headers) HTTP proxy. What's more, in this scenario the server isn't a WebSocket server, either, it's an HTTP server. So what Web Socket says is irrelevant. Note that there are *still* HTTP/1.0 proxies in deployment that don't know about hop by hop headers. It seems highly unlikely that they would act as working WebSocket proxies, though, right? WebSockets, even when initiating the connection by talking to an HTTP server and then switching to WebSockets, isn't layered on HTTP. It's just doing the bare minimum required to allow the HTTP server to understand the request and get out of the way. Once the handshake is complete, there is no HTTP anywhere on the stack at all. Its not doing the bare minimum. The bare minimum would be to accept the valid HTTP transforms which the Internet _will_ perform on the handshake. Discard those useless transforms and validate the handshake status line. The bare minimum is the least amount of processing possible. What you describe is _more_ processing, not less. Therefore it's not the minimum. The bare minimum would be to just start the tcp connection with 'websocket/1.0\r\n' Stop pretending to be HTTP: use port 80. Our whole point has been, if you want to Use HTTP Upgrade, Do It Correctly. If you want to Use port 80, just do that. I want to just use port 80, and I want to make it possible for a suitably configured HTTP server to pass connections over to WebSocket servers. It seems to me that using something that looks like an HTTP Upgrade is better than just having a totally unrelated handshake, but I guess maybe we should just reuse port 80 without doing anything HTTP-like at all. For now I haven't changed the handshake, since it's not clear that what benefit it would bring, other than not looking like HTTP, which seems at most a theoretical advantage, and at worst a disadvantage. But for 2 you need to use HTTP, which essentially
Re: WebSockets negotiation over HTTP
On 13/10/2009, at 10:23 PM, Ian Hickson i...@hixie.ch wrote: I want to just use port 80, and I want to make it possible for a suitably configured HTTP server to pass connections over to WebSocket servers. It seems to me that using something that looks like an HTTP Upgrade is better than just having a totally unrelated handshake, but I guess maybe we should just reuse port 80 without doing anything HTTP-like at all. To be clear, upgrade is appropriate for changing an existing connection over to a new protocol (ie reusing it). To pass a request over to a different server, a redirect would be more appropriate (and is facilitated by the new uri scheme). (Ian I don't have your draft in front of me, so this isnt a comment on it necessarily, just a general statement). Cheers,
Re: WebSockets negotiation over HTTP
On Wed, 2009-10-14 at 09:59 +1100, Mark Nottingham wrote: On 13/10/2009, at 10:23 PM, Ian Hickson i...@hixie.ch wrote: I want to just use port 80, and I want to make it possible for a suitably configured HTTP server to pass connections over to WebSocket servers. It seems to me that using something that looks like an HTTP Upgrade is better than just having a totally unrelated handshake, but I guess maybe we should just reuse port 80 without doing anything HTTP-like at all. To be clear, upgrade is appropriate for changing an existing connection over to a new protocol (ie reusing it). To pass a request over to a different server, a redirect would be more appropriate (and is facilitated by the new uri scheme). Yup; and the major issue here is that websockets *does not want* the initial handshake to be HTTP. Rather it wants to be something not-quite HTTP, specifically reject a number of behaviours and headers that are legitimate HTTP. -Rob signature.asc Description: This is a digitally signed message part
Re: WebSockets negotiation over HTTP
Catching up to Ian's -48 draft*, I don't think there's much of a problem here -- or at least the spec can be brought into alignment with HTTP with a few small changes. However, the comment about upgrade vs. redirect stands (see below). Section 4.1 describes the handshake from the client side. It requires the client to send a request that's a subset of HTTP; this doesn't conflict with HTTP in and of itself. It also constrains the responses to the request that the client can expect, but that's OK because at this point we shouldn't be talking HTTP any more. It would be nice if clients were explicitly allowed to send other headers, e.g., Referer or User-Agent, but it's not critical. Also, by its nature this protocol is going to be fragile on non-CONNECTed HTTP connections, but Ian has already acknowledged this. Section 5.1 describes the handshake from the server side. It doesn't place any requirements on the bytes received from the client, only on those sent by the server, so again this is a proper subset of HTTP. Section 5.2 does constrain the bytes the server accepts from the client, thereby conflicting with HTTP, but only in some small details. In particular, it makes HTTP header field-names case-sensitive, and requires certain arrangements of whitespace in them. Ian, if you can address these small things in section 5.2 it would help. The other aspect here is that you're really not using Upgrade in an appropriate fashion; as mentioned before, its intended use is to upgrade *this* TCP connection, not redirect to another one. If you really want to just redirect all of the time, you'd be much better off doing a normal 3xx redirect to something with a ws: or wss: URL scheme -- it would avoid a lot of the fragility we've been concerned about on the HTTP side. Cheers, * Ian, are you just trying to exceed 100 drafts, thereby crashing the IETF? :) On 14/10/2009, at 10:07 AM, Robert Collins wrote: On Wed, 2009-10-14 at 09:59 +1100, Mark Nottingham wrote: On 13/10/2009, at 10:23 PM, Ian Hickson i...@hixie.ch wrote: I want to just use port 80, and I want to make it possible for a suitably configured HTTP server to pass connections over to WebSocket servers. It seems to me that using something that looks like an HTTP Upgrade is better than just having a totally unrelated handshake, but I guess maybe we should just reuse port 80 without doing anything HTTP-like at all. To be clear, upgrade is appropriate for changing an existing connection over to a new protocol (ie reusing it). To pass a request over to a different server, a redirect would be more appropriate (and is facilitated by the new uri scheme). Yup; and the major issue here is that websockets *does not want* the initial handshake to be HTTP. Rather it wants to be something not- quite HTTP, specifically reject a number of behaviours and headers that are legitimate HTTP. -Rob -- Mark Nottingham m...@yahoo-inc.com
Re: WebSockets negotiation over HTTP
On Wed, 14 Oct 2009 11:58:56 +1100, Mark Nottingham m...@yahoo-inc.com wrote: Catching up to Ian's -48 draft*, I don't think there's much of a problem here -- or at least the spec can be brought into alignment with HTTP with a few small changes. However, the comment about upgrade vs. redirect stands (see below). Thanks for pointing at the number, I was working off an older copy. :( (sorry Ian). By the looks of draft 48 HTTP will not break any more when WebSockets Upgrade is thrown over it. As you point out some small fragility points remain that may cause WebSockets to false-fail, they will not utterly break things. Taking a look at the current draft myself and interleaving (adding to) Marks comments Section 4.1 describes the handshake from the client side. It requires the client to send a request that's a subset of HTTP; this doesn't conflict with HTTP in and of itself. It also constrains the responses to the request that the client can expect, but that's OK because at this point we shouldn't be talking HTTP any more. 4.1.1 thru 4.1.10 are workable, modulo the same case-sensitive notion. Though as you point out being a _sent_ data format it's minor. 4.1.11 thru 4.1.13 are regarding the reply received. There has been much change to this section that appears to incorporate the points that have been mentioned over the last few months. 4.1.12 prescribes semi-implicitly that HTTP/1.0 and HTTP/1.2 etc are not compatible. Maybe thats what you want. *very* minor enhancement would be to make that explicitly stated. 4.1.13 still has a fragility issue in that it assumes the Upgrade: and Connection: headers will retain both their specific sending order and be the very first headers in the reply. It will work in most situations, but proxies which 'correct' the headers order to have Date: first will kill WebSockets. The detection of these headers could easily be combined with 4.1.23 requiring exactly one copy of each, which solves all these issues. FYI Squid is not strictly compliant in header handling and merely appends Date: where missing. This may change. I have observed one major ISP using proxies advertising themselves 'bcN' (I would assume that's BlueCoat 1,2,3 etc) which do corrections by pre-pending headers. 4.1 14 thru 4.1.23 appear to be a very conflated description of parsing the headers. It seems to me that referencing rfc2616 section 4.2 should be sufficient for the parse and do away with 4.1.15 through 4.1.21. Similar to the way 4.1.23 mentions www-auth Obtain [header array] in a manner consistent with the requirements for handling the headers in HTTP Mandating drop of connections not conforming to correct format of headers is implied and some bits are explicitly stated. That can be cleaned up and locked in by the above and adding a clear BNF like: (alpha|hyphen) colon space (ascii)* CRLF The above would also cover handling of LWS cases. Which are currently breaking WebSockets. (less important) As a minor issue, it explicitly specifies reading single bytes. I can see people interpreting that as preventing buffering of received data. It would be nice if clients were explicitly allowed to send other headers, e.g., Referer or User-Agent, but it's not critical. Also, by its nature this protocol is going to be fragile on non-CONNECTed HTTP connections, but Ian has already acknowledged this. That is implied by the mention of also adding www-authenticate and not prohibiting other headers sent following the WebSockets ones. The servers will now cope and discard according to 4.1 of the current draft. Section 5.1 describes the handshake from the server side. It doesn't place any requirements on the bytes received from the client, only on those sent by the server, so again this is a proper subset of HTTP. Section 5.2 does constrain the bytes the server accepts from the client, thereby conflicting with HTTP, but only in some small details. In particular, it makes HTTP header field-names case-sensitive, and requires certain arrangements of whitespace in them. Ian, if you can address these small things in section 5.2 it would help. 5.1/5.2 seems to have been adapted to take into account our earlier discussions and now appear to be workable. Thank you Ian. The other aspect here is that you're really not using Upgrade in an appropriate fashion; as mentioned before, its intended use is to upgrade *this* TCP connection, not redirect to another one. If you really want to just redirect all of the time, you'd be much better off doing a normal 3xx redirect to something with a ws: or wss: URL scheme -- it would avoid a lot of the fragility we've been concerned about on the HTTP side. Cheers, * Ian, are you just trying to exceed 100 drafts, thereby crashing the IETF? :) In conclusion. Hooray! nearly there :) Amos
Re: WebSockets negotiation over HTTP
On Fri, 14 Aug 2009, Amos Jeffries wrote: Being sensitive to whether the server replies 101 Blah versus 101 blah absolutely cripples WebSockets. We want to help you fix this problem. I still do not understand why anything gets crippled. Maybe you could show an example of how you expect this problem to occur? Your protocol definition used byte-level. At the byte-level 'b' (0x97 IIRC) does not equal 'B' (0x65 IIRC). Thus the response is a different byte pattern and a failed WebSocket connection. Sure, but why does this cripple Web Sockets? A compliant Web Socket server won't send back a lower-case b here. (This has nothing to do with bytes vs text, by the way; we can just as easily have case-insensitivity with a byte-based definition. Indeed, a later part of the handshake does exactly that.) One very real example of this would be the web server or an fully WebSocket capable intermediary sending back bytes Your spec section 3.1 sub 12 says: 12. Read the first 85 bytes from the server. If the connection closes before 85 bytes are received, or if the first 85 bytes aren't exactly equal to the following bytes, then fail the Web [ note the words _exactly equal_ ] Socket connection and abort these steps. 48 54 54 50 2f 31 2e 31 20 31 30 31 20 57 65 62 20 53 6f 63 6b 65 74 20 50 72 6f 74 6f 63 6f 6c 20 48 61 6e 64 73 68 61 6b 65 0d 0a 55 70 67 72 61 64 65 3a 20 57 65 62 53 6f 63 6b 65 74 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 55 70 67 72 61 64 65 0d 0a example #1 suppose there was an intermediary translating websockets-over-http to websockets-port-81 which used HTTP to format said headers of confirmation. Here is the problem I have with this: Why would we suppose the existence of such an intermediary? Why would anyone ever implementa WebSocket-specific proxy? In all other ways it is fully WebSockets compliant. But sends byte 18 as 73 (s) instead of 53 (S). Boom! The entire application is not WebSockets compliant and will fail every single transaction that goes through it. Nobody would ever ship such a proxy, since the bug would be immediately detected with the most rudimentary of testing. This seems like a benefit to me, not a problem. example #2 is where the traffic is processed by an HTTP-only intermediary which sees the 'Upgrade:' header and flags the connection for transparent pass-thru (This by the way is the desirable method of making Squid support WebSockets). Being a good HTTP relay it accepts these bytes: HTTP/1.1 101 Web Socket Protocol Handshake Upgrade: WebSocket Connection: Upgrade It violates HTTP by omitting the Via and other headers your spec omits to handle. And passes these on: HTTP/1.1 101 Web Socket Protocol Handshake Connection: Upgrade Upgrade: WebSocket then moves to tunnel mode for you. Why would it violate HTTP in all the ways you mention? If it goes to such extreme lengths to have transparent pass-through to the point of violating HTTP, why would it then go out of its way to reorder header lines? From my perspective, such a proxy would raise all kinds of alarm bells to me, and I would be _glad_ that the connection failed. If it didn't, I wouldn't be sure we could trust the rest of the data. This is because if it doesn't, how can it say 'I won't upgrade'. It can just not upgrade. Returning anything but the correct handshake will be treated as a failed connection by the WebSocket client. Correct. So why would it need to say I won't upgrade? To inform MITM that the upgrade is not going to happen and the links they have open maybe used for other HTTP things without wasting network resources tearing them down and rebuilding. The MITM isn't the WebSocket client. In this situation, it's a (non-compliant, since it forwarded by-hop headers) HTTP proxy. What's more, in this scenario the server isn't a WebSocket server, either, it's an HTTP server. So what Web Socket says is irrelevant. On Thu, 30 Jul 2009, Robert Collins wrote: Suppose we had no handshake at all, and that there was no data framing, so that as soon as we connected to a port, we could send arbitrary data down. A Web page, say evil.example.net, could open a Web Socket connection to http://www.corp.example.com/, send it a GET request for /secret-plans, and then forward the contents of the file to a remote host. If they could then trick someone on example.com's intranet to look at this file, and assuming www.corp.example.com did nothing more than rely on connectivitity for authentication (pretty common in small intranets), then evil.example.net could steal the company's secret plans. Can't the web page just send an ajax request to corp.example.com anyway? It can't read the response, no.
Re: WebSockets negotiation over HTTP
On Fri, 2009-09-04 at 01:44 +, Ian Hickson wrote: One very real example of this would be the web server or an fully WebSocket capable intermediary sending back bytes ... example #1 suppose there was an intermediary translating websockets-over-http to websockets-port-81 which used HTTP to format said headers of confirmation. Here is the problem I have with this: Why would we suppose the existence of such an intermediary? Why would anyone ever implementa WebSocket-specific proxy? Welcome to the internet :). Seriously, Why would we suppose the existence of an intermediary that intercepts and MITM's SSL connections? Or HTTP - its even got a defined proxy facility, there is no need to take over and insert different behaviour, is there? We *should* assume that firewall vendors and ISP's will do arguably insane things, because they have time and again in the past. example #2 is where the traffic is processed by an HTTP-only intermediary which sees the 'Upgrade:' header and flags the connection for transparent pass-thru (This by the way is the desirable method of making Squid support WebSockets). Being a good HTTP relay it accepts these bytes: HTTP/1.1 101 Web Socket Protocol Handshake Upgrade: WebSocket Connection: Upgrade It violates HTTP by omitting the Via and other headers your spec omits to handle. And passes these on: HTTP/1.1 101 Web Socket Protocol Handshake Connection: Upgrade Upgrade: WebSocket then moves to tunnel mode for you. Why would it violate HTTP in all the ways you mention? If it goes to such extreme lengths to have transparent pass-through to the point of violating HTTP, why would it then go out of its way to reorder header lines? Because sysadmins do this! Don't ask us to justify the weird and wonderful things we run into. Nearly daily we have users asking for help doing similar things in #squid. From my perspective, such a proxy would raise all kinds of alarm bells to me, and I would be _glad_ that the connection failed. If it didn't, I wouldn't be sure we could trust the rest of the data. Again, welcome to the internet :P. Seriously, if its not digitally signed, you *can't* trust the rest of the data. The MITM isn't the WebSocket client. In this situation, it's a (non-compliant, since it forwarded by-hop headers) HTTP proxy. What's more, in this scenario the server isn't a WebSocket server, either, it's an HTTP server. So what Web Socket says is irrelevant. Note that there are *still* HTTP/1.0 proxies in deployment that don't know about hop by hop headers. ... WebSockets, even when initiating the connection by talking to an HTTP server and then switching to WebSockets, isn't layered on HTTP. It's just doing the bare minimum required to allow the HTTP server to understand the request and get out of the way. Once the handshake is complete, there is no HTTP anywhere on the stack at all. Its not doing the bare minimum. The bare minimum would be to accept the valid HTTP transforms which the Internet _will_ perform on the handshake. Discard those useless transforms and validate the handshake status line. The bare minimum is the least amount of processing possible. What you describe is _more_ processing, not less. Therefore it's not the minimum. The bare minimum would be to just start the tcp connection with 'websocket/1.0\r\n' Stop pretending to be HTTP: use port 80. Our whole point has been, if you want to Use HTTP Upgrade, Do It Correctly. If you want to Use port 80, just do that. On Thu, Jul 30 2009, Henrik Nordstrom wrote: But for 2 you need to use HTTP, which essentially boils down to defining that one may switch a webserver to use WebSockets by using the HTTP Upgrade mechanism as defined by HTTP. I understand that you disagree with my interpretation, but my interpretation is that this is exactly what the spec does already. At this point I think we need to go to the HTTP-WG and discuss further. Most of us are already there... 5) Specific mention is made to ignore non-understood headers added randomly by intermediaries. So long as that happens after the handshake, that's ok, but we can't allow that inside the handshake, it would allow for smuggling data through, If having this view then you CAN NOT use HTTP for the Upgrade handshake, and MUST use another port and other protocol signatures. IANA expert review has informed me that I must use ports 80 and 443, so there I don't have a choice here. Whats the message id what list? I'm extremely happy to jump into that conversation. -Rob signature.asc Description: This is a digitally signed message part
Re: WebSockets negotiation over HTTP
tor 2009-07-30 klockan 08:21 + skrev Ian Hickson: 1) CONNECT and HTTP Upgrade are optional, and independent. One may be used or the other. They may be both tried in any order. That doesn't sound specific enough to get interoperable behaviour. It seems like we'd want to define exactly what the sequence of events should be (as the draft does now, for instance), rather than leaving it open. The proposal is that you have the following: 1. A definition of the independent WebSockets protocol, being a bidirectional octet stream protocol per your specification. 2. Profiles defining how to establish a WebSockets transport in HTTP infrastructure if needed, covering both normal and proxied setups. For '1' we don't care what you do, just as we don't care about any other non-HTTP protocol (SMTP, IRC, SNMP, SMB, whatever). But for 2 you need to use HTTP, which essentially boils down to defining that one may switch a webserver to use WebSockets by using the HTTP Upgrade mechanism as defined by HTTP. If you want to first try port 81, then try port 815, then try another port, then we are talking about at least _three_ connection attempts, each of which could have multisecond latency. That simply isn't workable. Well, you are basically trying to do firewall avoidance. That in itself isn't workable and will require a bit of work to be successful. If you were not trying to do firewall avoidance then none of this trying different ports thing would be needed and the server would specify THE method to use from start. 3) The specific order of bytes is not mentioned _anywhere_ in the new text. That seems like a problem, not a benefit. It's a benefit. HTTP is not an exact octet sequence protocol, but still have very well defined message syntax and boundaries. What happens after the HTTP upgrade have completed (101 HTTP response seen) is up to you, but before that the HTTP Upgrade sequence is HTTP if you at all is to use an HTTP Upgrade sequence for switching to WebSockets (another port is most likely better). The HTTP Upgrade method is in HTTP terms just a request please can we switch to protocol X instead?, and this question and it's answer is defined by HTTP terms. What happens after that is however of no concern to HTTP and is specified by whatever protocol the connection got upgraded to. 4) The order of headers _received_ is not mentioned past the 101 / 4xx /5xx line. HTTP varies order in-transit. I'd feel much more confident with more than one line's worth of handshake. The 101 is more than one line. A minimal 101 response for switching to WebSockets is: HTTP/1.1 101 Switchng Protocols Upgrade: WebSockets/1.0 [empty line] But as already said there MAY be additional headers, and the Reason-Phrase MAY be different, intended for humans and SHOULD NOT be interpreted by machines. For example the following response is equal to the above: HTTP/1.1 101 Protocoles de commutation Server: WonderfulMagicServer/4.2 (WonderOS, MagicLogic/4.3) SomeHeaderDefinedByWonderfulMagic: SomeValue Authenticate-Info: Upgrade: WebSockets/1.0 [empty line] Additionally this is just the negotiation for switching protocol. What follows after this is mostly intended to be self-descriptive and not relying much on the properties of the HTTP Upgrade handshake. But the server MAY require HTTP authentication to accept the Upgrade. 5) Specific mention is made to ignore non-understood headers added randomly by intermediaries. So long as that happens after the handshake, that's ok, but we can't allow that inside the handshake, it would allow for smuggling data through, If having this view then you CAN NOT use HTTP for the Upgrade handshake, and MUST use another port and other protocol signatures. effectively faking the handshake with unexpecting servers. (I also don't really understand the point. If there are intermediaries adding data, then frankly we probably _do_ want the connection to fail.) Faking the handshake to an unsuspecting server is equally hard even if you use a exact octet sequence. A HTTP server not supporting WebSockets can not respond with the 101 mentioned above unless it supports WebSockets or the attacker has full control over the server down to exact octet sequence of the response. If the server does not accept the upgrade you may see a 200, 3xx, 4xx or 5xx response, but not 101. Some of those response codes means that the client is expected to take certain actions like trying again with proper authentication (i.e. 401), but most are just different variants of not supported in the context of WebSockets. See for example the Authenticate-Info header mentioned above for an example. The handshake absolutely must be the very first byte, otherwise you can just trick the remote end into sending back the appropriate bytes for the handshake half-way through what it thinks is an unrelated part of the connection, depending on what the remote end's protocol is.
