Re: [whatwg] [hybi] WebSockets: UDP
On Fri, 11 Jun 2010 00:21:38 +0200, Mark Frohnmayer mark.frohnma...@gmail.com wrote: TorqueSocket is not in the same category as RakNet or OpenTNL Ah, sorry I got the names mixed up, I meant to say RakNet/OpenTNL and not RakNet/TorqueSocket. I'd recommend doing some real-world testing for max packet size. Back when the original QuakeWorld came out it started by sending a large connect packet (could be ~8K) and a good number of routers would just drop those packets unconditionally. The solution (iirc) was to keep all packet sends below the Ethernet max of 1500 bytes. I haven't verified this lately to see if that's still the case, but it seems real-world functionality should be considered. Absolutely, that's why the path-MTU attribute was suggested. The ~64k limit is an absolute limit though at which sends can be rejected immediately without even trying. If WebSocket supports an encrypted and unencrypted mode, why would the real-time version not support data security and integrity? The reasoning was that if you do need data security and integrity the secure websocket over TCP uses the same state-of-the-art implementation as the browsers already have implemented. Secure connections over UDP would either require a full TCP over UDP implementation (to use TLS) or a second implementation that would need to be maintained. That implementation would be either a very complex piece or software or clearly inferior to that users are accustomed to. So what's a good use-case where you want a secure connection over UDP and cannot use a second TLS connection? Client puzzles allow the host to allocate zero resources for a pending connection until it knows that the source address of the client request is valid and that the client has done some work; you could still take a similar (though not client computationally expensive) approach by having the host hash the client identity (IP/port) with a server-generated secret. Any approach that allocates memory or does work on the host without verifying the client source address first is vulnerable to a super-trivial DOS attack (connection depletion before even any bandwidth overwhelm). Right, this is probably an area that needs to be looked more carefully at if/when real work is started on a spec. I'd propose that doing this in the javascript level would result in unnecessary extra overhead (sequence numbering, acknowledgements) that could easily be a part of the underlying protocol. Having implemented multiple iterations of a high-level networking API, the notification function is a critical, low-overhead tool for making effective higher-level data guarantees possible. Yes, no doubt it's useful for those implementing higher level APIs. As usual it's a matter of at what level to place the API. -- Erik Möller Core Developer Opera Software
Re: [whatwg] [hybi] WebSockets: UDP
On Fri, 11 Jun 2010 06:25:41 +0200, Lars Eggert lars.egg...@nokia.com wrote: Hi, on a purely managerial level, let me point out that this work is far beyond the current charter of the HYBI WG. This defines an entirely new protocol, and will definitely require a charter discussion. (If there is community/developer interest, we should by all means have that discussion.) Lars Sure, this just felt like the appropriate venue for these initial discussions. If not please say so and we'll move it somewhere else. From my personal experience when discussing this with developers and people from adjacent industries there's significant interest in the _final product_ but not so much interest in getting directly involved in the process. -- Erik Möller Core Developer Opera Software
Re: [whatwg] [hybi] WebSockets: UDP
On Fri, Jun 11, 2010 at 3:18 AM, Erik Möller emol...@opera.com wrote: Absolutely, that's why the path-MTU attribute was suggested. The ~64k limit is an absolute limit though at which sends can be rejected immediately without even trying. Ah, gotcha. I was trying to separate the cases of MTU before fragmentation and maximum packet size (fragmented) before some host along the way silently discards the packet. The reasoning was that if you do need data security and integrity the secure websocket over TCP uses the same state-of-the-art implementation as the browsers already have implemented. Secure connections over UDP would either require a full TCP over UDP implementation (to use TLS) or a second implementation that would need to be maintained. That implementation would be either a very complex piece or software or clearly inferior to that users are accustomed to. So what's a good use-case where you want a secure connection over UDP and cannot use a second TLS connection? I guess I was thinking more in the direction of a well conceived, well reviewed protocol (TLS-level of scrutiny) for encryption of the unreliable packet stream - a secure data stream is a useful primitive and would otherwise have to be implemented at the application level. I'd rather have security done well at the platform level (what developers expect from a modern browser). I'm not sure I'd agree that it has to be super complex in order to be effective, and could likely use much of the same functionality (ciphers, etc). I'm also not assuming a second message channel -- if a developer is using an API like TNL or RakNet that handles higher level data and message guarantees then the addition of multiple message streams is unneeded complexity. Yes, no doubt it's useful for those implementing higher level APIs. As usual it's a matter of at what level to place the API. Agreed. This goes back to Phil's point that UDP by itself isn't a great primitive to build on -- the key element for real-time applications is that dropped packets shouldn't stall the communication stream nor be automatically retransmitted. A slightly higher level approach (connection handshake, security, etc), as long as its overhead is low, could be a more effective foundation to build on. Cheers, Mark
Re: [whatwg] [hybi] WebSockets: UDP
On Thu, Jun 10, 2010 at 12:35 PM, Erik Möller emol...@opera.com wrote: Regarding the discussions on at what level the API of a UDP-WebSocket should be: One of the most important aspects to remember are that for this to be interesting to application developers we need all the browser vendors to support this feature in a compatible way. Therefore it doesn't seem reasonable to standardize and spec a higher level network API akin to RakNet / Torque Socket and hope all vendors will be willing to spend the (quite large amount of) resources required for their own implementation of TCP over UDP, bandwidth throttling etc. In our opinion we're much better off standardizing a minimal UDP-like socket. For most application developers it TorqueSocket is not in the same category as RakNet or OpenTNL -- the point of the TorqueSocket effort was to find the web equivalent of a minimal UDP'ish socket (understanding of course that it would be in the context of a connected packet stream). RakNet and OpenTNL provide higher level data guarantees, RPC, object state replication, etc. TorqueSocket is a simple send/recv packet protocol - it does no TCP over UDP or bandwidth throttling. As discussed the following features/limitations are suggested: -Same API as WebSockets with the possible addition of an attribute that allows the application developer to find the path MTU of a connected socket. -Max allowed send size is 65,507 bytes. I'd recommend doing some real-world testing for max packet size. Back when the original QuakeWorld came out it started by sending a large connect packet (could be ~8K) and a good number of routers would just drop those packets unconditionally. The solution (iirc) was to keep all packet sends below the Ethernet max of 1500 bytes. I haven't verified this lately to see if that's still the case, but it seems real-world functionality should be considered. Some additional points that were suggested on this list were: -Key exchange and encryption If you do want to have key exchange and encryption you really shouldn't reinvent the wheel but rather use a secure WebSocket connection in addition to the UDP-WebSocket. Adding key exchange and encryption to the UDP-WebSocket is discouraged. If WebSocket supports an encrypted and unencrypted mode, why would the real-time version not support data security and integrity? -Client puzzles to reduce connection depletion/CPU depletion attacks to the handshake. If the goal is to prevent DOS attacks on the accepting server this seems futile. Client puzzles only raises the bar ever so slightly for an attacker so this is also discouraged. Client puzzles allow the host to allocate zero resources for a pending connection until it knows that the source address of the client request is valid and that the client has done some work; you could still take a similar (though not client computationally expensive) approach by having the host hash the client identity (IP/port) with a server-generated secret. Any approach that allocates memory or does work on the host without verifying the client source address first is vulnerable to a super-trivial DOS attack (connection depletion before even any bandwidth overwhelm). -Packet delivery notification to be a part of the API. Again this is believed to be better left outside the UDP-WebSockets spec and implemented in javascript if the application developer requires it. I'd propose that doing this in the javascript level would result in unnecessary extra overhead (sequence numbering, acknowledgements) that could easily be a part of the underlying protocol. Having implemented multiple iterations of a high-level networking API, the notification function is a critical, low-overhead tool for making effective higher-level data guarantees possible. Regards, Mark
Re: [whatwg] [hybi] WebSockets: UDP
During the Opera Network Seminar held in Oslo this week I discussed the possible addition of a new wsd: URL scheme to WebSockets that would allow relaxing the packet resends and enable demanding real-time applications to be written. I'd like to summarize some of the conclusions a few of us came to when discussing this (informally). Regarding the discussions on at what level the API of a UDP-WebSocket should be: One of the most important aspects to remember are that for this to be interesting to application developers we need all the browser vendors to support this feature in a compatible way. Therefore it doesn't seem reasonable to standardize and spec a higher level network API akin to RakNet / Torque Socket and hope all vendors will be willing to spend the (quite large amount of) resources required for their own implementation of TCP over UDP, bandwidth throttling etc. In our opinion we're much better off standardizing a minimal UDP-like socket. For most application developers it seems likely they will be able to work with a mix of XMLHttpRequest, WebSockets and this new UDP-WebSocket to achieve the same functionality provided by those higher level APIs. If deemed necessary for an application the higher level network API can be written in JavaScript and work on-top of the much smaller hopefully cross-browser compatible UDP-WebSocket API. As discussed the following features/limitations are suggested: -Same API as WebSockets with the possible addition of an attribute that allows the application developer to find the path MTU of a connected socket. -Max allowed send size is 65,507 bytes. -Socket is bound to one remote address at creation and stays connected to that host for the duration of its lifetime. -IP Broadcast/Multicast addresses are not valid remote addresses and only a set range of ports are valid. -Reliable handshake with origin info (Connection timeout will trigger close event.) -Automatic keep-alives (to detect force close at remote host and keep NAT traversal active) -Reliable close handshake -Sockets open sequentially (like current DOS protection in WebSockets) or perhaps have a limit of one socket per remote host. -Cap on number of open sockets per host and global user-agent limit. Some additional points that were suggested on this list were: -Key exchange and encryption If you do want to have key exchange and encryption you really shouldn't reinvent the wheel but rather use a secure WebSocket connection in addition to the UDP-WebSocket. Adding key exchange and encryption to the UDP-WebSocket is discouraged. -Client puzzles to reduce connection depletion/CPU depletion attacks to the handshake. If the goal is to prevent DOS attacks on the accepting server this seems futile. Client puzzles only raises the bar ever so slightly for an attacker so this is also discouraged. -Packet delivery notification to be a part of the API. Again this is believed to be better left outside the UDP-WebSockets spec and implemented in javascript if the application developer requires it. Best Regards, -- Erik Möller Core Developer Opera Software
Re: [whatwg] [hybi] WebSockets: UDP
On Wed, 02 Jun 2010 01:07:48 +0200, Mark Frohnmayer mark.frohnma...@gmail.com wrote: Glad to see this discussion rolling! For what it's worth, the Torque Sockets design effort was to take a stab at answering this question -- what is the least-common-denominator webby API/protocol that's sufficiently useful to be a common foundation for real time games. I did the first stab at porting OpenTNL (now tnl2) atop it; from my reading of the RTP protocol that should easily layer as well, but it would be worth getting the perspective of some other high-level network stack folks (RakNet, etc). For those who missed Mark's initial post on the subject his TorqueSocket API can be found here: http://github.com/nardo/torque_sockets/raw/master/TorqueSocket_API.txt Perhaps we could communally have a look at how this compares to other common network libraries to find a least common denominator of functionality? Only feedback here would be I think p2p should be looked at in this pass -- many client/server game instances are peers from the perspective of the hosting service (XBox Live, Quake, Half-Life, Battle.net) -- forcing all game traffic to pass through the hosting domain is a severe constraint. My question -- what does a webby p2p solution look like regarding Origin restrictions, etc? Although it sure complicates things I don't see why WebSockets couldn't be extended to allow peer-to-peer connections if we really wanted to. User agent A and B connects to WebSocket server C who keeps a list of clients connected to it. A and B are informed of the id's of connected clients and both decide to set up a peer-to-peer connection. A and B simultaneously call socket.connect_to_peer() passing a peer id and the call returns a new WebSocket in connecting mode. The user agents communicate with the server over the server-socket, retrieves the necessary information to connect to the peer including what origin to expect and off we go. (Server implementations would of course be free to not support this if they choose) As long as the usual DOS/tamper protection is there it should be possible... then there's the success rate of UDP NAT hole punching... -- Erik Möller Core Developer Opera Software
Re: [whatwg] [hybi] WebSockets: UDP
On Tue, Jun 1, 2010 at 1:02 PM, Erik Möller emol...@opera.com wrote: I was hoping to be able to avoid looking at what the interfaces of a high vs low level option would look like this early on in the discussions, but perhaps we need to do just that; look at Torque, RakNet etc and find a least common denominator and see what the reactions would be to such an interface. Game companies are pretty restrictive about what they discuss, but I think I know enough game devs to at least get some good feedback on what would be required to make it work well with their engine/game. Glad to see this discussion rolling! For what it's worth, the Torque Sockets design effort was to take a stab at answering this question -- what is the least-common-denominator webby API/protocol that's sufficiently useful to be a common foundation for real time games. I did the first stab at porting OpenTNL (now tnl2) atop it; from my reading of the RTP protocol that should easily layer as well, but it would be worth getting the perspective of some other high-level network stack folks (RakNet, etc). I suspect they prefer to be empowered with UDP rather than boxed into a high level protocol that doesn't fit their needs but I may be wrong. If you put it like that, I don't see why anybody would not want to be empowered :-) Yeah I wouldn't put it like that when asking :) I'm really not trying to sell my view, I just like to see real browser gaming in a not too distant future. Hmm... given the number of different approaches to higher-level game networking I'd hate to see a high-level straightjacket where a well-conceived low level API could easily support all of the existing solutions out there. The more complex the API the larger the attack surface at the trusted level and the more difficulty in getting existing stakeholders (game and browser makers) on board. So, what would the minimal set of limitations be to make a UDP WebSocket browser-safe? -No listen sockets Only feedback here would be I think p2p should be looked at in this pass -- many client/server game instances are peers from the perspective of the hosting service (XBox Live, Quake, Half-Life, Battle.net) -- forcing all game traffic to pass through the hosting domain is a severe constraint. My question -- what does a webby p2p solution look like regarding Origin restrictions, etc? -No multicast -Reliable handshake with origin info -Automatic keep-alives -Reliable close handshake While we're at it, I'd add key exchange, encryption and client puzzles to reduce connection depletion/CPU depletion attacks to the handshake. The protocol you seem to be aiming for isn't UDP -- rather it's more like an connected, unreliable packet stream between hosts. -Socket is bound to one address for the duration of its lifetime -Sockets open sequentially (like current DOS protection in WebSockets) -Cap on number of open sockets per server and total per user agent A single UDP socket can host multiple connections (indexed by packet source address), so even a modest limit on actual number of sockets wouldn't be a big impediment. I'd also advocate for packet delivery notification to be a part of the API -- with a known success or failure status on packet delivery many of the higher level data transmission policies become trivial, and should be essentially zero overhead at the protocol level. Without notification the higher level code has to do manual packet acknowledgement as Phil mentioned. Regards, Mark
Re: [whatwg] [hybi] WebSockets: UDP
On Tue, Jun 1, 2010 at 4:07 PM, Mark Frohnmayer mark.frohnma...@gmail.com wrote: On Tue, Jun 1, 2010 at 1:02 PM, Erik Möller emol...@opera.com wrote: So, what would the minimal set of limitations be to make a UDP WebSocket browser-safe? -No listen sockets Only feedback here would be I think p2p should be looked at in this pass -- many client/server game instances are peers from the perspective of the hosting service (XBox Live, Quake, Half-Life, Battle.net) -- forcing all game traffic to pass through the hosting domain is a severe constraint. My question -- what does a webby p2p solution look like regarding Origin restrictions, etc? Unix domain sockets allow you to pass file descriptors between processes. It might be interesting to pass a WebSocket endpoint across a WebSocket. If the clients can punch through NATs, it becomes a direct peer-to-peer connection, otherwise it gets proxied through the server. Probably makes implementations excessively complicated, though. UDP-style would be easier (no need to worry about data received by the server after it initiates pushing the endpoint to the other client - just drop it on the floor). -scott
Re: [whatwg] [hybi] WebSockets: UDP
On Tue, Jun 1, 2010 at 4:35 PM, l.w...@surrey.ac.uk wrote: On 2 Jun 2010, at 00:07, Mark Frohnmayer wrote: A single UDP socket can host multiple connections (indexed by packet source address), so even a modest limit on actual number of sockets wouldn't be a big impediment. Um, NAT? You would want to index by the NAT'd address. In the case of peer introduction and connection a third party is needed to provide the external send-to address. Is that what you were asking? Regards, Mark
Re: [whatwg] [hybi] WebSockets: UDP
On Tue, Jun 1, 2010 at 5:12 PM, Mark Frohnmayer mark.frohnma...@gmail.comwrote: On Tue, Jun 1, 2010 at 4:35 PM, l.w...@surrey.ac.uk wrote: On 2 Jun 2010, at 00:07, Mark Frohnmayer wrote: A single UDP socket can host multiple connections (indexed by packet source address), so even a modest limit on actual number of sockets wouldn't be a big impediment. Um, NAT? You would want to index by the NAT'd address. In the case of peer introduction and connection a third party is needed to provide the external send-to address. In some cases you need to use UPNP to NAT, but in general the 3rd party connection facilitator will help. The UPNP is mostly needed so that clients can _host_, which is not the goal here. If we assume a public, carefully set up UDP host, then nearly anyone can connect if UDP is allowed at all. No NAT is required in this case. And I think this is the common case, since we are not trying to run service hosts in the browser at this time. If you have any sort of connection identifier (typically port will be different even if IP is not), then you can multiplex by that. (Also, hi! This is my first post. I'm Ben Garney, I worked at PushButton Labs on Flash game technology (www.pushbuttonengine.com, www.pushbuttonlabs.com). Naturally, seeing browser capabilities expand either by plugin or native capabilities is exciting. Before I worked at PBL I worked with Mark at GarageGames on networking technology, among other things.) Ben
