Thanks for your quick and detailed answers! I will follow your advices and lower the server cycle, it indeed seems to be a good way to have a better overall reactivity. As for compression, it may be interesting only for sending maps, but I will put this at the end of the network to-do list :)
Just one additional question: do you know approximately the bandwidth usage on the server side for the games you have work on? My target is to be below 50 kB/s upload for 8 players and below 100 kB/s for 16 players, to be able to host such a server on a "classic" Internet line (in France at least, I do not know the max upload rate for other countries...) Best regards, Jeremy Richert On Wed, Dec 4, 2013 at 9:16 AM, <[email protected]> wrote: > Send ENet-discuss mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.cubik.org/mailman/listinfo/enet-discuss > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ENet-discuss digest..." > > Today's Topics: > > 1. Bandwidth/latency optimization (J?r?my Richert) > 2. Re: Bandwidth/latency optimization (Blair Holloway) > 3. Re: Bandwidth/latency optimization (Lee Salzman) > > > ---------- Forwarded message ---------- > From: "Jérémy Richert" <[email protected]> > To: [email protected] > Cc: > Date: Tue, 3 Dec 2013 22:45:49 +0100 > Subject: [ENet-discuss] Bandwidth/latency optimization > Hi all, > > I am currently developing a multiplayer FPS, and I am wondering how to > tune ENet to have a network engine as optimized as possible on a bandwidth > usage and a latency point of view. > I would like to have your thougts on the following aspects: > > ---------------- > 1. Packet size > ---------------- > > I was wondering whether it was better to send large packets or to divide > them into small packets. > > Pros of the large packets : > - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for IPv4, > 10 for ENet) > > Pros of the small packets : > - When a packet is lost, less data is lost > - When a reliable packet is lost, resending it requires less bandwidth > - Lower latency > > From what I have read, most people agree that it is recommended to send > small packets to avoid packet splitting. This means that the application > has to ensure that the size of the data sent does not exceed the MTU. Also, > as the MTU depends on the router, some people recommended to have packets > that will never be splitted, i.e. <= 576 bytes. > What is your experience on this point? > For now I have capped the data size to 1500, but I am thinking of reducing > it for a better latency. Does anyone know the typical packet loss rate? > > ---------------- > 2. Compression > ---------------- > > Has anyone used compression in a network engine? If yes, which compression > algorithm? What was the average gain? > I have read that John Carmack used the Huffman compression in the Quake 3 > network engine because it was well suited for network data compression, but > I still need to find some time to implement it in my program and do some > tests. > > ---------------- > 3. Channels > ---------------- > > What is your network channel policy? How many channels do you use? What do > you send on each channel? > At the moment I have 2 channels: one for sending unreliable data (a lot), > another one for reliable data. > I have chosen this organization to avoid blocking the unreliable data > while waiting for an ACK for the reliable data. I am thinking of adding > another channel to send high priority reliable data, but I am not sure of > the benefit as I already group the reliable data before sending them to > limit the blocking. It may be useful if the packet loss rate is too high. > > ---------------- > 4. Server cycle > ---------------- > > I know this aspect depends on the game type, but I would be interested in > knowing how your applications work on this. > On my side, based on Valve's introduction to network concepts (+ some > readings on the UT and Quake network engines), I have decided to implement > a 50-ms cycle on the server side. This means that the servers only updates > the simulation and notifies the clients each 50 ms. Meanwhile, it only > reads the network messages to empty the network event buffer and to handle > the reliable packet sending. > On the client side, I have introduced a voluntary delay of 70 ms, which > goes unnoticed on a user point of view, but helps a lot for overall > fluidity as the client will almost always have the following world update > (unless a packet is lost). > I will soon try to increase the server cycle time to 60 ms to gain 10-20% > of bandwidth. I also plan to separate the display delay of the player (it > will stay to 70 ms) and the rest of the world (increased to 100 ms). > However, I am afraid of the impact on reactivity. > > Does anyone have a similar architecture? What is your experience on > timings? > If not, has anyone already developed a FPS or a game very dependant on the > network speed? If yes, what are your advices? > > ---------------- > 5. Other > ---------------- > > If anyone has some useful advices on how to improve the network engine of > a game/application, I would be pleased to hear it (well, to read it at > least). > > > Thanks in advance for all your experience sharing. > > Best regards, > Jeremy Richert > > > ---------- Forwarded message ---------- > From: Blair Holloway <[email protected]> > To: Discussion of the ENet library <[email protected]> > Cc: > Date: Tue, 3 Dec 2013 15:16:54 -0800 > Subject: Re: [ENet-discuss] Bandwidth/latency optimization > Here are my thoughts: > > 1) Packet size > > Smaller is nearly universally better, especially if any of your data are > reliable, as your packets may quickly be eaten up by resent data over poor > connections. > > A second advantage of smaller data sizes is that you can generally have a > higher number of maximum players, or more stuff happening in your game. > > 2) Compression > > Of all the games I've worked on, we only ever used bit-packing or a range > encoder to compress the data; we never used a more general-purpose > compression algorithm. > > One exception was a game whose packets often contained a lot of repeated > string data. We generated a dictionary at runtime using previously seen > strings, and used indices into that dictionary during future transmissions. > > In general, the best "compression" we implemented was always "remove the > need for X to be sent". :) > > 3) Channels > > Can you envision a need up-front for high-priority reliable data? I've > never encountered such a case myself, but I *have* found cases where > certain unreliable data need to be prioritized ahead of others. Sometimes > this can be implemented as simply as "put the high priority stuff in the > packet first; continue until pre-set limit reached". > > 4) Server cycle > > The only thing I can really say here is that the smaller simulation time > step your server takes, the lesser your latency will be. Increasing the > timestep will save you bandwidth, but it can make your game seem laggier. > For an FPS, you probably want to aim for an even higher packet rate -- 30 > packets per second wouldn't be unusual. > > Cheers, > > - Blair > > > > On Tue, Dec 3, 2013 at 1:45 PM, Jérémy Richert > <[email protected]>wrote: > >> Hi all, >> >> I am currently developing a multiplayer FPS, and I am wondering how to >> tune ENet to have a network engine as optimized as possible on a bandwidth >> usage and a latency point of view. >> I would like to have your thougts on the following aspects: >> >> ---------------- >> 1. Packet size >> ---------------- >> >> I was wondering whether it was better to send large packets or to divide >> them into small packets. >> >> Pros of the large packets : >> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for IPv4, >> 10 for ENet) >> >> Pros of the small packets : >> - When a packet is lost, less data is lost >> - When a reliable packet is lost, resending it requires less bandwidth >> - Lower latency >> >> From what I have read, most people agree that it is recommended to send >> small packets to avoid packet splitting. This means that the application >> has to ensure that the size of the data sent does not exceed the MTU. Also, >> as the MTU depends on the router, some people recommended to have packets >> that will never be splitted, i.e. <= 576 bytes. >> What is your experience on this point? >> For now I have capped the data size to 1500, but I am thinking of >> reducing it for a better latency. Does anyone know the typical packet loss >> rate? >> >> ---------------- >> 2. Compression >> ---------------- >> >> Has anyone used compression in a network engine? If yes, which >> compression algorithm? What was the average gain? >> I have read that John Carmack used the Huffman compression in the Quake 3 >> network engine because it was well suited for network data compression, but >> I still need to find some time to implement it in my program and do some >> tests. >> >> ---------------- >> 3. Channels >> ---------------- >> >> What is your network channel policy? How many channels do you use? What >> do you send on each channel? >> At the moment I have 2 channels: one for sending unreliable data (a lot), >> another one for reliable data. >> I have chosen this organization to avoid blocking the unreliable data >> while waiting for an ACK for the reliable data. I am thinking of adding >> another channel to send high priority reliable data, but I am not sure of >> the benefit as I already group the reliable data before sending them to >> limit the blocking. It may be useful if the packet loss rate is too high. >> >> ---------------- >> 4. Server cycle >> ---------------- >> >> I know this aspect depends on the game type, but I would be interested in >> knowing how your applications work on this. >> On my side, based on Valve's introduction to network concepts (+ some >> readings on the UT and Quake network engines), I have decided to implement >> a 50-ms cycle on the server side. This means that the servers only updates >> the simulation and notifies the clients each 50 ms. Meanwhile, it only >> reads the network messages to empty the network event buffer and to handle >> the reliable packet sending. >> On the client side, I have introduced a voluntary delay of 70 ms, which >> goes unnoticed on a user point of view, but helps a lot for overall >> fluidity as the client will almost always have the following world update >> (unless a packet is lost). >> I will soon try to increase the server cycle time to 60 ms to gain 10-20% >> of bandwidth. I also plan to separate the display delay of the player (it >> will stay to 70 ms) and the rest of the world (increased to 100 ms). >> However, I am afraid of the impact on reactivity. >> >> Does anyone have a similar architecture? What is your experience on >> timings? >> If not, has anyone already developed a FPS or a game very dependant on >> the network speed? If yes, what are your advices? >> >> ---------------- >> 5. Other >> ---------------- >> >> If anyone has some useful advices on how to improve the network engine of >> a game/application, I would be pleased to hear it (well, to read it at >> least). >> >> >> Thanks in advance for all your experience sharing. >> >> Best regards, >> Jeremy Richert >> >> _______________________________________________ >> ENet-discuss mailing list >> [email protected] >> http://lists.cubik.org/mailman/listinfo/enet-discuss >> >> > > > ---------- Forwarded message ---------- > From: Lee Salzman <[email protected]> > To: Discussion of the ENet library <[email protected]> > Cc: > Date: Wed, 4 Dec 2013 10:16:40 +0200 > Subject: Re: [ENet-discuss] Bandwidth/latency optimization > My experiences as for Cube/Sauerbraten: > > 1. packet size > > Do not hand ENet lots of tiny packets. This is bad. Use the ENet packets > as batches, i.e. take a bunch of your little user-level with their own > specialized little headers (in Sauerbraten these headers are only 1 byte > per packet!). However, keep these batch sizes underneath the ENet MTU with > some small reserved space for ENet headers, i.e. peer->mtu - 100 should be > good for limiting. > > The reason this became important for Sauerbraten is we were sending *lots* > of these, so ENet packet headers would have become a significant fraction > of the total size, since the payloads of these packets are small. That > Sauerbraten's internal protocol can get this down to 1 byte sub-header > essentially gets rid of that issue. These are then batched into > peer->mtu-100 ENet packets as described above to avoid triggering ENet's > transparent fragmentation system. > > 2. compression > > Pack your data intelligently in the first place, and you won't get much if > any benefit from compression. At least Sauerbraten's packets get at most > only 5-10% more out of ENet's range coder because we encode, say, integers > using an extension scheme that uses as few bytes as necessary (not unlike > how, say, UTF8 works). So, just be as smart as possible on the encoding up > front, and you can mostly avoid compression and save a lot of the CPU time > that you would have normally spent on that. > > Failing that, others have been happy enough to stick with ENet's built-in > range coder. > > 3. channels > > Nothing wrong with a 2 channel setup, and is almost what I use for > Sauerbraten. I send bulk movement data on a channel that mostly is only > unreliable. Occasionally reliable stuff gets sequenced into that, but very > rarely (only for special events like teleports or jumppads). The other > channel I use for reliable stuff like gun shots, chat, etc. There are other > schemes like how Quake 3 works that can be used to send even that > 'reliable' data as unreliable, but it wasn't terribly worth it for > Sauerbraten's case. > > I also have another channel or two I use for file transfers and things > that I don't want stalling the flow of everything else for long durations. > > 4. server-cycle > > Sauerbraten just because of its insane movement speed uses 33ms (~30 Hz). > Whereas for variants with slightly saner move speed, like Red > Eclipse/Tesseract/AssaultCube, I have preferred to settle on 40ms (25 Hz). > What matters most for this is the quality of your physics interpolation and > also that you do a lot of client-side prediction rather than waiting on > round-trips, then these small changes in rate don't matter quite as much, > but higher fidelity of input to the interpolation always helps. > > > > > On Tue, Dec 3, 2013 at 11:45 PM, Jérémy Richert <[email protected] > > wrote: > >> Hi all, >> >> I am currently developing a multiplayer FPS, and I am wondering how to >> tune ENet to have a network engine as optimized as possible on a bandwidth >> usage and a latency point of view. >> I would like to have your thougts on the following aspects: >> >> ---------------- >> 1. Packet size >> ---------------- >> >> I was wondering whether it was better to send large packets or to divide >> them into small packets. >> >> Pros of the large packets : >> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for IPv4, >> 10 for ENet) >> >> Pros of the small packets : >> - When a packet is lost, less data is lost >> - When a reliable packet is lost, resending it requires less bandwidth >> - Lower latency >> >> From what I have read, most people agree that it is recommended to send >> small packets to avoid packet splitting. This means that the application >> has to ensure that the size of the data sent does not exceed the MTU. Also, >> as the MTU depends on the router, some people recommended to have packets >> that will never be splitted, i.e. <= 576 bytes. >> What is your experience on this point? >> For now I have capped the data size to 1500, but I am thinking of >> reducing it for a better latency. Does anyone know the typical packet loss >> rate? >> >> ---------------- >> 2. Compression >> ---------------- >> >> Has anyone used compression in a network engine? If yes, which >> compression algorithm? What was the average gain? >> I have read that John Carmack used the Huffman compression in the Quake 3 >> network engine because it was well suited for network data compression, but >> I still need to find some time to implement it in my program and do some >> tests. >> >> ---------------- >> 3. Channels >> ---------------- >> >> What is your network channel policy? How many channels do you use? What >> do you send on each channel? >> At the moment I have 2 channels: one for sending unreliable data (a lot), >> another one for reliable data. >> I have chosen this organization to avoid blocking the unreliable data >> while waiting for an ACK for the reliable data. I am thinking of adding >> another channel to send high priority reliable data, but I am not sure of >> the benefit as I already group the reliable data before sending them to >> limit the blocking. It may be useful if the packet loss rate is too high. >> >> ---------------- >> 4. Server cycle >> ---------------- >> >> I know this aspect depends on the game type, but I would be interested in >> knowing how your applications work on this. >> On my side, based on Valve's introduction to network concepts (+ some >> readings on the UT and Quake network engines), I have decided to implement >> a 50-ms cycle on the server side. This means that the servers only updates >> the simulation and notifies the clients each 50 ms. Meanwhile, it only >> reads the network messages to empty the network event buffer and to handle >> the reliable packet sending. >> On the client side, I have introduced a voluntary delay of 70 ms, which >> goes unnoticed on a user point of view, but helps a lot for overall >> fluidity as the client will almost always have the following world update >> (unless a packet is lost). >> I will soon try to increase the server cycle time to 60 ms to gain 10-20% >> of bandwidth. I also plan to separate the display delay of the player (it >> will stay to 70 ms) and the rest of the world (increased to 100 ms). >> However, I am afraid of the impact on reactivity. >> >> Does anyone have a similar architecture? What is your experience on >> timings? >> If not, has anyone already developed a FPS or a game very dependant on >> the network speed? If yes, what are your advices? >> >> ---------------- >> 5. Other >> ---------------- >> >> If anyone has some useful advices on how to improve the network engine of >> a game/application, I would be pleased to hear it (well, to read it at >> least). >> >> >> Thanks in advance for all your experience sharing. >> >> Best regards, >> Jeremy Richert >> >> _______________________________________________ >> ENet-discuss mailing list >> [email protected] >> http://lists.cubik.org/mailman/listinfo/enet-discuss >> >> > > _______________________________________________ > ENet-discuss mailing list > [email protected] > http://lists.cubik.org/mailman/listinfo/enet-discuss > >
_______________________________________________ ENet-discuss mailing list [email protected] http://lists.cubik.org/mailman/listinfo/enet-discuss
