I don't have exact numbers to hand, but the last game I did work on was eight players, with around 100kbps uplink (12 kilobytes a second). Note that this was a distributed, peer-to-peer simulation, so that cost was the same for all the clients, as well.
Since you have an authoritative server, a larger target number makes sense. I expect that your target of 50 kilobytes can be met for 8 players, provided that your player updates are small (24-32 bytes per player per packet). - Blair On Thu, Dec 5, 2013 at 10:24 AM, Jérémy Richert <[email protected]>wrote: > Thanks for your quick and detailed answers! > I will follow your advices and lower the server cycle, it indeed seems to > be a good way to have a better overall reactivity. > As for compression, it may be interesting only for sending maps, but I > will put this at the end of the network to-do list :) > > Just one additional question: do you know approximately the bandwidth > usage on the server side for the games you have work on? > My target is to be below 50 kB/s upload for 8 players and below 100 kB/s > for 16 players, to be able to host such a server on a "classic" Internet > line (in France at least, I do not know the max upload rate for other > countries...) > > Best regards, > Jeremy Richert > > > > On Wed, Dec 4, 2013 at 9:16 AM, <[email protected]> wrote: > >> Send ENet-discuss mailing list submissions to >> [email protected] >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://lists.cubik.org/mailman/listinfo/enet-discuss >> or, via email, send a message with subject or body 'help' to >> [email protected] >> >> You can reach the person managing the list at >> [email protected] >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of ENet-discuss digest..." >> >> Today's Topics: >> >> 1. Bandwidth/latency optimization (J?r?my Richert) >> 2. Re: Bandwidth/latency optimization (Blair Holloway) >> 3. Re: Bandwidth/latency optimization (Lee Salzman) >> >> >> ---------- Forwarded message ---------- >> From: "Jérémy Richert" <[email protected]> >> To: [email protected] >> Cc: >> Date: Tue, 3 Dec 2013 22:45:49 +0100 >> Subject: [ENet-discuss] Bandwidth/latency optimization >> Hi all, >> >> I am currently developing a multiplayer FPS, and I am wondering how to >> tune ENet to have a network engine as optimized as possible on a bandwidth >> usage and a latency point of view. >> I would like to have your thougts on the following aspects: >> >> ---------------- >> 1. Packet size >> ---------------- >> >> I was wondering whether it was better to send large packets or to divide >> them into small packets. >> >> Pros of the large packets : >> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for IPv4, >> 10 for ENet) >> >> Pros of the small packets : >> - When a packet is lost, less data is lost >> - When a reliable packet is lost, resending it requires less bandwidth >> - Lower latency >> >> From what I have read, most people agree that it is recommended to send >> small packets to avoid packet splitting. This means that the application >> has to ensure that the size of the data sent does not exceed the MTU. Also, >> as the MTU depends on the router, some people recommended to have packets >> that will never be splitted, i.e. <= 576 bytes. >> What is your experience on this point? >> For now I have capped the data size to 1500, but I am thinking of >> reducing it for a better latency. Does anyone know the typical packet loss >> rate? >> >> ---------------- >> 2. Compression >> ---------------- >> >> Has anyone used compression in a network engine? If yes, which >> compression algorithm? What was the average gain? >> I have read that John Carmack used the Huffman compression in the Quake 3 >> network engine because it was well suited for network data compression, but >> I still need to find some time to implement it in my program and do some >> tests. >> >> ---------------- >> 3. Channels >> ---------------- >> >> What is your network channel policy? How many channels do you use? What >> do you send on each channel? >> At the moment I have 2 channels: one for sending unreliable data (a lot), >> another one for reliable data. >> I have chosen this organization to avoid blocking the unreliable data >> while waiting for an ACK for the reliable data. I am thinking of adding >> another channel to send high priority reliable data, but I am not sure of >> the benefit as I already group the reliable data before sending them to >> limit the blocking. It may be useful if the packet loss rate is too high. >> >> ---------------- >> 4. Server cycle >> ---------------- >> >> I know this aspect depends on the game type, but I would be interested in >> knowing how your applications work on this. >> On my side, based on Valve's introduction to network concepts (+ some >> readings on the UT and Quake network engines), I have decided to implement >> a 50-ms cycle on the server side. This means that the servers only updates >> the simulation and notifies the clients each 50 ms. Meanwhile, it only >> reads the network messages to empty the network event buffer and to handle >> the reliable packet sending. >> On the client side, I have introduced a voluntary delay of 70 ms, which >> goes unnoticed on a user point of view, but helps a lot for overall >> fluidity as the client will almost always have the following world update >> (unless a packet is lost). >> I will soon try to increase the server cycle time to 60 ms to gain 10-20% >> of bandwidth. I also plan to separate the display delay of the player (it >> will stay to 70 ms) and the rest of the world (increased to 100 ms). >> However, I am afraid of the impact on reactivity. >> >> Does anyone have a similar architecture? What is your experience on >> timings? >> If not, has anyone already developed a FPS or a game very dependant on >> the network speed? If yes, what are your advices? >> >> ---------------- >> 5. Other >> ---------------- >> >> If anyone has some useful advices on how to improve the network engine of >> a game/application, I would be pleased to hear it (well, to read it at >> least). >> >> >> Thanks in advance for all your experience sharing. >> >> Best regards, >> Jeremy Richert >> >> >> ---------- Forwarded message ---------- >> From: Blair Holloway <[email protected]> >> To: Discussion of the ENet library <[email protected]> >> Cc: >> Date: Tue, 3 Dec 2013 15:16:54 -0800 >> Subject: Re: [ENet-discuss] Bandwidth/latency optimization >> Here are my thoughts: >> >> 1) Packet size >> >> Smaller is nearly universally better, especially if any of your data are >> reliable, as your packets may quickly be eaten up by resent data over poor >> connections. >> >> A second advantage of smaller data sizes is that you can generally have a >> higher number of maximum players, or more stuff happening in your game. >> >> 2) Compression >> >> Of all the games I've worked on, we only ever used bit-packing or a >> range encoder to compress the data; we never used a more general-purpose >> compression algorithm. >> >> One exception was a game whose packets often contained a lot of repeated >> string data. We generated a dictionary at runtime using previously seen >> strings, and used indices into that dictionary during future transmissions. >> >> In general, the best "compression" we implemented was always "remove the >> need for X to be sent". :) >> >> 3) Channels >> >> Can you envision a need up-front for high-priority reliable data? I've >> never encountered such a case myself, but I *have* found cases where >> certain unreliable data need to be prioritized ahead of others. Sometimes >> this can be implemented as simply as "put the high priority stuff in the >> packet first; continue until pre-set limit reached". >> >> 4) Server cycle >> >> The only thing I can really say here is that the smaller simulation time >> step your server takes, the lesser your latency will be. Increasing the >> timestep will save you bandwidth, but it can make your game seem laggier. >> For an FPS, you probably want to aim for an even higher packet rate -- 30 >> packets per second wouldn't be unusual. >> >> Cheers, >> >> - Blair >> >> >> >> On Tue, Dec 3, 2013 at 1:45 PM, Jérémy Richert <[email protected] >> > wrote: >> >>> Hi all, >>> >>> I am currently developing a multiplayer FPS, and I am wondering how to >>> tune ENet to have a network engine as optimized as possible on a bandwidth >>> usage and a latency point of view. >>> I would like to have your thougts on the following aspects: >>> >>> ---------------- >>> 1. Packet size >>> ---------------- >>> >>> I was wondering whether it was better to send large packets or to divide >>> them into small packets. >>> >>> Pros of the large packets : >>> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for >>> IPv4, 10 for ENet) >>> >>> Pros of the small packets : >>> - When a packet is lost, less data is lost >>> - When a reliable packet is lost, resending it requires less bandwidth >>> - Lower latency >>> >>> From what I have read, most people agree that it is recommended to send >>> small packets to avoid packet splitting. This means that the application >>> has to ensure that the size of the data sent does not exceed the MTU. Also, >>> as the MTU depends on the router, some people recommended to have packets >>> that will never be splitted, i.e. <= 576 bytes. >>> What is your experience on this point? >>> For now I have capped the data size to 1500, but I am thinking of >>> reducing it for a better latency. Does anyone know the typical packet loss >>> rate? >>> >>> ---------------- >>> 2. Compression >>> ---------------- >>> >>> Has anyone used compression in a network engine? If yes, which >>> compression algorithm? What was the average gain? >>> I have read that John Carmack used the Huffman compression in the Quake >>> 3 network engine because it was well suited for network data compression, >>> but I still need to find some time to implement it in my program and do >>> some tests. >>> >>> ---------------- >>> 3. Channels >>> ---------------- >>> >>> What is your network channel policy? How many channels do you use? What >>> do you send on each channel? >>> At the moment I have 2 channels: one for sending unreliable data (a >>> lot), another one for reliable data. >>> I have chosen this organization to avoid blocking the unreliable data >>> while waiting for an ACK for the reliable data. I am thinking of adding >>> another channel to send high priority reliable data, but I am not sure of >>> the benefit as I already group the reliable data before sending them to >>> limit the blocking. It may be useful if the packet loss rate is too high. >>> >>> ---------------- >>> 4. Server cycle >>> ---------------- >>> >>> I know this aspect depends on the game type, but I would be interested >>> in knowing how your applications work on this. >>> On my side, based on Valve's introduction to network concepts (+ some >>> readings on the UT and Quake network engines), I have decided to implement >>> a 50-ms cycle on the server side. This means that the servers only updates >>> the simulation and notifies the clients each 50 ms. Meanwhile, it only >>> reads the network messages to empty the network event buffer and to handle >>> the reliable packet sending. >>> On the client side, I have introduced a voluntary delay of 70 ms, which >>> goes unnoticed on a user point of view, but helps a lot for overall >>> fluidity as the client will almost always have the following world update >>> (unless a packet is lost). >>> I will soon try to increase the server cycle time to 60 ms to gain >>> 10-20% of bandwidth. I also plan to separate the display delay of the >>> player (it will stay to 70 ms) and the rest of the world (increased to 100 >>> ms). However, I am afraid of the impact on reactivity. >>> >>> Does anyone have a similar architecture? What is your experience on >>> timings? >>> If not, has anyone already developed a FPS or a game very dependant on >>> the network speed? If yes, what are your advices? >>> >>> ---------------- >>> 5. Other >>> ---------------- >>> >>> If anyone has some useful advices on how to improve the network engine >>> of a game/application, I would be pleased to hear it (well, to read it at >>> least). >>> >>> >>> Thanks in advance for all your experience sharing. >>> >>> Best regards, >>> Jeremy Richert >>> >>> _______________________________________________ >>> ENet-discuss mailing list >>> [email protected] >>> http://lists.cubik.org/mailman/listinfo/enet-discuss >>> >>> >> >> >> ---------- Forwarded message ---------- >> From: Lee Salzman <[email protected]> >> To: Discussion of the ENet library <[email protected]> >> Cc: >> Date: Wed, 4 Dec 2013 10:16:40 +0200 >> Subject: Re: [ENet-discuss] Bandwidth/latency optimization >> My experiences as for Cube/Sauerbraten: >> >> 1. packet size >> >> Do not hand ENet lots of tiny packets. This is bad. Use the ENet packets >> as batches, i.e. take a bunch of your little user-level with their own >> specialized little headers (in Sauerbraten these headers are only 1 byte >> per packet!). However, keep these batch sizes underneath the ENet MTU with >> some small reserved space for ENet headers, i.e. peer->mtu - 100 should be >> good for limiting. >> >> The reason this became important for Sauerbraten is we were sending >> *lots* of these, so ENet packet headers would have become a significant >> fraction of the total size, since the payloads of these packets are small. >> That Sauerbraten's internal protocol can get this down to 1 byte sub-header >> essentially gets rid of that issue. These are then batched into >> peer->mtu-100 ENet packets as described above to avoid triggering ENet's >> transparent fragmentation system. >> >> 2. compression >> >> Pack your data intelligently in the first place, and you won't get much >> if any benefit from compression. At least Sauerbraten's packets get at most >> only 5-10% more out of ENet's range coder because we encode, say, integers >> using an extension scheme that uses as few bytes as necessary (not unlike >> how, say, UTF8 works). So, just be as smart as possible on the encoding up >> front, and you can mostly avoid compression and save a lot of the CPU time >> that you would have normally spent on that. >> >> Failing that, others have been happy enough to stick with ENet's built-in >> range coder. >> >> 3. channels >> >> Nothing wrong with a 2 channel setup, and is almost what I use for >> Sauerbraten. I send bulk movement data on a channel that mostly is only >> unreliable. Occasionally reliable stuff gets sequenced into that, but very >> rarely (only for special events like teleports or jumppads). The other >> channel I use for reliable stuff like gun shots, chat, etc. There are other >> schemes like how Quake 3 works that can be used to send even that >> 'reliable' data as unreliable, but it wasn't terribly worth it for >> Sauerbraten's case. >> >> I also have another channel or two I use for file transfers and things >> that I don't want stalling the flow of everything else for long durations. >> >> 4. server-cycle >> >> Sauerbraten just because of its insane movement speed uses 33ms (~30 Hz). >> Whereas for variants with slightly saner move speed, like Red >> Eclipse/Tesseract/AssaultCube, I have preferred to settle on 40ms (25 Hz). >> What matters most for this is the quality of your physics interpolation and >> also that you do a lot of client-side prediction rather than waiting on >> round-trips, then these small changes in rate don't matter quite as much, >> but higher fidelity of input to the interpolation always helps. >> >> >> >> >> On Tue, Dec 3, 2013 at 11:45 PM, Jérémy Richert < >> [email protected]> wrote: >> >>> Hi all, >>> >>> I am currently developing a multiplayer FPS, and I am wondering how to >>> tune ENet to have a network engine as optimized as possible on a bandwidth >>> usage and a latency point of view. >>> I would like to have your thougts on the following aspects: >>> >>> ---------------- >>> 1. Packet size >>> ---------------- >>> >>> I was wondering whether it was better to send large packets or to divide >>> them into small packets. >>> >>> Pros of the large packets : >>> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for >>> IPv4, 10 for ENet) >>> >>> Pros of the small packets : >>> - When a packet is lost, less data is lost >>> - When a reliable packet is lost, resending it requires less bandwidth >>> - Lower latency >>> >>> From what I have read, most people agree that it is recommended to send >>> small packets to avoid packet splitting. This means that the application >>> has to ensure that the size of the data sent does not exceed the MTU. Also, >>> as the MTU depends on the router, some people recommended to have packets >>> that will never be splitted, i.e. <= 576 bytes. >>> What is your experience on this point? >>> For now I have capped the data size to 1500, but I am thinking of >>> reducing it for a better latency. Does anyone know the typical packet loss >>> rate? >>> >>> ---------------- >>> 2. Compression >>> ---------------- >>> >>> Has anyone used compression in a network engine? If yes, which >>> compression algorithm? What was the average gain? >>> I have read that John Carmack used the Huffman compression in the Quake >>> 3 network engine because it was well suited for network data compression, >>> but I still need to find some time to implement it in my program and do >>> some tests. >>> >>> ---------------- >>> 3. Channels >>> ---------------- >>> >>> What is your network channel policy? How many channels do you use? What >>> do you send on each channel? >>> At the moment I have 2 channels: one for sending unreliable data (a >>> lot), another one for reliable data. >>> I have chosen this organization to avoid blocking the unreliable data >>> while waiting for an ACK for the reliable data. I am thinking of adding >>> another channel to send high priority reliable data, but I am not sure of >>> the benefit as I already group the reliable data before sending them to >>> limit the blocking. It may be useful if the packet loss rate is too high. >>> >>> ---------------- >>> 4. Server cycle >>> ---------------- >>> >>> I know this aspect depends on the game type, but I would be interested >>> in knowing how your applications work on this. >>> On my side, based on Valve's introduction to network concepts (+ some >>> readings on the UT and Quake network engines), I have decided to implement >>> a 50-ms cycle on the server side. This means that the servers only updates >>> the simulation and notifies the clients each 50 ms. Meanwhile, it only >>> reads the network messages to empty the network event buffer and to handle >>> the reliable packet sending. >>> On the client side, I have introduced a voluntary delay of 70 ms, which >>> goes unnoticed on a user point of view, but helps a lot for overall >>> fluidity as the client will almost always have the following world update >>> (unless a packet is lost). >>> I will soon try to increase the server cycle time to 60 ms to gain >>> 10-20% of bandwidth. I also plan to separate the display delay of the >>> player (it will stay to 70 ms) and the rest of the world (increased to 100 >>> ms). However, I am afraid of the impact on reactivity. >>> >>> Does anyone have a similar architecture? What is your experience on >>> timings? >>> If not, has anyone already developed a FPS or a game very dependant on >>> the network speed? If yes, what are your advices? >>> >>> ---------------- >>> 5. Other >>> ---------------- >>> >>> If anyone has some useful advices on how to improve the network engine >>> of a game/application, I would be pleased to hear it (well, to read it at >>> least). >>> >>> >>> Thanks in advance for all your experience sharing. >>> >>> Best regards, >>> Jeremy Richert >>> >>> _______________________________________________ >>> ENet-discuss mailing list >>> [email protected] >>> http://lists.cubik.org/mailman/listinfo/enet-discuss >>> >>> >> >> _______________________________________________ >> ENet-discuss mailing list >> [email protected] >> http://lists.cubik.org/mailman/listinfo/enet-discuss >> >> > > _______________________________________________ > ENet-discuss mailing list > [email protected] > http://lists.cubik.org/mailman/listinfo/enet-discuss > >
_______________________________________________ ENet-discuss mailing list [email protected] http://lists.cubik.org/mailman/listinfo/enet-discuss
