[hlds_announce] Public steam client released with server browser changes

[Fletcher Dunn (via hlds_announce list)]

The steam client was updated with the changes to the server browser protocol 
that have been discussed.

More info here:
https://steamcommunity.com/discussions/forum/14/2974028351344359625/

The latest binaries for all gameserver platforms, with experimental features to 
harden against server browser packet attacks, can be obtained here:
http://media.steampowered.com/apps/steamworks/steam_bins_6246211.zip

___
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/

[hlds_announce] Beta server binaries available for testing

[Fletcher Dunn (via hlds_announce list)]

Hello!

Some beta steam binaries are available for testing.  They have several 
improvements meaningful to dedicated servers, mostly related to handling of 
server browser packets such as the A2S_INFO packet that has been discussed.

If you do not set any environment variables, the new binaries have the 
following change:


*Improved challenge generation.  (Challenges are generated using 
siphash, so there is no table to search or fill up.)

There are also some environment variables that can be set to active new 
features and behaviour:


*STEAM_GAMESERVER_A2S_INFO_REQUIRE_CHALLENGE=1.  This will opt into the 
new behavior we have been discussing for A2S_INFO packets.  It is primarily 
intended for testing 3rd party query clients.  It is *not* currently compatible 
with the main steam client, so it should not enabled be otherwise!  It is 
compatible with the current steam beta client.


*STEAM_GAMESERVER_RATE_LIMIT_200MS=N.  This will drop any 
connectionless packets (A2S_INFO, A2S_RULES, A2S_PLAYERS) from a given IP after 
more than N are received in a 200ms window.  By default, this rate limiting is 
disabled, but a reasonable value might be somewhere around 25-75 range.


*STEAM_GAMESERVER_PACKET_HANDLER_NO_IPC.  If this variable is set, then 
the Steamworks packet handling calls will use a "fast path".  The design of the 
Steamworks SDK did not have dedicated servers in mind, and most API calls are 
serialized over an IPC, and in fact execute in the steam client process.  A 
dedicated server does not communicate with a steam client (there usually isn't 
one running), but to keep the code simple, the same basic design is used - 
there are two threads, and all API calls are serialized and executed in the 
Steam thread.  This common architecture makes all access to the steam related 
data structures essentially single-threaded, vastly simplifying the code.  But 
it adds some overhead for calls, and for the packet handling in particular that 
overhead can be significant.  Turning on the environment variable will bypass 
this serialization and context switch, which makes these calls, much, much 
faster.  Note that this only affects a server in "shared socket mode" - meaning 
the game port and the query port are the same.

You can download the appropriate binaries here:
http://media.steampowered.com/apps/steamworks/steam_bins_6243860.zip

If you have an opportunity to try out these changes, please let me know if you 
are able to tell any differences, or run into any problems.  If this testing 
goes well and does not uncover any issues, we will probably change the defaults 
to enable the IPC bypass and enable some rate limiting at a conservative rate.

NOTE: CSGO servers sometimes process A2S_INFO in the engine, based on the 
values of host_info_show, host_players_show, and host_rules_show.  The 
corresponding packets will use go through these steamclient binaries, if 
host_info_show=2, host_players_show=2, or host_rules_show is true, respectively.
___
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/

[hlds_announce] RE: [External Mail] Re: [Csgo_servers] RE: Changes to A2S_INFO - take 2

[Fletcher Dunn (via hlds_announce list)]

Yes, the A2S_INFO w/ challenge just appends the 4 byte challenge after “TSource 
Engine Query\0”.  PLUS in the future there may be other stuff!  Specifically, a 
future update to the protocol may append a server-assigned “revision number” of 
the game data (map name, server name, tags, etc) obtained from the master 
server.  The game server may then omit those details in its reply that have not 
changed since that revision.  This is totally optional.  The server can 
continue to provide all the details if it wishes.  (Or if some middle box is 
caching them.)  But if it wishes to send a smaller reply, it can do so.

Unfortunately, my testing revealed that at least one relatively large game 
hosting provider was blocking packets that had only 4 extra 0’s appended to the 
original A2S_INFO request.  So for that hosting provider, at least, there 
doesn’t appear to be any change that could be made to the original packet that 
would not get blocked.  And that is just the first example of very aggressive 
filtering brought to our attention, because it was a relatively large user of 
this service, from a partner with whom we happen to be in frequent 
communication.  There are presumably many other hosting providers and server 
operators that have already (quite reasonably) taken draconian steps such as 
this to protect their servers against abuse of this ancient and poorly-designed 
protocol from the early 2000’s.

This is why “Just tell the game hosting providers to change their filters” is 
not a serious alternative.  There are too many of them, and there is not a 
clear channel through which the communication could be made to reach all of 
them.  This forum is perhaps the best, and many did not get the first message.  
Even now, they are slow to respond when I was in direct communication when 
them.  So “Just make them change the filters” in practice means to go ahead and 
break games, and use player reports of the game being broken to get the 
attention of the operators.  Personally I find that approach unacceptable.  
This latest proposal is definitely not the ideal design one would make if one 
were starting from scratch with no existing users.  But it seems to be a good 
path forward to address the reflection attack vulnerability, given the current 
state of affairs and the constraints that are in place given how many users of 
this protocol there are.  This plan puts the control in the hands of the server 
operators.  If compatibility with third party clients is important to them, 
they can retain the current behavior indefinitely.  It’s up to them.  The 
official client will be ready to prove they are not spoofing soon, and they can 
request that proof when they decide they value protection against the 
reflection amplification over compatibility.

From: csgo_serv...@list.valvesoftware.com  
On Behalf Of David Parker
Sent: Thursday, December 3, 2020 8:46 PM
To: csgo_serv...@list.valvesoftware.com
Cc: hlds_announce@list.valvesoftware.com
Subject: [External Mail] Re: [Csgo_servers] RE: Changes to A2S_INFO - take 2

Thanks for the updates.  I assume this means that the "TSource Engine Query\0" 
remains in place, and the challenge response (when provided) gets appended 
after the trailing null byte?

Also, this may be a silly question, but is there a smaller padded packet size 
that could be used, which doesn't trigger DDoS protections but is still big 
enough that it makes a reflection attack less appealing to the asshats who 
launch them?  Just curious.

Thanks,
Dave

On Thu, Dec 3, 2020 at 10:54 PM Fletcher Dunn - fletcherd at 
valvesoftware.com (via csgo_servers list) 
mailto:csgo_serv...@list.valvesoftware.com>>
 wrote:
A steam client beta has been released:

https://steamcommunity.com/groups/SteamClientBeta/announcements/detail/2896341257765264787

It understands how to respond if the server issues a challenge in response to 
an A2S_INFO request.  Importantly because of the existing filtering environment 
servers run in, the client will behave EXACTLY as it did before, until the 
server replies in the new method.  
(https://twitter.com/ZPostFacto/status/1334700095221104640)

The protocol is now as follows:


•Client will send the exact A2S_INFO packet that it has always sent, no 
more, no less.

•A new server will reply with a challenge, using the same S2C_CHALLENGE 
packet that’s used for the A2S_PLAYERS and A2S_RULES packets.  (Indeed, if a 
client is quick enough, it can use the same challenge for multiple requests.)

•Now, a client will send a A2S_INFO with the challenge appended.  Also: 
DO NOT ASSUME THAT ANY EXTRA BYTES AFTER THE CHALLENGE ARE INVALID.  This is 
reserved for future expansion to the protocol!  There are some more protocol 
changes in development right now designed to have the client obtain more 
information from the master server, thus reducing the amount of information 
that must come from the server.  Those improvements won’t be 

[hlds_announce] RE: Changes to A2S_INFO - take 2

[Fletcher Dunn (via hlds_announce list)]

A steam client beta has been released:

https://steamcommunity.com/groups/SteamClientBeta/announcements/detail/2896341257765264787

It understands how to respond if the server issues a challenge in response to 
an A2S_INFO request.  Importantly because of the existing filtering environment 
servers run in, the client will behave EXACTLY as it did before, until the 
server replies in the new method.  
(https://twitter.com/ZPostFacto/status/1334700095221104640)

The protocol is now as follows:


*Client will send the exact A2S_INFO packet that it has always sent, no 
more, no less.

*A new server will reply with a challenge, using the same S2C_CHALLENGE 
packet that's used for the A2S_PLAYERS and A2S_RULES packets.  (Indeed, if a 
client is quick enough, it can use the same challenge for multiple requests.)

*Now, a client will send a A2S_INFO with the challenge appended.  Also: 
DO NOT ASSUME THAT ANY EXTRA BYTES AFTER THE CHALLENGE ARE INVALID.  This is 
reserved for future expansion to the protocol!  There are some more protocol 
changes in development right now designed to have the client obtain more 
information from the master server, thus reducing the amount of information 
that must come from the server.  Those improvements won't be possible if 
assumptions are made about packet sizes!

I'll post again when there are server binaries available that can opt into the 
new behavior, fixing the reflection attack vulnerability.  You will not want to 
opt in until all clients you care about are speaking the new protocol.  For 
steam clients, that will probably at least a couple of weeks.

Please share this with any authors of third party clients that you know!


From: csgo_serv...@list.valvesoftware.com 
Sent: Thursday, December 3, 2020 2:42 PM
To: 'hlds_announce@list.valvesoftware.com' 
; csgo_serv...@list.valvesoftware.com
Subject: [Csgo_servers] Changes to A2S_INFO - take 2

The previous change to pad the server browser query A2S_INFO packets has 
triggered some aggressive Anti-DDoS filters for some games.  This change was 
made to address a reflection amplification attack in the protocol.  So it looks 
like we will need to address the vulnerability by securing the response with a 
challenge, in the same way that the A2S_PLAYERS and A2S_RULES queries work.  
We'll be releasing a new client soon that sends the small A2S_INFO packets 
again, but also understands how to reply to a server that replies with a 
challenge instead of the data.  This protocol does make it more complicated to 
write a custom client for the protocol (although not drastically so), and means 
that the query traffic cannot be trivially filtered at the edge.  
Unfortunately, it looks like in the current environment, that is what we need 
to do.

Further bulletins as events warrant.

___
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/
___
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/

[hlds_announce] Changes to A2S_INFO - take 2

[Fletcher Dunn (via hlds_announce list)]

The previous change to pad the server browser query A2S_INFO packets has 
triggered some aggressive Anti-DDoS filters for some games.  This change was 
made to address a reflection amplification attack in the protocol.  So it looks 
like we will need to address the vulnerability by securing the response with a 
challenge, in the same way that the A2S_PLAYERS and A2S_RULES queries work.  
We'll be releasing a new client soon that sends the small A2S_INFO packets 
again, but also understands how to reply to a server that replies with a 
challenge instead of the data.  This protocol does make it more complicated to 
write a custom client for the protocol (although not drastically so), and means 
that the query traffic cannot be trivially filtered at the edge.  
Unfortunately, it looks like in the current environment, that is what we need 
to do.

Further bulletins as events warrant.
___
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/

[hlds_announce] Steam Client beta released with changes to the A2S_INFO protocol

[Fletcher Dunn (via hlds_announce list)]

A Steam client beta has just been released, with the following change notes 
relevant to the A2S_INFO discussion:

Server Browser

*   Server browser packets (A2S_INFO, A2S_PLAYER, A2S_RULES) sent by 
clients will now be at least 1200 bytes. (For more details, see 
https://steamcommunity.com/discussions/forum/14/2989789048633291344/) Third 
party tools that send these packets are especially encouraged to read this 
thread.)

*   Improved gameserver challenge generation to harden against certain DoS 
attacks

https://steamcommunity.com/groups/SteamClientBeta/announcements/detail/2896339990496271925

Also, if you use the steamclient.dll/.so with that beta, you can activate the 
new, stricter message handling on the gameserver by setting the environment 
variable STEAM_GAMESERVER_MIN_CONNECTIONLESS_PACKET_SIZE=1200

This is just the beta steam client, not the full public release.  Only a set of 
users will be using this client, and we are not quite to deployment step #1 
described below.



From: Fletcher Dunn
Sent: Monday, November 16, 2020 4:47 PM
To: 'hlds_announce@list.valvesoftware.com' 

Subject: RFC: Changes to the A2S_INFO protocol

Hello!

Over the next couple of months we will be releasing some changes to how servers 
and clients using steamclent.so/dll handle the venerable Source engine A2S_INFO 
message used by the server browser.  This includes the Steam client server 
browser, all Source engine games, and all Steam games using the 
ISteamMatchmaking API.  The purpose of these changes is a long overdue fix for 
a reflection attack vulnerability.

This email is to let you know what those plans are and to solicit your 
feedback.  Fixing the vulnerability requires changing the protocol and will 
necessarily break existing third party utilities that speak the protocol.

Currently, the A2S_INFO packet looks like this:
4 bytes: 0x
1 byte: 0x54  (A2S_INFO packet type identifier)
20 bytes: "Source Engine Query\0"

The proposal is for clients to modify the A2S_INFO  packet they send in one of 
two ways:

Option 1: Pad the message with zeros, so that the request is larger than the 
reply.  The passes size is TBD, but it will probably be at least 800 bytes, and 
perhaps as high as 1200.  Feedback is requested concerning this size.

Option 2: Append a 4-byte anti-spoofing challenge obtained using the existing 
A2S_PLAYER or A2S_RULES messages.

Note that both options produce a packet that is acceptable to the current code 
in steamclient.so/dll.  However, any custom handlers might have stricter 
behavior, and will need to be updated to be aware than "extra" data might 
appear after the end of the magic string in packets from legitimate clients.

Once all clients are using one of these two options, a server wishing to avoid 
being vulnerable to reflection attacks could drop any A2S_INFO packets below a 
minimum size without a challenge.

The changes would be deployed as follows:


1.) First, we will release a new Steam client that sends A2S_INFO messages 
padding to a minimum size.  (Option #1 above.)  Since it takes time for Steam 
client updates to roll out to all Steam users, and for third parties to change 
their code to make queries in the new format, we will not change the server to 
require the new format by default.  However, the server code will be updated to 
look for an environment variable that can be used to opt into the new, stricter 
behavior.  This is so that third parties can test their clients to make sure 
they are compliant with the new server code.

2.) As more clients upgrade to the new code and third party tools are 
updated to send queries in the new format, server operators may elect to opt 
into the new behavior at their discretion using the environment variable.

3.) After some time has passed (and we have posted several warnings on this 
mailing list), we will ship a new steamclient.so/.dll that has the strict 
behavior enabled by default.  A different environment variable can be used to 
use the older, more permissive behaviour.

If you have any concerns or feedback about this change, please post it to 
hlds_announce@list.valvesoftware.com.
  After feedback has been collected and details finalized, I'll post again with 
more technical details about the changes that are going to be made.

___
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/

[hlds_announce] RFC: Changes to the A2S_INFO protocol

[Fletcher Dunn (via hlds_announce list)]

Hello!

Over the next couple of months we will be releasing some changes to how servers 
and clients using steamclent.so/dll handle the venerable Source engine A2S_INFO 
message used by the server browser.  This includes the Steam client server 
browser, all Source engine games, and all Steam games using the 
ISteamMatchmaking API.  The purpose of these changes is a long overdue fix for 
a reflection attack vulnerability.

This email is to let you know what those plans are and to solicit your 
feedback.  Fixing the vulnerability requires changing the protocol and will 
necessarily break existing third party utilities that speak the protocol.

Currently, the A2S_INFO packet looks like this:
4 bytes: 0x
1 byte: 0x54  (A2S_INFO packet type identifier)
20 bytes: "Source Engine Query\0"

The proposal is for clients to modify the A2S_INFO  packet they send in one of 
two ways:

Option 1: Pad the message with zeros, so that the request is larger than the 
reply.  The passes size is TBD, but it will probably be at least 800 bytes, and 
perhaps as high as 1200.  Feedback is requested concerning this size.

Option 2: Append a 4-byte anti-spoofing challenge obtained using the existing 
A2S_PLAYER or A2S_RULES messages.

Note that both options produce a packet that is acceptable to the current code 
in steamclient.so/dll.  However, any custom handlers might have stricter 
behavior, and will need to be updated to be aware than "extra" data might 
appear after the end of the magic string in packets from legitimate clients.

Once all clients are using one of these two options, a server wishing to avoid 
being vulnerable to reflection attacks could drop any A2S_INFO packets below a 
minimum size without a challenge.

The changes would be deployed as follows:


1.) First, we will release a new Steam client that sends A2S_INFO messages 
padding to a minimum size.  (Option #1 above.)  Since it takes time for Steam 
client updates to roll out to all Steam users, and for third parties to change 
their code to make queries in the new format, we will not change the server to 
require the new format by default.  However, the server code will be updated to 
look for an environment variable that can be used to opt into the new, stricter 
behavior.  This is so that third parties can test their clients to make sure 
they are compliant with the new server code.

2.) As more clients upgrade to the new code and third party tools are 
updated to send queries in the new format, server operators may elect to opt 
into the new behavior at their discretion using the environment variable.

3.) After some time has passed (and we have posted several warnings on this 
mailing list), we will ship a new steamclient.so/.dll that has the strict 
behavior enabled by default.  A different environment variable can be used to 
use the older, more permissive behaviour.

If you have any concerns or feedback about this change, please reply to this 
steam post:
https://steamcommunity.com/discussions/forum/14/2989789048633291344/

After feedback has been collected and details finalized, I'll post again with 
more technical details about the changes that are going to be made.
___
To unsubscribe, edit your list preferences, or view the list archives,
please visit:
https://list.valvesoftware.com/