I don't think EA is going to hunt me down for spilling almost decade-old history, so...
I worked on The Sims Online and Ultima X Odyssey (sadly cancelled). Both were sharded the "boring way" - each shard was a separate city/realm with an entirely separate set of character accounts. If a shard was too full, it was closed to new players. Each TSO shard (IIRC, there were 10 shards at launch) was a cluster of 20 top-of-the-line (at the time) 1U servers. Each shard was capable of supporting 1k-2k peak simultaneous users... rather a disappointing number for all that hardware, actually. The biggest bottleneck were the "simulators" which actually ran code hacked out of The Sims PC game to try to make the world a little more interesting than a chat room (it didn't). Those poor boxes typically ran at load averages > 500, sometimes > 1000. I know less about the UXO cluster since the core team was in Austin (I was at EARS by then) and the product never made it to launch. We never really found out how it would perform in the real world. It was doubtlessly less CPU intensive than TSO but it was built with the same ancient toolkit that EA inherited from some ill-considered acquisition a half-decade prior. You can probably get a shudder out of most EA Online veterans just by mentioning Aries/Voltron. At any rate, the architecture was pretty similar so it's another "big cluster of machines". I didn't work directly on the game engine (just the login servers, account management stuff, online community, etc) so I'd be talking out of my ass if I offered advice on how to actually build one today... except one thing: You want a garbage-collected language. C++ is a horrid tool for the job. I don't think they ever got all the memory leaks out of the servers... and segfaults take down a whole world. On the other hand, Go looks like a language specifically designed for MMORPG programming. Jeff On Wed, Dec 21, 2011 at 11:42 AM, Ikai Lan (Google) <[email protected]> wrote: > Hey Jeff, > > Just out of curiosity, what did you shard by? Realm/region? Do the shards > just run on gigantic machines? Does the gating factor tend to be amount of > combat or players? I don't know if you can share any of this stuff - I just > love hearing war stories. The game development world is one I know very > little about. > > Fun fact: Alfred on the datastore team knows a TON about video game > development. =) > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > > On Wed, Dec 21, 2011 at 9:12 AM, Jeff Schnitzer <[email protected]> wrote: >> >> I used to work on MMO games at EA. Not on the the core game engines, >> but close enough to understand how they work. >> >> Assuming you want to create an MMORPG (see: the subject line), there's >> no way in hell this will work. The persistent-world state is composed >> of every player, every monster, and what they are doing at any moment. >> Every time a user clicks the mouse or hits a key on the keyboard it >> changes the PSW state. Every "tick" of a timer can change the PSW >> state. You can't just read and write this to the datastore - the data >> volumes would be insane and even if you could afford the bill, the >> latency will *destroy* your user experience. >> >> MMORPG games, in my experience, are generally implemented as an >> in-memory game state that gets written to a database at intervals. >> The in-memory state of a shard is very long-lived; a badly-timed >> reboot of a shard usually kicks everyone off the world. You could do >> this with a GAE backend but they aren't reliable enough - users get >> *very* pissy about getting booted off the server in the middle of >> swordfight. >> >> Another thing is that the MMORPG games I'm familiar with have been >> implemented as more-or-less single-threaded async systems. There are >> a lot of good reasons to do this. Maybe you can make a different >> approach work but when you're trying to squeeze tens of thousands of >> active players into a single PSW you pretty quickly hit the >> scalability limits of synchronous architectures. >> >> So really, if you are serious about building an MMORPG, you do not >> want to use Appengine. If you're seriously thinking about using >> Appengine, I can tell you right now that 1) MMORPG games are hard to >> implement and 2) you don't know enough about how to design them to >> make this work. Keep researching, there's a lot of literature out >> there... and a few opensource projects to learn from. Even a few >> frameworks (commercial and not). They will not run on GAE. >> >> Note that this doesn't encompass all MMO games; you could build a >> simple turn-based game on GAE quite easily. But MMORPGs are a >> different beast. >> >> Jeff >> >> On Wed, Dec 21, 2011 at 5:07 AM, Andrius A <[email protected]> wrote: >> > You can increase entity update limit by using MS datastore and shards. >> > This >> > way you can have a rate of 100/s for your logical single instance. But >> > don't >> > forget that MS is not reliable, neither memcache or backends. You will >> > face >> > a nightmare trying to make it work fast and reliable. >> > >> > On Dec 20, 2011 7:38 PM, "Kaan Soral" <[email protected]> wrote: >> >> >> >> My initial idea was to keep everything in one datastore entity, in a >> >> blob >> >> property, players would post changes to this entity, and when they >> >> request >> >> the game state, they would also get contents of this item >> >> But I totally forgot about the 5/s datastore write limit for a single >> >> entity >> >> >> >> So even with short-polling, it wouldn't work >> >> >> >> Backend + short-polling sounds like a great idea now to me, although I >> >> didn't quite get the git idea >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "Google App Engine" group. >> >> To view this discussion on the web visit >> >> https://groups.google.com/d/msg/google-appengine/-/vobs7E5lllYJ. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/google-appengine?hl=en. >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> We are the 20% >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- We are the 20% -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
