Query performance is generally a function of how many objects have to be
returned in the simple case, which seems to be what you are doing here. How
many PlayerGameStates exist per User?
Queries work like this:
- iterate through the index to find all PlayerGameStates with the property
given to find the keys of all PlayerGameStates matching this property
- run the query for all objects
Are PlayerGameStates child entities of a User? I have to double check, but I
believe this could reduce the App Engine datastore's ability to parallelize
the queries-by-key, though the effect of this should be negligible if you
are only fetching 5 objects.
The reason I mentioned "denormalizing" is because the concept of
normalization doesn't make as much sense with App Engine's datastore. We
don't provide referential integrity, for instance, and transactions exist
around entity groups, not globally. You don't pay a penalty for storing data
that is sparse - some User entities can be huge, some can be tiny. You
should optimize for reads. Doing multiple writes and the risk of data
getting out of sync can suck, but as long as it is recoverable you should be
okay. Contrast the performance of the query above with storing
PlayerGameStates inside User. You'd do this:
- Get user by key
That's it! This is really fast, so you don't have to go through doing an
unnecessary query - you just pay your cost at write time, possibly
asynchronously.
On Tue, May 4, 2010 at 12:14 PM, Philip Tucker <ptuc...@gmail.com> wrote:
> I don't think duplicating the data won't work because then I'll be
> doing 2 writes for every turn of the game, and I need that to be fast.
> I don't mind if fetching a user's game list is a little bit slow, but
> 5-20 seconds is too much. I can denormalize all the data such that
> Game 2 fields for each entry in PlayerGameState if that helps, but I'd
> still like to know why it's so slow now.
>
> I do have appstat installed. For one particular misbehaving request,
> the query fetches 5 PlayerGameState records. It iterates through the
> first 4 taking ~20 ms each, then I see a RunQuery taking 8 seconds.
>
> I don't see an easy way to export appstat data, but I can include more
> details if necessary.
>
> Here's the query code in question.
>
> Set<Key> myUserCache = Sets.newHashSet();
> Query query = pm.newQuery(PlayerGameState.class);
> query.declareParameters(
> "com.google.appengine.api.datastore.Key userKeyParam");
> query.setFilter("userKey == userKeyParam");
> List<PlayerGameState> playerStates =
> (List<PlayerGameState>) query.execute(playerKey);
> List<Game> games = Lists.newArrayList();
> for (PlayerGameState playerState : playerStates) {
> Game game = playerState.getGame();
> myUserCache.add(game.getPlayerState().getUserKey());
> myUserCache.add(game.getOpponentState().getUserKey());
> games.add(game);
> }
>
>
>
> On May 3, 1:16 am, "Ikai L (Google)" <ika...@google.com> wrote:
> > Indexes don't work the same way they do in relational databases. If you
> can
> > fetch a value with a query, that likely means an index exists on that
> > property. In a relational database, when an index is not present, the
> > database will do a full table scan.
> >
> > A fetch by Key is always fast. Have you considered storing multiple games
> > inside a single Game instance? Or simply storing the data you need both
> in
> > the Game instances? If you want to cut down on request latency, do as
> many
> > by key gets as possible. Don't try to approach the modeling from a
> > normalization perspective. How many entities are you retrieving? And have
> > you tried looking at what's happening via AppStats yet (
> http://code.google.com/appengine/docs/java/tools/appstats.html)?
> >
> >
> >
> >
> >
> > On Fri, Apr 30, 2010 at 1:54 AM, Philip Tucker <ptuc...@gmail.com>
> wrote:
> > > I've got a Game object that includes two PlayerGameState objects. Both
> > > are persistence-capable. My query fetches all PlayerGameState objects
> > > for a particular user, then I get the associated game and other
> > > player. The initial fetch is always fast, and the entire operation is
> > > generally lass than a second. But periodically, maybe 5 of the time,
> > > it takes much longer, from 5-20 seconds. It's not fetching the
> > > PlayerGameState objects that's slow, it's accessing
> > > PlagerGameState.game and Game.playerStates.
> >
> > > I shouldn't need to add an index on the keys, should I? I'd like to
> > > fetch the Game objects directly but as far as I can tell we can't
> > > query on a joined table; ie, I can't do something like "select from
> > > Game, PlayerGameState join on PlayerGameState.game where
> > > PlayerGameState.userKey == userKeyParam".
> >
> > > @PersistenceCapable(identityType = IdentityType.APPLICATION)
> > > public class Game {
> > > @PrimaryKey
> > > @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
> > > private Key key;
> >
> > > @Persistent(mappedBy = "game")
> > > @Element(dependent = "true")
> > > private List<PlayerGameState> playerStates;
> >
> > > ...
> > > }
> >
> > > @PersistenceCapable(identityType = IdentityType.APPLICATION)
> > > public class PlayerGameState {
> > > @SuppressWarnings("unused")
> > > @PrimaryKey
> > > @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
> > > private Key key;
> >
> > > @Persistent
> > > private Key userKey;
> >
> > > @Persistent
> > > private Game game;
> >
> > > ...
> > > }
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Google App Engine for Java" group.
> > > To post to this group, send email to
> > > google-appengine-j...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com><google-appengine-java%2B
> unsubscr...@googlegroups.com>
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine-java?hl=en.
> >
> > --
> > Ikai Lan
> > Developer Relations, Google App Engine
> > Twitter:http://twitter.com/ikai
> > Delicious:http://delicious.com/ikailan
> >
> > ----------------
> > Google App Engine links:
> > Blog:http://googleappengine.blogspot.com
> > Twitter:http://twitter.com/app_engine
> > Reddit:http://www.reddit.com/r/appengine
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> > To post to this group, send email to
> google-appengine-j...@googlegroups.com.
> > To unsubscribe from this group, send email to
> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com>
> .
> > For more options, visit this group athttp://
> groups.google.com/group/google-appengine-java?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to
> google-appengine-j...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
>
--
Ikai Lan
Developer Relations, Google App Engine
Twitter: http://twitter.com/ikai
Delicious: http://delicious.com/ikailan
----------------
Google App Engine links:
Blog: http://googleappengine.blogspot.com
Twitter: http://twitter.com/app_engine
Reddit: http://www.reddit.com/r/appengine
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
