Like appengine, big query is a bit of a work in progress and always moving. We use big query to store data for reports, however it does have some gotchas: It doesn't like many small writes (I.e streaming records), this has a huge performance impact on queries. (Note this may also have changed since I last looked) There are limits on queries, for example they only allowed queries to return over 250mb in result sets in the last 6 or so months. I'm not sure what the upper bound is now.
If you want to use GAE + BQ, you're pretty much at square one - you need to store all requests temporarily and process a batch, just like GCS. The good news is GCS and BQ work pretty well together. I would recommend BQ as a good tool for querying through your millions of rows in reasonable time (not sure about billions - I have yet to see an example of a truly 'big data' BQ usage. The 250mb limit sort of put me off on BQ and true big data). Regarding using the data store, I have hit a scenario before where the indexes got so big for entity insertion that putting a small number (say 25 entities) hit deadline limits. I interpret this to mean that basically there is an effective upper limit on the amount of data you can store in the data store for a given Kind, and youll hit that limit sooner if you have many search dimensions (ie indexes). I would be interested in hearing what you finally end up choosing, please repost after you decide, with a little bit of explanation on why. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
