Thanks Nickolaus for sharing your experience. I've just come across this simple yet very elegant solution by Nacho Coloma to break through the 10m time limit of taks queues:
http://blog.extrema-sistemas.com/long-tasks-in-app-engine-for-java/ https://github.com/icoloma/queue4gae I'm going to play with it, it looks like that's what I need On Tuesday, 16 December 2014 11:35:13 UTC+13, Nickolas Daskalou wrote: > > We use BigQuery (Python) to analyse visitor and click data on followus.com > pages. > > From what we've seen, BigQuery queries scale really well over large > datasets and complex queries. > > However, there is an overhead to each BigQuery query which makes even > simple queries over small datasets take a couple of seconds. > > So keep that in mind. > > There is also the ability to parallelise multiple different queries in > order to reduce total query time. > > We skip the Google Cloud Storage step and use streaming inserts > <https://cloud.google.com/bigquery/streaming-data-into-bigquery> with a > combination of Memcache (App Engine side) and insertIds (BigQuery side) to > avoid duplicate inserts. This could work for you too if your rows are not > too large. > > Since BigQuery is append-only, if you want it to replace the Datastore for > queries, you will need to add versioning to each row you insert into > BigQuery, and construct a query which only considers the latest version of > an entity/record. > > Nick > > > On 16 December 2014 at 08:05, Emanuele Ziglioli < > the...@emanueleziglioli.it <javascript:>> wrote: >> >> Hi Gilberto, >> >> quick question: do you think BigQuery could possibly replace the >> Datastore for queries? >> A big Datastore pain point is the fact for each query requires an index, >> while BigQuery doesn't have this restriction. >> Do you think it would be feasible for a GAE app to internally redirect >> client requests to BigQuery? >> >> I'm tempted to add support for BigQuery to Siena (a java ORM), that would >> be a big win for this project I keep maintaining for our own use: >> https://github.com/siena/siena/tree/async >> >> Emanuele >> >> >> On Friday, 12 December 2014 06:17:22 UTC+13, Gilberto Torrezan Filho >> wrote: >>> >>> Actually I just migrated my statistics job from MapReduce to BigQuery >>> (using the Datastore -> Cloud Storage -> BigQuery pattern) =) >>> >>> I strongly recommend the book "Google BigQuery Analytics" from Jordan >>> Tigani and Siddartha Naidu if you plan to use or know more about BigQuery. >>> I got mine at I/O this year (the last book in the box) =) >>> >>> BigQuery is awesome but have its quirks - the append-only tables is just >>> one of them. You have to shape your business logic to handle that before >>> starting to heavily use it. >>> >>> If you don't need statistics, you probably don't need BigQuery. >>> >>> The sad part is that I spent more than 2 months tweaking and improving >>> my whole pipeline stack trying to get a better performance (or >>> cost-effectiveness), when I could just be using BigQuery to solve my >>> problems. Anyway, it was a good lesson. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to google-appengi...@googlegroups.com <javascript:>. >> To post to this group, send email to google-a...@googlegroups.com >> <javascript:>. >> Visit this group at http://groups.google.com/group/google-appengine. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
