Datastore is not designed for large scale data aggregation. Its emphasize is on fast and horizontally scalable reads. Though it *can* be used to query over large datasets, it is nowhere near optimized in the same way that SQL or SQL-like technologies are. For full dataset joins and aggregation, BigQuery really shines. The other options mentioned above Postgres and MapReduce would also be appropriate depending on your data. I would strongly suggest using BigQuery as it comes with a Datastore loading solution <https://cloud.google.com/bigquery/loading-data-cloud-datastore> and is designed for precisely what you describe. In addition, reading all entities in order to aggregate some data from them will be far more costly with Datastore than the equivalent job in BigQuery.
In your earlier comments, you mention that you cannot use BigQuery due to an update and delete requirement. Data can be updated in BigQuery <https://cloud.google.com/bigquery/docs/updating-data>. Do such update capabilities meet the requirements you mentioned above? If not, what is ability do you think is missing from these update and delete features? Lastly, in your example case, you mentioned that a given query could pull anywhere from 10 to 10,000 entities. How many entities are there in total? Knowing the total count of entities through which your query must search is crucial to understanding the likely costs of queries in time and quota usage. How many total entities do you expect to have in a given dataset within the next 6-12 months based on recent month's growth? The answer to this should help you decide whether to invest time into replicating data into a aggregation tool like BigQuery and building useful queries for reporting. On Wednesday, February 8, 2017 at 4:46:28 AM UTC-5, Rajesh Gupta wrote: > > All of the above solutions involve more expertise and time, which is not > suitable for small teams. Ofcourse, they achieve good results for large > data and big companies. Small companies will loose the benefit of using > the appengine paas with such wide solutions. > It is beneficial to provide some aggregation layer on top of datastore > with built in fast google technologies. > Interested developers will choose to use it. > > Regards, > Rajesh > *www.VeersoftSolutions.com <http://www.veersoftsolutions.com/>* > *www.GainERP.com <https://www.gainerp.com/>* > *Accounting/Inventory/Orders/Sales/Purchase on Google Cloud Platform and > Mobile* > > On Sat, Jan 28, 2017 at 12:48 AM, Jim <jeb62...@gmail.com> wrote: > >> Another option is to use map-reduce against your datastore tables for >> aggregation of truly 'big' data sets. It's nowhere near as flexible as >> some of the other options mentioned here, but if your requirements are >> fairly static it works great and will allow you to keep your data in one >> place. >> >> >> >> On Thursday, January 26, 2017 at 2:52:06 PM UTC-6, George (Cloud Platform >> Support) wrote: >>> >>> How large can your sales invoice data get in the end? The solutions >>> recommended above may work well for relatively small volumes. If you need >>> to process terabytes of data in the end, Cloud Bigtable >>> <https://cloud.google.com/bigtable/docs/overview> might prove speedier >>> and cost less overall. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to google-appengine+unsubscr...@googlegroups.com. >> To post to this group, send email to google-appengine@googlegroups.com. >> Visit this group at https://groups.google.com/group/google-appengine. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/9fe3d5f5-1995-461b-85cf-1006877f3dad%40googlegroups.com >> >> <https://groups.google.com/d/msgid/google-appengine/9fe3d5f5-1995-461b-85cf-1006877f3dad%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/0cbca628-9706-4f0e-ac49-714227d9005a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
