[google-appengine] Re: Datastore Read and Writes Statstics
Is there an update on this? Have we filed an issue? Our quota depleted so fast when we use map-reduce... On Sep 29, 7:19 pm, djidjadji wrote: > It would be really nice if the WriteOps and ReadOps are logged for > each request. Like the loading_request and ms time that is added to a > line in the log. > That way you can download the logs and find out which request is > making the most Ops and perhaps needs a look at if it can be > optimized. > > Op 22 september 2011 14:29 heeft Vivek Puri het > volgende geschreven: > > > > > > > > > For "Estimated Charges Under New Pricing", my app is showing $40/per > > day for Writes and $13/day for Reads. $40 worth of writes is 40 > > million write operations. Right now i have not idea which table is > > being written to most and contributing to that number. Can AppEngine > > team provide statistics on these write and read operations. I would > > hope this is something similar to existing "Datastore Statistics" > > tool. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore Read and Writes Statstics
I'll take a shot at mapping these. datastore_v3.Put will result in one Entity Put + One Index Writes for each index field that was added (in case of new records), or Two Index Writes for each index field that was modified (an index delete for the old index value and an index write for the new value). datastore_v2.Get is equivalent to Entity Fetch datastore_v3.RunQuery is equivalent to a Query. It seems that app stats was written to reflect on the costs in the old pricing model. It'll be nice it is redesigned to be consistent with the new pricing model. I don't know who supports this tool, but can anyone in the GAE team tell us if there are plans on updating it? Is app stats open source, allowing anyone to take a stab at it? On Thu, Sep 22, 2011 at 9:47 AM, Vivek Puri wrote: > As per this doc: > http://code.google.com/appengine/kb/postpreviewpricing.html#operations_charged_for > > There are 3 categories of Datastore operations: > Write operations (Entity Put, Entity Delete, Index Write) Each of > these operations will cost $0.10 per 100k operations. > Read operations (Query, Entity Fetch) Each of these operations will > cost $0.07 per 100k operations. > Small operations (Key Fetch, Id Allocation) Each of these operations > will cost $0.01 per 100k operations > > From App Stats, i see the following datastore related operations: > datastore_v3.Get > datastore_v3.RunQuery > datastore_v3.Put > datastore_v3.AllocateIds > datastore_v3.BeginTransaction > datastore_v3.Commit > > Can someone from AppEngine team let me know under which category do > the above operations fall under. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore Read and Writes Statstics
Just to jump in here, I also agree that we could do with an easy way of identifying where costs are coming from. I like appstats, but it gives you a detail level view for one request. Perhaps information like this might be useful for people Entity Name - Total Writes Total Reads Total Small Operations It could also give stats on indexes on this entity. (As an easy way to remind new people that they really need to keep an eye on indexes) J. ps.. I usually caveat my posts with this - I don't have a live app, so don't listen overly to my words. On Sep 22, 3:47 pm, Vivek Puri wrote: > As per this > doc:http://code.google.com/appengine/kb/postpreviewpricing.html#operation... > > There are 3 categories of Datastore operations: > Write operations (Entity Put, Entity Delete, Index Write) Each of > these operations will cost $0.10 per 100k operations. > Read operations (Query, Entity Fetch) Each of these operations will > cost $0.07 per 100k operations. > Small operations (Key Fetch, Id Allocation) Each of these operations > will cost $0.01 per 100k operations > > From App Stats, i see the following datastore related operations: > datastore_v3.Get > datastore_v3.RunQuery > datastore_v3.Put > datastore_v3.AllocateIds > datastore_v3.BeginTransaction > datastore_v3.Commit > > Can someone from AppEngine team let me know under which category do > the above operations fall under. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore Read and Writes Statstics
As per this doc: http://code.google.com/appengine/kb/postpreviewpricing.html#operations_charged_for There are 3 categories of Datastore operations: Write operations (Entity Put, Entity Delete, Index Write) Each of these operations will cost $0.10 per 100k operations. Read operations (Query, Entity Fetch) Each of these operations will cost $0.07 per 100k operations. Small operations (Key Fetch, Id Allocation) Each of these operations will cost $0.01 per 100k operations >From App Stats, i see the following datastore related operations: datastore_v3.Get datastore_v3.RunQuery datastore_v3.Put datastore_v3.AllocateIds datastore_v3.BeginTransaction datastore_v3.Commit Can someone from AppEngine team let me know under which category do the above operations fall under. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore Read and Writes Statstics
Rishi Thank you for your suggestion. I have been using it. But for a large app, it is a manual way of finding out which table is getting hit most. Also, if tables have few fields with indexed=false, it becomes really hard to figure out the true cost of writes across tables. On Sep 22, 8:57 am, Rishi Arora wrote: > There was a similar question just a few hours ago on this forum: "Re: > Datastore Reads and Writes Crazy High" > > Here's my answer: > > I used the appstats tool to figure out where all my datastore writes were > coming from. Super easy to configure, and barely any processing overhead. > > http://code.google.com/appengine/docs/python/tools/appstats.htmlhttp://googleappengine.blogspot.com/2010/03/easy-performance-profilin... > > The only problem I had with this tool was it made my memory usage grow > exponentially for a cron job that my backend was executing. The backend > does a lot of work, and this job usually lasts 3 minutes, even though it > happens only once every hour. But since this memory explosion was limited > to a single type of request, I was able to exclude it from appstats by using > appstats_FILTER_LIST in appengine_config.py. > > Hope this helps. > > > > > > > > On Thu, Sep 22, 2011 at 7:29 AM, Vivek Puri wrote: > > For "Estimated Charges Under New Pricing", my app is showing $40/per > > day for Writes and $13/day for Reads. $40 worth of writes is 40 > > million write operations. Right now i have not idea which table is > > being written to most and contributing to that number. Can AppEngine > > team provide statistics on these write and read operations. I would > > hope this is something similar to existing "Datastore Statistics" > > tool. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
