[google-appengine] Re: Large number of datastore puts
Thanks a lot for the reply -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/8hHDnGaLZzkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Large number of datastore puts
On Tue, Sep 6, 2011 at 10:05 PM, Jason Collins jason.a.coll...@gmail.com wrote: We are seeing a lot more datastore write operations than we can account for (375M / day). Still trying to get to the bottom of it because it makes for a scary line item on the new sample bills. Divide 375M writes by number of request to get the average writes per request. Use code similar to the following to log requests with more than average writes: https://gist.github.com/715284 Should home in on the write-happy code pretty quickly. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Large number of datastore puts
We also have a large number of Datastore Write ops in our sample bill, and we figured out that it was likely due to properties with indexed=True thanks to the Quota Details admin page that gives more detail: Datastore CPU Time 0% 431.12 of Unlimited CPU hours Datastore Entity Fetch Ops 0% 8,288,728 of Unlimited Datastore Entity Put Ops0% 4,159,838 of Unlimited Datastore Entity Delete Ops 0% 1,072,478 of Unlimited Datastore Index Write Ops 0% 37,251,177 of Unlimited Datastore Query Ops 0% 642,110 of Unlimited Datastore Key Fetch Ops 0% 4,403,176 of Unlimited Here we clearly see that it's Datastore Index Write Ops that is responsible for an expensive bill. We will see soon the impact of having indexed=False on most properties. Hope it helps On 7 sep, 13:23, Stephen sdeasey+gro...@gmail.com wrote: On Tue, Sep 6, 2011 at 10:05 PM, Jason Collins jason.a.coll...@gmail.com wrote: We are seeing a lot more datastore write operations than we can account for (375M / day). Still trying to get to the bottom of it because it makes for a scary line item on the new sample bills. Divide 375M writes by number of request to get the average writes per request. Use code similar to the following to log requests with more than average writes: https://gist.github.com/715284 Should home in on the write-happy code pretty quickly. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Large number of datastore puts
We are seeing a lot more datastore write operations than we can account for (375M / day). Still trying to get to the bottom of it because it makes for a scary line item on the new sample bills. We haven't looked closely at read operations yet because the writes dwarfs it. This blog post outlines some unexpected read statistics: http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ . I think that using offset with your queries (instead of cursors) could lead to an inflated number of reads. Aside, I'm really surprised that datastore reads cost 70% of a write. I would have expected perhaps an order of magnitude cheaper. j On Sep 6, 7:57 am, Richard Druce contactd...@gmail.com wrote: With the appengine pricing changes, we've been paying attention to our datastore puts. According to the pricing comparison chart we're making 2.18 million puts a day. This seems a lot higher than expected. We receive about 0.6 queries per second which means that each request is making about 60 puts!! Using the sample code for db profilinghttp://code.google.com/appengine/articles/hooks.htmlwe measured this for a day and the most we counted was ~14,000 which seems more reasonable. Does anyone have experience with something similar on their site? Thanks, Richard -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Large number of datastore puts
Also, not setting indexed=False on your attributes leads to an unexpectedly high number of datastore writes. Each attribute without indexed=False will yield two more datastore write operations (asc, desc), I believe. Even still, we're trying to account for the large number of datastore write ops j On Sep 6, 3:05 pm, Jason Collins jason.a.coll...@gmail.com wrote: We are seeing a lot more datastore write operations than we can account for (375M / day). Still trying to get to the bottom of it because it makes for a scary line item on the new sample bills. We haven't looked closely at read operations yet because the writes dwarfs it. This blog post outlines some unexpected read statistics:http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine... . I think that using offset with your queries (instead of cursors) could lead to an inflated number of reads. Aside, I'm really surprised that datastore reads cost 70% of a write. I would have expected perhaps an order of magnitude cheaper. j On Sep 6, 7:57 am, Richard Druce contactd...@gmail.com wrote: With the appengine pricing changes, we've been paying attention to our datastore puts. According to the pricing comparison chart we're making 2.18 million puts a day. This seems a lot higher than expected. We receive about 0.6 queries per second which means that each request is making about 60 puts!! Using the sample code for db profilinghttp://code.google.com/appengine/articles/hooks.htmlwemeasured this for a day and the most we counted was ~14,000 which seems more reasonable. Does anyone have experience with something similar on their site? Thanks, Richard -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Large number of datastore puts
A datastore write op != an entity put/delete. It is actually (entity + |index deltas|). For example if you have a Kind with 3 indexed properties and 1 composite index the number of write ops to put a new entity would be: 1 entity + 1 kind index + (1 ascending + 1 descending) * 3 indexed properties + 1 composite index value = 9 write ops if you change a single property on an existing entity it will be: 1 entity + (2 ascending changes + 2 descending changes) * 1 indexed property changed + 2 composite index values changed = 7 write ops (the change must remove the old value and add the new value for each index) deleting the entity will cost the same a creating it (9 write ops) It is important to note that these costs have not changed in the new pricing model, they were just obfuscated in the old model in the form of cpu hours. Hope this clears it up. - Alfred On Tue, Sep 6, 2011 at 2:09 PM, Jason Collins jason.a.coll...@gmail.comwrote: Also, not setting indexed=False on your attributes leads to an unexpectedly high number of datastore writes. Each attribute without indexed=False will yield two more datastore write operations (asc, desc), I believe. Even still, we're trying to account for the large number of datastore write ops j On Sep 6, 3:05 pm, Jason Collins jason.a.coll...@gmail.com wrote: We are seeing a lot more datastore write operations than we can account for (375M / day). Still trying to get to the bottom of it because it makes for a scary line item on the new sample bills. We haven't looked closely at read operations yet because the writes dwarfs it. This blog post outlines some unexpected read statistics: http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine... . I think that using offset with your queries (instead of cursors) could lead to an inflated number of reads. Aside, I'm really surprised that datastore reads cost 70% of a write. I would have expected perhaps an order of magnitude cheaper. j On Sep 6, 7:57 am, Richard Druce contactd...@gmail.com wrote: With the appengine pricing changes, we've been paying attention to our datastore puts. According to the pricing comparison chart we're making 2.18 million puts a day. This seems a lot higher than expected. We receive about 0.6 queries per second which means that each request is making about 60 puts!! Using the sample code for db profilinghttp:// code.google.com/appengine/articles/hooks.htmlwemeasured this for a day and the most we counted was ~14,000 which seems more reasonable. Does anyone have experience with something similar on their site? Thanks, Richard -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Large number of datastore puts
Thanks Alfred. That's the clearest definition of datastore write operations I've seen to-date. The delta aspect is super-cool and I'm glad to hear it, however, it makes my accounting for our large number of datastore write operations even more out of whack - while we might be updating entities, we're certainly not changing many/all of the indexed properties. The search continues j On Sep 6, 4:18 pm, Alfred Fuller arfuller+appeng...@google.com wrote: A datastore write op != an entity put/delete. It is actually (entity + |index deltas|). For example if you have a Kind with 3 indexed properties and 1 composite index the number of write ops to put a new entity would be: 1 entity + 1 kind index + (1 ascending + 1 descending) * 3 indexed properties + 1 composite index value = 9 write ops if you change a single property on an existing entity it will be: 1 entity + (2 ascending changes + 2 descending changes) * 1 indexed property changed + 2 composite index values changed = 7 write ops (the change must remove the old value and add the new value for each index) deleting the entity will cost the same a creating it (9 write ops) It is important to note that these costs have not changed in the new pricing model, they were just obfuscated in the old model in the form of cpu hours. Hope this clears it up. - Alfred On Tue, Sep 6, 2011 at 2:09 PM, Jason Collins jason.a.coll...@gmail.comwrote: Also, not setting indexed=False on your attributes leads to an unexpectedly high number of datastore writes. Each attribute without indexed=False will yield two more datastore write operations (asc, desc), I believe. Even still, we're trying to account for the large number of datastore write ops j On Sep 6, 3:05 pm, Jason Collins jason.a.coll...@gmail.com wrote: We are seeing a lot more datastore write operations than we can account for (375M / day). Still trying to get to the bottom of it because it makes for a scary line item on the new sample bills. We haven't looked closely at read operations yet because the writes dwarfs it. This blog post outlines some unexpected read statistics: http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine... . I think that using offset with your queries (instead of cursors) could lead to an inflated number of reads. Aside, I'm really surprised that datastore reads cost 70% of a write. I would have expected perhaps an order of magnitude cheaper. j On Sep 6, 7:57 am, Richard Druce contactd...@gmail.com wrote: With the appengine pricing changes, we've been paying attention to our datastore puts. According to the pricing comparison chart we're making 2.18 million puts a day. This seems a lot higher than expected. We receive about 0.6 queries per second which means that each request is making about 60 puts!! Using the sample code for db profilinghttp:// code.google.com/appengine/articles/hooks.htmlwemeasured this for a day and the most we counted was ~14,000 which seems more reasonable. Does anyone have experience with something similar on their site? Thanks, Richard -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
