[google-appengine] Re: Datastore operations are seriously expensive
If you're churning through your datastore ops because you have 100 indexes, you would have churned through your CPU quota before. The problem is you're trying to update too many indexes. The only solution is don't. Not true. I had an application that I optimized in CPU usage and it was well below the limits every day. I only had two entities with a few indexes each (three and four respectively). The latter is updated frequently (several times a day). I (currently) have about 4000 entities which required approximately 20 write operations on each update. This was no problem under the old billing model! I threw out some of the indexes but on the cost of usability. Now I'm down to 8 write ops for a new entity and 6 for an update. I got those figures from the dev environment, but what I'd really like to see is read and write ops in the production logs, since the new billing isn't focused around CPU cycles anymore. My two cents. Volker -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore operations are seriously expensive
On Mon, Nov 14, 2011 at 4:27 AM, Volker Thiel v.th...@kosatec.de wrote: If you're churning through your datastore ops because you have 100 indexes, you would have churned through your CPU quota before. The problem is you're trying to update too many indexes. The only solution is don't. Not true. I had an application that I optimized in CPU usage and it was well below the limits every day. I only had two entities with a few indexes each (three and four respectively). The latter is updated frequently (several times a day). I (currently) have about 4000 entities which required approximately 20 write operations on each update. This was no problem under the old billing model! I threw out some of the indexes but on the cost of usability. Now I'm down to 8 write ops for a new entity and 6 for an update. I got those figures from the dev environment, but what I'd really like to see is read and write ops in the production logs, since the new billing isn't focused around CPU cycles anymore. My two cents. I don't mean to argue that the price hasn't gone up... just that it bills more or less the same way as before (which hid datastore ops in api_cpu_ms). They just tripled (or more) the price for these ops. Sounds like your app is in the margin of what was cheap before and expensive now. Read and write ops in the production logs would be rad. Is there an issue I can star? Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore operations are seriously expensive
I hear what you're saying, but at that rate I'm pretty much rewriting my application plus cost isn't even known. And I like the datastore. The problem as always is how the app was written. Users can update one field at a time on a data grid and save it back via ajax, so that's seriously not efficient from a cost pov if they're doing a few fields. The model they're writing to has more than 100 indexes (property indexes both directions + composite indexes). So I need to look at de-indexing fields and removing composite indexes given some of the datastore goodies that have come on stream in the past year. There's also a problem with another model with 3 large key lists that's written a lot. I don't know how the cost on that is even calculated tbh. The fundamental problem is I lack numbers on how much each request is costing me so I don't know where to focus my efforts. It's mostly guesswork and intuition. There could be a big elephant I'm missing. I'd really love to pre-purchase datastore operations in the same vein as instances. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Wgyduglk9KMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore operations are seriously expensive
After managing my instances, I had a similar sticker shock with data store usage. However, the app stats utility, even though its really built around the old pricing model, was invaluable in telling me what my heavy hitters were in terms of data store ops. I implemented caching for my most important routinely accessed data, and that helped a great deal. Also important was to completely move off less critical data from the data store to external storage. This increased CPU and instance time for me, because I implemented it using URL fetch, but in the end, saved a lot of costs. On Tue, Nov 8, 2011 at 2:01 AM, Kenneth kennet...@aladdinschools.comwrote: I hear what you're saying, but at that rate I'm pretty much rewriting my application plus cost isn't even known. And I like the datastore. The problem as always is how the app was written. Users can update one field at a time on a data grid and save it back via ajax, so that's seriously not efficient from a cost pov if they're doing a few fields. The model they're writing to has more than 100 indexes (property indexes both directions + composite indexes). So I need to look at de-indexing fields and removing composite indexes given some of the datastore goodies that have come on stream in the past year. There's also a problem with another model with 3 large key lists that's written a lot. I don't know how the cost on that is even calculated tbh. The fundamental problem is I lack numbers on how much each request is costing me so I don't know where to focus my efforts. It's mostly guesswork and intuition. There could be a big elephant I'm missing. I'd really love to pre-purchase datastore operations in the same vein as instances. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Wgyduglk9KMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore operations are seriously expensive
link does not work -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/JHLT-C_FgEwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore operations are seriously expensive
FWIW, your app would have been a problem under the old pricing model as well. The only difference WRT the datastore is that instead of being charged datastore operations you were charged a magic number of api_cpu_ms per operation. The pricing model is pretty much the same, it's just more expensive. If you're churning through your datastore ops because you have 100 indexes, you would have churned through your CPU quota before. The problem is you're trying to update too many indexes. The only solution is don't. BTW you only incur costs for index changes; if you re-put an entity without changing the index, you don't get charged for an index update. Jeff On Tue, Nov 8, 2011 at 4:01 AM, Kenneth kennet...@aladdinschools.com wrote: I hear what you're saying, but at that rate I'm pretty much rewriting my application plus cost isn't even known. And I like the datastore. The problem as always is how the app was written. Users can update one field at a time on a data grid and save it back via ajax, so that's seriously not efficient from a cost pov if they're doing a few fields. The model they're writing to has more than 100 indexes (property indexes both directions + composite indexes). So I need to look at de-indexing fields and removing composite indexes given some of the datastore goodies that have come on stream in the past year. There's also a problem with another model with 3 large key lists that's written a lot. I don't know how the cost on that is even calculated tbh. The fundamental problem is I lack numbers on how much each request is costing me so I don't know where to focus my efforts. It's mostly guesswork and intuition. There could be a big elephant I'm missing. I'd really love to pre-purchase datastore operations in the same vein as instances. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Wgyduglk9KMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Datastore operations are seriously expensive
Here is an article that can help you reduce the # of composite indexes you use in some cases: http://code.google.com/appengine/articles/indexselection.html On Tue, Nov 8, 2011 at 4:16 PM, Jeff Schnitzer j...@infohazard.org wrote: FWIW, your app would have been a problem under the old pricing model as well. The only difference WRT the datastore is that instead of being charged datastore operations you were charged a magic number of api_cpu_ms per operation. The pricing model is pretty much the same, it's just more expensive. If you're churning through your datastore ops because you have 100 indexes, you would have churned through your CPU quota before. The problem is you're trying to update too many indexes. The only solution is don't. BTW you only incur costs for index changes; if you re-put an entity without changing the index, you don't get charged for an index update. Jeff On Tue, Nov 8, 2011 at 4:01 AM, Kenneth kennet...@aladdinschools.com wrote: I hear what you're saying, but at that rate I'm pretty much rewriting my application plus cost isn't even known. And I like the datastore. The problem as always is how the app was written. Users can update one field at a time on a data grid and save it back via ajax, so that's seriously not efficient from a cost pov if they're doing a few fields. The model they're writing to has more than 100 indexes (property indexes both directions + composite indexes). So I need to look at de-indexing fields and removing composite indexes given some of the datastore goodies that have come on stream in the past year. There's also a problem with another model with 3 large key lists that's written a lot. I don't know how the cost on that is even calculated tbh. The fundamental problem is I lack numbers on how much each request is costing me so I don't know where to focus my efforts. It's mostly guesswork and intuition. There could be a big elephant I'm missing. I'd really love to pre-purchase datastore operations in the same vein as instances. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Wgyduglk9KMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore operations are seriously expensive
It's sad how we find ourselves thinking about not implementing stuff, because of the pricing... The new pricing Model had to be implemented, we can all agree on that... but the pricing itself is too high! Does Google servers cost that much more than Amazon's? I'm sure that's not the case. On Nov 7, 2:48 pm, Kenneth kennet...@aladdinschools.com wrote: I've been monitoring my app today for cost and the instances are so far fairly ok. However I'm getting absolutely hammered on the datastore operations. (yes, this was in the billing preview) I found this gem with seems to have been updated since I last looked at it. This may be old news to people but I just discovered it. http://code.google.com/appengine/kb/postpreviewpricing.html#operation... I was under the impression that if you did an entity get by key it was a small operation, apparently not. If you put a new entity with 4 indexed properties that's actually 10 writes! If you update those 4 properties 18 writes! If you delete that entity 10 writes again! If you have a composite index you're screwed even more. I understand how the datastore works and all but that's pretty crazy. Every time I look at a property and think maybe I won't index that one I think actually no, I can see how I might need that sometime. Damn. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Datastore operations are seriously expensive
Maybe Google Cloud SQL will better suit your dataset. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cc22RSpIbToJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
