In this one case we had to change all of the items in the listproperty. In our most common case we might have to add and delete a couple items to the list property every once in a while. That would still cost us well over $1,000 each time.
Most of the reasons for this type of data in our product is to compensate for the fact that there isn't full text search yet. I know they are beta testing full text, but I'm still worried that that also might be too expensive per write. On Jan 5, 6:54 am, Richard Watson <richard.wat...@gmail.com> wrote: > A couple thoughts. > > Maybe the GAE team should borrow the idea of spot prices from Amazon. > That's a great way to have lower-priority jobs that can run when there are > instances available. We set the price we're willing to pay, if the spot > cost drops below that, we get the resources. It creates a market where more > urgent jobs get done sooner and Google makes better use of quiet periods. > > On your issue: > Do you need to update every entity when you do this? How many items on the > listproperty need to be changed? Could you tell us a bit more of what the > data looks like? > > I'm thinking that 14 million entities x 18 items each is the amount of > entries you really have, each distributed across at least 3 servers and > then indexed. That seems like a lot of writes if you're re-writing > everything. It's likely a bad idea to rely on an infrastructure change to > fix this (recurring) issue, but there is hopefully a way to reduce the > amount of writes you have to do. > > Also, could you maybe run your mapreduce on smaller sets of the data to > spread it out over multiple days and avoid adding too many instances? Has > anyone done anything like this? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
