Hi Steve, I would certainly agree about some sort of ability to guarantee low- volume high-importance storage. I had an idea that it might be able to be implemented in the form of something along the lines of 1MB unevictable memcache storage; we wouldn't be able to store much there, but we would absolutely guarantee that whatever we store there will be able to be retrieved until we flush it or delete the keys. I also like your idea of some sort of task queue with certain guarantees attached to it at the cost of it being called rarely and with smaller amounts of payload. Alternatively I wouldn't mind seeing some sort of pay option where you could pay for guarantees on latency and write availability. This might be covered in the Business SLA, though my cursory impression was that it doesn't actually guarantee things it simply reimburses you if they have significant outages.
Something that occurred to me that might help you was I was actually considering de-isolating the datastore as a single point of failure by additionally sending out those really high importance writes via XMPP to some other server. The other server could be EC2, it could even be a private home server(it's not going to get hammered since we're talking very low volume). It would then back up those records even if the datastore is having latency/write issues and they could be processed in the near future. So in this sense, you could even query the private server via XMPP and get a response regarding the payment record without GAE having put the record to the datastore yet. I'm not sure about this solution, it just occurred to me and I thought it might have some benefits. Your solution sounds good too, though I suppose you might even be able to combine them. Thanks for the tips! Robert: regarding using memcache, I do plan to use the memcache in this sense to see what's been sold and what hasn't been, but I assume because of the volatility of the memcache there's no way to rely on it absolutely to check this. Which of course goes back to my prior point that I'd love to see some sort of very constrained, low volume segment of memcache be non-volatile for purposes like this. Alternatively, what about the ability to at least see some record of even just the names of tasks yet to be executed in the task queue? This way I could construct a record of what's been sold regardless of datastore performance and whether or not the memcache has evicted it. -Richard On Mar 3, 10:53 am, stevep <prosse...@gmail.com> wrote: > Hi Richard, > > Thought I'd comment since I started this thread. Think Wim and Robert > have helped already more than I could. So just some thoughts from me. > > First: I hope any GAE engineers reading this thread might have a > thought sparked about task queue featured needed for highly important, > lower-volume tasks such as yours. > > My problem is that I've got to record a one-time purchase payment. > Yours is more complicated. My issue was the all the steps needed to > setup and maintain a payment record for a Paypal for an Express > Checkout purchase (digital good). > > Rather than risk any loss of the update in the handler functions > (which already has to deal with sending urlfetch calls to Paypal), I > send the responses from Paypal to a high-priority task queue for the > put()s. > > This high-priority queue is pretty strict about very lightweight > imports, and record updates (i.e. no complex indexes on records being > updated). Hopefully this keeps issues related to slower task handlers > low. Look to Wim's link in this thread for many helpful points about > "handler weight control". > > Since I can't move forward with Express Checkout until the payment > record has been put() by the high-priority queue, the client starts to > send a request to the on-line handler once every second (with a nice > dialog explaining the delay for the customer). If the new payment > record has not been put() after 10 client calls, then I go into damage > control on the client side. Hopefully this happens very infrequently. > > The alternative for me is to have the on-line handler send the > urlfetch call to Paypal, put() the new payment record upon PP's > response, and then respond back to the client with "result=OK". If all > this happens, then the client knows to proceed because the initial > payment put() has been done. I just felt that there was a greater risk > of failure having all this functionality in the one on-line handler > function. Perhaps with HR, this is unfounded, but I'm not sure there's > enough anecdotal evidence to support this yet. > > This of course encumbers the user with an additional dialog and a bit > more wait time, but I don't think it is too much as many purchase > processes on-line take a bit of time. Plus, if we can't get this all > done within the allotted time, the damage control is very favorable to > the customer and penalizes me which is as it should be (Google of > course getting its cut either way -- actually a bit more should GAE be > running slowly thereby expending more CPU cycles than normal :-). -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
