We are running auction sites in GAE, they have timer and with each user bid price increases. It deals with multiple bids by using sharded counters in MS datastore, because its much more faster and consistent. Auction data processing is done by backends and saved into memcache which is read by frontends via ajax requests.
About 60-75% is paid for instances, before new pricing this kind of site in GAE was cheap to host. Big pain here is various issues with GAE inside, especially requests failing to load, often datastore read deadlines, unexplained requests latency within GAE, all backends suddenly restarting in the same time.. How can you build something reliable in GAE by using these api's all together? backends + taskqueue + memcache + datastore? Try to fit in your mind, because I can't - any of them can fail any time, tasks can be executed multiple times in the queue, memcache flushes any time and datastore reads die? And again, errors within GAE are too frequent, services die too often, requests suddenly break or become too slow, admin console works like a snail and crashes too often. On 6 December 2011 14:59, Jeff Schnitzer <j...@infohazard.org> wrote: > It sounds like you would be pushing any architecture to the limits. A few > thousand requests per second with shared state is a nontrivial problem, > especially if you have a poor read/write ratio. If you have a long-polling > scenario you may have chosen the wrong platform; this is something that > node.js or twisted python or equivalent asynchronous architectures excel > at. But then expect to do a lot of work setting up and maintaining a fleet > of servers; it isn't a one-man job anymore. > > What percentage of your (present) cost is instance hours vs datastore > operations? > > > and finally I shouldn't be paying for something called experimental! > > Maybe you shouldn't be *using* something called experimental. You're > utilizing GAE in a way that is not efficient. Possibly your application is > one that cannot be run efficiently on GAE; it sounds like an async > appserver would be more appropriate. But maybe it can be done. Perviously > Google hid the real cost of your app from you; now you're getting charged > based on a closer approximation of the "real cost" of service. > > If you want help fitting your application to GAE's architecture then you > need to post a lot more details about what you are doing. This post was > the first time in your countless messages that you included enough > information for any of us to even get a glimpse. Seriously, we (the > community) are willing to help, but you need to actually want help. > > Jeff > > On Tue, Dec 6, 2011 at 9:08 AM, Andrius A <andriu...@gmail.com> wrote: > >> Did you Simon reduced the costs by 1.5-2x? I dont know what are you >> building but if I had a blog site running in GAE I wouldn't be complaining. >> >> I think I am pushing GAE to the limits, my sites do a lot of ajax >> requests with some processing on the server (1 request per user each >> second) and I have thousands of users, you would say that I should use >> channels? Yes, I thought so, but can't since there is no way to broadcast >> the same message to all users and looping through each channel will delay >> sending message! You would then say use backends to off load sending >> message through backends? Ohh yes, but backends do not support channels! So >> backends are really good, you can have them running on background and doing >> time consuming work which actually saves your money! But wait, they are not >> reliable, they restart few times per day and this is what google says in >> documentation, so you need to build some mechanism on top to make sure when >> backends restart your app do not collapse! >> So far I have moved all processing to backends and made all data to be >> passed to user requests via memcache and believe me this is where it gets >> more complicated. You later realise its nearly impossible to have backends >> and frontedns working in local environtment, so develpment process and >> debugging is really really painful. Also backends in production restart few >> times per day so reliability is very low, so on top you build few master >> backends who look after each other and launch other backends when needed >> through task queue, and let this make more complicated, we know that task >> queue can execute in some occasions the same task few times (which is more >> frequent to happen within backends) and then you rely on memcache, and lets >> make it a bit more complicated - you know that memcache can be flushed any >> time, so then you have datastore which can also time out and then to be it >> the most complicated to handle - request can occasionaly fail to load. Do >> you see the point? You have a so many API's, we are being charged for them >> but they are too unreliable, thats totoally not right. I understand that >> nothing is reliable, anything could happen, but at least backends should >> not restart few times per day, datastore shoudn't time out every hour, >> requests shouldn't fail every day! and finally I shouldn't be paying for >> something called experimental! >> >> >> On 6 December 2011 11:55, Simon Knott <knott.si...@gmail.com> wrote: >> >>> Andrius, >>> >>> I'm curious as to what changes you've made to your application to fit in >>> with the new pricing model? >>> >>> Whilst a lot of people said they were facing 10x price increase (some >>> were 100x) when the model was first announced, a lot of those people have >>> since come back and stated how they've managed to reduce that to 1.5-2x by >>> re-writing their code. >>> >>> Cheers, >>> Simon >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-appengine/-/A7uwTwRi69wJ. >>> >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > > -- > We are the 20% > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
