Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
On 11 September 2011 18:17, Gerald Tan wrote: > I'm ok with that. cool > > Your next concern would be the Datastore Reads. I think you should be able > to bring that down a lot by using memcache and/or combining your monitors > into a few entities. I'm not familiar with python, but I believe you can use > pickle to serialize an array of monitors into a blob which you can store as > one entity. You may need to split your list of monitors into a few groups to > stay under the max entity size of 1MB, but this will drastically cut down on > the number of reads that you need, especially if you combine with a > memcache. Oh, I don't need to visit all the monitors, far from it. It's just a bit of crap code doing that which needs to be refactored out entirely. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/3QXRW2UJcB8J. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
I'm ok with that. Your next concern would be the Datastore Reads. I think you should be able to bring that down a lot by using memcache and/or combining your monitors into a few entities. I'm not familiar with python, but I believe you can use pickle to serialize an array of monitors into a blob which you can store as one entity. You may need to split your list of monitors into a few groups to stay under the max entity size of 1MB, but this will drastically cut down on the number of reads that you need, especially if you combine with a memcache. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/3QXRW2UJcB8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
Oh wow, you're absolutely right. Going back to the billing on the 4th of September (after I changed Max Idle Instances and before I made any code changes), I was already seeing the full price drop. So, I didn't need to make the changes to spread out my tasks; that dropped the blue line down, but I don't pay for the blue line. So optimisation is even easier than I thought. Gerald, I'll quote you in an update to my post, if you're ok with that. On 11 September 2011 02:04, Gerald Tan wrote: > Nice blog Emlyn. > > The reason why your Frontend Instance hours are lower than you expected is > because you assumed that you will be billed for the area under the BLUE line > in the Instance graph. It's not. You are being billed for the area under the > YELLOW line (Active Instance) PLUS your Max Idle Instance setting. So your > Active Instances is hovering at around ~0.72, and I assume you have set your > application's Max Idle Instance to 1. Therefore ~1.72 * 24 = ~41.28 Instance > Hours > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/dWoTZKzCy7kJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
Nice blog Emlyn. The reason why your Frontend Instance hours are lower than you expected is because you assumed that you will be billed for the area under the BLUE line in the Instance graph. It's not. You are being billed for the area under the YELLOW line (Active Instance) PLUS your Max Idle Instance setting. So your Active Instances is hovering at around ~0.72, and I assume you have set your application's Max Idle Instance to 1. Therefore ~1.72 * 24 = ~41.28 Instance Hours -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/dWoTZKzCy7kJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
+1 , very good read! -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/7qPNiFG2oVIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
I've just posted the last of what became 4 posts in this series. http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ http://point7.wordpress.com/2011/09/04/appengine-tuning-1/ http://point7.wordpress.com/2011/09/07/appengine-tuning-an-instance-of-success/ http://point7.wordpress.com/2011/09/10/appengine-tuning-schlemiel-youre-fired/ tl;dr is, that my pricing's back down really low, things have worked out. btw I've had great feedback, tips and techniques from this community. Thanks! I think that, regarding longevity of a tech, the culture that builds around a it is just as important as the tech itself. All signs are that AppEngine is going to be a long term viable platform. On 3 September 2011 19:46, Emlyn wrote: > Hi all, > > I don't think I've posted here before, but I've been an appengine user > for a while now (closing on 2 years? Is that even possible?). And like > many, I had a rude shock with the new pricing (going from $0.50/day to > $50/day). > > However, I dug into what I'm actually being charged for, and I think > it's all actually in my control to sort out, and that in itself is > sort of fascinating. I wrote a long blog post on this, which people > might find interesting. > > The Amazing Story Of Appengine And The Two Orders Of Magnitude > http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ > > I'd be really grateful for feedback, especially if I've gotten > anything wildly wrong. I haven't actually made any of the changes that > I've foreshadowed in the post, that's for the next day or two, and > I'll write a followup article on how it goes. > > Thanks in advance for having a look! > > -- > Emlyn > > http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google > Buzz posts, > comments and all. > http://point7.wordpress.com - My blog > Find me on Facebook and Buzz > -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
I've create this feature request for a way to create scheduler profiles and then filter traffic into those profiles. Star it if you think it would be useful. http://code.google.com/p/googleappengine/issues/detail?id=5775 Steve On Sep 3, 4:14 pm, Joshua Smith wrote: > This point is not so simple: > > > (and unfortunately may remain > > that way because instance optimization is markedly revenue and profit > > negative for GAE) > > It really is not in google's economic interest to have idle instances > running. More idle instances means more infrastructure cost, and, if you > charge for it, more user backlash. Idle instances are a necessary evil when > you want responsive, dynamic scaling. But technically, you only need as many > as it takes to satisfy the simple equation (rate of join * max wait) / > startup time. That is, you need to have enough on hand to handle the > incoming requests while you spin up some more. > > The key problem is that for probably all applications, the numerator in that > equation is completely different for tasks than it is for users browsing the > site. And it is often different for different kinds of tasks. (In fact, > it's often different for different users/urls/handlers as well, but sometimes > too much configurability is a bad thing.) > > The scheduler isn't as complicated as many people seem to believe. The only > real magic in it is the heuristics for computing the three values in that > equation when the user sets them to "automatic." The rest is simple trend > averaging. > > -Joshua -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
Fantastic post. Thanks for clarifying what an instance was! On Sep 3, 12:32 pm, stevep wrote: > I had spent most of the night wondering about task queue's apparent > inefficiencies under Scheduler, and had planned to open a topic > related to it, but then read Joshua's post. > > THIS point made by Joshua is THE HEART of the issue:'...it's a lot > more trouble than just having a way to say to the scheduler, "run > these when you have an idle instance with nothing better to do," or > "DO NOT spin up an instance just to handle this task"...' > > As Joshua suggests, there is likely a great deal of optimization > within an instance obtainable by elegantly utilizing TQs. Scheduler > today, however, seems quite inelegant (and unfortunately may remain > that way because instance optimization is markedly revenue and profit > negative for GAE). > > At this point, I would like to suggest that GAE employ a consultant > who is an expert with systems modeling with iThink. The current > opaqueness and complexity of Scheduler are an ideal situation for this > type of systems analysis.** > Link:http://www.iseesystems.com/softwares/Business/ithinkSoftware.aspx > > Please Google, can you give us some more help here. > > Thanks, > stevep > > ** not at all affiliated with iThink, but have developed some > financial analyses using iThink back in the day when I worked for HP. > It really is a great tool for situations like this, and you guys will > certainly find that there are some very experienced and capable > consultants who will do great things for you without too much time > needed from GAE personnel. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
BTW, I really like the blog post Emlyn. On Sun, Sep 4, 2011 at 11:54 PM, Alfred Fuller < arfuller+appeng...@google.com> wrote: > > > On Sat, Sep 3, 2011 at 8:23 AM, johnP wrote: > >> >> Which leads to a potential constructive suggestion. Maybe Goog can >> post a troubleshooting guide that lists different line-items in the >> new-style billing, and potential gotchas? An example in your blog is >> that sudden parallelism is costly and that making things serial is a >> good optimization. (Ironically, the opposite of what was being >> promoted earlier - mapreduce). Another factoid is that offset queries >> are costly. Using queries as a generator is costly, and you should >> fetch items at once. >> >> So it can look like this: >> >> Excessive instances cost? Look at these items: >> - sudden parallelism >> - idle instance setting >> - decrease response time >> - other? >> >> Excessive writes: >> - decrease unneeded indexes >> - >> >> Excessive Reads: >> - make sure your fetch() rather than looping through results >> > > If you look or use fetch the cost is the same (though fetch might be faster > as it will pick larger batch sizes) > ...if you *loop or... :-) > > - check offset queries >> > > I am sure this is the culprit in Emlyn's case. Use query cursors ( > http://code.google.com/appengine/docs/python/datastore/queries.html#Query_Cursors) > not offsets! :-) > > - etc... >> >> >> >> >> >> >> >> >> >> On Sep 3, 4:36 am, peterk wrote: >> > Very good read, thanks for posting. Will definitely be curious to see >> > how your changes improve things or otherwise. >> > >> > On Sep 3, 11:16 am, Emlyn wrote: >> > >> > >> > >> > >> > >> > >> > >> > > Hi all, >> > >> > > I don't think I've posted here before, but I've been an appengine user >> > > for a while now (closing on 2 years? Is that even possible?). And like >> > > many, I had a rude shock with the new pricing (going from $0.50/day to >> > > $50/day). >> > >> > > However, I dug into what I'm actually being charged for, and I think >> > > it's all actually in my control to sort out, and that in itself is >> > > sort of fascinating. I wrote a long blog post on this, which people >> > > might find interesting. >> > >> > > The Amazing Story Of Appengine And The Two Orders Of Magnitudehttp:// >> point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine... >> > >> > > I'd be really grateful for feedback, especially if I've gotten >> > > anything wildly wrong. I haven't actually made any of the changes that >> > > I've foreshadowed in the post, that's for the next day or two, and >> > > I'll write a followup article on how it goes. >> > >> > > Thanks in advance for having a look! >> > >> > > -- >> > > Emlyn >> > >> > >http://my.syyn.cc-Synchonise Google+, Facebook, WordPress and Google >> > > Buzz posts, >> > > comments and all.http://point7.wordpress.com-My blog >> > > Find me on Facebook and Buzz >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
On Sat, Sep 3, 2011 at 8:23 AM, johnP wrote: > > Which leads to a potential constructive suggestion. Maybe Goog can > post a troubleshooting guide that lists different line-items in the > new-style billing, and potential gotchas? An example in your blog is > that sudden parallelism is costly and that making things serial is a > good optimization. (Ironically, the opposite of what was being > promoted earlier - mapreduce). Another factoid is that offset queries > are costly. Using queries as a generator is costly, and you should > fetch items at once. > > So it can look like this: > > Excessive instances cost? Look at these items: > - sudden parallelism > - idle instance setting > - decrease response time > - other? > > Excessive writes: > - decrease unneeded indexes > - > > Excessive Reads: > - make sure your fetch() rather than looping through results > If you look or use fetch the cost is the same (though fetch might be faster as it will pick larger batch sizes) - check offset queries > I am sure this is the culprit in Emlyn's case. Use query cursors ( http://code.google.com/appengine/docs/python/datastore/queries.html#Query_Cursors) not offsets! :-) - etc... > > > > > > > > > > On Sep 3, 4:36 am, peterk wrote: > > Very good read, thanks for posting. Will definitely be curious to see > > how your changes improve things or otherwise. > > > > On Sep 3, 11:16 am, Emlyn wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > I don't think I've posted here before, but I've been an appengine user > > > for a while now (closing on 2 years? Is that even possible?). And like > > > many, I had a rude shock with the new pricing (going from $0.50/day to > > > $50/day). > > > > > However, I dug into what I'm actually being charged for, and I think > > > it's all actually in my control to sort out, and that in itself is > > > sort of fascinating. I wrote a long blog post on this, which people > > > might find interesting. > > > > > The Amazing Story Of Appengine And The Two Orders Of Magnitudehttp:// > point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine... > > > > > I'd be really grateful for feedback, especially if I've gotten > > > anything wildly wrong. I haven't actually made any of the changes that > > > I've foreshadowed in the post, that's for the next day or two, and > > > I'll write a followup article on how it goes. > > > > > Thanks in advance for having a look! > > > > > -- > > > Emlyn > > > > >http://my.syyn.cc-Synchonise Google+, Facebook, WordPress and Google > > > Buzz posts, > > > comments and all.http://point7.wordpress.com-My blog > > > Find me on Facebook and Buzz > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
+1 Also, the ability to make a spider bot wait, without spawning a new task. Although that may hurt search engine placement. It is a conflict of interest for google to charge for instances, and then send their bots around to run up your costs. Google should be able to add some logic to check if the new instance were caused by googlebot or the task queue, and make that instance free. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
Here's the next post, showing the results of just changing the performance sliders. Again, hard data. graphs, all the good stuff. http://point7.wordpress.com/2011/09/04/appengine-tuning-1/ On 3 September 2011 19:46, Emlyn wrote: > Hi all, > > I don't think I've posted here before, but I've been an appengine user > for a while now (closing on 2 years? Is that even possible?). And like > many, I had a rude shock with the new pricing (going from $0.50/day to > $50/day). > > However, I dug into what I'm actually being charged for, and I think > it's all actually in my control to sort out, and that in itself is > sort of fascinating. I wrote a long blog post on this, which people > might find interesting. > > The Amazing Story Of Appengine And The Two Orders Of Magnitude > http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/ > > I'd be really grateful for feedback, especially if I've gotten > anything wildly wrong. I haven't actually made any of the changes that > I've foreshadowed in the post, that's for the next day or two, and > I'll write a followup article on how it goes. > > Thanks in advance for having a look! > > -- > Emlyn > > http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google > Buzz posts, > comments and all. > http://point7.wordpress.com - My blog > Find me on Facebook and Buzz > -- Emlyn http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google Buzz posts, comments and all. http://point7.wordpress.com - My blog Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
I made a very similar request back in June -- there was zero reaction from Google. My suggestion is a little different -- provide a separate slider for "maximum latency for task queue requests". -Sergey -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ZgT491lCdLUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
This point is not so simple: > (and unfortunately may remain > that way because instance optimization is markedly revenue and profit > negative for GAE) It really is not in google's economic interest to have idle instances running. More idle instances means more infrastructure cost, and, if you charge for it, more user backlash. Idle instances are a necessary evil when you want responsive, dynamic scaling. But technically, you only need as many as it takes to satisfy the simple equation (rate of join * max wait) / startup time. That is, you need to have enough on hand to handle the incoming requests while you spin up some more. The key problem is that for probably all applications, the numerator in that equation is completely different for tasks than it is for users browsing the site. And it is often different for different kinds of tasks. (In fact, it's often different for different users/urls/handlers as well, but sometimes too much configurability is a bad thing.) The scheduler isn't as complicated as many people seem to believe. The only real magic in it is the heuristics for computing the three values in that equation when the user sets them to "automatic." The rest is simple trend averaging. -Joshua -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
I had spent most of the night wondering about task queue's apparent inefficiencies under Scheduler, and had planned to open a topic related to it, but then read Joshua's post. THIS point made by Joshua is THE HEART of the issue:'...it's a lot more trouble than just having a way to say to the scheduler, "run these when you have an idle instance with nothing better to do," or "DO NOT spin up an instance just to handle this task"...' As Joshua suggests, there is likely a great deal of optimization within an instance obtainable by elegantly utilizing TQs. Scheduler today, however, seems quite inelegant (and unfortunately may remain that way because instance optimization is markedly revenue and profit negative for GAE). At this point, I would like to suggest that GAE employ a consultant who is an expert with systems modeling with iThink. The current opaqueness and complexity of Scheduler are an ideal situation for this type of systems analysis.** Link: http://www.iseesystems.com/softwares/Business/ithinkSoftware.aspx Please Google, can you give us some more help here. Thanks, stevep ** not at all affiliated with iThink, but have developed some financial analyses using iThink back in the day when I worked for HP. It really is a great tool for situations like this, and you guys will certainly find that there are some very experienced and capable consultants who will do great things for you without too much time needed from GAE personnel. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
You can pull from task queues instead of having them push, right? That could help a lot where task queues are the source of instance spin up. I'd even be happy to dedicated a back-end to task queue pull-work if it was necessary - at least that is totally under your control. On Sep 3, 5:05 pm, Joshua Smith wrote: > I also identified task queues as the source of my excessive instances. I > suspect this is a quite common issue, and together with datastore access bugs > (sorry, but you never should have written it like that) like the one you > found, are conspiring to make a lot of these crazy new billing numbers. > > I think a lot of people just need to take a deep breath and look at their > apps the way you did. It's certainly a lot less work than migrating to EC2. > > On Sep 3, 2011, at 6:16 AM, Emlyn wrote: > > > > > > > > > Hi all, > > > I don't think I've posted here before, but I've been an appengine user > > for a while now (closing on 2 years? Is that even possible?). And like > > many, I had a rude shock with the new pricing (going from $0.50/day to > > $50/day). > > > However, I dug into what I'm actually being charged for, and I think > > it's all actually in my control to sort out, and that in itself is > > sort of fascinating. I wrote a long blog post on this, which people > > might find interesting. > > > The Amazing Story Of Appengine And The Two Orders Of Magnitude > >http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine... > > > I'd be really grateful for feedback, especially if I've gotten > > anything wildly wrong. I haven't actually made any of the changes that > > I've foreshadowed in the post, that's for the next day or two, and > > I'll write a followup article on how it goes. > > > Thanks in advance for having a look! > > > -- > > Emlyn > > >http://my.syyn.cc- Synchonise Google+, Facebook, WordPress and Google > > Buzz posts, > > comments and all. > >http://point7.wordpress.com- My blog > > Find me on Facebook and Buzz > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
Which leads to a potential constructive suggestion. Maybe Goog can post a troubleshooting guide that lists different line-items in the new-style billing, and potential gotchas? An example in your blog is that sudden parallelism is costly and that making things serial is a good optimization. (Ironically, the opposite of what was being promoted earlier - mapreduce). Another factoid is that offset queries are costly. Using queries as a generator is costly, and you should fetch items at once. So it can look like this: Excessive instances cost? Look at these items: - sudden parallelism - idle instance setting - decrease response time - other? Excessive writes: - decrease unneeded indexes - Excessive Reads: - make sure your fetch() rather than looping through results - check offset queries - etc... On Sep 3, 4:36 am, peterk wrote: > Very good read, thanks for posting. Will definitely be curious to see > how your changes improve things or otherwise. > > On Sep 3, 11:16 am, Emlyn wrote: > > > > > > > > > Hi all, > > > I don't think I've posted here before, but I've been an appengine user > > for a while now (closing on 2 years? Is that even possible?). And like > > many, I had a rude shock with the new pricing (going from $0.50/day to > > $50/day). > > > However, I dug into what I'm actually being charged for, and I think > > it's all actually in my control to sort out, and that in itself is > > sort of fascinating. I wrote a long blog post on this, which people > > might find interesting. > > > The Amazing Story Of Appengine And The Two Orders Of > > Magnitudehttp://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine... > > > I'd be really grateful for feedback, especially if I've gotten > > anything wildly wrong. I haven't actually made any of the changes that > > I've foreshadowed in the post, that's for the next day or two, and > > I'll write a followup article on how it goes. > > > Thanks in advance for having a look! > > > -- > > Emlyn > > >http://my.syyn.cc-Synchonise Google+, Facebook, WordPress and Google > > Buzz posts, > > comments and all.http://point7.wordpress.com-My blog > > Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
Excellent information. This is very similar to my experience and I suspect explains most of the trouble with the new billing scheme. At least the datastore reads are very much within our control. As for keeping the instance count low, I still don't see that as very easy to do. I'd be very curious if you find a way to do it. I've been messing with the sliders for minimum pending latency to keep it above all my averages and I still end up with 9 instances always hanging around to handle a site that has basically NO traffic. I wish Google would explain more about what is happening within that scheduler and ultimately give us the control of when and how to start and stop instances. Something that can be so costly should not be left to a black box algorithm that is controlled by the side who profits from erring on keeping the instance count high. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/YvNbQd9hNkYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude
Very good read, thanks for posting. Will definitely be curious to see how your changes improve things or otherwise. On Sep 3, 11:16 am, Emlyn wrote: > Hi all, > > I don't think I've posted here before, but I've been an appengine user > for a while now (closing on 2 years? Is that even possible?). And like > many, I had a rude shock with the new pricing (going from $0.50/day to > $50/day). > > However, I dug into what I'm actually being charged for, and I think > it's all actually in my control to sort out, and that in itself is > sort of fascinating. I wrote a long blog post on this, which people > might find interesting. > > The Amazing Story Of Appengine And The Two Orders Of > Magnitudehttp://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine... > > I'd be really grateful for feedback, especially if I've gotten > anything wildly wrong. I haven't actually made any of the changes that > I've foreshadowed in the post, that's for the next day or two, and > I'll write a followup article on how it goes. > > Thanks in advance for having a look! > > -- > Emlyn > > http://my.syyn.cc- Synchonise Google+, Facebook, WordPress and Google > Buzz posts, > comments and all.http://point7.wordpress.com- My blog > Find me on Facebook and Buzz -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
