Re: [google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Hi Pol, that is unfortunately a bug in the status dashboard that we are working on... Greg On Wed, Sep 7, 2011 at 8:19 AM, Pol i...@pol-online.net wrote: Great to see the problem acknowledged, so why is the HDR line still all green at http://code.google.com/status/appengine? The HRD is having problems now (yes, not as bad pas a MS problems, but that's beside the point as the expectancies have been officially raised quite a bit for HRD), non-negligible problems, officially acknowledged problems, so not putting it in the status page is deceptive to say the least. That line ought to be orange. On Sep 6, 11:43 am, Gregory D'alesandre gr...@google.com wrote: Hello All, This is an issue right now related to higher latency on HRD, particularly affecting queries. We are working on potential fixes to mitigate the issue. Once we have determined the full scope of it, we will provide further information. This is not expected behavior for HRD and is indeed a recent issue, it has *not* been a slow steady increase based on load or a fundamental issue with the way HRD is built. The issue with M/S was that it heavily relied on a fundamentally inconsistent piece of infrastructure, this is not true with HRD. Greg On Tue, Sep 6, 2011 at 9:17 AM, Jason Collins jason.a.coll...@gmail.com wrote: Those trends are concerning. We're putting a large effort in to convert our app over to HR. Performance consistency is the only reason we're doing this. j On Sep 6, 7:57 am, Mike Wesner mbwes...@gmail.com wrote: Thanks for putting those together. It very clearly shows that recently things have changed. It would be nice to hear from Greg or someone about this. -Mike On Sep 6, 7:11 am, Simon Knott knott.si...@gmail.com wrote: I've attached graphs showing the daily stats for each of the HR operations for the last year, driven from the GAE Status stats. I'll grab the M/S datastore statistics, when I get the time, to give a comparison. https://lh4.googleusercontent.com/-MV_ixL3Jz0Y/TmYNsX9QEGI/AF.. . https://lh4.googleusercontent.com/-6yrWNd7wfxU/TmYNoldrFWI/AF.. . https://lh6.googleusercontent.com/-NiRHqyRtbro/TmYNgZLeRXI/AF.. . https://lh3.googleusercontent.com/-WGYYZUzXiAg/TmYNk35ZztI/AF.. . https://lh5.googleusercontent.com/-5GRqG-Sy17A/TmYN7E7C60I/AF.. . -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Great to see the problem acknowledged, so why is the HDR line still all green at http://code.google.com/status/appengine? The HRD is having problems now (yes, not as bad pas a MS problems, but that's beside the point as the expectancies have been officially raised quite a bit for HRD), non-negligible problems, officially acknowledged problems, so not putting it in the status page is deceptive to say the least. That line ought to be orange. On Sep 6, 11:43 am, Gregory D'alesandre gr...@google.com wrote: Hello All, This is an issue right now related to higher latency on HRD, particularly affecting queries. We are working on potential fixes to mitigate the issue. Once we have determined the full scope of it, we will provide further information. This is not expected behavior for HRD and is indeed a recent issue, it has *not* been a slow steady increase based on load or a fundamental issue with the way HRD is built. The issue with M/S was that it heavily relied on a fundamentally inconsistent piece of infrastructure, this is not true with HRD. Greg On Tue, Sep 6, 2011 at 9:17 AM, Jason Collins jason.a.coll...@gmail.comwrote: Those trends are concerning. We're putting a large effort in to convert our app over to HR. Performance consistency is the only reason we're doing this. j On Sep 6, 7:57 am, Mike Wesner mbwes...@gmail.com wrote: Thanks for putting those together. It very clearly shows that recently things have changed. It would be nice to hear from Greg or someone about this. -Mike On Sep 6, 7:11 am, Simon Knott knott.si...@gmail.com wrote: I've attached graphs showing the daily stats for each of the HR operations for the last year, driven from the GAE Status stats. I'll grab the M/S datastore statistics, when I get the time, to give a comparison. https://lh4.googleusercontent.com/-MV_ixL3Jz0Y/TmYNsX9QEGI/AF... https://lh4.googleusercontent.com/-6yrWNd7wfxU/TmYNoldrFWI/AF... https://lh6.googleusercontent.com/-NiRHqyRtbro/TmYNgZLeRXI/AF... https://lh3.googleusercontent.com/-WGYYZUzXiAg/TmYNk35ZztI/AF... https://lh5.googleusercontent.com/-5GRqG-Sy17A/TmYN7E7C60I/AF... -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
I've attached graphs showing the daily stats for each of the HR operations for the last year, driven from the GAE Status stats. I'll grab the M/S datastore statistics, when I get the time, to give a comparison. https://lh4.googleusercontent.com/-MV_ixL3Jz0Y/TmYNsX9QEGI/AFM/8Zs7Odu9iB4/hr_update.png https://lh4.googleusercontent.com/-6yrWNd7wfxU/TmYNoldrFWI/AFI/Ph43zC2bNgY/hr_delete.png https://lh6.googleusercontent.com/-NiRHqyRtbro/TmYNgZLeRXI/AFA/KXIeil4e__M/hr_get.png https://lh3.googleusercontent.com/-WGYYZUzXiAg/TmYNk35ZztI/AFE/nrDR_h1mB4M/hr_put.png https://lh5.googleusercontent.com/-5GRqG-Sy17A/TmYN7E7C60I/AFQ/42-RRlZEZLc/hr_query.png -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/VLBlQYnLRkgJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Thanks for putting those together. It very clearly shows that recently things have changed. It would be nice to hear from Greg or someone about this. -Mike On Sep 6, 7:11 am, Simon Knott knott.si...@gmail.com wrote: I've attached graphs showing the daily stats for each of the HR operations for the last year, driven from the GAE Status stats. I'll grab the M/S datastore statistics, when I get the time, to give a comparison. https://lh4.googleusercontent.com/-MV_ixL3Jz0Y/TmYNsX9QEGI/AF... https://lh4.googleusercontent.com/-6yrWNd7wfxU/TmYNoldrFWI/AF... https://lh6.googleusercontent.com/-NiRHqyRtbro/TmYNgZLeRXI/AF... https://lh3.googleusercontent.com/-WGYYZUzXiAg/TmYNk35ZztI/AF... https://lh5.googleusercontent.com/-5GRqG-Sy17A/TmYN7E7C60I/AF... -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Those trends are concerning. We're putting a large effort in to convert our app over to HR. Performance consistency is the only reason we're doing this. j On Sep 6, 7:57 am, Mike Wesner mbwes...@gmail.com wrote: Thanks for putting those together. It very clearly shows that recently things have changed. It would be nice to hear from Greg or someone about this. -Mike On Sep 6, 7:11 am, Simon Knott knott.si...@gmail.com wrote: I've attached graphs showing the daily stats for each of the HR operations for the last year, driven from the GAE Status stats. I'll grab the M/S datastore statistics, when I get the time, to give a comparison. https://lh4.googleusercontent.com/-MV_ixL3Jz0Y/TmYNsX9QEGI/AF... https://lh4.googleusercontent.com/-6yrWNd7wfxU/TmYNoldrFWI/AF... https://lh6.googleusercontent.com/-NiRHqyRtbro/TmYNgZLeRXI/AF... https://lh3.googleusercontent.com/-WGYYZUzXiAg/TmYNk35ZztI/AF... https://lh5.googleusercontent.com/-5GRqG-Sy17A/TmYN7E7C60I/AF... -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Hello All, This is an issue right now related to higher latency on HRD, particularly affecting queries. We are working on potential fixes to mitigate the issue. Once we have determined the full scope of it, we will provide further information. This is not expected behavior for HRD and is indeed a recent issue, it has *not* been a slow steady increase based on load or a fundamental issue with the way HRD is built. The issue with M/S was that it heavily relied on a fundamentally inconsistent piece of infrastructure, this is not true with HRD. Greg On Tue, Sep 6, 2011 at 9:17 AM, Jason Collins jason.a.coll...@gmail.comwrote: Those trends are concerning. We're putting a large effort in to convert our app over to HR. Performance consistency is the only reason we're doing this. j On Sep 6, 7:57 am, Mike Wesner mbwes...@gmail.com wrote: Thanks for putting those together. It very clearly shows that recently things have changed. It would be nice to hear from Greg or someone about this. -Mike On Sep 6, 7:11 am, Simon Knott knott.si...@gmail.com wrote: I've attached graphs showing the daily stats for each of the HR operations for the last year, driven from the GAE Status stats. I'll grab the M/S datastore statistics, when I get the time, to give a comparison. https://lh4.googleusercontent.com/-MV_ixL3Jz0Y/TmYNsX9QEGI/AF... https://lh4.googleusercontent.com/-6yrWNd7wfxU/TmYNoldrFWI/AF... https://lh6.googleusercontent.com/-NiRHqyRtbro/TmYNgZLeRXI/AF... https://lh3.googleusercontent.com/-WGYYZUzXiAg/TmYNk35ZztI/AF... https://lh5.googleusercontent.com/-5GRqG-Sy17A/TmYN7E7C60I/AF... -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Glad to hear Googler speak about it. Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/R3HOpbexI54J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Will datastore latency be covered by the SLA? All I'm seeing online is an old draft SLA: http://code.google.com/appengine/sla.html -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/O4Mw7RYuna4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Reviving this thread... GAE team: can we get an official comment on this really important issue? Thx On Sep 2, 1:14 pm, Pol i...@pol-online.net wrote: Our app has been on HRD since the beginning and indeed, the past few days for the time ever, I saw some timeouts. To be clear, not timeouts due to contention which we get as expected now and then on transactions, but timeout on the db itself. Our usage patterns haven't changed though, so it would expect it to be on GAE's end. On Sep 2, 10:02 am, JH ja...@mhztech.com wrote: I agree on both fronts. Yes, M/S was a JOKE. Once moving to HRD I have not seen a datastore timeout at all. Until earlier this week I saw a small series of them. I really hope we are not going back to what we are all used to on M/S. The old M/S problem days were always written off as well, it's good enough for a free service. Now this is hardly free service. In fact it is top dollar service, so I'd hope that datastore issue DO NOT RETURN. On Sep 2, 11:10 am, Robert Kluin robert.kl...@gmail.com wrote: I think the point of this post is the trend. At one time master-slave also performed quite well, then we'd see trends where latency increased until a blowup. These graphs show a similar pattern to that: increasing latencies over time. We would like to know that: 1) the trend won't continue, 2) we won't start seeing massive blowups. As for the SLA, see the exclusions section. SLAs are fairly worthless in my honest opinion anyway, so this probably isn't really something to worry about though. http://code.google.com/appengine/sla.html Robert On Fri, Sep 2, 2011 at 10:47, Vivek Puri v...@vivekpuri.com wrote: I will disagree with this post. We were on MS earlier and now on HRD. While on MS, we used to have datastore timeouts all the time and it was driving me insane looking at 20-30k error emails come in. After the move to HRD, everything is stable and there are no datastore timeouts anymore. I am happy with the move. On Sep 2, 11:15 am, Pol i...@pol-online.net wrote: This is really scary. I'm not so obsessed about the price increase as this is something you can control, but if the HRD starts becoming slower and slower, there's absolutely nothing you can do. According to these official graphs, it's literally 2X slower than 1 year ago. I'm OK with training some speed against increase reliability, the promise of HRD, but at some point, you need to draw a line: what tells us it's not going to take 500ms to do basic queries a year from now? App Engine team: what's your explanation for this? Are you going to put clauses in the SLA to guarantee max average get / put / query latencies? Seems to me that if you want to go enterprise-focus that's needed. On Sep 2, 6:23 am, Mike Wesner mbwes...@gmail.com wrote: Yes, I agree that it won't (can't) get as bad as m/s did get because HRD is awesome. My concern is mostly over how they provision and that the latency is that much higher than it was. Why does the load have such a big impact on latency of datastore query? We keep very detailed statistics about the latency of our services and its clear by our average latencies that it has creeped up over time. It is roughly 200ms higher than it was when we first moved over to HRD. Mike On Sep 2, 8:16 am, Tim Hoffman zutes...@gmail.com wrote: Hi Actually I would disagree on the M/S front, all of my apps are on M/S and even despite the recent issues M/S is way more stable than it was in 2008 and 2009, I mean WAY MORE STABLE ;-) Thats not to say more growing pains won't be felt. My guess is with a supposed imminent exodus of apps, we may say some performance improvements across the board ;-) Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Especially the datastore is getting quite a number of timeout these two days: http://code.google.com/status/appengine/detail/hr-datastore/2011/09/04#ae-trust-detail-hr-datastore-get-latency -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/MFzG_ppM18AJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
On Friday, 2 September 2011 23:10:30 UTC+7, Robert Kluin wrote: As for the SLA, see the exclusions section. SLAs are fairly worthless in my honest opinion anyway, so this probably isn't really something to worry about though. http://code.google.com/appengine/sla.html The SLA does not really offer me any peace of mind. If the datastore slows down to the point where requests are taking 5 seconds to return then my app is as good as dead. But the SLA doesn't cover performance problems - only errors. If my app becomes so slow that it starts to throw DEE's then that is probably considered a problem with the application code rather than an infrastructure error. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/vaMwJ2LpSUoJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
So you're saying stay on MS since it will become super stable when everyone has migrated to HR? :-) Look at Aug 29, http://code.google.com/status/appengine/detail/hr-datastore/2011/08/29#ae-trust-detail-hr-datastore-query-latency Bringing this back to billing, there's no sla on the datastore, so spikes like that are certainly making google money, which seems perverse. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Zorl3LKxNyAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
No, actually I migrated to HR earlier (was an awful experience with 30+hrs downtime. trial run took 8hrs only with the same data). I am just trying to give an signal to everyone before something bad could hit us again. Maybe GAE team could do sth before it happen. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/stVMQHLlJPcJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
I would be interested in what the gae team has to say about this. It seems exactly the same as the evolution of the master slave data store. Its latency creeped up over the first few years and became pretty unstable. Definitely concerning. Mike On Sep 1, 9:30 pm, Raymond C. windz...@gmail.com wrote: http://code.google.com/status/appengine/detail/hr-datastore/2011/09/0... Compare it to half a year before:http://code.google.com/status/appengine/detail/hr-datastore/2011/03/0... It seems to me that HRD was so stable because there was not much applications on it (just like back in the day M/S was not having so many downtime and performance issues) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Hi Actually I would disagree on the M/S front, all of my apps are on M/S and even despite the recent issues M/S is way more stable than it was in 2008 and 2009, I mean WAY MORE STABLE ;-) Thats not to say more growing pains won't be felt. My guess is with a supposed imminent exodus of apps, we may say some performance improvements across the board ;-) Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/luoCASgoBVkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Yes, I agree that it won't (can't) get as bad as m/s did get because HRD is awesome. My concern is mostly over how they provision and that the latency is that much higher than it was. Why does the load have such a big impact on latency of datastore query? We keep very detailed statistics about the latency of our services and its clear by our average latencies that it has creeped up over time. It is roughly 200ms higher than it was when we first moved over to HRD. Mike On Sep 2, 8:16 am, Tim Hoffman zutes...@gmail.com wrote: Hi Actually I would disagree on the M/S front, all of my apps are on M/S and even despite the recent issues M/S is way more stable than it was in 2008 and 2009, I mean WAY MORE STABLE ;-) Thats not to say more growing pains won't be felt. My guess is with a supposed imminent exodus of apps, we may say some performance improvements across the board ;-) Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
This is really scary. I'm not so obsessed about the price increase as this is something you can control, but if the HRD starts becoming slower and slower, there's absolutely nothing you can do. According to these official graphs, it's literally 2X slower than 1 year ago. I'm OK with training some speed against increase reliability, the promise of HRD, but at some point, you need to draw a line: what tells us it's not going to take 500ms to do basic queries a year from now? App Engine team: what's your explanation for this? Are you going to put clauses in the SLA to guarantee max average get / put / query latencies? Seems to me that if you want to go enterprise-focus that's needed. On Sep 2, 6:23 am, Mike Wesner mbwes...@gmail.com wrote: Yes, I agree that it won't (can't) get as bad as m/s did get because HRD is awesome. My concern is mostly over how they provision and that the latency is that much higher than it was. Why does the load have such a big impact on latency of datastore query? We keep very detailed statistics about the latency of our services and its clear by our average latencies that it has creeped up over time. It is roughly 200ms higher than it was when we first moved over to HRD. Mike On Sep 2, 8:16 am, Tim Hoffman zutes...@gmail.com wrote: Hi Actually I would disagree on the M/S front, all of my apps are on M/S and even despite the recent issues M/S is way more stable than it was in 2008 and 2009, I mean WAY MORE STABLE ;-) Thats not to say more growing pains won't be felt. My guess is with a supposed imminent exodus of apps, we may say some performance improvements across the board ;-) Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
I will disagree with this post. We were on MS earlier and now on HRD. While on MS, we used to have datastore timeouts all the time and it was driving me insane looking at 20-30k error emails come in. After the move to HRD, everything is stable and there are no datastore timeouts anymore. I am happy with the move. On Sep 2, 11:15 am, Pol i...@pol-online.net wrote: This is really scary. I'm not so obsessed about the price increase as this is something you can control, but if the HRD starts becoming slower and slower, there's absolutely nothing you can do. According to these official graphs, it's literally 2X slower than 1 year ago. I'm OK with training some speed against increase reliability, the promise of HRD, but at some point, you need to draw a line: what tells us it's not going to take 500ms to do basic queries a year from now? App Engine team: what's your explanation for this? Are you going to put clauses in the SLA to guarantee max average get / put / query latencies? Seems to me that if you want to go enterprise-focus that's needed. On Sep 2, 6:23 am, Mike Wesner mbwes...@gmail.com wrote: Yes, I agree that it won't (can't) get as bad as m/s did get because HRD is awesome. My concern is mostly over how they provision and that the latency is that much higher than it was. Why does the load have such a big impact on latency of datastore query? We keep very detailed statistics about the latency of our services and its clear by our average latencies that it has creeped up over time. It is roughly 200ms higher than it was when we first moved over to HRD. Mike On Sep 2, 8:16 am, Tim Hoffman zutes...@gmail.com wrote: Hi Actually I would disagree on the M/S front, all of my apps are on M/S and even despite the recent issues M/S is way more stable than it was in 2008 and 2009, I mean WAY MORE STABLE ;-) Thats not to say more growing pains won't be felt. My guess is with a supposed imminent exodus of apps, we may say some performance improvements across the board ;-) Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
I think the point of this post is the trend. At one time master-slave also performed quite well, then we'd see trends where latency increased until a blowup. These graphs show a similar pattern to that: increasing latencies over time. We would like to know that: 1) the trend won't continue, 2) we won't start seeing massive blowups. As for the SLA, see the exclusions section. SLAs are fairly worthless in my honest opinion anyway, so this probably isn't really something to worry about though. http://code.google.com/appengine/sla.html Robert On Fri, Sep 2, 2011 at 10:47, Vivek Puri v...@vivekpuri.com wrote: I will disagree with this post. We were on MS earlier and now on HRD. While on MS, we used to have datastore timeouts all the time and it was driving me insane looking at 20-30k error emails come in. After the move to HRD, everything is stable and there are no datastore timeouts anymore. I am happy with the move. On Sep 2, 11:15 am, Pol i...@pol-online.net wrote: This is really scary. I'm not so obsessed about the price increase as this is something you can control, but if the HRD starts becoming slower and slower, there's absolutely nothing you can do. According to these official graphs, it's literally 2X slower than 1 year ago. I'm OK with training some speed against increase reliability, the promise of HRD, but at some point, you need to draw a line: what tells us it's not going to take 500ms to do basic queries a year from now? App Engine team: what's your explanation for this? Are you going to put clauses in the SLA to guarantee max average get / put / query latencies? Seems to me that if you want to go enterprise-focus that's needed. On Sep 2, 6:23 am, Mike Wesner mbwes...@gmail.com wrote: Yes, I agree that it won't (can't) get as bad as m/s did get because HRD is awesome. My concern is mostly over how they provision and that the latency is that much higher than it was. Why does the load have such a big impact on latency of datastore query? We keep very detailed statistics about the latency of our services and its clear by our average latencies that it has creeped up over time. It is roughly 200ms higher than it was when we first moved over to HRD. Mike On Sep 2, 8:16 am, Tim Hoffman zutes...@gmail.com wrote: Hi Actually I would disagree on the M/S front, all of my apps are on M/S and even despite the recent issues M/S is way more stable than it was in 2008 and 2009, I mean WAY MORE STABLE ;-) Thats not to say more growing pains won't be felt. My guess is with a supposed imminent exodus of apps, we may say some performance improvements across the board ;-) Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
I agree on both fronts. Yes, M/S was a JOKE. Once moving to HRD I have not seen a datastore timeout at all. Until earlier this week I saw a small series of them. I really hope we are not going back to what we are all used to on M/S. The old M/S problem days were always written off as well, it's good enough for a free service. Now this is hardly free service. In fact it is top dollar service, so I'd hope that datastore issue DO NOT RETURN. On Sep 2, 11:10 am, Robert Kluin robert.kl...@gmail.com wrote: I think the point of this post is the trend. At one time master-slave also performed quite well, then we'd see trends where latency increased until a blowup. These graphs show a similar pattern to that: increasing latencies over time. We would like to know that: 1) the trend won't continue, 2) we won't start seeing massive blowups. As for the SLA, see the exclusions section. SLAs are fairly worthless in my honest opinion anyway, so this probably isn't really something to worry about though. http://code.google.com/appengine/sla.html Robert On Fri, Sep 2, 2011 at 10:47, Vivek Puri v...@vivekpuri.com wrote: I will disagree with this post. We were on MS earlier and now on HRD. While on MS, we used to have datastore timeouts all the time and it was driving me insane looking at 20-30k error emails come in. After the move to HRD, everything is stable and there are no datastore timeouts anymore. I am happy with the move. On Sep 2, 11:15 am, Pol i...@pol-online.net wrote: This is really scary. I'm not so obsessed about the price increase as this is something you can control, but if the HRD starts becoming slower and slower, there's absolutely nothing you can do. According to these official graphs, it's literally 2X slower than 1 year ago. I'm OK with training some speed against increase reliability, the promise of HRD, but at some point, you need to draw a line: what tells us it's not going to take 500ms to do basic queries a year from now? App Engine team: what's your explanation for this? Are you going to put clauses in the SLA to guarantee max average get / put / query latencies? Seems to me that if you want to go enterprise-focus that's needed. On Sep 2, 6:23 am, Mike Wesner mbwes...@gmail.com wrote: Yes, I agree that it won't (can't) get as bad as m/s did get because HRD is awesome. My concern is mostly over how they provision and that the latency is that much higher than it was. Why does the load have such a big impact on latency of datastore query? We keep very detailed statistics about the latency of our services and its clear by our average latencies that it has creeped up over time. It is roughly 200ms higher than it was when we first moved over to HRD. Mike On Sep 2, 8:16 am, Tim Hoffman zutes...@gmail.com wrote: Hi Actually I would disagree on the M/S front, all of my apps are on M/S and even despite the recent issues M/S is way more stable than it was in 2008 and 2009, I mean WAY MORE STABLE ;-) Thats not to say more growing pains won't be felt. My guess is with a supposed imminent exodus of apps, we may say some performance improvements across the board ;-) Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.
Our app has been on HRD since the beginning and indeed, the past few days for the time ever, I saw some timeouts. To be clear, not timeouts due to contention which we get as expected now and then on transactions, but timeout on the db itself. Our usage patterns haven't changed though, so it would expect it to be on GAE's end. On Sep 2, 10:02 am, JH ja...@mhztech.com wrote: I agree on both fronts. Yes, M/S was a JOKE. Once moving to HRD I have not seen a datastore timeout at all. Until earlier this week I saw a small series of them. I really hope we are not going back to what we are all used to on M/S. The old M/S problem days were always written off as well, it's good enough for a free service. Now this is hardly free service. In fact it is top dollar service, so I'd hope that datastore issue DO NOT RETURN. On Sep 2, 11:10 am, Robert Kluin robert.kl...@gmail.com wrote: I think the point of this post is the trend. At one time master-slave also performed quite well, then we'd see trends where latency increased until a blowup. These graphs show a similar pattern to that: increasing latencies over time. We would like to know that: 1) the trend won't continue, 2) we won't start seeing massive blowups. As for the SLA, see the exclusions section. SLAs are fairly worthless in my honest opinion anyway, so this probably isn't really something to worry about though. http://code.google.com/appengine/sla.html Robert On Fri, Sep 2, 2011 at 10:47, Vivek Puri v...@vivekpuri.com wrote: I will disagree with this post. We were on MS earlier and now on HRD. While on MS, we used to have datastore timeouts all the time and it was driving me insane looking at 20-30k error emails come in. After the move to HRD, everything is stable and there are no datastore timeouts anymore. I am happy with the move. On Sep 2, 11:15 am, Pol i...@pol-online.net wrote: This is really scary. I'm not so obsessed about the price increase as this is something you can control, but if the HRD starts becoming slower and slower, there's absolutely nothing you can do. According to these official graphs, it's literally 2X slower than 1 year ago. I'm OK with training some speed against increase reliability, the promise of HRD, but at some point, you need to draw a line: what tells us it's not going to take 500ms to do basic queries a year from now? App Engine team: what's your explanation for this? Are you going to put clauses in the SLA to guarantee max average get / put / query latencies? Seems to me that if you want to go enterprise-focus that's needed. On Sep 2, 6:23 am, Mike Wesner mbwes...@gmail.com wrote: Yes, I agree that it won't (can't) get as bad as m/s did get because HRD is awesome. My concern is mostly over how they provision and that the latency is that much higher than it was. Why does the load have such a big impact on latency of datastore query? We keep very detailed statistics about the latency of our services and its clear by our average latencies that it has creeped up over time. It is roughly 200ms higher than it was when we first moved over to HRD. Mike On Sep 2, 8:16 am, Tim Hoffman zutes...@gmail.com wrote: Hi Actually I would disagree on the M/S front, all of my apps are on M/S and even despite the recent issues M/S is way more stable than it was in 2008 and 2009, I mean WAY MORE STABLE ;-) Thats not to say more growing pains won't be felt. My guess is with a supposed imminent exodus of apps, we may say some performance improvements across the board ;-) Rgds Tim -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
