[google-appengine] Datastore Viewer Not Working
I've been trying to access our Datastore viewer in the AppEngine admin interface and it has been throwing an error for the better part of a week. Are there any known issues going on? We're not on the HRD and do have 250 million records if that makes a difference, appid is noticeorange. Thanks, -Casey -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/q-FruHifaVwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Datastore Small Operations and Nov 7th
App ID has been sent via private reply, but for anyone else looking around it's "onlyrounder-sc". Thanks for taking a look. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Dd6QvVMoJ4sJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Datastore Small Operations and Nov 7th
It appears that on the same day billing was enabled (Nov 7th), what counted as a "Datastore Small Operation" also changed. Is that true? In all of my billing previews before Nov 7th, the count was at an acceptable level, but started to climb on Nov 7th, topping out at around 100x what it was previously. Causing an accumulation of over $1,000 in bills that should have been less than a tenth of that. I left for vacation expecting the billing preview level to stay the same after billing was enabled, but appears that count() operations were not included prior to the Nov 7th update? Code wasn't changed since mid-October, and here are the counts for small operations (in millions): Nov 6th: 14 (Days before the 6th were at or under this level) Nov 7th: 10 Nov 8th: 41 Nov 9th: 118 Nov 10th: 192 Nov 11th: 462 Nov 12th: 632 And then it levels out here, using up all quota each day, which was my fault for leaving so high as I had imported a lot of data prior to leaving. Code didn't change, traffic patterns didn't change (there may have been a bit of a fight with spammers hammering the site a bit, but that doesn't appear to be a totally new thing - still trying to post-mortem that but very difficult weeks after the fact). Only my bill went crazy. All posts that talk about these operations changing reference these two URLs: http://code.google.com/appengine/docs/quotas.html#Resources http://code.google.com/appengine/docs/kb/billing.html#operations_charged_for One of which can't be found and the other doesn't seem to have any information related to this. Can someone clarify what changed and when? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Scheduler Oddities
Jon, Thank you for the detailed response and taking the time to look at my specific application. I'll take a look at the new billing statement in the morning (I'm central european time) and see what I can tell. On Sep 5, 10:49 pm, Jon McAlister wrote: > Hi millisecond, I can illuminate some of the issues here. > > As Johan said, the bill will be based on active-instances [the orange > line] plus max-idle-instances. That is correct. > > I can see though how your 09-01 billing report is confusing though, I > can explain that. Also, I'm very sorry that the billing reports are > delayed so long, we're working on that. > > The billing reports are computed on the usage for the day, based on > Pacific Standard Time. In your case, the setting of > max-idle-instances=5 was made at 2011/09/01-04:28:41. So, it applied > to the remaining 19.5 hours of the day, but not to the initial 4.5 > hours for the day. This is why it didn't drop as low as expected (it > dropped from ~3000 instance-hours to 764 instance hours, but obviously > it should be going lower than that). But, we can figure out the > average billed instance rate for the latter 19.5 hours of the day by > solving for: > > 3000 / 24 * 4.5 + N * 19.5 = 764 > > Which is N=10.3. As such, I would expect the 09-02 billing report to > have ~250 instance hours. We'll see in a few hours if this is correct > or not. > > The number 10.3 seems correct though. Your active-instances rate graph > is a sawtooth graph with a trough of 3 and a peak of 13. So, the > average of that graph, plus 5, coming out to 10.3, looks correct to > me. > > I'll check back in on you once the 09-02 billing report comes out in a > few hours for you. > > > > > > > > On Mon, Sep 5, 2011 at 7:12 AM, Johan Euphrosine wrote: > > The Max Idle instances slider help text states: > > > *You will not be charged for instances over the specified maximum* > > > So if you are, it is a billing bug and you should ask a refund. > > > Feel free to a production issue with your application id if you want > > us to track the instance inconsistencies for your application: > >http://code.google.com/p/googleappengine/issues/entry?template=Produc... > > > I would also suggest to fill a feature request for faster billing > > preview/reporting: > >http://code.google.com/p/googleappengine/issues/entry?template=Featur... > > > Hope that helps. > > > On Mon, Sep 5, 2011 at 4:00 PM, Millisecond wrote: > >> I agree that we shouldn't be, it just seems that we are. > > >> I'll just have to wait for more days to show up in Billing History > >> where I have screenshots of the dashboard available to compare > >> against. > > >> Frustrating to have to wait 5 days to see what something is going to > >> be billed at. We have basically 2 cycles to make changes and see the > >> effects before billing goes into effect and the two things biting us > >> are instance-hours and datastore writes. One is totally opaque and > >> the other appears to be inconsistently reported in the live graphs > >> (right now I have "20 Total" in text, 35 total / 12 active in the > >> graph). > > >> On Sep 5, 3:40 pm, Johan Euphrosine wrote: > >>> You should never be charged for more than: Active Instance + Max Idle > >>> instances, even if the scheduler keeps more than Max Idle instances > >>> around. > > >>> Setting Min-Pending-Latency to 60ms instruct the scheduler to wait at > >>> least 60ms if all the instances are busy before deciding to spawn a > >>> new instance for handling an incoming request. > > >>> You can maximize existing instance usage over new instance creation by > >>> increasing that value, but this could come at the expense of > >>> increasing request latency. > > >>> Hope that helps. > > >>> On Mon, Sep 5, 2011 at 3:12 PM, Millisecond wrote: > >>> > It looks like we are being charged at the higher rate of 30-35 > >>> > instance hours / hour even though only ~10 are active at a time and > >>> > max-idle is set to 5. Although it's still hard to tell as we're > >>> > behind ~5 days in billing history summaries. > > >>> > Is the current scheduler going to be changed before new-billing is > >>> > implemented? Seems like a must. > > >>> > On Sep 1, 4:48 pm, Millisecond wrote: > >>> >> The table at the top of the dashboard is showing 23 active instances, > >>> >>
Re: 回覆：Re: [google-appengine] Data Transfer to AWS
Update on our progress: -Currently rather busy optimizing what we can on GAE as we can't take a $7,000 / mo bill until the transfer is complete and it's going to take a non-trivial amount of time to move. -Very app-specific, but our data is really broken out into two categories, configuration and data. We're going to move the configuration over, start the servers on the new side and then have a process pull the data, taking however many days are required. User's will just see data appear over time. -Planning on using the remote API from a command-line utility (we'll probably have to write from scratch as it's so specific) to pull into SDB newer entries first. We'll run it on the AWS side to do the transfer. http://code.google.com/appengine/docs/java/tools/remoteapi.html -Biggest outstanding problem is id matching and allocation. We'll need to start new data records with an ID higher than what's allocated in GAE so the subsequent pull doesn't clobber those allocated in the meantime. Haven't tackled that yet, but assuming the solution won't be super hard. Also spending some time wishing that AppScale would run on Beanstalk / SDB, so our code changes would be super minimal. Digging through their code now to see how hard it would be to add Beanstalk / SDB support to their current setup. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Scheduler Oddities
I agree that we shouldn't be, it just seems that we are. I'll just have to wait for more days to show up in Billing History where I have screenshots of the dashboard available to compare against. Frustrating to have to wait 5 days to see what something is going to be billed at. We have basically 2 cycles to make changes and see the effects before billing goes into effect and the two things biting us are instance-hours and datastore writes. One is totally opaque and the other appears to be inconsistently reported in the live graphs (right now I have "20 Total" in text, 35 total / 12 active in the graph). On Sep 5, 3:40 pm, Johan Euphrosine wrote: > You should never be charged for more than: Active Instance + Max Idle > instances, even if the scheduler keeps more than Max Idle instances > around. > > Setting Min-Pending-Latency to 60ms instruct the scheduler to wait at > least 60ms if all the instances are busy before deciding to spawn a > new instance for handling an incoming request. > > You can maximize existing instance usage over new instance creation by > increasing that value, but this could come at the expense of > increasing request latency. > > Hope that helps. > > > > > > > > > > On Mon, Sep 5, 2011 at 3:12 PM, Millisecond wrote: > > It looks like we are being charged at the higher rate of 30-35 > > instance hours / hour even though only ~10 are active at a time and > > max-idle is set to 5. Although it's still hard to tell as we're > > behind ~5 days in billing history summaries. > > > Is the current scheduler going to be changed before new-billing is > > implemented? Seems like a must. > > > On Sep 1, 4:48 pm, Millisecond wrote: > >> The table at the top of the dashboard is showing 23 active instances, > >> QPS of 2.6 and a latency of 481ms. Which makes some sense as I'm > >> hovering between 40 and 80 QPS overall. > > >> I've set the scheduler to have max 5 idle instances and the min > >> latency to 60ms, but when I pull down the graph to the "Instances" > >> display, it has me hovering between ~5 and ~12 active and 30-35 > >> total. > > >> Is the graph just not accurate and we're billed for what's in the text > >> area? Are we charged for active instances or total instances or > >> instances in the text area? Why isn't it more-or-less "active + 5 as > >> max = total"? > > >> With a 3-4 day delay on my billing history reporting, this is going to > >> be very hard to tweak if I can't see correct numbers on the dashboard. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > Johan Euphrosine (proppy) > Developer Programs Engineer > Google Developer Relations -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Scheduler Oddities
It looks like we are being charged at the higher rate of 30-35 instance hours / hour even though only ~10 are active at a time and max-idle is set to 5. Although it's still hard to tell as we're behind ~5 days in billing history summaries. Is the current scheduler going to be changed before new-billing is implemented? Seems like a must. On Sep 1, 4:48 pm, Millisecond wrote: > The table at the top of the dashboard is showing 23 active instances, > QPS of 2.6 and a latency of 481ms. Which makes some sense as I'm > hovering between 40 and 80 QPS overall. > > I've set the scheduler to have max 5 idle instances and the min > latency to 60ms, but when I pull down the graph to the "Instances" > display, it has me hovering between ~5 and ~12 active and 30-35 > total. > > Is the graph just not accurate and we're billed for what's in the text > area? Are we charged for active instances or total instances or > instances in the text area? Why isn't it more-or-less "active + 5 as > max = total"? > > With a 3-4 day delay on my billing history reporting, this is going to > be very hard to tweak if I can't see correct numbers on the dashboard. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Data Transfer to AWS
App Rocket looks awesome. Assuming I can start it up in a non-default Python instance (I'm normally Java) the same way I've done for some of the Google-supplied python tools that'll really help. Thanks! -Casey On Sep 1, 7:23 pm, Waleed Abdulla wrote: > Another tool I used in the past to move my data into GAE (it works both > ways) is App Rocket. It uses timestamps to do real-time replication between > GAE and mysql. You might need to tweak it a little bit to fit your need > exactly, but it gets you most of the way there. > > https://github.com/k7d/approcket/wiki/ > > > > > > > > On Thu, Sep 1, 2011 at 10:15 AM, Daniel wrote: > > We will be facing the same problem as we leave app engine. I believe > > I'm going to attempt to use the bulk downloader to get the data to a > > local computer. I am slightly worried about the cost of transferring > > 11TB. I believe that the process make take a couple of days, so I plan > > on moving inbound records to the new database before I start the d/l, > > that way I can have reduced functionality over the data transfer > > period but no data will be lost. > > > -Daniel > > > On Sep 1, 8:22 am, Millisecond wrote: > > > Won't rehash the pricing discussions, but because of the pricing > > > changes and the way it's been handled we're 90% sure we're going to > > > move to AWS. > > > > Anyways... now I'm tasked with figuring out how to get our almost TB > > > of data over there either into SimpleDB or into an RDS instance, not > > > sure yet. > > > > Have other people done the move, how did it go? Did you pull from > > > AWS / push from GAE? Take an intermediate backup and then load from > > > some other mechanism (thinking S3) into SDB / RDS? > > > > Our app is adding over 1 GB / day with 20MM reads and 20MM writes, and > > > we'd rather not take it offline for too long... Thinking of some > > > crazy scheme based on descending keys to move data over from key A > > > "downwards", shut off app, move over everything from key A "upwards" > > > as we're mostly only writing new data. Maybe special-case a few > > > object classes and use timestamps to detect deltas. Realize it's > > > pretty app-specific, but wondering if other people have tackled that > > > same problem and what their experiences were. > > > > Thanks for any and all help, > > > -Casey > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Scheduler Oddities
The table at the top of the dashboard is showing 23 active instances, QPS of 2.6 and a latency of 481ms. Which makes some sense as I'm hovering between 40 and 80 QPS overall. I've set the scheduler to have max 5 idle instances and the min latency to 60ms, but when I pull down the graph to the "Instances" display, it has me hovering between ~5 and ~12 active and 30-35 total. Is the graph just not accurate and we're billed for what's in the text area? Are we charged for active instances or total instances or instances in the text area? Why isn't it more-or-less "active + 5 as max = total"? With a 3-4 day delay on my billing history reporting, this is going to be very hard to tweak if I can't see correct numbers on the dashboard. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Keep it short: Who is forced to leave GAE?
We're using Java and expect a 7x increase. With threading, and some optimizations we should bring that down to 3 or 4x - I guess we're a bit lucky, way less than a lot of people. But, when you're over $1k/mo already and a small business, it's really hard to take. And more than the cost, the way this whole thing has been handled is very odd. Will explore the options to go to another cloud provider, but what a pain in the arse to move a TB of data and a handful of apps with a collective ~50k LoC and 30 million reqs/day. Hopefully we can find a reasonable path forward. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Data Transfer to AWS
Won't rehash the pricing discussions, but because of the pricing changes and the way it's been handled we're 90% sure we're going to move to AWS. Anyways... now I'm tasked with figuring out how to get our almost TB of data over there either into SimpleDB or into an RDS instance, not sure yet. Have other people done the move, how did it go? Did you pull from AWS / push from GAE? Take an intermediate backup and then load from some other mechanism (thinking S3) into SDB / RDS? Our app is adding over 1 GB / day with 20MM reads and 20MM writes, and we'd rather not take it offline for too long... Thinking of some crazy scheme based on descending keys to move data over from key A "downwards", shut off app, move over everything from key A "upwards" as we're mostly only writing new data. Maybe special-case a few object classes and use timestamps to detect deltas. Realize it's pretty app-specific, but wondering if other people have tackled that same problem and what their experiences were. Thanks for any and all help, -Casey -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] C2DM Rate Limiting
Hey all, We're running into an odd issue, and not sure if it's us, the AppEngine group, or the C2DM Android group at Google so I thought I'd post here to start and see where we can get. We have implemented a C2DM server on AppEngine servicing several hundred of our clients' Android applications, sending a lot of messages. It was working quite well for a while, but now we're seeing a lot of "Error=QuotaExceeded" returns from the C2DM server. Seems like one of three causes: -Our back-off algorithms aren't correct. Hard to tell though as the google docs are very light on what's an acceptable rate: http://code.google.com/android/c2dm/index.html I have turned off all retry logic, only sending original messages and it still triggers, so guessing it's not a back-off problem. -We're just plain hitting a C2DM limit, but would be great to be able to talk to someone at Google-Android about raising it. -Google is limited based on source IP and lumping all GAE requests together in a DDoS prevention bucket. Which would mean that over time our ability to send messages from GAE would only become worse. At this point I'm just looking to eliminate possibilities and the third one there seems to be the easiest to rule out. Does anyone on the AppEngine team have insight into whether that might be what's happening? Thanks, -Casey -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Unscheduled downtime/problems lately
Since I posted that my dashboard has been wonky and the errors counting _backwards_. So, there are only ~2k errors at the moment, but there definitely were ~3500 when I first posted. Really don't think I'm going crazy but I do feel like it with the dashboard at times... -C On Jun 1, 3:41 pm, Millisecond wrote: > Yes, app id noticeorange. 3500+ latency related errors today in 2 > major and ~10 minor spikes. > > Most days since the 25th have had 5k+ errors, with several 20k+. > > It's usually related to ms/req spikes coinciding with the high latency > spikes on the query latency graph. > > -C > > On Jun 1, 3:34 pm, "Ikai L (Google)" wrote: > > > > > The post-mortem should address the outage as well as the latency leading up > > to the outage. > > > We're keeping an eye on latency now. Are you experience latency in your > > application? > > > On Tue, Jun 1, 2010 at 3:31 PM, Millisecond wrote: > > > Thanks Ikai, I look forward to reading the post-mortem. The last one > > > was indeed very good. > > > > Will the post-mortem address the dramatically increased latency since > > > the 25th? It's ongoing and has already caused tens of thousands of > > > errors on our site alone. As far as I can tell, there's been no > > > official communication regarding that. > > > > And it's still continuing with 1500ms+ for a lot of today: > > > >http://code.google.com/status/appengine/detail/datastore/2010/06/01#a... > > > > -C > > > > On Jun 1, 3:16 pm, "Ikai L (Google)" wrote: > > > > I just wanted to let you guys know to hang in there. We're working on a > > > > post-mortem describing what went wrong and steps we are taking to > > > > prevent > > > > this from happening again. I don't have an ETA for this yet, but we've > > > done > > > > this before ( > > >http://groups.google.com/group/google-appengine/browse_thread/thread/...) > > > > and are committed to as much transparency as we can with regards to > > > > these > > > > issues with you guys. > > > > > On Tue, Jun 1, 2010 at 2:59 PM, Mark wrote: > > > > > +1 as worried, I'm investing time in app engine and this is quite > > > > > discouraging to say the least! How often does this happen? (I don't > > > > > have a live app yet), but should I abandon app engine now before > > > > > getting myself in trouble? > > > > > > On May 27, 1:32 pm, Jody Belka wrote: > > > > > > And what do you know, no-one from Google has made any response > > > whatsoever > > > > > to > > > > > > this thread. And yet we know they are around, they occasionally do > > > write > > > > > > messages after all. Very disappointing. > > > > > > > On 25 May 2010 21:51, Waleed Abdulla wrote: > > > > > > > > It's definitely worrisome. My app is down at the moment, my task > > > queue > > > > > > > backlog is at 4 days, and I'm very disappointed. > > > > > > > > On Tue, May 25, 2010 at 12:46 PM, Flips > > > > > wrote: > > > > > > > >> These latency spikes are really annoying. Will the business app > > > engine > > > > > > >> edition use the same bigtable cluster as we do? > > > > > > > >> On May 25, 9:03 pm, James wrote: > > > > > > >> > +1 > > > > > > > >> > On May 25, 10:13 am, Jody Belka wrote:> From > > > > > reading > > > > > > >> this list, there appear to have been quite few incidents lately > > > > > > >> > > where multiple sites start reporting errors, with the status > > > page > > > > > > >> showing > > > > > > > >> > ... > > > > > > > >> -- > > > > > > >> You received this message because you are subscribed to the > > > > > > >> Google > > > > > Groups > > > > > > >> "Google App Engine" group. > > > > > > >> To post to this group, send email to > > > > > google-appeng...@googlegroups.com. > > > > > > >> To unsubscribe from this group, send email to > > > > > > >> google-appengine+unsubscr...@googlegroups.com > > > &g
[google-appengine] Re: Unscheduled downtime/problems lately
Yes, app id noticeorange. 3500+ latency related errors today in 2 major and ~10 minor spikes. Most days since the 25th have had 5k+ errors, with several 20k+. It's usually related to ms/req spikes coinciding with the high latency spikes on the query latency graph. -C On Jun 1, 3:34 pm, "Ikai L (Google)" wrote: > The post-mortem should address the outage as well as the latency leading up > to the outage. > > We're keeping an eye on latency now. Are you experience latency in your > application? > > > > > > On Tue, Jun 1, 2010 at 3:31 PM, Millisecond wrote: > > Thanks Ikai, I look forward to reading the post-mortem. The last one > > was indeed very good. > > > Will the post-mortem address the dramatically increased latency since > > the 25th? It's ongoing and has already caused tens of thousands of > > errors on our site alone. As far as I can tell, there's been no > > official communication regarding that. > > > And it's still continuing with 1500ms+ for a lot of today: > > >http://code.google.com/status/appengine/detail/datastore/2010/06/01#a... > > > -C > > > On Jun 1, 3:16 pm, "Ikai L (Google)" wrote: > > > I just wanted to let you guys know to hang in there. We're working on a > > > post-mortem describing what went wrong and steps we are taking to prevent > > > this from happening again. I don't have an ETA for this yet, but we've > > done > > > this before ( > >http://groups.google.com/group/google-appengine/browse_thread/thread/...) > > > and are committed to as much transparency as we can with regards to these > > > issues with you guys. > > > > On Tue, Jun 1, 2010 at 2:59 PM, Mark wrote: > > > > +1 as worried, I'm investing time in app engine and this is quite > > > > discouraging to say the least! How often does this happen? (I don't > > > > have a live app yet), but should I abandon app engine now before > > > > getting myself in trouble? > > > > > On May 27, 1:32 pm, Jody Belka wrote: > > > > > And what do you know, no-one from Google has made any response > > whatsoever > > > > to > > > > > this thread. And yet we know they are around, they occasionally do > > write > > > > > messages after all. Very disappointing. > > > > > > On 25 May 2010 21:51, Waleed Abdulla wrote: > > > > > > > It's definitely worrisome. My app is down at the moment, my task > > queue > > > > > > backlog is at 4 days, and I'm very disappointed. > > > > > > > On Tue, May 25, 2010 at 12:46 PM, Flips > > > > wrote: > > > > > > >> These latency spikes are really annoying. Will the business app > > engine > > > > > >> edition use the same bigtable cluster as we do? > > > > > > >> On May 25, 9:03 pm, James wrote: > > > > > >> > +1 > > > > > > >> > On May 25, 10:13 am, Jody Belka wrote:> From > > > > reading > > > > > >> this list, there appear to have been quite few incidents lately > > > > > >> > > where multiple sites start reporting errors, with the status > > page > > > > > >> showing > > > > > > >> > ... > > > > > > >> -- > > > > > >> You received this message because you are subscribed to the Google > > > > Groups > > > > > >> "Google App Engine" group. > > > > > >> To post to this group, send email to > > > > google-appeng...@googlegroups.com. > > > > > >> To unsubscribe from this group, send email to > > > > > >> google-appengine+unsubscr...@googlegroups.com > > > > >> e...@googlegroups.com> > e...@googlegroups.com> > > > e...@googlegroups.com> > > > > > >> . > > > > > >> For more options, visit this group at > > > > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > > > "Google App Engine" group. > > > > > > To post to this group, send email to > > google-appengine@googlegroups.com > > > > . > > > > > > To unsubscribe from this group, send email to > > >
[google-appengine] Re: Unscheduled downtime/problems lately
Thanks Ikai, I look forward to reading the post-mortem. The last one was indeed very good. Will the post-mortem address the dramatically increased latency since the 25th? It's ongoing and has already caused tens of thousands of errors on our site alone. As far as I can tell, there's been no official communication regarding that. And it's still continuing with 1500ms+ for a lot of today: http://code.google.com/status/appengine/detail/datastore/2010/06/01#ae-trust-detail-datastore-query-latency -C On Jun 1, 3:16 pm, "Ikai L (Google)" wrote: > I just wanted to let you guys know to hang in there. We're working on a > post-mortem describing what went wrong and steps we are taking to prevent > this from happening again. I don't have an ETA for this yet, but we've done > this before > (http://groups.google.com/group/google-appengine/browse_thread/thread/...) > and are committed to as much transparency as we can with regards to these > issues with you guys. > > > > > > On Tue, Jun 1, 2010 at 2:59 PM, Mark wrote: > > +1 as worried, I'm investing time in app engine and this is quite > > discouraging to say the least! How often does this happen? (I don't > > have a live app yet), but should I abandon app engine now before > > getting myself in trouble? > > > On May 27, 1:32 pm, Jody Belka wrote: > > > And what do you know, no-one from Google has made any response whatsoever > > to > > > this thread. And yet we know they are around, they occasionally do write > > > messages after all. Very disappointing. > > > > On 25 May 2010 21:51, Waleed Abdulla wrote: > > > > > It's definitely worrisome. My app is down at the moment, my task queue > > > > backlog is at 4 days, and I'm very disappointed. > > > > > On Tue, May 25, 2010 at 12:46 PM, Flips > > wrote: > > > > >> These latency spikes are really annoying. Will the business app engine > > > >> edition use the same bigtable cluster as we do? > > > > >> On May 25, 9:03 pm, James wrote: > > > >> > +1 > > > > >> > On May 25, 10:13 am, Jody Belka wrote:> From > > reading > > > >> this list, there appear to have been quite few incidents lately > > > >> > > where multiple sites start reporting errors, with the status page > > > >> showing > > > > >> > ... > > > > >> -- > > > >> You received this message because you are subscribed to the Google > > Groups > > > >> "Google App Engine" group. > > > >> To post to this group, send email to > > google-appeng...@googlegroups.com. > > > >> To unsubscribe from this group, send email to > > > >> google-appengine+unsubscr...@googlegroups.com > > >> e...@googlegroups.com> > e...@googlegroups.com> > > > >> . > > > >> For more options, visit this group at > > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to google-appengine@googlegroups.com > > . > > > > To unsubscribe from this group, send email to > > > > google-appengine+unsubscr...@googlegroups.com > > > e...@googlegroups.com> > e...@googlegroups.com> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com > e...@googlegroups.com> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Query Latency, etc
We're going the wrong direction, with regard to available information, query latency graph is now broken again today. Going back in time, the only day that has a functioning query latency graph in the last 10 (!) is May 26th. http://code.google.com/status/appengine/detail/datastore/2010/05/26#ae-trust-detail-datastore-query-latency -C On May 28, 9:39 am, Bay wrote: > There just needs to be more information. This is a service people pay > money for. Where are the explanations of todays errors? Adding an > issue at the tracker almost always goes unnoticed - writing in these > (three different) forum groups almost always goes unnoticed. > > Can we expect these 500 error that have been coming and going for > months to be fixed one day, or are we supposed to pull our apps and > rewrite them into something that is less scalable but -actually- > working? This is not supposed to be hard. Everybody are confined to > java and python - nothing more, errors can be contained. No > filesystem, everything in a database which prohibit any operation that > look like join and has indexes for everything that can be fetched > except if one actually knows the unique key. With this in mind, things > are simple. Its not supposed to be that hard - if you can't do what > companies that allow SQL and multiple languages can, then please buy a > hosting company and get some best practices from there. You've got the > money - you've got our money, as we pay for the service. If you need > more money to make it work upp the prices a little, because -random- > 500 errors are unacceptable for everybody. > > A few answers and a few pieces of information would be nice. Thank > you. > > On May 28, 6:04 pm, Millisecond wrote: > > > > > The query latency has been red on and off all morning, requests are > > returning 500's and not counting as errors (saw another thread I was > > going to reference about other people seeing this as well, but can't > > find it now), but no alerts/status > > updates:http://code.google.com/status/appengine/detail/datastore/2010/05/28#a... > > > The only day marked as 'service disruption' was May 25th, but we've > > thrown 75,000+ errors in the past week due to latency, not including > > May 25th. > > > Billing is missing and no word from Google on whether it's > > intentional:http://groups.google.com/group/google-appengine/browse_thread/thread/... > > > Would be great to hear about how this is all just a temporary thing, > > what the problem is, and what Google's plan is to fix it. But, at the > > moment, this feels like "normal operation" of AppEngine and nobody at > > Google is concerned which really has me concerned. > > > -C -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Query Latency, etc
The query latency has been red on and off all morning, requests are returning 500's and not counting as errors (saw another thread I was going to reference about other people seeing this as well, but can't find it now), but no alerts/status updates: http://code.google.com/status/appengine/detail/datastore/2010/05/28#ae-trust-detail-datastore-query-latency The only day marked as 'service disruption' was May 25th, but we've thrown 75,000+ errors in the past week due to latency, not including May 25th. Billing is missing and no word from Google on whether it's intentional: http://groups.google.com/group/google-appengine/browse_thread/thread/beaba487d51f6339/6f48263f24ca553d?lnk=gst&q=billing#6f48263f24ca553d Would be great to hear about how this is all just a temporary thing, what the problem is, and what Google's plan is to fix it. But, at the moment, this feels like "normal operation" of AppEngine and nobody at Google is concerned which really has me concerned. -C -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Billing missing?
We're seeing (and wondering) the same things On May 26, 3:26 pm, jay wrote: > Hey all, I just noticed that the billing info for my app is missing > and it's billing status is Free. > > Is this just some compensation for the downtime or is something wrong > with my app? > > Jay. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: "Request was aborted after waiting too long"
Odd, our app had gotten to a point where it was working pretty well until this. Since maintenance yesterday it's been all over the place, not even sure how many errors it's throwing, the counters keep resetting. Sometimes on their own, and once on deploying a new version with 1.3.3.1. But about 500 errors in the last 15 minutes. App id: noticeorange -C On May 6, 12:03 pm, bsb wrote: > And for me, it's magically working flawlessly again today. Google, did > you do something? That would be good. If you didn't, it means the > service is extremely unreliable: That'd be really bad... > > Ben > > On May 6, 8:02 pm, Joshua Smith wrote: > > > > > > > +1 Here is my error: > > > 05-06 11:00AM 46.656 / 500 10544ms 0cpu_ms 0kb Mozilla/5.0 (Macintosh; U; > > Intel Mac OS X 10_6_3; en-us) AppleWebKit/531.22.7 (KHTML, like Gecko) > > Version/4.0.5 Safari/531.22.7,gzip(gfe),gzip(gfe) > > 216.230.77.82 - - [06/May/2010:11:00:57 -0700] "GET / HTTP/1.1" 500 0 - > > "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us) > > AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 > > Safari/531.22.7,gzip(gfe),gzip(gfe)" "fix3dpdf.v-central.com" > > W05-06 11:00AM 57.200 > > Request was aborted after waiting too long to attempt to service your > > request. This may happen sporadically when the App Engine serving cluster > > is under unexpectedly high or uneven load. If you see this message > > frequently, please contact the App Engine team. > > > On May 6, 2010, at 12:53 PM, bFlood wrote: > > > > my task queue is filled with these today. each task had been paired > > > back to 20 small entity puts. > > > > On May 6, 11:42 am, mkmanning wrote: > > >> One of my apps started getting this yesterday and the problem > > >> continued all day, and is still there this a.m. This is happening any > > >> time you visit the app. The error log for "/" shows: > > > >> "Request was aborted after waiting too long to attempt to service your > > >> request. This may happen sporadically when the App Engine serving > > >> cluster is under unexpectedly high or uneven load. If you see this > > >> message frequently, please contact the App Engine team." > > > >> This started after my app had been up and running for months, with no > > >> changes; worked the day before, then dead. > > > >> Nothing in the system status shows what the cause may be or what (if > > >> anything) is being done about this. > > > >> On May 5, 12:56 pm, bsb wrote: > > > >>> I also get the same error with cron jobs in another app of mine. This > > >>> has only started happening recently. Can someone @ Google please take > > >>> a look at this? > > > >>> On May 5, 10:18 am, Benjamin Schuster-Böckler > > >>> wrote: > > > I'm getting loads of these. It's ok with the deferred tasks, as they > > are rescheduled, but the cron jobs (to be run every 10min) just don't > > run if this happens, causing unpredictable behaviour of my app. > > Effectually, the cron only runs eery few hours, because most of the > > time, this error occurs. What can I do to fix this? Do I really need > > to have a dummy task to constantly hammer instances so the app stays > > loaded? > > > Thanks in advance, > > Ben > > > -- > > You received this message because you are subscribed to the Google > > Groups "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. > > > >>> -- > > >>> You received this message because you are subscribed to the Google > > >>> Groups "Google App Engine" group. > > >>> To post to this group, send email to google-appeng...@googlegroups.com. > > >>> To unsubscribe from this group, send email to > > >>> google-appengine+unsubscr...@googlegroups.com. > > >>> For more options, visit this group > > >>> athttp://groups.google.com/group/google-appengine?hl=en. > > > >> -- > > >> You received this message because you are subscribed to the Google > > >> Groups "Google App Engine" group. > > >> To post to this group, send email to google-appeng...@googlegroups.com. > > >> To unsubscribe from this group, send email to > > >> google-appengine+unsubscr...@googlegroups.com. > > >> For more options, visit this group > > >> athttp://groups.google.com/group/google-appengine?hl=en. > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine" group. > > > To post to this group, send email to google-appeng...@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com. > > > For more options, visit this group > > > athttp://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are
[google-appengine] Re: 1500 ms query latency
It's broken for me as well. And fwiw, this latency is driving me crazy. -C On Mar 7, 9:02 pm, Patrick Twohig wrote: > On this note, I'm not actually able to see the query latency chart. Is that > something broken for me or everybody else? > > http://code.google.com/status/appengine/detail/datastore/2010/03/07#a... > > On Fri, Mar 5, 2010 at 11:05 AM, Ikai L (Google) wrote: > > > > > > > Patrick, let us know if you do find any glaring differences. Nothing > > we've done yet has demonstrated that JDO/JPA introduces anything > > beyond negligible overhead to queries, but we're always in need of > > more good data. > > > On Thu, Mar 4, 2010 at 11:47 PM, Patrick Twohig > > wrote: > > > Ikai, > > > > Off hand, no. Since I started using the low-level API, I've added lots > > of > > > caching and the like so anything I could pull out of my code wouldn't > > > exactly be a fair comparison. If I come across anything, I'll post it. > > > When I was using JDO, I didn't see much that let me tune how aggressively > > it > > > fetched entities from the datastore, which may have contributed to the > > > frustrated I was having. I have had reasonably good success being able > > to > > > adjust the fetching options manually. > > > > Pat. > > > > On Thu, Mar 4, 2010 at 11:56 AM, Ikai L (Google) > > wrote: > > > >> Patrick, do you have any examples of queries implemented using JDO/JPA > > >> versus the low-level API that can demonstrate that JDO/JPA queries > > >> introduce overhead? If you can post them, the datastore team will take > > >> a look at it. > > > >> On Thu, Mar 4, 2010 at 9:49 AM, Patrick Twohig > > >> wrote: > > >> > Just a heads up, if you're using GAE/J and I've had much better > > >> > performance > > >> > with the low-level API over JDO/JPA. > > > >> > Pat. > > > >> > -- > > >> > Patrick H. Twohig. > > > >> > Namazu Studios > > >> > P.O. Box 34161 > > >> > San Diego, CA 92163-4161 > > > >> > -- > > >> > You received this message because you are subscribed to the Google > > >> > Groups > > >> > "Google App Engine" group. > > >> > To post to this group, send email to > > google-appeng...@googlegroups.com. > > >> > To unsubscribe from this group, send email to > > >> > google-appengine+unsubscr...@googlegroups.com > >> > e...@googlegroups.com> > > . > > >> > For more options, visit this group at > > >> >http://groups.google.com/group/google-appengine?hl=en. > > > >> -- > > >> Ikai Lan > > >> Developer Programs Engineer, Google App Engine > > >>http://googleappengine.blogspot.com|http://twitter.com/app_engine > > > >> -- > > >> You received this message because you are subscribed to the Google > > Groups > > >> "Google App Engine" group. > > >> To post to this group, send email to google-appeng...@googlegroups.com. > > >> To unsubscribe from this group, send email to > > >> google-appengine+unsubscr...@googlegroups.com > >> e...@googlegroups.com> > > . > > >> For more options, visit this group at > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > > Patrick H. Twohig. > > > > Namazu Studios > > > P.O. Box 34161 > > > San Diego, CA 92163-4161 > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine" group. > > > To post to this group, send email to google-appeng...@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine+unsubscr...@googlegroups.com > > e...@googlegroups.com> > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Engine > >http://googleappengine.blogspot.com|http://twitter.com/app_engine > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com > e...@googlegroups.com> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Patrick H. Twohig. > > Namazu Studios > P.O. Box 34161 > San Diego, CA 92163-4161 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
