[google-appengine] Re: Hi, GAE team, I beg you, please don't spawn any dynamic instances for my project, is it possible?
Maybe related:
Our instance count exploded to over 100 instances while we normally use 2
or 3. It happened about 6 hours ago, and average latency went to 2s instead
of 300ms. It's now slowly going back to normal.
Also, the memcache status page shows a spike. I'm guessing App Engine has
another of these days, and it might not be anything in your code at all.
On Friday, January 24, 2014 12:47:20 PM UTC+1, Tapir wrote:
Here are two screnshots of instance lists, http://imgur.com/a/Ts6O0,
in both cases, gae stole me about 20 minutes computing money.
On Friday, January 24, 2014 7:40:02 PM UTC+8, Tapir wrote:
My settings works well before I converted projects into the modules style.
But after the conversion, the app engine scheduler behavior changes.
It seems the app engine scheduler treats modules style apps and old style
apps differently.
On Friday, January 24, 2014 7:36:38 PM UTC+8, Tapir wrote:
On Friday, January 24, 2014 6:48:28 PM UTC+8, Ian Marshall wrote:
Have you tried something like (we cannot see all your settings):
appengine-web-app xmlns=http://appengine.google.com/ns/1.0;
[...]
threadsafetrue/threadsafe
automatic-scaling
min-idle-instances1/min-idle-instances
max-idle-instancesautomatic/max-idle-instances
min-pending-latency15.0s/min-pending-latency
max-pending-latencyautomatic/max-pending-latency
max-concurrent-requests50/max-concurrent-requests
/automatic-scaling
/appengine-web-app
Thanks for the suggestion.
Ok, I set max-pending-latency and max-concurrent-requests as your
example. But I doubt there are any differences from the last version. In
fact max-concurrent-requests should be about 5.
The min-pending-latency value is still 2.7 seconds, for I hope dynamic
instances still can be spawned WHEN THEY ARE REALLY NEEDED.
I still keep max-idle-instances as 1 for I don't want to there are 2 and
more resident instances.
Here is the full settings:
?xml version=1.0 encoding=utf-8?
appengine-web-app xmlns=http://appengine.google.com/ns/1.0;
application{{app_id}}/application
moduledefault/module
version{{app_version}}/version
!-- loading faster --
precompilation-enabledtrue/precompilation-enabled
!-- warmup --
warmup-requests-enabledtrue/warmup-requests-enabled
!-- concurrentcy --
threadsafetrue/threadsafe
!-- concurrentcy --
sessions-enabledtrue/sessions-enabled
inbound-services
servicechannel_presence/service
/inbound-services
!-- scale settings --
instance-classF1/instance-class
automatic-scaling
min-idle-instances1/min-idle-instances
max-idle-instances1/max-idle-instances
min-pending-latency2730ms/min-pending-latency
max-pending-latencyautomatic/max-pending-latency
max-concurrent-requests50/max-concurrent-requests
/automatic-scaling
!-- Configure java.util.logging --
system-properties
property name=java.util.logging.config.file
value=WEB-INF/logging.properties/
/system-properties
static-files
include path=/favicon.ico /
include path=/crossdomain.xml /
include path=/css/**.* expiration=1d/
include path=/swfs/**.swf expiration=365d /
include path=/js/**.js expiration=1d /
include path=/images/**.ico expiration=1d /
include path=/images/**.png expiration=1d /
include path=/images/**.gif expiration=1d /
include path=/images/**.jpg expiration=1d /
include path=/htmls/**.i* expiration=1d /
/static-files
/appengine-web-app
to minimise the number of application instances spawned?
On Friday, 24 January 2014 02:17:50 UTC, Tapir wrote:
For a website with 1000 requests per days (300 pages), I have tried my
best to limit the number of simultaneous running instances.
Here is the settings:
?xml version=1.0 encoding=utf-8 standalone=no?
appengine-web-app xmlns=http://appengine.google.com/ns/1.0;
applicationyouyards/application
!--Other settings here...--
automatic-scaling
min-idle-instances1/min-idle-instances
max-idle-instances1/max-idle-instances
min-pending-latency2.7s/min-pending-latency
/automatic-scaling
/appengine-web-app
But you scheduler will always spawn useless dynamic instances so that
the Frontend Instance Hours will always exceeds 28 hours each day.
Hi, gea team, I beg beg beg you, pls help tweak the settings so that
the Frontend Instance Hours for my project each days will be less than
28
hours. Thank thank thanks!
On Friday, January 24, 2014 9:43:44 AM UTC+8, Tapir wrote:
All the dynamic instances you spawned all just handle only one
/_ah/warmup request.
After this request, they do nothing but still running there for many
minutes even hours!
You are wasting/stealing my money!
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send
[google-appengine] Re: 100 Email Quota
It's surprisingly easy to set up a payment plan with Mailgun or Sendgrid. If it's urgent, you can be up and running in an hour or two. We did that a year ago and have never looked back. On Friday, December 6, 2013 5:31:32 PM UTC+1, Peabody Wormsworth wrote: Hi, I have an application that sends out an email when a user submits to a form. The free quota limit is 100 emails per day. I reached the limit and want to increase it. So I signed up for paid service. Now I read elsewhere that the email quota limit will not be increase until after 1 full payment cycle. But this is not possible. Because the rest of my application does not extend beyond any of the free quota limits. So I will always be blocked at 100 emails and I will never be charged a payment cycle. Is there a way to make a payment to Google... like as a donation or something... in order to generate a payment cycle against my card in order to begin the process of being charged for emails beyond the 100 limit? Or how else can I send out more then 100 emails? It seems that perhaps the limit should be raised to 200 emails per day when a user enters paid service. In this way, it is possible for me to exceed the limit, incur a billing cycle and then request a much higher limit. I am not spamming. I am sending emails requested by users through a form. Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Application down for almost 2 hours now. Error code 104, and tons of memcache errors
Hi there, our application (small-improvements-hrd) has been misbehaving for two hours now. We didn't update it in 5 days, and we've not seen such a bad downtime in a year. Memcache seems to be mostly unavailable, so many requests take 20 to 30 seconds. Some requests terminate very quickly: A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104) And we see this one a lot 1 more Caused by: com.google.appengine.api.memcache.MemcacheServiceException: Memcache put: Set failed to set 1 keys: SystemSettings(1) at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$8.transform(AsyncMemcacheServiceImpl.java:655) at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$8.transform(AsyncMemcacheServiceImpl.java:629) at com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.convertResponse(MemcacheServiceApiHelper.java:57) at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.wrap(MemcacheServiceApiHelper.java:101) at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.wrap(MemcacheServiceApiHelper.java:96) at com.google.appengine.api.utils.FutureWrapper.wrapAndCache(FutureWrapper.java:53) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:86) at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:26) at com.google.appengine.api.memcache.MemcacheServiceImpl.putAll(MemcacheServiceImpl.java:115) at com.googlecode.objectify.cache.EntityMemcache.getAll(EntityMemcache.java:230) at com.googlecode.objectify.cache.CachingAsyncDatastoreService.get(CachingAsyncDatastoreService.java:253) at com.googlecode.objectify.impl.AsyncObjectifyImpl.get(AsyncObjectifyImpl.java:68) at com.googlecode.objectify.impl.AsyncObjectifyImpl.find(AsyncObjectifyImpl.java:201) at com.googlecode.objectify.impl.AsyncObjectifyImpl.get(AsyncObjectifyImpl.java:110) at com.googlecode.objectify.impl.AsyncObjectifyImpl.get(AsyncObjectifyImpl.java:143) at com.googlecode.objectify.impl.ObjectifyImpl.get(ObjectifyImpl.java:81) at com.praisemanager.service.CompanyService.getSystemSettings(CompanyService.java:720) at Any help would be greatly appreciated, because our customers are (rightfully) very unhappy. (We'd be on a paid support plan if it was available in our country, but it was not last time we checked) Kind regards Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Application down for almost 2 hours now. Error code 104, and tons of memcache errors
https://lh3.googleusercontent.com/-3OJ2o9KkGYs/UqEkFP9hIxI/AGU/gycl82E6tLw/s1600/dashboard-1.jpg -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Application outage for 25 minutes now, Error code 121
Our SAAS application is throwing very unspecific errors since about 20 minutes on about 50% of the requests (which means we're basically offline). Our application ID is small-improvements-hrd. A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. (Error code 121) Tried restarting the instances, but it didn't help. The system status page is normal unfortunately. We didn't upload any new code for the past 5 days, and everything in the admin console looks just normal. Pingdom verifies that the site is essentially down. We'd be on a paid support plan, if it existed for our country (Germany). But since we cannot get official support, I'm posting to the forums in the hope of someone being able to help? Kind regards Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Announcing a credit for App Engine applications with new custom domains
This is great news! It will be even greater if this also removes any subdomain limits along the way. Our customers love to get a subdomain when they purchase our product, but from what I can tell the limit right now is 20 subdomains per SNI, and you can only have 20 SNIs. Which means, if we continue to be successful, we're out of subdomains next year.. Anyway, great news that you're working on it in the first place! On Tuesday, June 18, 2013 1:09:41 AM UTC+2, Andrew Jessup wrote: Hey pdnsk, that's what we are working towards, but we're not quite there yet. In the meantime we're offering a one time credit for users who need to purchase a one-user license of Google Apps for business. On Jun 16, 2013 3:53 PM, pdknsk pdk...@gmail.com javascript: wrote: It is not clear from your wording what the solution will be. If it completely removes Google Apps from the domain setup process, it's time to rejoice and dance in the streets! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com javascript:. To post to this group, send email to google-a...@googlegroups.comjavascript: . Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: security issue (strange behavior)
The easiest way to battle unsophisticated spam is to add a invisible honeypot field to your form. Ignore any posts for which the robot filled the field. http://stackoverflow.com/questions/8873961/how-do-i-add-honey-pot-fields-to-my-forms Other than that, you will have to live with bots crawling your site. I wouldn't say that 100 comments per day is a lot. :) On Saturday, April 13, 2013 4:37:01 AM UTC+2, @Mlaynes wrote: forgot to say .. now, in my graphs you can see few empty recordings, but that's because many of them I've deleted .. and during the attack was averaging 100-200 sex ads per day on average, which of course also I've erased... :/( help me please ! @Mlaynes -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Every day around 9.10 AM Brussels time, huge drop in GAE performances
It is indeed a bit embarrassing that nobody at Google picks this up. I only visit the forum every two or three days and I noticed this thread. How hard can it be for someone at Google to take a look. But still. I'd also urge you to provide a lot more information. Show us data from external sources (Pingdom, NewRelic etc). Run Appstats and post the results. Provide more information that highlights better how severe the problem is. Show everyone that you have done your homework and that it's basically impossible to be your own fault. It's good practice, and it will make it more probable that someone at Google picks this up. (I'd also add increase to F4 unless you already have, since it's better to pay a bit extra than to be down, and see if that hels) Cheers, Per On Monday, February 11, 2013 10:28:34 PM UTC+1, gafal wrote: I'm afraid to see the same thing happening tomorrow. Today my app has almost been unavailable for 30 minutes! Could anyone from Google react on this? Le lundi 11 février 2013 11:32:31 UTC+1, Sam a écrit : One of the reason for this could be that the billing cycle for the day ends/starts at that time. Everyday, we see that all running Instances are shut down new ones go up at this time. Sameer On Mon, Feb 11, 2013 at 2:25 PM, Philip Kilner ph...@xfr.co.uk wrote: Hi Again, On 11/02/13 08:33, Philip Kilner wrote: There was certainly something odd going on, around 8am UK time today - from my PoV, it started at or a whisker before 8am GMT, and lasted c.20 mins. Spoke too soon - am still getting anomalous behaviour, specifically spinning (un-served) requests, and console reports no instances - looks like it's having trouble spinning up instances to serve requests. It's improving over time, but is still not what I'd expect. Based on what I'm seeing, I'm astonished that this does not show as a spike in the latency on the status page. -- Regards, PhilK e: ph...@xfr.co.uk - m: 07775 796 747 'work as if you lived in the early days of a better nation' - alasdair gray -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@**googlegroups.com. To post to this group, send email to google-a...@googlegroups.**com. Visit this group at http://groups.google.com/** group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en . For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out . -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Every day around 9.10 AM Brussels time, huge drop in GAE performances
While I haven't seen that problem, I'd love to learn more about it. Do you have more information other than it being slow? Did you try running appstats? Is it db-related, memcache, CPU slowness, or maybe even session-related? Google may find out by looking at your app, but us others cannot, and I'd love to hear what that actual issue is, in case we run into something similar. On Wednesday, February 6, 2013 9:12:40 AM UTC+1, gafal wrote: Happening again today 6th February. Can anyone at Google just care? It would really be appreciated... Le mardi 5 février 2013 09:18:55 UTC+1, gafal a écrit : I've been experiencing this for almost a week now. Requests take 10x longer than usual!! It seems to start around 9 and stop around 9:30... my app id is myagendapro Can anyone have a look? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Re: Console issues?
We're also still experiencing problems. Latency has increased by about 30% : from 850ms pre incident per Pingdom to roughly 1100ms currently. We're getting some errors in the logs (not as many as during the incident fortunately, and not as many as Mark), but we're also getting reports from users about errors which we cannot find in the logfiles. It's hard to estimate how many of those there are, since GAE doesn't seem to log to the logfile when it's serving our static error-html. App ID is small-improvements-hrd, a Java application on HRD. Cheers, Per On Tuesday, January 15, 2013 10:07:16 PM UTC+1, Mark wrote: Hi Chris No my application is still not functioning correctly. Yes, my ID is bedbuzzserver. I have attached a screenshot of my latest log console (as of 1.06PM). As you can see, almost every request has either: This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. Request was aborted after waiting too long to attempt to service your request. A problem was encountered with the process that handled this request, causing it to exit. I haven't uploaded new code in months, and this used to be working fine.- Mark On Tuesday, January 15, 2013 12:13:20 PM UTC-8, Chris Ramsdale wrote: Mark, Your application should be functioning correctly. Let me know if this isn't the case. Also, I'm assuming that this is app ID: bedbuzzserver. -- Chris On Tue, Jan 15, 2013 at 12:05 PM, Mark markrobe...@gmail.com wrote: My app is totally down. I have been complaining about my response times getting worse and worse on this thread, https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/cJcOrOE4JDcand also in a bug I created. Why is it Google doesn't listen, until it is too late? Also for the love of everything good - please update the App Engine Status page to reflect real life. For me, for the past few weeks App Engine has been unstable. On Tuesday, January 15, 2013 11:37:05 AM UTC-8, Chris Ramsdale wrote: We continue to work at a resolution in regard to current App Engine issue. Error rates should continue to decline for affected applications and we will provide another status update by 12:00 PM PST. -- Chris Product Manager, Google App Engine On Tue, Jan 15, 2013 at 11:00 AM, Chris Ramsdale cram...@google.comwrote: We are still working to resolve the issue related to Google App Engine serving. At this point error error rates for affected applications should be declining. We will provide another status update by 11:30 AM PST. -- Chris Product Manager, Google App Engine On Tue, Jan 15, 2013 at 10:29 AM, Chris Ramsdale cram...@google.comwrote: We are currently experiencing an issue with Google App Engine and some applications are experiencing elevated latency levels and error rates. For everyone who is affected, we apologize - we know you count on Google to work for you and we're working hard to restore normal operation. We will provide an update by 11 AM PST with current details, and if available an estimated time for resolution. -- Chris Product Manager, Google App Engine On Tue, Jan 15, 2013 at 10:26 AM, Alexandru Farcaş alex@expert-group.biz wrote: For my application the number of paying instances has increased to 30 (from 3) and is rising... On Tuesday, January 15, 2013 8:23:41 PM UTC+2, Jim Briggs wrote: Similar problems. No instances are accessible and deployments fail. Latest error due to an error reading the deployed files. On Tuesday, January 15, 2013 1:20:04 PM UTC-5, Joakim wrote: This is causing my instances to fail to start within 60 seconds. I currentyl have zero instances, and every attempt to start a new one fails. On Tuesday, January 15, 2013 7:12:30 PM UTC+1, hyperflame wrote: On Tuesday, January 15, 2013 11:58:11 AM UTC-6, Janusz Skonieczny wrote: The issue may be wider. My application started to show default error and timeout error static pages (those setup in app.yamls error_handlers) +1. Seeing a massive spike in latency on my end. Servlets that normally handle requests in 0.5 seconds are taking 4+ seconds to complete. Instance startup times are up 30%. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/**msg/google-appengine/-/**AWkkFhNEG-QJhttps://groups.google.com/d/msg/google-appengine/-/AWkkFhNEG-QJ . To post to this group, send email to google-a...@googlegroups.**com. To unsubscribe from this group, send email to google-appengi...@** googlegroups.com. For more options, visit this group at http://groups.google.com/** group/google-appengine?hl=enhttp://groups.google.com/group/google-appengine?hl=en . -- You received this message because you are subscribed
[google-appengine] Re: Error deploying
Yes, you're not alone :) I'm trying to mitigate the Taskqueue-Problem with a quick code-change... but I run into the same error as you are. Cheers, Per On Tuesday, December 18, 2012 7:42:37 PM UTC+1, Thiago Catoto wrote: Hi, Is someone having issues deploying a new version on GAE? I keep getting this and allways got the same error: === 04:34 PM Cloned 600 files. 04:35 PM Uploading 1 files and blobs. 04:39 PM Rolling back the update. Error 500: --- begin server output --- htmlhead meta http-equiv=content-type content=text/html;charset=utf-8 title500 Server Error/title /head body text=#00 bgcolor=#ff h1Error: Server Error/h1 h2The server encountered an error and could not complete your request.pIf the problem persists, please A HREF= http://code.google.com/appengine/community.html;report/A your problem and mention this error message and the query that caused it./h2 h2/h2 /body/html --- end server output --- Password for tca...@gmail.com javascript:: If deploy fails you might need to 'rollback' manually. The Make Symlinks... menu option can help with command-line work. *** appcfg.py has finished with exit code 1 *** === any tips?? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/v0a51E7z3YsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Server Error. Application Down. Java. HRD.
Thankfully I saw the error while testing a newly deployed version, and before taking it live. That would have been very embarrassing -- the day before had already encuntered a 30 minute window of up to 20% of memcache errors, which affected our clients as well (in the form of 500 error pages, which IMO should not happen due to Memcache in the first place, but that's another story). http://code.google.com/status/appengine/detail/memcache/2012/12/05#ae-trust-detail-memcache-get-error_rate I can't help it, but I have to (yet again) complain about reliability of App Engine. It's a marvellous piece of technology, and feature-wise it has come such a long way since I started using it two years ago. But it's causing me more and more headache the more mature it gets. I was expecting it to be the other way round when I started. I'm doing a presentation about GAE at the Berlin Java User Group next week. While I will praise all the cool features of App Engine, I have the feeling that I won't convert a single attendee once I mention the errors we keep running into. Which is sad, because I do want GAE to prosper. Cheers, Per On Thursday, December 6, 2012 6:08:51 AM UTC+1, Takashi Matsuo (Google) wrote: Hi prashant, It's good to see the comment on the issue saying that your app is now OK. Thanks for reporting! On Wed, Dec 5, 2012 at 7:32 PM, prashant prashan...@gmail.comjavascript: wrote: Users are impacted. Production issue has been raised. http://code.google.com/p/googleappengine/issues/detail?id=8511 Can you please check. We are getting this exception a 500 error. java.lang.NoClassDefFoundError: Could not initialize class com.google.appengine.api.datastore.DatastoreAttributes at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl.getDatastoreAttributes(AsyncDatastoreServiceImpl.java:985) at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl.getDatastoreType(AsyncDatastoreServiceImpl.java:977) at com.google.appengine.api.datastore.AsyncDatastoreServiceImpl.get(AsyncDatastoreServiceImpl.java:284) at com.google.appengine.api.datastore.DatastoreServiceImpl$1.runInternal(DatastoreServiceImpl.java:78) at com.google.appengine.api.datastore.DatastoreServiceImpl$1.runInternal(DatastoreServiceImpl.java:75) at com.google.appengine.api.datastore.TransactionRunner.runInTransaction(TransactionRunner.java:31) at com.google.appengine.api.datastore.DatastoreServiceImpl.get(DatastoreServiceImpl.java:75) at com.google.appengine.api.datastore.DatastoreServiceImpl.get(DatastoreServiceImpl.java:62) at com.google.apphosting.runtime.jetty.DatastoreSessionStore.getSession(DatastoreSessionStore.java:82) at com.google.apphosting.runtime.jetty.SessionManager.loadSession(SessionManager.java:338) Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/P9EyMaLNEt4J. To post to this group, send email to google-a...@googlegroups.comjavascript: . To unsubscribe from this group, send email to google-appengi...@googlegroups.com javascript:. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Takashi Matsuo | Developers Advocate | tma...@google.com javascript: -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/yIu2J1rgY1YJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: instance scheduling mystery
Hi Peter, I currently have the pending latency slider set to 14.9 secs min and 15.0 secs max. I have also set it to auto/auto per tips in other threads but can't tell a difference. That's most likely your problem. Check out this video: http://www.youtube.com/watch?feature=player_embeddedv=zQ5_47zy4bY#! from 25:45 onwards. You don't want to micro-manage the scheduler too much, or it will get confused. The scheduler may not be great for low usage apps, but it can do a lot better than what you're seeing. I'd recommend only having one resident instance. We're using 1 - automatic for instances and auto - auto for pending latency. We have about 1-2 requests per second during the week, but maybe only 0.2 on the weekend, so its close enough to what you have. Cheers, Per On Thursday, December 6, 2012 10:19:03 PM UTC+1, Peter Warren wrote: Despite having very low request volume (about 1 request / 8 seconds), low latency (1 sec avg) and having 3 dedicated instances, our application still suffers from frequent warmup requests. See attached charts for requests, instances, and latency. All our app does is serve static resources and items from memcache, and I don't see any OOM exceptions in our logs, so I don't think we're churning instances because of memory limits. I figured I could try brute-forcing things and upped our dedicated # of instances to 5. That didn't help at all and only costs us more money. Look at the attached instances chart. Google is now using an avg of about 8 instances to serve exactly the same volume as 5 instances handled previously. It does look like our latency decreased a bit, but we still get tons of warmup requests. I currently have the pending latency slider set to 14.9 secs min and 15.0 secs max. I have also set it to auto/auto per tips in other threads but can't tell a difference. So 2 questions: 1) why does Google need more than 3 instances to serve 1 relatively low-load request every 8 seconds??? I would think 3 dedicated instances should be far more than sufficient for 1 request / 8 seconds. 2) how come the number of instances seems unrelated to load? I would think that more dedicated servers would mean fewer dynamic instances would be created (assuming of course that the dedicated servers are under a manageable load). That doesn't seem to be the case. Thanks for any insights! Peter -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/yS977aD3Z-gJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Latency doubled since Nov 19 -- our Pingdom stats look pretty awful
Okay, I'll continue my conversation with myself here. Someone fixed it! It's not unicorn land yet, but *substantially* faster. Hovering around 800ms according to Pingdom now. Which I guess is fine: Pingdom requests include session creation cost each time, so regular requests would be faster. Well done Googlers! I'm wondering what latencies others are seeing if measured from the outside? Anyone care to share their secrets? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-99DUs63HAYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE keeps starting new instances causing lots of user facing loading requests
We had a very similar problem (see my other post). It's resolved since two days or so, after some quite bumpy weeks. Latency is down, and spikyness of the dashboard charts is down too, everything runs a lot smoother since yesterday. Maybe we're just lucky, but maybe also something has actually been reconfigured and we're just the first who moved to new smarter settings, and you'll follow soon. Do you guys happen to use https by the way, and do you track latency externally like with Pingdom or so? Cheers, Per On Wednesday, November 28, 2012 1:32:25 AM UTC+1, Nik Eleftheriou wrote: I can confirm that we are seeing same behaviour in our app. we can usually serve everyone with 6 instances, but in the last 6 weeks something has been changed and even though at any one time we have 6-7 instances, we have 1000 warmups a day. so definitely something caused the scheduler to change behaviour as we only had 10 warmups a day before Oct. We've not made any code changes to our app in 5 months. nik. On Saturday, November 17, 2012 3:31:42 PM UTC, Francois Masurel wrote: After checking my logs for warming requests I can see that new instances have been started every 3 to 4 minutes for the last hours. Is this normal behavior ? On Saturday, November 17, 2012 4:21:28 PM UTC+1, Francois Masurel wrote: I've notice lots of user facing loading requests since a few hours (days?). Just check the screenshot below. Two new instances have been started to serve only 13 (12 + 1) requests though another instance was already running and available and obviously was not overloaded as it had only served 105 requests during the last hour : https://lh4.googleusercontent.com/-UxI0M8fnvw0/UKep9Wkb_VI/wN0/wxKaPhZvXlI/s1600/Instances+-+VinoCities+-+Google+Chrome.jpg My application instance settings are all set to automatic and pending latency set to 10s : https://lh4.googleusercontent.com/-TxgVTyTvqJM/UKeqaNNQvBI/wN8/fKUP3ftluk8/s1600/Application+Settings+-+VinoCities+-+Google+Chrome.jpg Did something change on GAE side recently ? These user facing loading requests are really killing my business as my users are getting really upset about it. Thanx for your help. François App Id = vncts1 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ey3AsWmJ6DwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Random Server Errors which don't show up in logs, only on the dashboard statistics (and when you run into them)
https://lh4.googleusercontent.com/-_3no8_wP8W4/UK_4ZzGFX-I/AE8/uUCoytxQw6g/s1600/Dashboard+-+Small+Improvements-2.png I just read Christina's postmortem analysis on the Downtime forum. It's interesting that the errors I reported above started around the 19th (when the configuration was changed) and ended on the 22nd (after the outage). https://lh4.googleusercontent.com/-_3no8_wP8W4/UK_4ZzGFX-I/AE8/uUCoytxQw6g/s1600/Dashboard+-+Small+Improvements-2.png https://lh4.googleusercontent.com/-_3no8_wP8W4/UK_4ZzGFX-I/AE8/uUCoytxQw6g/s1600/Dashboard+-+Small+Improvements-2.png Who knows, maybe the entire outage could have been avoided by analysing our bug report in time :) Well, at least the error I described above is now gone, without any action on our end. Cheers, Per On Wednesday, November 21, 2012 3:16:24 AM UTC+1, Per wrote: AppID is small-improvements-hrd. We are seeing plenty of server errors which leave no traces in the log files, but we and our users keep running into them, and it simply shows as Server Error written in black on a white empty screen. This seems to have started about 6 to 8 hours ago. First we also had other server issues (causing 500 errors) but after restarting our servers since memory consumption seemed high, those 500-errors went away. Now we're stuck with server errors that we cannot properly identify and that are hard to report. Any ideas? Like always, I'd like to point out that I wouldn't be posting such a trivial support request to the forums, but since I've rarely (never?) heard back from previous support requests, I guess this is the best way. See Brandon's post for stronger wording :) Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_DfCtiWzIQYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Latency doubled since Nov 19 -- our Pingdom stats look pretty awful
Hi App Engine team, my Pingdom stats for a JSON request that does nothing at all has doubled (on average) since November 19. Since your post-mortem for the recent outage mentions you made a poor performance config-change on that day, I'm pretty sure it has to do with it: * Our app (small-improvements-hrd) is on a custom domain, and it's on https * Appstats confirms that these no-op requests do nothing (except create a session) and they take some 100 to 200ms to do so * The time in the log file is shown as 500 to 2500ms though * And Pingdom consistently shows 200 to 2500ms as well * Our average per Pingdom was 700ms until November 19, and now it's 1400ms. https://lh3.googleusercontent.com/-z3biVzfKyqk/ULAWzA-2jAI/AFM/RZC5_qkkg-E/s1600/Response+Time+Report+noop.png It's as if one extra roundtrip was added somewhere. Maybe the algorithm you described in the post-mortem *still* does something very inefficient? Just to be sure, our other Pingdom check (which pulls our website front page) has also doubled. But the no-op is probably more relevant, since it really does nothing at all except return 1 byte. https://lh5.googleusercontent.com/-7N_St4j2kN0/ULAXJ7FZudI/AFU/yBIF62LKHHU/s1600/Response+Time+Report+website.png Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ZjsqBXwt_h8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Latency doubled since Nov 19 -- our Pingdom stats look pretty awful
And for the big picture, here's our performance stats for the past 1.5 years: https://lh5.googleusercontent.com/-TYq3YsOfgL8/ULAbUiqW3KI/AFk/zbPWOlqZlG0/s1600/Response+Time+Report.png Naturally we *did* make changes to the landing page in those 18 months, so of course it has to be taken with a grain of salt. And we added https 6 months ago. However, both our Java Profiler and Appstats tell us the landing page has no major flaws. So a large share of the slowdown will have to do with App Engine IMO. Also note that we switched to F2 and then F4 over time, and we always have at least 2 instances running now. While both measures smoothen out the spikyness of the stats (and things go really bad when we move to F2 or just one VM) these expensive actions have almost no effect on the longer term trend either. Latency continues to go up. Any help or suggestions would of course be greatly appreciated! Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/FsdUfR9ywaEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Looks like appengine is down
Wholeheartedly agree. Can't blame them that the system has issues every now and then. I'm sure it's a massively complex system, and other products have their issues too. But what's the point in showing a Google-branded generic error message to our clients. The nonchalant Try again in 30 seconds. That's all we know is adding insult to injury. When the next outage strikes, I'd like to see something like The application hosted in our data center can currently not be reached. We apologise. We're working on restoring our client's service. Please try again later. The team at Google. Cheers, Per On Friday, November 23, 2012 2:24:19 AM UTC+1, Brandon Wirtz wrote: http://appengine.google.com/ 500s and All of our sites are down. So is https://developers.google.com/*appengine*/kb/*status*https://developers.google.com/appengine/kb/status ** So is code.google.com/*status*/*appengine* * * Happy Thanksgiving. Aren’t you glad you are on appengine where you can call someone 24/7 and get a reassurance that someone knows things are going wrong and they are working on it? And you can get a status page that says when the issue will be resolved, and a notification that your site is down so you don’t have to get a call from your CEO? Don’t you enjoy the time with your family on the holiday knowing that you can have peace of mind knowing that you have great uptime. And with Black Friday and Cyber Monday aren’t you glad you can trust that Google has prepared for the surge in traffic that is coming and that your service will be up? Yeah, Me Niether. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/fKsrD_7UHUIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: This is my app!
We're neighbours! On Thursday, October 18, 2012 10:34:02 AM UTC+2, pdknsk wrote: You cannot easily tell, but I have it on good authority that an instance of one of my apps is currently hosted in the first (from the top) green wired board, second rack from the right. http://www.google.com/about/datacenters/gallery/#/all/20 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/MdPK_ZmWjMgJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Hundreds of 500-pages out of the blue. All related to database errors (An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get())
This error has been striking again, and our customers had the impression the site was down for about two hours. Our customers are complaining and I am really unhappy about this too. I've raised an issue on the tracker (http://code.google.com/p/googleappengine/issues/detail?id=8233). I know that it's impossible to build an error-free system, and I applauded the most recent 1.7.2 release for all the bugfixes it contained. But several hours of database problems, for the fourth time in about one week, is starting to hurt. I'd like to urge you to continue down the path of fixing bugs above all. Reliability and availability are the only concerns I have about App Engine. I'm entirely satisfied with the existing feature set, and happy to integrate external services (e.g. for email sending) when reaching limitations of app engine. But reliability of the core service and database cannot be outsourced, and having to write post mortems (or reimbursing clients that are affected) more than once every few months is not an option. Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/IQ8hmHV0TJkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Hundreds of 500-pages out of the blue. All related to database errors (An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get())
Hi Takashi, Here is the last request of one of these spikes, just search backwards from there. 59.167.133.100 - - [03/Oct/2012:16:56:02 -0700] GET / HTTP/1.1 500 0 http://small-improvements.com/; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4 www.small-improvements.com ms=2463 cpu_ms=2210 exit_code=103 instance=00c61b117ca2c5dd4a6002dd13f39c6296da66 https://appengine.google.com/instances?app_id=s%7Esmall-improvements-hrdversion_id=1.362133182361985490key=00c61b117ca2c5dd4a6002dd13f39c6296da66#00c61b117ca2c5dd4a6002dd13f39c6296da66 Also, you have my application ID (small-improvements-hrd), and I gave you the general time. The log file is full of these errors, as you can see from the screenshot spikes. I have no idea why you wouldn't be able to find these? Does your search tool really yield no errors? Then it must be fundamentally broken. If I run a search I get hundreds of stacktraces. Of course I also raised a production error too, which has the exact time. I would have sent you a bug tracker ID for my production issue, if your bug tracker had such a thing. Every support system I've worked with in the past 10 years sends IDs in an email receipt. Yours does not. It seems to be GAE policy to keep customers in the dark, and this makes it a lot harder to tell you about issues -- but you do have my application ID and my name, so it shouldn't be hard to retrieve it that way either. Cheers, Per On Thursday, October 4, 2012 9:22:39 AM UTC+2, Takashi Matsuo (Google) wrote: Hi Per, Sorry, but I'm not able to locate the log you pasted. Can you confirm the actual time(or time period) of the occurrence? Do you still see the issue? If yes, can you file a production issue on our issue tracker with more details? -- Takashi On Thu, Oct 4, 2012 at 5:07 AM, Per per.fr...@gmail.com javascript:wrote: https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg I only realised after posting the other thread (about the response time increase) that we're actually encountering hundreds and hundreds of database errors. As always, the log file doesn't tell me anything. It's the generic DATASTORE_V3.Get() all over. We didn't make any deployment, we didnd't change any code recently, the load pattern didn't change, and there is no common trend either. The errors occur for an hour or so, strike everywhere (dozens of different clients are affected) and then the problem disappears for a few hours. Here's the stacktrace again (from about 90 minutes ago): https://www.small-improvements.com/dashboard javax.servlet.ServletException: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:268) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447) at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) at java.lang.Thread.run(Thread.java:679) Caused by: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:464) at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:392) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:781) at com.google.tracing.TraceContext$TraceContextRunnable$1.run
Re: [google-appengine] Hundreds of 500-pages out of the blue. All related to database errors (An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get())
Hi Takashi, you're right, it seems to have stopped now. The response time is still a bit high, so I'm fearing the error could strike again, but at least right now nothing bad is happening. I'm always searching using 'HTTP/1.1 500' to identify failed requests, and I ignore the error level. Maybe I have to rework our logging in that regard, sorry if that confused you. I guess I'll stop using the raise production error feature then, I thought it was the preferred way of raising problems these days. Glad to hear it's not! And thanks for passing this on the engineering team :) Cheers, Per On Thursday, October 4, 2012 1:03:20 PM UTC+2, Takashi Matsuo (Google) wrote: Hi Per, I have informed this issue to the engineering team. Thanks for the report. Anyway, it seems that now the error stopped. Can you confirm? -- Takashi On Thu, Oct 4, 2012 at 7:34 PM, Takashi Matsuo tma...@google.comjavascript: wrote: Hi Per, I couldn't find the logs because I searched for 'Error' severity. Sorry about that. Now I can see your warning logs, escalating it to the engineering team. If you want to have a tracker ID, please use our issue tracker at: http://code.google.com/p/googleappengine/issues/list As you say, the 'Report production issue' feature in the admin console lacks a tracking feature, so we're re-thinking about this tool. -- Takashi On Thu, Oct 4, 2012 at 6:41 PM, Per per.fr...@gmail.com javascript:wrote: Hi Takashi, Here is the last request of one of these spikes, just search backwards from there. 59.167.133.100 - - [03/Oct/2012:16:56:02 -0700] GET / HTTP/1.1 500 0 http://small-improvements.com/; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4 www.small-improvements.com ms=2463 cpu_ms=2210 exit_code=103 instance=00c61b117ca2c5dd4a6002dd13f39c6296da66 https://appengine.google.com/instances?app_id=s%7Esmall-improvements-hrdversion_id=1.362133182361985490key=00c61b117ca2c5dd4a6002dd13f39c6296da66#00c61b117ca2c5dd4a6002dd13f39c6296da66 Also, you have my application ID (small-improvements-hrd), and I gave you the general time. The log file is full of these errors, as you can see from the screenshot spikes. I have no idea why you wouldn't be able to find these? Does your search tool really yield no errors? Then it must be fundamentally broken. If I run a search I get hundreds of stacktraces. Of course I also raised a production error too, which has the exact time. I would have sent you a bug tracker ID for my production issue, if your bug tracker had such a thing. Every support system I've worked with in the past 10 years sends IDs in an email receipt. Yours does not. It seems to be GAE policy to keep customers in the dark, and this makes it a lot harder to tell you about issues -- but you do have my application ID and my name, so it shouldn't be hard to retrieve it that way either. Cheers, Per On Thursday, October 4, 2012 9:22:39 AM UTC+2, Takashi Matsuo (Google) wrote: Hi Per, Sorry, but I'm not able to locate the log you pasted. Can you confirm the actual time(or time period) of the occurrence? Do you still see the issue? If yes, can you file a production issue on our issue tracker with more details? -- Takashi On Thu, Oct 4, 2012 at 5:07 AM, Per per.fr...@gmail.com wrote: https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg I only realised after posting the other thread (about the response time increase) that we're actually encountering hundreds and hundreds of database errors. As always, the log file doesn't tell me anything. It's the generic DATASTORE_V3.Get() all over. We didn't make any deployment, we didnd't change any code recently, the load pattern didn't change, and there is no common trend either. The errors occur for an hour or so, strike everywhere (dozens of different clients are affected) and then the problem disappears for a few hours. Here's the stacktrace again (from about 90 minutes ago): https://www.small-**improvements.com/dashboard https://www.small-improvements.com/dashboard javax.servlet.**ServletException: com.google.apphosting.api.**ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). at com.google.apphosting.runtime.**jetty.AppVersionHandlerMap.**handle(AppVersionHandlerMap.**java:268) at org.mortbay.jetty.handler.**HandlerWrapper.handle(**HandlerWrapper.java:152) at org.mortbay.jetty.Server.**handle(Server.java:326) at org.mortbay.jetty.**HttpConnection.handleRequest(**HttpConnection.java:542) at org.mortbay.jetty.**HttpConnection$RequestHandler.**headerComplete(HttpConnection.**java:923) at com.google.apphosting.runtime.**jetty.RpcRequestParser.**parseAvailable(**RpcRequestParser.java:76
[google-appengine] Average response time went from 400ms to 800ms about 36 hours ago, and is still up there
Hi GAE team, there might be some issue with recent maintenance you're doing under the covers. We didn't change any code, the load pattern didn't change, and additionally we're seeing a several bugs: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.RunQuery(). I know that hiccups can happen on GAE, so it's not like I'm flagging this instantly. But the change doesn't go away, and 36h of decreased performance is starting to hurt. Here's one of the error stacktraces I'm seeing: Caused by: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.RunQuery(). at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:781) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.rpcFinished(RpcStub.java:823) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.success(RpcStub.java:808) at com.google.net.rpc3.impl.client.RpcClientInternalContext.runCallbacks(RpcClientInternalContext.java:893) at com.google.net.rpc3.impl.client.RpcClientInternalContext.finishRpcAndNotifyApp(RpcClientInternalContext.java:798) at com.google.net.rpc3.impl.client.RpcNetChannel.afterFinishingActiveRpc(RpcNetChannel.java:1063) at com.google.net.rpc3.impl.client.RpcNetChannel.finishRpc(RpcNetChannel.java:911) at com.google.net.rpc3.impl.client.RpcNetChannel.handleResponse(RpcNetChannel.java:2265) at com.google.net.rpc3.impl.client.RpcNetChannel.messageReceived(RpcNetChannel.java:2068) at com.google.net.rpc3.impl.client.RpcNetChannel.access$2000(RpcNetChannel.java:143) at com.google.net.rpc3.impl.client.RpcNetChannel$TransportCallback.receivedMessage(RpcNetChannel.java:3127) at com.google.net.rpc3.impl.client.RpcChannelTransportData$TransportCallback.receivedMessage(RpcChannelTransportData.java:599) at com.google.net.rpc3.impl.wire.RpcBaseTransport.receivedMessage(RpcBaseTransport.java:417) at com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260) at com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121) at com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:578) at com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:1002) at com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:884) at com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:136) at com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1855) And here's a picture of the response time increase, taken about 24 hours ago: https://lh4.googleusercontent.com/--Lf5oZY3xCc/UGyR0weY_SI/AEE/AZeG_6QToNM/s1600/Dashboard+-+Small+Improvements.jpg Application ID is small-improvements-hrd, we're on HRD, and we were on F2 when this happened. Changing to F4 didn't make a difference, neither did restarting the instances. Which would have surprised me anyway, since it seems to be database related. I'd investigate further, but I'm on vacation. I hope this can be solved by moving us to a different cluster for now? Kind regards, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cyeWlHvzoDYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Hundreds of 500-pages out of the blue. All related to database errors (An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get())
(UdrpcTransport.java:265) at com.google.apphosting.runtime.udrpc.UdrpcTransport.readPackets(UdrpcTransport.java:217) at com.google.apphosting.runtime.udrpc.UdrpcTransport$1.run(UdrpcTransport.java:81) at com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260) at com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121) at com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:578) at com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:1002) at com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:884) at com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:136) at com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1855) And this is the error spike distribution today: https://lh4.googleusercontent.com/-CQmIN9Qm_cY/UGyaOGMP9iI/AEU/zeYKUS7dxkc/s1600/Dashboard+-+Small+Improvements-1.jpg I find it very frustrating that GAE is so inconsistent. We had severe issues two months ago and had to reimburse several clients for downtimes caused by very similar problems. Then it was fine for 2 months, and just when I go on vacation, it breaks down again. Anything I can do on my end? Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/U9KKQ-WMZ98J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Google App Engine SDK 1.7.2 is out
Thanks Takashi and everyone on the GAE team. It's great to see how many bugfixes you squeezed in, and I'd like to applaud the effort! Keep it up. Shiny new toys get people more excited and are probably harder to justify to certain stakeholders, but as a customer I can sleep a lot better at night knowing that quality is high on your agenda. Releases like this give me the confidence I need to be recommending GAE to others again. So kudos to you all for such a boring yet bold release! :) Cheers, Per On Thursday, September 20, 2012 12:26:40 PM UTC+2, Takashi Matsuo (Google) wrote: Hi everyone, We're happy to announce that we released the 1.7.2 SDK. This release includes several new features including the trusted tester program of App Engine outbound socket, as well as huge amount of bug fixes. For more details, please read our blog post and release notes. Announcement: http://googleappengine.blogspot.com.au/2012/09/app-engine-172-released.html Release notes for Java: http://code.google.com/p/googleappengine/wiki/SdkForJavaReleaseNotes Release notes for Python: http://code.google.com/p/googleappengine/wiki/SdkReleaseNotes Enjoy! -- Takashi Matsuo | Developers Advocate | tma...@google.com javascript: -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Mn75qzVakQAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: APP DOWN: App suddenly no longer starts, no code changes
Hi Peter, I'm sure it's extremely(!) hard to host this many applications reliably and at a sane cost. I won't try to make any suggestions about that. But what concerns me is the support situation on App Engine. It shouldn't be too hard to monitor a low-bandwidth forum like this, and to provide somewhat timely feedback when downtime is being reported by some of the most senior users, during a time when you're actually rolling out changes. I can understand that you don't want to turn this into an official support forum for all the dumb question we see here, and I can understand that you want to encourage us to purchase premium accounts. But I feel you're hurting your own business by not responding swiftly to posts like these. We've been through similar issues with our application, and the support situation is really the one thing that stops me from recommending App Engine wholeheartedly. I bet I'm not the only one. :) When discussing your strategies, it would be great if you could consider improved responsiveness too. Cheers, Per On Thursday, September 13, 2012 8:12:35 PM UTC+2, psm wrote: Jeff, these are good ideas and suggestions. we are working on a number of different strategies to ameliorate these issues. some of the items you are suggesting are already in progress, and others besides. and i agree that this is a general philosophical challenge with PaaS. on GAE we now regularly serve several hundreds of thousands of applications, so it is indeed a challenge to handle the long tail problem. we are aware of this, and you should expect us to be rolling out a number of things to address it. in fact, we expect to make our experience of running this large workload over a long period of time into an advantage with GAE. Peter S Magnusson (GAE Eng Dir) On Wednesday, September 12, 2012 5:35:39 PM UTC-7, Jeff Schnitzer wrote: On Wed, Sep 12, 2012 at 3:12 PM, Kaan Soral kaan...@gmail.com wrote: This is why I love App Engine, when a problem occurs instead of having a heart attack or committing suicide, you can just wait for it to be resolved. Hmmm. This really unfortunately timed incident may have cost us an important client, so I'm not feeling the love. I have quite a lot of experience building and running large online systems prior to embracing GAE and my products have never had as much downtime as I've had over the last year. It hasn't always been Google's fault (the entire .st registry going down for 8+ hours really sucked[1]) but it usually has been. See: * Instance startup time ballooning by 3X and hitting deadlines (multiple occasions) * GAE blocking CloudFlare with an undocumented security system * This incident, where Java instances started mysteriously failing Would waiting have fixed these issues? I'm not convinced. Google may have smart people running GAE but they aren't watching _my_ app, they're just watching for an uptick in the number of complaints. If you're doing something slightly unusual (say, running a CF reverse proxy), you might be statistical noise. Apparently this Java problem _was_ widespread, but I had no way of knowing that. GAE's value proposition is that it's better to have Google's smart engineers building and maintaining your infrastructure. But my site would be more reliable if I had one dumb person (possibly me) who cares specifically about _my_ infrastructure. I've screwed up deployments and upgrades in production before, but at least I'm aware when changes happen, get immediate feedback, and can fix the problem right then and there. With GAE, the only thing I can do when my alarms go off is to whine as loudly as possible. But there is no feedback! I have no way of knowing if Google is working on the problem or if they're still waiting for more complaints that will never materialize. Will I be down for 15 minutes, 1 hour, 2 hours, 8 hours, forever? How long do you want to wait? This feels like a fundamental flaw in the PaaS concept, destined to produce multiple-hour downtimes at irregular intervals. The feedback loop is too slow (and lossy if the problem is not widespread). There's no amount of QA or testing that will prevent failures in a system as big as complicated as GAE. So the only reasonable option is to get that feedback loop shorter. How can that happen? Some ideas: * Google could announce when they are rolling out changes. I don't need release notes (although it would be nice to know what to watch for) but I'd like to know when I should pay extra attention. Or not schedule client demos. Facebook does something like this, rolling out platform changes on specific days of the week (which I long ago stopped caring about). * Google could make extra support channels available during this time. Hell, use twitter. Think of us as your QA staff - if we see something amiss, we'd like to let
[google-appengine] Re: Catching bounced emails
Hi Blaine, GAE cannot do this, but you can use external mailers (sendgrid, mailgun, etc) which detect bounced mails, and then you can either collect the list of bounces, or you can provide web hooks so the external service calls your application, letting you know about the bounces. It's really simple to integrate, and while it costs a little extra, you'll get really good service right now, and some features (like not resending to already bounced addresses) that you might never get on GAE. Cheers, Per On Tuesday, August 14, 2012 9:31:19 PM UTC+2, Blaine Garrett wrote: I'd like to be able to catch bounced emails and handle them on the server. I have customers that send documents to users and need to be notified if the email doesn't go through (i.e. incorrect email address, mailbox full, etc). Looking at the below docs, it appears you can directly email an appspot and handle it. However, can the address that bounces are sent to be specified? https://developers.google.com/appengine/docs/python/mail/receivingmail http://code.google.com/p/googleappengine/issues/detail?id=1800#c22 Blaine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/aXOtNF6JVI8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Our application is suffering from 4 different kinds of 500 errors, plus memory spikes. Are we the 1.7.1 guinea pig?
Hi, Our application all of a sudden throws errors that I've not seen before. While we continuously deploy new minor features, we've not made any substantial changes that would explain either of these: * Datastore access misbehaving: Nested in javax.servlet.ServletException: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get().: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:452) at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:382) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:782) * Got out of Memory errors while saving the session (outside our control) java.lang.OutOfMemoryError at java.util.zip.Deflater.init(Native Method) at java.util.zip.Deflater.init(Deflater.java:177) at java.util.zip.GZIPOutputStream.init(GZIPOutputStream.java:63) at java.util.zip.GZIPOutputStream.init(GZIPOutputStream.java:75) at com.google.apphosting.runtime.HttpCompression.compress(HttpCompression.java:51) at com.google.apphosting.runtime.HttpCompression.attemptCompression(HttpCompression.java:205) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:482) at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455) at com.google.tracing.TraceContext.runInContext(TraceContext.java:695) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) at java.lang.Thread.run(Thread.java:679) * com.google.appengine.api.memcache.MemcacheServiceException: Memcache put: Error setting single item (_ahsahRx7dGRBNmJh6XPmzlfTg) -- This one is thrown so deep inside the bowels of Jetty that it's outside our application's reach, and thus a memcache error takes down our application once again: http://code.google.com/p/googleappengine/issues/detail?id=6463 * URL fetch throws errors we've not seen before: Caused by: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to URLFETCH.Fetch(). Also, memory is all of a sudden rising a lot, every now and then. Resetting the instance with the most instances helps for a while, but the problems resume a few hours later. I'd be tempted to think that the memory issues can cause some of the above errors (but not all). It's been a while since 1.7.0, so I wouldn't be surprised of this is related to internal 1.7.1 testing. If this is the case, please stop and move us back to a regular 1.7.0 environment. Cheers, Per PS: I've raised production issues, but since I have never heard back from a single one *ever*, I'm posting to this forum to state how unhappy I am. It seems to be the only way to get attention (yeah, I've tried getting premium support, but it's not available in Germany...) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/2mhoZ1NyFa8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Frameworks on GAE
I doubt that Google will commit to any official statement. So here's my 2 cents, from a low-usage, rather complex B2B application. Frameworks are okay, to a certain degree, but be prepared to rework if you're successful We're using one heavy framework (Apache Wicket) and plenty of the regular libs and frameworks like JodaTime, Objectify, OpenID, LessCss (which depends on Rhino), Bouncy Castle etc. It's not a super-heavy app, but it's not trivial anymore either. We had to make certain modifications to make things work fine on GAE, conduct some profiling, etc, but mostly we're just writing business logic. I guess we would have had to spend some time on any platform we chose, no matter if on AWS, or Heroku, or anything else. Most of the issues we encountered were quite obvious right from the start. (Most notably, Wicket is really not that suited for GAE, but it made feature-development super-fast, so we kept it.) Some other issues only started appearing after switching to the HR datastore, but since that's the default these days, you'll probably encounter those issues right away as well. So I'd recommend you write a prototype, cram in as much of the technology you want to, and keep it running for a couple of days or weeks. Make sure you're mostly happy with performance and startup time. There are times when GAE behaves oddly and is 2-3 times slower than normal. Memcaches is usually super-reliable, until it some day isn't. So, you need to expect the unexpected, and optimise your requests so they are always reasonably fast. We aim for 500ms or less, so on a bad day if a request takes 2 or even 4 seconds, our clients may be unhappy, but not rioting. A request that's typically taking 2 seconds is too slow though (IMO). If your prototype has that kind of performance, rethink it. If you're happy with the prototype and want to move forward, I'd suggest also spending some time ensuring your service and database layers have clean interfaces. I never had to move away from GAE yet, but the recent Memcache issues had me very worried and looking for alternatives. Email, Cron, Memcache, Deploy-scripts, etc are great, but these are the things you can probably rework in a couple of weeks, even later on, especially since you can split the work between your team to a certain extent. But if the entire application (and not just a db-layer) was strongly dependent on GAE specifics, you might find it hard to port it over. There are times when things change, and then I'm happy we don't have sudden load spikes, but that thanks to a sales process that takes weeks, we know what's coming. If we see a certain part of the application is slower than it should be, we spend a couple of days improving it. That's something that simply doesn't work in Consumer apps. If you're getting featured, and it's your once-in-a-lifetime, then you probably have to heed Brandon's wise words and optimise (very) early, and not rely on frameworks, But if it's a normal B2B application, I think you can use a framework or two. Just make sure you test it with a prototype, yadda yadda etc. If you're super-successful, you'll hopefully make enough money in B2B-land, so you can afford paying someone to drop a framework that helped you kickstart your project faster. That's my strategy at least :) Cheers Per On Sunday, July 22, 2012 11:32:41 AM UTC+2, glimmung wrote: Hi All, I've been reading, initially with amusement but more recently with concern, the dialogue (for want of a better word) between Brandom Wirtz and Jeff Schnitzer re. startup time/optimisation. Brandom has now made the following very strong statement: NO FRAMEWORKS. NONE. Deal with it. This leads me to ask the Google team for their position on this: Is it your position that GAE is an unsuitable platform for framework-driven apps? I'm using a framework, and trust the framework's authors to optimise their part of the piece as much as possible, but I'm paid to solve business problems, and am not about to dive into that timesink of esoterica. It's outside my skill-set, and properly so in my view. I've only really tinkered with GAE so far, and this is putting me off investing more time in what is starting to look like a risky platform for me. So Google peeps, if I want to write B2B apps using a framework, am I your market or not? Or is it your position that the low-level optimisation to balance start-up time and hosting cost will always be required? -- Cheers, PhilK -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/OX-pCOHb8qYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group
[google-appengine] Re: Task Guarantee?
If you need to be super-certain, then you will always want to have a status flag that tells you if you have already processed some entity or not. It's also useful for cases where you cannot use transactions (a task might die in the middle as well of course) Well, at least that's how we do it :) Whenever something is important, we want to track some kind of status anyhow. One of these statuses would be STATUS_UNPROCESSED or so. Cheers, Per On Wednesday, July 11, 2012 1:55:37 PM UTC+2, Richard Arrano wrote: Hello, I have been designing my app with the notion in mind that even named tasks may execute more than once, but I only recently came to realize that a task may not execute at all. I have a task that operates on a subset of my entities and it's absolutely imperative that all members of this subset get processed and saved. I originally thought named tasks would help accomplish this, but this does not seem to be the case. Is there any way to guarantee that I process these entities? I also considered a cron job that checks every couple of minutes to check for unprocessed entities(since a cron job will kick off the initial task) but I was hoping for a slightly more elegant solution. Thanks, Richard -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/UNIfi6NldFYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Memcache errors break the GAE-internal session handling. Our application is almost unusable now. Help, please?
Memcache errors are frequently breaking the sessions-persistence, as there doesn't seem to be any try-catch in the GAE session handling. This results in tons of 500 errors for our end users. This seems like fundamental flaw in the framework to me. Uncaught exception from servlet com.google.appengine.api.memcache.MemcacheServiceException: Memcache put: Error setting single item (_ahsvuHmc0fDcNMbpwqBsbrJ4Q) at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$7.transform(AsyncMemcacheServiceImpl.java:426) at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$7.transform(AsyncMemcacheServiceImpl.java:418) at com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.convertResponse(MemcacheServiceApiHelper.java:60) at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.wrap(MemcacheServiceApiHelper.java:112) at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.wrap(MemcacheServiceApiHelper.java:105) at com.google.appengine.api.utils.FutureWrapper.wrapAndCache(FutureWrapper.java:57) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:98) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90) at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28) at com.google.appengine.api.memcache.MemcacheServiceImpl.put(MemcacheServiceImpl.java:81) at com.google.apphosting.runtime.jetty.MemcacheSessionStore.saveSession(MemcacheSessionStore.java:39) at com.google.apphosting.runtime.jetty.SessionManager$AppEngineSession.save(SessionManager.java:164) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:41) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:249) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:477) at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455) at com.google.tracing.TraceContext.runInContext(TraceContext.java:695) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) at java.lang.Thread.run(Thread.java:679) No matter what causes the Memcache error, the exception should get caught by the framework. Or there should be a way to catch it from the application side. I've tried unsuccessfully to apply the LogAndContinue-Errorhandler. Is there some way to disable memcache for session-handling maybe? Or is it possible to copy select GAE source code into your own application, applying your own error-handler some way? I'd do anything to make our application work again. I've raised a Production Issue, I've reported this in the forum. And other suggestions? I'd go premium if I had some kind of reassurance that it wouldn't simply add insult to injury. Application ID is small-improvements. Kind regards, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web
Re: [google-appengine] Need help or consulting: Request was aborted errors
It's obviously very hard to diagnose this remotely. But if it's spending 8 out of 15 seconds waiting, then what about the other 7 seconds, where exactly are they lost? There's no point arguing that this should work (of course it should), IMO you need to start analysing. Maybe you did already, but it's not obvious from your posts what you did already, and what you didn't. In another thread you mentioned you didn't use appstats for a long time - I'd do that first thing. Also, I'd add very detailed manual logging, so you know what bits run fast, and which ones don't. I've encountered plenty of GAE glitches in the past, but also quite a few things were our own fault. I'm not sure how performance analysis works on Python, but in Java it would be time to bring on the good old Profiler. I'd make sure that on localhost a request really just takes a few miliseconds, even under load, and then take it live, to a loadtesting environment and see what happens there under growing load. Given the numbers you posted, I'd not be surprised if some lockingsynchronizing is part of the problem, but that's just a guess. Find out what operation really takes the most time, or at what load things start going wrong. On Monday, July 9, 2012 4:26:01 AM UTC+2, Richard wrote: Yep, I know about that. The internal limit is 1000msec, but you should aim for 750msec just to 'be safe'. Here are my timings: ms=15376 cpu_ms=37 api_cpu_ms=17 cpm_usd=0.003428 pending_ms=8265 37 msec of CPU after waiting 8.2 seconds to execute. Some math: - 500 incoming requests - 40 servers = ~ 15 requests/server ... but each request takes ~55 msec of CPU + API That's 825 msec for THE WHOLE LOT ! So why on earth is there 8 second of waiting before that request was serviced ? -R On Sunday, July 8, 2012 10:13:28 PM UTC-4, Per wrote: Not sure where I saw it, but I believe an application must respond within a second at most *on average*, or GAE reserves the right to throttle it. So, if you have some kind of slowdown inside your app, then requests start piling up. I'm not a Python developer, but if you had the equivalent of a synchronised map in there, with lots of concurrent access, then you might end up in a situation like this. Not sure about what's really to blame of course, but I'd strongly recommend setting up a controlled loadtesting environment (just copy the app, and start firing requests at it). Add plenty of logging to your app, enable appstats, and then slowly increase the load. Maybe even strip the application down, starting only with read access. I'm sure you will find something, and I would love to hear what it was! I wish you luck! On Monday, July 9, 2012 3:19:04 AM UTC+2, Richard wrote: I could if memcache actually worked. But it does not. I originally tried to use it and found that I could not push the game state to memcache and then have the other instances pull it. They would get versions of it that were up to 5 minutes old. My timings are 5 second windows. 5 secs to submit all scores 5 secs to reap scores and calc leaderboards 5 secs to fan out results to clients Experience shows that memcache is just broken for that sort of timing. As for using Go instead of Python. I am not sure I follow why Go should be better. The lag is not coming from CPU or queries. Right now I am running 50 instances to serve 500 game clients. $48 for the last 18 hours. 11% of my requests result in Request was aborted. Yeah, that is 12 THOUSAND fails. Back in the year 1995, ftp.cdrom.com could serve 2000 clients simultaneously on a Pentium Pro 200MHz and I cannot serve 20 clients on a 500Mhz virtual box ? I still contend there is some internal throttling going on somewhere. -R On Sunday, July 8, 2012 6:23:02 PM UTC-4, Kyle Finley wrote: Richard, Another option would be to move the Game State request to a Gohttps://developers.google.com/appengine/docs/go/overview instance, either as a backendhttps://developers.google.com/appengine/docs/go/config/backendsor as a separate version. I believe a single Go instance should be able the handle 500 request / second. You could then share the Game State between the Python version and the Go version through Memcache, cacheing to instance memory every 5 sec. - Kyle -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_V2FQCZb_sQJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Memcache errors break session-saving. Should fail silently IMO. Any workaround?
Hi Joakim,
replacing the session with a cookie is unfortunately not an option for us,
our web framework stores a lot of stuff in the session.
I don't even want to use memcache at all, the session ends up in the
datastore anyway, so if I had a choice, I would disable memcache here. It's
just built that way in GAE. I'd do anything to stop our users from
getting 500-errors, and I'd like a way to ignore memcache errors here.
I read that the sessions get stored into memcache namespace _ahs, so I
tried to register a LogAndContinue-Errorhandler for that namespace, but
unfortunately that doesn't work either, at least not this way:
MemcacheServiceFactory.getMemcacheService(_ahs).setErrorHandler(new
LogAndContinueErrorHandler(Level.INFO) {
@Override
public void handleDeserializationError(InvalidValueException
thrown) {
log.error(gotcha!,thrown);
super.handleDeserializationError(thrown);
}
@Override
public void handleServiceError(MemcacheServiceException thrown)
{
log.error(gotcha!,thrown);
super.handleServiceError(thrown);
}
});
Any idea how else to register a handler to prevent this from killing our
requests?
Cheers,
Per
On Friday, July 6, 2012 7:24:26 PM UTC+2, Joakim wrote:
If silent failure on write is to be acceptable, we need a way to make sure
following requests do not read session from memcache, as it could return an
old value. I am unaware of any way to accomplish this reliably without
hitting the datastore, at which point you might as well not have the
session in memcache at all.
One possible alternative is to replace the entire session with storing
encrypted data in a cookie, though this only works for smaller amounts of
data as the maximum size for an entire cookie is generally said to be 4095
bytes.
Just my 2 cents.
Joakim
On Wednesday, July 4, 2012 11:38:34 PM UTC+2, Per wrote:
When using sessions, these are (also) stored in memcache by GAE.
Unfortunately, it seems like any memcache hiccup can crash the process,
turning an otherwise fine page into a 500 page on the way out.
To me that looks like a design flaw. After all, Memcache is optional, and
its expected to be unavailable or breaking every now and then. As of 1.6
you can even provide a policy to silently ignore memcache errors inside
your application. We're using
setErrorHandler(ErrorHandlers.getConsistentLogAndContinue(Level.SEVERE))
with Objectify just fine, but it doesn't seem to have any effect on the
Session-Save process. This seems to be entirely outside our control.
These errors are really annoying some of our clients. Is there maybe some
kind of other error handler I'm missing?
Here's the stracktrace for reference:
com.google.appengine.api.memcache.MemcacheServiceException: Memcache put:
Error setting single item (_ahsOH7nik8bCRTbIXblKurKQQ)
at
com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$7.transform(AsyncMemcacheServiceImpl.java:426)
at
com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$7.transform(AsyncMemcacheServiceImpl.java:418)
at
com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.convertResponse(MemcacheServiceApiHelper.java:60)
at
com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.wrap(MemcacheServiceApiHelper.java:112)
at
com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.wrap(MemcacheServiceApiHelper.java:105)
at
com.google.appengine.api.utils.FutureWrapper.wrapAndCache(FutureWrapper.java:57)
at
com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:98)
at
com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
at
com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28)
at
com.google.appengine.api.memcache.MemcacheServiceImpl.put(MemcacheServiceImpl.java:81)
at
com.google.apphosting.runtime.jetty.MemcacheSessionStore.saveSession(MemcacheSessionStore.java:39)
at
com.google.apphosting.runtime.jetty.SessionManager$AppEngineSession.save(SessionManager.java:164)
at
com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:41)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle
[google-appengine] Re: managing bulk emails from app engine app
Hi Vik, technical details aside, keep in mind that the spam filter settings on App Engine can be pretty strict. We have been sending mails for 18 months now, increasing the number gradually, and we also used DKIM signing. Still, suddenly some time last week our mails stopped getting sent. From all I can tell some Google mechanism decided that we're a spammer. We have inquired with Google instantly, but of course the GAE customer service is poor as always and never got back to us. So we switched to an external email provider now (mailgun, but there are dozens out there), and I'm quite happy with that, since tracking of emails is a lot more convenient too. It's not free though. Long story short, if you all of a sudden send 10k mails, be prepared to get filtered by Google because they think you're a spammer. Make sure to spread the load, and track what you have sent already, so you know what you'd need to resend if push comes to shove. Cheers, Per On Saturday, July 7, 2012 7:19:12 PM UTC+2, Vivek Kumar wrote: Hello We have around 1 registered users with us who are blood donors. We want to send them periodic emails like every month to remind them or their account status etc. Obviously, a single request cannot process so many emails. So, what is the suggested way to handle this use case without hitting request time limits etc on app engine? our app is in GAE for java Vik -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/CZHqGIOGAAUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Need help or consulting: Request was aborted errors
Sounds odd. I'd also suggest trying F4 (just for the fun of it, even if the
chance is low that you'll find something, it won't hurt to try it for a few
hours).
Did you check out appstats yet?
Also, in my experience the whole fiddling-with-sliders was never
productive, I'd suggest resetting to automatic/automatic values, it just
seemed more reliable to me.
You didn't mention if you had checked the quota details screen, there are a
couple of per-minute quotas that seem to bite people out of the blue. I
once had a backup-job that was stuck and cost me lots and lots of money, it
was running in the background on the built-in version, maybe something like
this can also contribute to throttling the core application?
Well, admittedly we're all poking in the dark here, maybe you can detail
what else you have tried and where you've looked already?
All the best
Per
On Sunday, July 8, 2012 10:14:05 PM UTC+2, Richard wrote:
I have an app that produces: request was aborted errors. My users are
getting pissed. So am I. I thought App Engine was meant to scale ?
The app is running up to 40 instances to serve 500 users. All 500 users
will query the app within a 5 second window (like a mini-DDOS). It is
python 2.7 and marked as thread-safe. Why one instance can only serve
under 20 users, I don't know ! Average memory usage is 44MB on F1
instances.
Latency is set at max 500msec. I have tried setting idle instances to
20-Automatic. Billing has been around $30 for the last 12 hours. However
this is CRAZY. It cost me $30 to push out 1.25GB of data to under 500
users (500 is the peak!) ?!?
The stupid part is the code looks like this:
global GAME_DATA
global GAME_DATA_TIMESTAMP
delta = datetime.datetime.now() - GAME_DATA_TIMESTAMP
if delta.total_seconds() 5.:
# return cached data
game = GAME_DATA
else:
# get new
cg_ref = CurrentGame.get_by_key_name(current_game_key_name)
game = cg_ref.k
GAME_DATA = game
GAME_DATA_TIMESTAMP = datetime.datetime.now()
data = {'s': 'OK', 'g': game.board }
self.response.out.write(json.dumps(data))
Yes, that's right... we don't actually do 500 db queries...
At this point, I think there is some internal throttle with GAE that only
allows a limited number of http connections at once and just kills off the
rest or something.
Can only explain what is going on here, or point me to a competent
consultant who I can pay to tell me how to fix this ?
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/nsTNa0Ub8PEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Need help or consulting: Request was aborted errors
Not sure where I saw it, but I believe an application must respond within a second at most *on average*, or GAE reserves the right to throttle it. So, if you have some kind of slowdown inside your app, then requests start piling up. I'm not a Python developer, but if you had the equivalent of a synchronised map in there, with lots of concurrent access, then you might end up in a situation like this. Not sure about what's really to blame of course, but I'd strongly recommend setting up a controlled loadtesting environment (just copy the app, and start firing requests at it). Add plenty of logging to your app, enable appstats, and then slowly increase the load. Maybe even strip the application down, starting only with read access. I'm sure you will find something, and I would love to hear what it was! I wish you luck! On Monday, July 9, 2012 3:19:04 AM UTC+2, Richard wrote: I could if memcache actually worked. But it does not. I originally tried to use it and found that I could not push the game state to memcache and then have the other instances pull it. They would get versions of it that were up to 5 minutes old. My timings are 5 second windows. 5 secs to submit all scores 5 secs to reap scores and calc leaderboards 5 secs to fan out results to clients Experience shows that memcache is just broken for that sort of timing. As for using Go instead of Python. I am not sure I follow why Go should be better. The lag is not coming from CPU or queries. Right now I am running 50 instances to serve 500 game clients. $48 for the last 18 hours. 11% of my requests result in Request was aborted. Yeah, that is 12 THOUSAND fails. Back in the year 1995, ftp.cdrom.com could serve 2000 clients simultaneously on a Pentium Pro 200MHz and I cannot serve 20 clients on a 500Mhz virtual box ? I still contend there is some internal throttling going on somewhere. -R On Sunday, July 8, 2012 6:23:02 PM UTC-4, Kyle Finley wrote: Richard, Another option would be to move the Game State request to a Gohttps://developers.google.com/appengine/docs/go/overview instance, either as a backendhttps://developers.google.com/appengine/docs/go/config/backendsor as a separate version. I believe a single Go instance should be able the handle 500 request / second. You could then share the Game State between the Python version and the Go version through Memcache, cacheing to instance memory every 5 sec. - Kyle -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iAnWI4OPNRsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Memcache errors break session-saving. Should fail silently IMO. Any workaround?
When using sessions, these are (also) stored in memcache by GAE. Unfortunately, it seems like any memcache hiccup can crash the process, turning an otherwise fine page into a 500 page on the way out. To me that looks like a design flaw. After all, Memcache is optional, and its expected to be unavailable or breaking every now and then. As of 1.6 you can even provide a policy to silently ignore memcache errors inside your application. We're using setErrorHandler(ErrorHandlers.getConsistentLogAndContinue(Level.SEVERE)) with Objectify just fine, but it doesn't seem to have any effect on the Session-Save process. This seems to be entirely outside our control. These errors are really annoying some of our clients. Is there maybe some kind of other error handler I'm missing? Here's the stracktrace for reference: com.google.appengine.api.memcache.MemcacheServiceException: Memcache put: Error setting single item (_ahsOH7nik8bCRTbIXblKurKQQ) at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$7.transform(AsyncMemcacheServiceImpl.java:426) at com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$7.transform(AsyncMemcacheServiceImpl.java:418) at com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.convertResponse(MemcacheServiceApiHelper.java:60) at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.wrap(MemcacheServiceApiHelper.java:112) at com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.wrap(MemcacheServiceApiHelper.java:105) at com.google.appengine.api.utils.FutureWrapper.wrapAndCache(FutureWrapper.java:57) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:98) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90) at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28) at com.google.appengine.api.memcache.MemcacheServiceImpl.put(MemcacheServiceImpl.java:81) at com.google.apphosting.runtime.jetty.MemcacheSessionStore.saveSession(MemcacheSessionStore.java:39) at com.google.apphosting.runtime.jetty.SessionManager$AppEngineSession.save(SessionManager.java:164) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:41) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:249) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:477) at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455) at com.google.tracing.TraceContext.runInContext(TraceContext.java:695) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) at java.lang.Thread.run(Thread.java:679) And here's the code I tried, but which didn't help: MemcacheService service = MemcacheServiceFactory.getMemcacheService(); service.setErrorHandler(new LogAndContinueErrorHandler(Level.SEVERE)); Any help would be appreciated! Kind regards, Per -- You received this message because you
[google-appengine] Re: Is strong symmetric encryption possible on GAE at all (using Java)? I've searched for hours now...
Success! SDK 1.7 enables you to register your own JCE provider, so it now works with Bouncy Castle for instance. Just make sure to disable precompilation, since it breaks signed JARs. We're now encrypting our main content columns using AES-256, and this (in combination with SSL) makes me very happy! Also, encryption is surprisingly fast, a lot faster than on my local MBP. There's really no excuse to not encrypt your data anymore. Thanks for solving the underlying issue, despite there being relatively few votes. Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/RaJSSoSUEWUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Cannot purchase VIP SSL: Limit reached, contact SSL trusted testers support
Hi, I was so excited about the new SSL VIP, I wanted to purchase right away. Unfortunately, a strange error message stops me: https://lh6.googleusercontent.com/-RCO4OkSVxMw/T-s8tNiGODI/ADk/H49MVBYGUjs/s1600/Google+Apps.jpg I have no clue who the Trusted Testers support team is. When I last inquired (by email) to the address which signed me up to the TT, I didn't hear back. Well, I *never* hear back from Google when sending emails to you. So I'm putting this into the forum right away :) Any idea? Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/D_Xh-oQ_kD0J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: App Engine 1.7.0 Release is Out!
AWESOME news! A big thank you to the entire App Engine team. This is the release I've been waiting for. VIP at last! And at a very affordable price tag too. Now, the only thing that's left would be proper support, e.g. replying on the forums, responding to emails, replying on SO. With all these improvements in place, you might even be able to spare a developer to rework the FAQs :) Cheers, Per On Wednesday, June 27, 2012 1:42:16 AM UTC+2, Takashi Matsuo (Google) wrote: Hi App Engine users, We've just released the release of the 1.7.0 SDK. Release notes for Java: http://code.google.com/p/googleappengine/wiki/SdkForJavaReleaseNotes Release notes for Python: http://code.google.com/p/googleappengine/wiki/SdkReleaseNotes Release notes for Go: http://code.google.com/p/googleappengine/wiki/SdkForGoReleaseNotes Enjoy! -- Takashi Matsuo -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Tu_-R_0NUsgJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Is strong symmetric encryption possible on GAE at all (using Java)? I've searched for hours now...
Hi all, I'm not a crypto-enthusiast, so maybe there is something that I'm missing. But the usual suspects like the Sun JCE unlimited strength policy files don't work for obvious reasons, and Bounce Castle does not either. I got Jasypt to work, using the StrongTextEncryptor, which is a wrapper for PBEWithMD5AndTripleDES, but I read that DES is not considered strong enough anymore, so I was hoping for AES with more than 128 bit (or something else that's considered good enough these days). I came across dozens of questions about this topic, but no answers unfortunately. Just fluffy assumptions, go figure it out suggestions, etc. So I'm wondering, maybe it's simply not possible? Is it entirely dependent on the JCE unlimited strength files? (please star http://code.google.com/p/googleappengine/issues/detail?id=2889 in that case) Or is there some other way to get strong symmetric encryption off the ground? Any help would be greatly appreciated! Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-l-kGzq5nWIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Is strong symmetric encryption possible on GAE at all (using Java)? I've searched for hours now...
Oh wait! Maybe SDK 1.7 has the answer! Too late to try it now (2am) but I'll check tomorrow and update this question with my findings! That would be awesome news. Totally missed that announcement detail. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/uXiLEKiPfUkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Java instance startup time out of control
Hi Richard, IMO it's not important to put all JAR files into one. It's important to reduce your absolute number of files. Each initial file access costs a little time, but it doesn't really matter if you access 30 jar files or 1. What makes a difference is wether App Engine needs to load 2000 class files one by one, or if they are all inside a handful of jars. I'd also consider moving all other files, like property-files and HTML files (in case your app parses them) into those jars. Made a huge difference for me. Good luck! Per On Saturday, June 23, 2012 7:22:34 AM UTC+2, Richard Watson wrote: Hi Will, I also tried bundling (most [1]) jars into one but it didn't seem to move the needle at all, once classes were jarred. I did perceive a lower initial-RAM level - I think it was about 4 or 5 megs lower but I didn't test that too carefully. Is your load-time difference compared to unjarred classes, or to multiple jars only? I did think to try bundling only the jars my app would need on startup, to reduce the overall initial load. Does Java have to inspect the contents of all jar files to figure out where required classes are? Richard [1] GAE deployment complained my 1-jar solution was too big, so I wrote an ant task to jar-up only the jars below a certain size and leaving very few bigger ones. Went from 50+ to about 8. But again, no perceived load-time improvement once the classes were jarred. On Friday, June 22, 2012 8:35:11 AM UTC+2, Will Rayner wrote: Hi all, I've also been battling with with java warmup times. Last week I had startup time of at least 37 seconds. Now it's hovering around 16. My performance improvements were made by bundling all my dependencies together into a single jar. I've been using the excellent gradle gae plugin (https://github.com/bmuschko/gradle-gae-plugin), which integrates with https://github.com/musketyr/gradle-fatjar-plugin/. This could easily be integrated with an existing gradle project in under an hour. We're using Resteasy, Htmleasy, soy templates, hibernate orm and validator. There were about 60 jars in my WEB-INF/lib. Regards, Will Rayner On Friday, June 22, 2012 1:18:51 PM UTC+10, Thomas Wiradikusuma wrote: I have updated http://code.google.com/p/googleappengine/issues/detail?id=7706 with this information. On Monday, 18 June 2012 11:44:29 UTC+8, Takashi Matsuo (Google) wrote: On Mon, Jun 18, 2012 at 9:57 AM, Thomas Wiradikusuma wrote: Just my 2 cents, If indeed our app needs to be single-JARred and obfuscaticated (at least removing unused code), IMO that feature should be baked in the tool. Probably triggered with extra flag. I think this is also a good feedback especially if creating the single JAR contributes the performance. I'd appreciate it if you could file an issue. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/H1eqfW8o0vQJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.7.0 Prerelease Now Available
Hi Marzia, - The Java Datastore API now supports Or queries. This sounds interesting. Do you have some preliminary documentation for this feature? Kind regards, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/9nLLBK6wqpQJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Java instance startup time out of control
Hi Jeff,
just as a comparison, we have 33 classes that get initialised with
Objectify, and it takes merely 2 seconds. We had all the problems you
mentioned prior to jaring things up, and since then performance has been
improved vastly. I thought Google had improved the problem (they made some
comment about this 9 months ago or so, when we posted our summary at
http://www.small-improvements.com/app-engine-performance-tuning) but maybe
they haven't.
We're also using Apache Wicket, and while initialising the mounted paths,
we're actually referring to hundreds of additional Java files, which again
refer to some 2 to 5 inner classes each. This was *killing* us before some
nice fellow suggested jarring the classes, and now even that merely takes 4
seconds during startups.
Maybe it also helps a little that we're on F4 these days, but all the
performance tuning we did back then was on F1. I'm guessing you're on F4
anway, if you're this desperate.
BTW, I had also considered stripping unused classes from the jars, but it
was really the class *loading*, not the parsing of Jar files, that was
causing the slowdown. I'm guessing it had to do with file access
ultimately, and that each file access on the VM needs to be verified by the
secure Classloader, and that this is simply tons more efficient if you're
just looking at the same jar file all the time.
So, add that target to your ant file, and let us know how you go! :)
target name=createjar depends=copycerts description=Creates a jar
from the classes folder
jar jarfile=${libs}/small-improvements.jar basedir=${classes}/
delete dir=${classes}/
/target
Cheers,
Per
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/LkQhIAFwLWoJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Java instance startup time out of control
Hi Jeff, sounds awful. Even 20s on a good day is a lot IMO. We're using Wicket, which is on the heavy side, and still the startup typically only takes 10 to 15s, and that's including rendering an initial page too (so it basically involves firing up all subsystems too, loading all classes, contacting the database several times, filling some caches, etc). I'm guessing you're experiencing some new release testing by the Google Team, so that might account for a certain slowdown. We've seen our average latency increase from 250ms to 380ms over night, 2 days ago. Maybe your slowdown is related to a similar service change. But it sounds like your application startup is too slow to begin with. Do you have logging in there which can pinpoint what parts take how much time? Would be interested to learn about that. Cheers, Per On Saturday, June 16, 2012 11:56:33 PM UTC+2, Jeff Schnitzer wrote: We're having a big problem with instance startup time. It varies between 20s and 60+ seconds, and lately it's tending towards the high end. We're starting to experience downtime because instances get deadlined before they go active. This app is well optimized for GAE. There's no classpath scanning and it doesn't try to eagerly load data. On a good day it starts in 20s... so at this point there's not really much I can do. I have a cron task that performs a db cleanup once a minute, and since crons can run over 60s I can eventually get one instance started, which is enough to serve traffic at the moment. But at this point I can no longer deploy code over old versions because the appserver restart will fail. Please help. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/CzSFxHEhj3QJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Memcache problem on a production application
Hi Mathieu, not sure if you're aware of this (it took me a while as well) but you need to design your application so it doesn't flinch even when memcache is unavailable. Try-catch everything. We had an issue at the same time as you as well, and it turned out that in one part of the application we had also forgotten to try-catch memcache errors. Just keep improving until it's 100% robust. Because memcache just isn't. A large scale downtime would probably show up on the system status page, but local issues won't. Cheers, Per On Wednesday, May 16, 2012 7:14:49 PM UTC+2, Mathieu Clavel wrote: Memcache is now working as expected. Problem was from 16h to 18h (UTC+2) Mathieu Le mercredi 16 mai 2012 17:49:45 UTC+2, Mathieu Clavel a écrit : Hello, My production application is currently having memcache problem. Nothing is visible in the global appengine system status : https://code.google.com/status/appengine/ Error is : Caused by: com.google.appengine.api.memcache.MemcacheServiceException: Memcache put: Error setting single item (...) Nothing is working : Jasper also has error : org.apache.jasper.runtime.JspFactoryImpl internalGetPageContext: Exception initializing page context com.google.appengine.api.memcache.MemcacheServiceException: Memcache put: Error setting single item (_ahsXuelLhc5ClPxy8Dra2vPUQ) My application name is oav-agz. Regards, Mathieu -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/eG0cP9poQPgJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Calculation of frontend hours seems off by almost factor 2.
Hi Gregory, if I'm doing something wrong here then I'd love to learn how you do it. I tried setting the max idle instances to 1 (didn't help, except increase latency), I set the pending queue to 1s and 2s and 3s (didn't help, but increased the latency). If there is a way to limit the instances reliably, I would love to hear how you do it. We have maybe 0.5 to 1 requests per second on average (the QPS up there in the screenshot is only a snapshot for the past minute). Our average response time is around 300ms. Occasionally more, occasionally less. It's not great, but it's still a lot lower than the 1s limit for continuous usage. We do use the memcache where we can, but it's limited to some 10MB to 30MB for us at the moment, and the hit ratio is always around 65%, so we do have to access the datastore more than we want to. We're also denormalising plenty of data for faster access, but it would substantially slow us down if we had to denormalise *everything* that is displayed on a page. I don't think it should be a problem to do 2 to 3 queries in a single request, maybe 5 datastore gets plus a few memcache gets, should it? Given that we're developing a complex application (www.small-improvements.com) and not just a number crunching app, this seems not too bad. And performance-wise we're happy, it's just the cost that's prohibitive. Anyway, I would love to hear how you do it, and maybe a screenshot of your system (and the ratio of requests to instances) would be interesting. Cheers, Per On Tuesday, May 15, 2012 11:01:26 AM UTC+2, Gregory Nicholas wrote: Not directly answering the question, I'm posing another, why the eff are you running instances in such a manner for such little qps Also, I can also suggest that at that low of qps, you could stand to substantially optimize the work you're doing reading data from the data store, which will also help minimize the frontend instance workload -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ht06S7VVAkkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Scheduler/billing changes in 1.6.5? My daily cost went from $5 to $35.
Oh my Goodness! You're right. I never noticed the new built-in version. Plus I never realised there were failed backup-tasks that were retrying all over again (for 20 days! 500 retries!) to do a backup. This meant two instances were permanently doing stuff. Also this explains the unusually high number of datastore operations. Stupid me! But stupid GAE user interface too. How about adding a few words to the backup initiated confirmation dialog? Like these tasks will run in a built-in version, so you can track progress and billing over there? Something along the lines. There's plenty of whitespace on that screen. Anyway, thanks Takashi for pointing me in the right direction! Per On Tuesday, May 15, 2012 5:33:47 PM UTC+2, Takashi Matsuo (Google) wrote: Hi Per, Thanks for sending the details. First thing comes to mind is that the culprit might be instances on a version 'ah-builtin-python-bundle'. This version is currently dedicated to the 'Datastore Admin' feature. I think you are running some task like backup/copy/deletion in 'Datastore Admin', right? I hope it could explain the situation you are experiencing right now. Please let me know if that isn't the case. Thanks, -- Takashi On Tue, May 15, 2012 at 7:06 AM, Per per.fragem...@gmail.com wrote: https://lh3.googleusercontent.com/-qa-cjA-cUbM/T7GBSShwGrI/ADY/OYljQYgcczc/s1600/changed-2-weeks-ago.jpg Hi Takashi, I waited with my response because I didn't want to jump to conclusions. Yes, I have been experimenting with some settings every now and then, limiting instances, playing with the pending sliders etc. But all it did was change how quickly instances were created or collected (while always increasing latency). It didn't change how the billed instances were calculated. Above is the chart of my applications' instances for the past month. (The editor didn't allow me to place it at the right location...) It's rather easy to spot when things changed. We used to get billed for roughly one instance on average. Yes, we did have more than one running, but it was (and is!) always one instances that handles 95% to 99% of the load. So it seemed only just that Google would only charge for the main instances. I never asked for an instance that just sits there, but I didn't mind it while it was free :) So now being charged for two or three instances, when only one is really doing anything, seems like a major change that should be documented. Okay, maybe it's just our application, but our pricing has increased steeply. I just posted another message about the frontend hour calculation. Both issues combined seem to have led to a pricing increase from $5 to $10 before, to now $30 to $50. Our request per second have increased moderately, we may have made some requests slower, we might have slightly different usage patterns. But I cannot see a reason for a price increase this steep. Any help or insight would be appreciated. Our app ID is small-improvements-hrd Kind regards, Per On Friday, April 27, 2012 7:02:26 AM UTC+2, Takashi Matsuo (Google) wrote: On Fri, Apr 27, 2012 at 10:16 AM, Per per.fragem...@gmail.com wrote: Hi team, previously, the scheduler used to spin up 3 instances for our application, only used 1 out of the 3, but at least we didn't have to pay for the unused instances. There was always a big gap between total instances and billed instances. Wasn't it because that you set Max Idle Instances at that time? If you set Max Idle Instances to Automatic, the total instances and billed instances should be the same. However, as of yesterday, the scheduler continues to spin up 3 instances on average, but we have to pay for all of them. We're at maybe 1 request per second on average, and it's all handled just fine by one instance. We're on F4, so having to pay for 2 mostly unused instances hurts. Since this seems to coincide with the 1.6.5 release, I'm tempted to think that there's something changed in the background, and it would be great to learn more about the new suggested way. When we tried to limit idle instances last time, all we got was instance churn and bad latency. So we're on automatic/automatic these days, and I'd prefer not to have to experiment again. Some advice would be great. Again, I'm wondering when you tried to limit the idle instances. Generally speaking, you can not get everything. If you want a better performance, you'll need to pay more, and vice versa. However, if you're really certain that the behavior of our scheduler has significantly changed, please let me know your app-id and detailed explanation(it can be off-list), so I can look into it further. That aside 1.6.5 is great of course! :) Thanks! -- Takashi Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App
[google-appengine] Calculation of frontend hours seems off by almost factor 2.
Hi Google team, according to my understanding the F4 instances incur 4 instance hours per actual hour. Assuming an average of 2 instances over the past 14 hours, we should be at 2*14*4 = 112 instance hours, not at 205. I understand that this data may not be exact, so maybe we should be at 125 or even 140 or so. Or am I mistaken? https://lh6.googleusercontent.com/-7D0y1QsVR2g/T7F9XDMQbCI/ADM/lX6_EjfvimU/s1600/instances2.jpg This has been going on for about 2 weeks now, and our bills are getting very high, given that we rarely exceeds 0.8 requests per second. I raised two production issues (on Thursday and Friday) but I haven't received confirmation even wether the issues have been created, so I'm trying the forum. We used to be at $5 to $10per day, now we're at $30 to $50. Some of it might be due to increased usage, but we certainly didn't quadruple load. It would be great to get some insight into this. Our app ID is small-improvements-hrd Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/57lww16rkA0J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Scheduler/billing changes in 1.6.5? My daily cost went from $5 to $35.
https://lh3.googleusercontent.com/-qa-cjA-cUbM/T7GBSShwGrI/ADY/OYljQYgcczc/s1600/changed-2-weeks-ago.jpg Hi Takashi, I waited with my response because I didn't want to jump to conclusions. Yes, I have been experimenting with some settings every now and then, limiting instances, playing with the pending sliders etc. But all it did was change how quickly instances were created or collected (while always increasing latency). It didn't change how the billed instances were calculated. Above is the chart of my applications' instances for the past month. (The editor didn't allow me to place it at the right location...) It's rather easy to spot when things changed. We used to get billed for roughly one instance on average. Yes, we did have more than one running, but it was (and is!) always one instances that handles 95% to 99% of the load. So it seemed only just that Google would only charge for the main instances. I never asked for an instance that just sits there, but I didn't mind it while it was free :) So now being charged for two or three instances, when only one is really doing anything, seems like a major change that should be documented. Okay, maybe it's just our application, but our pricing has increased steeply. I just posted another message about the frontend hour calculation. Both issues combined seem to have led to a pricing increase from $5 to $10 before, to now $30 to $50. Our request per second have increased moderately, we may have made some requests slower, we might have slightly different usage patterns. But I cannot see a reason for a price increase this steep. Any help or insight would be appreciated. Our app ID is small-improvements-hrd Kind regards, Per On Friday, April 27, 2012 7:02:26 AM UTC+2, Takashi Matsuo (Google) wrote: On Fri, Apr 27, 2012 at 10:16 AM, Per per.fragem...@gmail.com wrote: Hi team, previously, the scheduler used to spin up 3 instances for our application, only used 1 out of the 3, but at least we didn't have to pay for the unused instances. There was always a big gap between total instances and billed instances. Wasn't it because that you set Max Idle Instances at that time? If you set Max Idle Instances to Automatic, the total instances and billed instances should be the same. However, as of yesterday, the scheduler continues to spin up 3 instances on average, but we have to pay for all of them. We're at maybe 1 request per second on average, and it's all handled just fine by one instance. We're on F4, so having to pay for 2 mostly unused instances hurts. Since this seems to coincide with the 1.6.5 release, I'm tempted to think that there's something changed in the background, and it would be great to learn more about the new suggested way. When we tried to limit idle instances last time, all we got was instance churn and bad latency. So we're on automatic/automatic these days, and I'd prefer not to have to experiment again. Some advice would be great. Again, I'm wondering when you tried to limit the idle instances. Generally speaking, you can not get everything. If you want a better performance, you'll need to pay more, and vice versa. However, if you're really certain that the behavior of our scheduler has significantly changed, please let me know your app-id and detailed explanation(it can be off-list), so I can look into it further. That aside 1.6.5 is great of course! :) Thanks! -- Takashi Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Takashi Matsuo | Developer Advocate | tmat...@google.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/7AtnmRTCRToJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Should my memcache stats worry me? Oldest item 10-20 minutes, 10mb size, 65% hit rate. Seems low..?
Hi all, I never really paid attention to the memcache stats, but after my app felt slow the past couple of days I started checking the stats more regularly. It's been between 5mb and 15mb the last three days, there are between 200 and 1000 items in it, and the oldest item was between 5 minutes and 20 minutes old. The hit rate is between 60% and 70%. We typically have between 1 and 3 instances running, with maybe 0.5 requests per second on average. Based on gut feeling I'd say it would be beneficial if the memcache wasn't evicted that quickly, we should see higher hit rates if items were allowed to stick around for an hour or so. I'm curious, what are everyone else's experiences? Or is there maybe even some suggested approach to increase this? I searched, but couldn't find anything relevant, only complaints about memcache failures :) Cheers, Per (small-improvements-hrd) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Sfe1mOxZqWsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Scheduler/billing changes in 1.6.5? My daily cost went from $5 to $35.
Hi team, previously, the scheduler used to spin up 3 instances for our application, only used 1 out of the 3, but at least we didn't have to pay for the unused instances. There was always a big gap between total instances and billed instances. However, as of yesterday, the scheduler continues to spin up 3 instances on average, but we have to pay for all of them. We're at maybe 1 request per second on average, and it's all handled just fine by one instance. We're on F4, so having to pay for 2 mostly unused instances hurts. Since this seems to coincide with the 1.6.5 release, I'm tempted to think that there's something changed in the background, and it would be great to learn more about the new suggested way. When we tried to limit idle instances last time, all we got was instance churn and bad latency. So we're on automatic/automatic these days, and I'd prefer not to have to experiment again. Some advice would be great. That aside 1.6.5 is great of course! :) Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The instance scheduler still has problems
The scheduler is most likely made with larger applications in mind (hundreds of instances). I saw plenty of isses when I tried to optimise for few instances. I had very similar issues as you had, when I was on a 2/2 setting for instance. Basically I wanted to have 2 instances all the time, and I tried 2/2, 2/3, 3/3, 1/3, and 1/2. But what works best (by far!) for me at the moment is the setting automatic/3. I decided to use the F4 instance as well. It now turns out I always have 2 or 3 instances running, but only get charged for one :) And very few spin-up and spin-down happening. And consistently fast responses too. Every app is different, so do experiment a bit. On Thursday, April 5, 2012 4:30:00 AM UTC+2, Tapir wrote: In my previous post, http://groups.google.com/group/google-appengine/browse_thread/thread/c5ab4ec85c5b5525/b8f97af3ddcdcaa1?lnk=gstq=charming#b8f97af3ddcdcaa1 I said after I rewrote the custom warm-up servlet, I found it works like a charming. Yes, the warm-up cases really becomes fewer, but now I find it still exits. After some research, I found the behavior of the instance scheduler is uncertain. Here is a case: one resident instance is idle, a new request coming, the idle resident instance will handle it. At the same time, another new request coming, the scheduler wait pending latency time, the idle instance is still not free, so a new instance is started, which needs 20-30 seconds to warmup. After ***N seconds***, the scheduler found the new instance is still not warmed-up and found the resident instance is free now, so the scheduler let the resident instance also hand the second new request. The implementation is not bad, but the strange thing is ***N seconds*** varies from 1 second to 30 seconds. That is why some warm-up requests will be still encountered. btw, I found affer I rewrote the custom warm-up servlet, 99% percent cases of the effort to create a new instance are useless. The new created instance will be shutdown without handling any user requests. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/PdQTyE299w4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: how do I get my 'Recipients Emailed' quota back after hrd migration?
Hi Jose, thanks for that link. I just did the migration to HRD, and it was not at all obvious to me that this limit gets re-introduced despite having been a paying GAE user for a year now. The migration guide says to update billing, which I did. A link to the URL mentioned by Jose would have been very helpful. How about placing that link into http://code.google.com/appengine/docs/adminconsole/migration.html ? Cheers, Per On Saturday, March 17, 2012 4:21:41 AM UTC+1, Jose Montes de Oca wrote: Hi, Please fill out this form for your use case: https://docs.google.com/a/google.com/spreadsheet/viewform?formkey=dENESzEwNzJiZEdpQkRzTl9RNElVWUE6MQ#gid=0 Hope this helps! On Friday, March 16, 2012 8:22:09 AM UTC-7, Karl Rosaen wrote: Ok, I know see that the limit is 100 until the first charge is cleared: http://code.google.com/appengine/docs/quotas.html but given we just migrated an app, waiting for up to a week for the first charge to go through is not really acceptable... I'll try to file a production ticket about this, but I would hope that the migration tool, when used with an app in good standing, could transfer the email quotas to avoid disruption of sending emails. On Friday, March 16, 2012 11:14:11 AM UTC-4, Karl Rosaen wrote: Hi, I just migrated our app to the high replication datastore, billing enabled and everything. I can't seem to adjust or opt in to pay more for the mail api quotas, and the default limit of 100 recipients emailed is not nearly enough and it's disrupting our application. In our previous app, our quota for 'Recipients Emailed' was 1,771,789, I think I opted in to pay more for that limit a while back. Anyways, how do I get my quota back? We're willing to pay! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/c5onwEdK2A4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Fwd: 1.6.4 Prerelease Now Available
- Billed applications which have specified additional logs storage over 1 GB are now being charged for that storage at $0.24/GB/month. What's actually happening is that we are hiding logs, but once 1.6.4 is released, we will be deleting logs greater than 1gb. With 1.6.3 we released some settings for people to tweak Log Retention in Application Settings ( http://googleappengine.blogspot.com/2012/02/app-engine-163-released.html). Hi Ikai, definitely noticed. Excellent! Also, fairly modest pricing. Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/gnDObs8D9SMJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Do maintentance periods really have to be during core working hours?
Although I'm overall happy with GAE, and I know that the maintenance periods only affect M/S. And I don't mind the occasional one either. But I find it increasingly frustrating that they always have to be scheduled in the middle of the week, and during a time that most of our US clients would still call core working hours. I am now pasting the third downtime notice in a month onto our application's dashboard. Yeah I know that one maintenance period didn't actually happen, but it was cancelled too late, all our clients had already gotten our downtime warning, which is almost as bad. I am wondering why these maintenance periods cannot be done on the weekend, or at least later during the day? How about 6pm to 8pm PST for instance? That's still late at night in Europe, it's just about as annoying for APAC, but at least it's a lot better for US users. And it's not *that* late so that East Coast Googlers have to get up in the dead of the night either. Or am mistaken and the typical usage pattern is after hours, because nobody except for us dared to use GAE for an enterprise application? :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bK7kFbD_8E4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Application down, no graceful degradation despite checking the capabilities API
Hi there, during previous maintenance periods our application degraded gracefully, as we do check the Capabilities API before each request, and don't write anything to the datastore if the Capabilities API tells us that it's not available. This time around though I just get a Google Exception, and the stacktrace doesn't look like our code is being called at all. Are we doing something wrong, any suggestions how to avoid this next time? Or is it a Google problem? Uncaught exception from servlet com.google.apphosting.api.ApiProxy$CapabilityDisabledException: The API call datastore_v3.Put() is temporarily unavailable: Datastore writes are temporarily unavailable. Please see http://code.google.com/status/appengine for more information. at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:746) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455) at com.google.tracing.TraceContext.runInContext(TraceContext.java:695) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.rpcFinished(RpcStub.java:788) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.success(RpcStub.java:773) at com.google.net.rpc3.impl.client.RpcClientInternalContext.runCallbacks(RpcClientInternalContext.java:888) at com.google.net.rpc3.impl.client.RpcClientInternalContext.finishRpcAndNotifyApp(RpcClientInternalContext.java:793) at com.google.net.rpc3.impl.client.RpcNetChannel.afterFinishingActiveRpc(RpcNetChannel.java:1049) at com.google.net.rpc3.impl.client.RpcNetChannel.finishRpc(RpcNetChannel.java:895) at com.google.net.rpc3.impl.client.RpcNetChannel.handleResponse(RpcNetChannel.java:2248) at com.google.net.rpc3.impl.client.RpcNetChannel.messageReceived(RpcNetChannel.java:2060) at com.google.net.rpc3.impl.client.RpcNetChannel.access$2000(RpcNetChannel.java:147) at com.google.net.rpc3.impl.client.RpcNetChannel$TransportCallback.receivedMessage(RpcNetChannel.java:3097) at com.google.net.rpc3.impl.client.RpcChannelTransportData$TransportCallback.receivedMessage(RpcChannelTransportData.java:602) at com.google.net.rpc3.impl.wire.RpcBaseTransport.receivedMessage(RpcBaseTransport.java:420) at com.google.net.rpc3.impl.wire.RpcClientTcpTransport.parseOneMessage(RpcClientTcpTransport.java:773) at com.google.net.rpc3.impl.wire.RpcClientTcpTransport.parseMessages(RpcClientTcpTransport.java:651) at com.google.net.rpc3.impl.wire.RpcClientTcpTransport.access$100(RpcClientTcpTransport.java:38) at com.google.net.rpc3.impl.wire.RpcClientTcpTransport$1.dataReceived(RpcClientTcpTransport.java:293) at com.google.net.async3.SocketConnection.handleNetworkReadEvent(SocketConnection.java:843) at com.google.net.async3.SocketConnection.access$400(SocketConnection.java:40) at com.google.net.async3.SocketConnection$NetworkReadHandlerImpl.run(SocketConnection.java:872) at com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260) at com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121) at com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:576) at com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:997) at com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:878) at com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:134) at com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1833) Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/fY3zdxODD8sJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Application down, no graceful degradation despite checking the capabilities API
Ok, turns out this is only happening when you already have a session. The capability check does work if you don't have a session. So it's not as bad, but I'm still puzzled. Any ideas? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Y0onWpsO7XAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: What's going on with the high latency ?
Without doubt, plenty of people will now advise you to upgrade to the HRD for christ's sake, and then you'll read today's latest post about entity kinds gone missing during the migration, and then you'll be back at square 1. At least that's my situation right now, again. But yeah, your only choice is really to find a way to migrate to the HRD. On Feb 28, 8:31 pm, Alain de Raynal aderay...@gmail.com wrote: http://code.google.com/status/appengine/detail/datastore/2012/02/28#a... my apps have been unreachable for the past 30 minutes... please help -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: What is a good pattern to display and sort long lists? Work around the 1MB memcache limit? Or something else?
Thanks so much Sameer and Robert, the zipping really blows me away! Since Snappy and Ning Compress don't work on GAE, I resorted to plain DeflaterOutputStream, and it turns out it's fast and efficient as well. It's shrinking some 1.4 Megabyte of list data to 150K within 200ms (on F4, admittedly), which is outstanding (remember, this only happens on admin pages where the admin doesn't mind waiting a second or two). It's so fast I'm even considering to use it on the session itself (we use Wicket, which stores page-representations in the session, which means we may reach the 1M limit if we don't evict old pages quickly). Brilliant hint. I'll probably still need to implement the bucketing that Jeff mentions, but not for the next 6 months. Happy Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] What is a good pattern to display and sort long lists? Work around the 1MB memcache limit? Or something else?
Hi all, say I have a few hundred users in one organisation, and I want to display them in a list. I want to be able to sort them by user name, or manager, or some other criteria. Right now I load them from the database all in one go, sort them, store them in memcache, and then during pagination, I load them from that memcache each time the user clicks forth or back. When the user changes the ordering, I load the list from memcache, re-order, and store it back. This works fine for 1000 users (each of them has some 15 fields). Now if I had a few thousand users, I'd have to make sure the user objects are not too big, or the list will be more than 1 megabyte. I can denormalise into a slimmer user object, which makes database access faster too. But what if I want to display 10.000 or more users -- Each user object can't have more than 100 bytes then, because of the memcache limit of 1MB. But that's not even enough for even the most basic fields. Is the solution to split the list into several memcache blocks? If so, are there any libraries or tools that support this? Or am I barking up the wrong tree, should I deal with this in an entirely different way? I did some googling, but the main questions (and answer) was always around speeding up the initial database call. This is working fine for us, and it doesn't matter if it takes a few seconds either. Feedback from people who have dealt with similar issues would be greatly appreciated! :) Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: What is a good pattern to display and sort long lists? Work around the 1MB memcache limit? Or something else?
Thanks, hadn't thought about the zipping yet. Any recommendations? The query cursor would work fine if there was no sorting involved - but since this is not the only list to be sorted, and since each sort requires a separate index, the number of indexes would get out of control quickly. We're already at some 100 indexes, and I'd rather not create new ones unless really needed. Cheers, Per On Feb 7, 11:36 am, Sameer Lodha sams...@gmail.com wrote: One tip is to Zip data before you put it into memcache. That way you can easily store 4x to 10x more information. But again there would be limit to how much you can save. Beyond that, I think the better approach would be to use Query Cursor to get just the required data (to be displayed) at runtime. Regards, Sam On Tue, Feb 7, 2012 at 3:39 PM, Per per.fragem...@gmail.com wrote: Hi all, say I have a few hundred users in one organisation, and I want to display them in a list. I want to be able to sort them by user name, or manager, or some other criteria. Right now I load them from the database all in one go, sort them, store them in memcache, and then during pagination, I load them from that memcache each time the user clicks forth or back. When the user changes the ordering, I load the list from memcache, re-order, and store it back. This works fine for 1000 users (each of them has some 15 fields). Now if I had a few thousand users, I'd have to make sure the user objects are not too big, or the list will be more than 1 megabyte. I can denormalise into a slimmer user object, which makes database access faster too. But what if I want to display 10.000 or more users -- Each user object can't have more than 100 bytes then, because of the memcache limit of 1MB. But that's not even enough for even the most basic fields. Is the solution to split the list into several memcache blocks? If so, are there any libraries or tools that support this? Or am I barking up the wrong tree, should I deal with this in an entirely different way? I did some googling, but the main questions (and answer) was always around speeding up the initial database call. This is working fine for us, and it doesn't matter if it takes a few seconds either. Feedback from people who have dealt with similar issues would be greatly appreciated! :) Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Frontend Instance Class (Default F1? 128MB?) All my instances have been running around 256MB
I'm confused by these changes too. Although I'm happy to have choice (pay more for more speed sounds awesome) this leaves an odd aftertaste After all the communication issues around the recent pricing changes, shouldn't it be quite obvious that people will ask you about the details anyway? Wouldn't it help to spend a couple minutes upfront, explaining this change/new feature better? Surely you have spent many hours working on the implementation, why not share some additional insight? By default, all applications use the basic frontend instance setting of 128MB memory and 600MHz CPU. So, if F1 is what we've been using the past months, why don't you just say so. And if it happened to be F2, then people will find out eventually, and you should say to too. I can see a mysterious drop in performance of my application since the 22nd of November. I have not analysed it yet (it only occurred to me now, since I was busy), but it would help a lot if you'd take the guesswork out of these announcements -- because otherwise I am somewhat tempted to believe that we've been on something similar to F2 and will now have to pay more to achieve the same old performance. Please tell me it's not like this, and I'll shut up instantly :) Cheers, Per On Dec 14, 12:03 pm, Scott Murphy sc...@pixoto.com wrote: Hi, I am confused. Did the default used to be F2? All my instances run 128MB of ram. How is this possible? Does this mean that part of the memory is swapped to disk? If I choose a larger instance class say (F4), will my Java Heap size automatically change to allow up to 512MB heap? Or is this extra memory not usable by the app? 0..0 ms108112:44:46170.1 MBytes[image: Resident Icon]Resident0..0 ms 27701 day, 21:07:27190.0 MBytes[image: Resident Icon]Resident0.0171905.0 ms 14101 day, 5:35:31182.7 MBytes[image: Resident Icon]Resident0.4331114.4 ms 402102:49:43239.3 MBytes[image: Dynamic Icon]Dynamic0..0 ms109701:00:15203.4 MBytes[image: Dynamic Icon]Dynamic0..0 ms12500:12:01197.4 MBytes[image: Dynamic Icon]Dynamic0.2331295.7 ms48500:22:12192.7 MBytes[image: Dynamic Icon]Dynamic0.4671081.0 ms696734:08:09245.1 MBytes[image: Dynamic Icon] Dynamic0.3831486.6 ms367314:03:36243.4 MBytes[image: Dynamic Icon]Dynamic 0..0 ms641304:15:35243.5 MBytes[image: Dynamic Icon]Dynamic0.0503124.3 ms600:02:07149.5 MBytes[image: Dynamic Icon]Dynamic0.05030326.0 ms200:00:40136.2 MBytes[image: Dynamic Icon]Dynamic0.5671189.9 ms551503:33:30244.8 MBytes[image: Dynamic Icon]Dynamic -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: a happy developer
I agree. I still think some communication issues could have been handled better, but the technology is pretty awesome, and getting better all the time! Also, everything has been steadily growing faster for us, I really enjoy how responsive our application is these days. Keep up the amazing work! Cheers, Per On Nov 30, 10:59 pm, alex a...@cloudware.it wrote: I've seen many people bashing GAE dev team, especially lately. What I want to say here is, I've been using it for years now and it's pretty awesome what you guys have been doing and where you've gone today. Of course there are bugs and stuff you need to find workarounds for but it is nowhere near compared to the burden of managing a cluster of EC2 on AWS or pay a double to Heroku or RackSpace even if they're great platforms too. I'm using GAE whenever I can and wherever it fits but I often find myself stuck with companies that by all means want to keep their data on the servers behind their desks (almost literally) and that's where one truly understands what it means to design, maintain and keep up with any sorts of upgrades of a multitenant infrastructure. IT'S HARD. So, this is from a (very happy) developer on GAE, AWS, Heroku, RightScale and an admin of a custom designed in-house multitenant infrastructure to GAE team: you're doing an awesome job. Keep it up! alex. PS Brandon, please no self-promotional replies with 10k-a-day clients and porn sites. thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Short-notice reschedule of maintenance mode
Hi there, our application is unfortunately unusable during maintenance mode, so we add large banners informing our clients about upcoming maintenance, a few days ahead. We don't have the capacity to upgrade to HRD at the moment (it still seems somewhat risky), but that's okay. It's not so great though that maintenance gets cancelled one hour ahead of time, and then rescheduled to next week, meaning we now have to add these banners again and look like we don't really know what we're doing. Maybe you could inform us a bit earlier than 1 hour ahead next time? Totally understood that this *can* happen, and I don't want to sound too negative either, just saying it's not ideal for us, and possibly to others too who rely on the datastore for most operations in the application. Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Billing history stopped 5 day ago
Same problem here. Two of my apps are stuck 5 days ago. But the one that I disabled the always-on option is showing a 2 day old billing, and that's actually already taking into account my change in the billing (new pricing==1 cent, but only because it's my QA environment that's pretty idle). Maybe you have to change the pricing a little bit to kick off the job for the billing history. Cheers, Per On Sep 7, 11:51 am, Alex Popescu a...@mypopescu.com wrote: In the light of the new billing details, I've made some changes to the Max Idle Instances and Min Pending Latency (as per the optimization documentation) just to see how these would reflect in my bill. Anyways my billing history has stopped 5 days ago on Sept. 2nd. I'm really confused on how to proceed right now. I know that I have to get ready for the moment the new pricing becomes effective, but if I don't have the tools to estimate my costs and understand what optimizations I can do, then I don't seem to be left with much to do. Could anyone clarify why billing history has stopped? Also, how soon changes to the above configurations would be reflected in the billing history? tia, A:// PS: Application ID: twimblr. I do have other applications each with its own different billing status (most recent being from Sept.5th, which is OKish). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: How do I set maximum total instances to 1 ? OR How do I run my app for free for 24 hrs a day ?
I changed my QA environment down from the always-on option, and set max-idle==1, and I set the pending latency to 300ms. The instance was up for 24h (restarted about 11h ago the last time): It's just a QA box, but the cron jobs run and seem to keep it alive well. E.g. no latency either. My charge under the new pricing regime is 1ct for the day. Even if there were a bit more load, I expect additional instances would get spun up and down again pretty quickly, and not cost me (or you) much. All good! Cheers, Per On Sep 7, 11:12 am, Gerald Tan woefulwab...@gmail.com wrote: I believe even if you have additional instances spawned for you, you are only charged for 1 instance (which should be your max idle instance setting) + active instances, which should be a very small number. Now you will see either one of two situations: 1) You have enough time when there are 0 instances up to make up for the active instance numbers, in which case your app should have no problem serving requests all day 2) You have an instance up all the time, in which case your app will be automatically shutdown when you run out of quota near the end of the billing day, probably for around 30mins to an hour depending on your active instance time, until the daily quota resets. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Will you prohibit to keep the free instance up and running permanently? Or is it a best practice now?
Hi, I just disabled the always on option for my QA application, wanting to experiment with the new pricing model. Instead, I set the max-idle to 1, and I defined a 300ms latency in the application settings. The QA box is rarely used, but since the Cron-Jobs keep running every 5 minutes to clean up stuff, there was always some base load. I checked out the stats, and it turns out there was always one instance up in the past 24 hours, and the currently active instance is 11h old. That's perfect for me: the less instance restarts we have, the faster the app will react (since startup takes some 5 to 10 seconds). Overall, this QA box cost me a mere 1ct under the new pricing. Great news! Is this too good to be true? When App Engine started out, new instances got killed after 60seconds, and it was officially prohibited to ping instances to keep it alive. Will that happen again? Or is it fine to consume the free 1 instance permanently? Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Will you prohibit to keep the free instance up and running permanently? Or is it a best practice now?
It's M/S, but thanks for the quick answer. Very appreciated! Cheers, Per On Sep 7, 6:25 pm, Jon McAlister jon...@google.com wrote: Hi Per, I'm going to make a wild guess here and assume you are using the High Replication Datastore. Apps which have selected this are presently running on a set of machines that have the properties you describe. Rather than having idle instance processes evicted after some number of minutes, they hang around for hours. This is because of the present ratio of cpu to memory pressure on those machines; essentially, it's a by-product of the present supply and demand for that subset of apps and machines. It will certainly change but probably not for several months yet. Eventually it will have the same behavioral patterns as you see for M/S-D apps. If you'd like to keep one instance up always though, when the billing rollout is complete you'll be able to set min-idle-instances=1. On Wed, Sep 7, 2011 at 4:56 AM, Per per.fragem...@gmail.com wrote: Hi, I just disabled the always on option for my QA application, wanting to experiment with the new pricing model. Instead, I set the max-idle to 1, and I defined a 300ms latency in the application settings. The QA box is rarely used, but since the Cron-Jobs keep running every 5 minutes to clean up stuff, there was always some base load. I checked out the stats, and it turns out there was always one instance up in the past 24 hours, and the currently active instance is 11h old. That's perfect for me: the less instance restarts we have, the faster the app will react (since startup takes some 5 to 10 seconds). Overall, this QA box cost me a mere 1ct under the new pricing. Great news! Is this too good to be true? When App Engine started out, new instances got killed after 60seconds, and it was officially prohibited to ping instances to keep it alive. Will that happen again? Or is it fine to consume the free 1 instance permanently? Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Google, What are you doing? (suggestion)
Full agreement here. For us it's not the pricing increase that makes us feel so uneasy, a price hike was to be expected. It's the seeming lack of experience with enterprise products and how to communicate upfront that makes me afraid. I doubt that other Google products, e.g. Adwords, would make such drastic changes at such short notice. At my previous employer, one of the five Company Values was Don't f*ck the customer. It applied to every single employee, no matter if in engineering, sales or marketing, no matter if you're the VP or the intern, and it really kept everyone honest. People make mistakes (we sure did), you can't please everyone (we certainly didn't) but at least you have to try real hard. I'd like to see more of that from the product management team here. I'm on Java, but the Python 2.7 discussion looks like a big failure in management to me. This does not seem like the best way to treat your customers. When you said second half of 2011, why didn't you pick mid October or mid November instead if you didn't have solutions to these problems yet. Could have saved you a lot of frustration from all I can see. Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Our new performance tuning best practices document
Hi all, August was a pretty tough month for us, since the 1.5.2 release and the datastore issues really exposed some of our inefficient code. But we've learned and improved, and our app is faster than ever. Check out our new best practice document: http://www.small-improvements.com/app-engine-performance-tuning Nothing earth-shattering, but upfront knowledge would have saved us a lot of work. Happy to add more hints if you have any. Thanks again to Simon Knott for suggesting to check out JARing classes to improve speed. You're our hero! Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] How to deal with the current Memcache outages?
Hi, i spent quite some time optimising datastore access since it was so unreliabale in the past weeks. Now using memcache intensely throughout the application. Only to find out that memcache has been flaky for some 4 hours now, with error rates between 10% and 70%. What's the recommended strategy to dealing with a situation like this? Since the memcache isn't entirely down, I don't want to take the app down just just yet. But it would be great if I could at least show an automated warning sign (this happened over night). The Capabilities API only has enabled or disabled -- it would be nice if you could query the error rate just as it shows up on the system status. Or do some other tools or libs exist (I'm on Java..)? I'm open to other ideas, how does everyone else deal with this outages and error rates? Any other suggestions would be greatly appreciated. Cheers, Per PS: While I'm at it, let me also express my frustration at the system status page. Only 10 minutes ago there was a 70% error rate spike, but the overall status is Green again. I wouldn't be surprised if tomorrow the system status is all Green for today as well, despite these serious problems. I understand that errors occur (and that you're working hard at fixing them), but it leaves an odd aftertaste if these errors get glossed over after a while. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: How to deal with the current Memcache outages?
Actually, forget it, my mistake. I think I was simply not catching enough exceptions while putting stuff into the memcache. Obviously, you should never reply on the memcache being there at all, and just go to the datastore instead. Must drink more coffee before posting. Sorry about that! Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Classloading still way too slow. Any recommended practices? (yes, I am using warmup requests already)
Actually, you're right, never mind the behind the scenes business. But if your internal tests confirm that JARs are faster to load, please add a note to the docs, this would have saved me a lot of time experimenting with various workarounds. Cheers, Per On Aug 20, 12:21 am, Jose Montes de Oca jfmontesde...@google.com wrote: Hi Peter, I think we don't make this for you behind the scenes because there is an specific use case where this would not work: For files that you want to override that are already in a jar file, this files should remain in the WEB-INF/classes folder. one alternative would be to modified the jar you are overriding and replace the clase with your own. Best, Jose Montes de Oca -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Classloading still way too slow. Any recommended practices? (yes, I am using warmup requests already)
Simon, you are my hero!! I had been deploying the classes and the html raw, e.g. from web-inf/ classes, but it turns out it's 20 to 100 times slower this way!! I packaged everything up in one neat jar, and now my application flies again. Even deployment is faster than before. I guess it wasn't such a problem prior to 1.5.2 because the 3 resident instances would warm up rather quickly, but the new post 1.5.2 scheme spins up new instances all the time, so you get this problem all the time. Hello Googlers, in case your reading, IMO this belongs in the manual. The difference between the local devserver and the actual production system is really massive in this respect, and it's been haunting me for a while now. A simple bullet-point please deploy jars only would be nice. Cheers, Per On Aug 16, 5:30 pm, Simon Knott knott.si...@gmail.com wrote: Have you tried deploying the classes as a JAR? I only ask, because I seem to remember someone posting recently who was complaining that file reading on GAE appeared to be slow - maybe it's not the classloading which is actually taking the time, but reading the class files from disk (which seems a bit ridiculous really). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Classloading still way too slow. Any recommended practices? (yes, I am using warmup requests already)
The classes above took anything from 100ms to 500ms under normal conditions, and could take up to several seconds when things got really pear-shaped. Now that everything is jarred up, the classloading happens in 20ms to 30ms on average, pretty much as it does on the local devserver. I am sure it will also be slower under big stress, but still, that's a substantial improvement already. Adding it to the deployment tool would be great IMO. I was worried that jarring everything up would make deployment slow (always deploying a big 1MB jar instead of a couple classfiles only) but to my surprise it's a lot faster this way. Deployments are fun again! :) On Aug 19, 10:38 pm, Ikai Lan (Google) ika...@google.com wrote: We suspected this was the case but this is good information. Do you have specific numbers? If not, that's fine. I'll do the measurements myself. If it's significant, it could even be worth building into the deployment tool itself. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Aug 19, 2011 at 1:35 PM, Simon Knott knott.si...@gmail.com wrote: Wow, glad it worked so well! Time to go and Jar my classes up as well... -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-WlJUmVddaYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Classloading still way too slow. Any recommended practices? (yes, I am using warmup requests already)
After using appstats for weeks now, reducing queries and offloading a lot to memcache, our application is pretty fast despite the abysmal performance of the datastore. Once it's been serving for a few minutes, that is. But: Very many pages of our application take several seconds (4 to 6s!) to load for the first time, even despite warmup requests and several other pages having been executing on the same instance already. Appstats clearly shows that the API calls are not to blame, and running a local profiler doesn't indicate any problems either. I added some more classloading (Class.forName()) to the warmup requests, and it turns out classloading is frequently the cause. Sometimes it takes 100ms to load 20 large classes (and their anonymous inner classes). But sometimes it takes 8 seconds! Occasionally it takes more than 15s, and the warmup request dies! I'm now considering to let the warmup request call additional webpages, so that these request warm up the classes instead. However, given that these new http requests might end up on *other* instances, it's not really a great option. It will increase probablilty of classes being loaded, but it's not a guarantee. So my question: Is there a recommended way of loading classes before the user tries to get them? Is there a way to chain warmup requests for instance, e.g. a guarantee that the next request will be on the same instance? That would help a lot. Cheers, Per PS: As much as I love App Engine and the power it brings, I find it extremely annoying to have to fiddle with unreliable classloading performance, and queries that take 300ms on average. Maybe I was overly ambitious in coding a complex application on App Engine, maybe App engine's goal is really just outsourcing large volume processing (and not rich user-centered web applications), but the warning signs are clearly missing then. IMO, speed and reliability are much more important than anything else, and as much as I'd love to see SSL support for custom domains, fulltext search, etc I'd really prefer these core problems solved first. Just my 2 cents.. :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Classloading still way too slow. Any recommended practices? (yes, I am using warmup requests already)
Hi Simon,
these classes are all our own and they come from web-inf classes. I
double checked that it actually doesn't load anything else, using the -
verbose option locally.
Here's the code snippet showing just the first batch. This can take up
to 8 seconds to load remotely (but also be as fast as 120ms), while it
takes only 20ms to 50ms locally...
try {
Class.forName(com.praisemanager.web.general.panels.comment.view.CommentPanel);
Class.forName(com.praisemanager.web.general.panels.comment.view.CommentListPanel);
Class.forName(com.praisemanager.web.general.panels.comment.view.ViewRepliesAjaxLink);
Class.forName(com.praisemanager.web.general.panels.comment.view.CommentRestrictionsPanel);
Class.forName(com.praisemanager.web.general.panels.comment.ReplyCommentPanel);
Class.forName(com.praisemanager.web.FeedbackRequestPage);
Class.forName(com.praisemanager.web.EditCommentPage);
Class.forName(com.praisemanager.web.general.panels.comment.view.UnveilAnonymousAjaxLink);
Class.forName(com.praisemanager.web.general.panels.characters.PersonTab);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
Cheers,
Per
On Aug 16, 2:27 pm, Simon Knott knott.si...@gmail.com wrote:
Are these classes your own, or from third-party libraries?
Do you deploy your own classes inside a JAR, or just within WEB-INF classes?
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE Starts dynamic instances for no reason... causing unnecessary cold start delays
Hi Jon and Ikai, I've been seeing crazy amounts of warmup requests all week. For instance, while one request was being served, 5 to 7 instances started up *simultaneously*, although there was no other load. Now, the *next* request was almost guaranteed to hit one of the new instances, which are warmed up, but not 100% hot... and thus takes a long time to load... which spins up a few *new* instances. The app used to have a startup time of 5 to 6 seconds, and it used to respond within 400ms to 800ms, with only a few heavy pages being a lot slower. Not a lightning fast app, but it worked really well before 1.5.2. Now response times have dropped by factor 3 to 5, which takes even normal requests into a really awkward 2 to 3 second range. This has been plaguing the application all week, and I had no clue what was going on. I figured out that I needed to limit the the idle VMs to 3 to, so I'd at least avoid having 3-7 new idle VMs per request and always hitting luke-warm instances. That's better, but there is still an awful lot of spinning up and spinning down going on, and every 3rd or 4th request or so still goes to a warm-but-not-hot VM, taking 3 seconds or so. I don't really care about the billing at the moment, but it is really embarrassing me in front of our customers to have such a slow application all of a sudden. I've been working the weekend on some general profiling to mitigate, but please make this stop, or give us more control so we can determine how soon a new instance is created (in my case, not so soon...) Does this also change the recommended approach to warmup-request in general? So far, I had only warmed the most crucial parts of the app, since the doc states that requests may as well hit a cold instance, and you wouldn't want to wait for 20s until really the entire app was piping hot. Instead of doing an extreme 20 seconds warm-up, the instances would gradually hotten up with each new request coming in. But now instances go up and down a lot, so each one is barely luke- warm, and thus slower on average. I'd change the warmup sequence to 20s of heavy exercises if it helped, but that would spin up even more instances... oh dear. :) Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Thursday 7/14 Downtime
Hi Nick, my application has just been stuck and throwing exceptions for about an hour. I understand that you need to schedule maintenance periods at short notice, but as far as I can tell, this was announced 2 days ahead in the forum, but not posted into the downtime calendar like I'd expect. I have 5 notification (email AND sms) on that calendar because I really need to communicate downtime to our customers upfront. Did I miss something obvious? Even now I can find the downtime announcements from April and May in that calendar, but not today's downtime. I just realised there's a new sentence in your recent announcement: Most importantly, sign up to receive notices from the App Engine To my knowledge that was not mentioned in the previous announcements, the previous announcements only directed us to the calendar. I get the feeling this downtime was not handled as well as it should have been. But, I'm happy to learn, please let me know what the best way is to stay up to date reliably. Cheers, Per On Jul 13, 7:14 am, Nick Johnson (Google) nick.john...@google.com wrote: Hi Ryan, On Wed, Jul 13, 2011 at 12:33 AM, Ryan Goldstein r...@moberg.com wrote: Regarding https://groups.google.com/d/msg/google-appengine-downtime-notify/C7d8... ... 1. I assume the subject stating July 15 is incorrect, and the real date is Thursday, July 14, as in the body. That's correct. 2. Will this downtime affect apps that don't use the datastore (i.e. only use the Google Spreadsheets API to read/write data)? Only insofar as it will also result in Memcache being cleared. Other than that, your app will continue to serve as normal. -Nick Johnson Thanks, -Ryan -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Sdy_q1wx20cJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Error when deploying to dev server
I am seeing the same error after upgrading from 1.3.0 to 1.3.1. Strange. Did you find a solution? Cheers, Per -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Error: Server Error from Android phone, no log
Thanks for the reply. Currently I am unable to reproduce the error. I will investigare it further if it reappears and post here. Cheers, Per Sandström On Nov 24, 1:09 am, Ikai L (Google) ika...@google.com wrote: I'd like to try to replicate this error, but I am getting a 401 when I attempt to POST arbitrary data to that url: curl -d a=1http://pg-sandstrom-guestbook.appspot.com/checklists What you should try to do is replicate the error using command line tools or by monitoring the low level request. If this URL works in a browser but not from an Android application, you should try to trace the request and find the difference between the requests. 2009/11/23 Per Sandström pg.sandst...@gmail.com Hi! This is the exact message I retrieve when my servlet is the way I want it to be: htmlhead meta http-equiv=content-type content=text/html;charset=utf-8 title500 Server Error/title /head body text=#00 bgcolor=#ff h1Error: Server Error/h1 h2The server encountered an error and could not complete your request.pIf the problem persists, please A HREF=http:// code.google.com/appengine/community.htmlreport/A your problem and mention this error message and the query that caused it./h2 h2/h2 /body/html But still, there are no entries in my admin panel-log under errors. However, I just noticed that they appear under requests like this: 11-23 03:04AM 23.201 /checklists 500 41499ms 175cpu_ms 175api_cpu_ms 0kb Apache-HttpClient/UNAVAILABLE (java 1.4),gzip(gfe) See details 81.224.65.211 - - [23/Nov/2009:03:05:04 -0800] POST /checklists HTTP/1.1 500 0 - Apache-HttpClient/UNAVAILABLE (java 1.4),gzip(gfe) They take up 4ms CPU! I tried decreasing the amount of code that I executed to find the problem, but it only helped when I didnt access the permanent storage. So I tried deleting everything that I had stored, and after that it worked fine even after putting in new entries. However, my datastore only held ~40 entries. There are absolutely no reason for this kind of delay on my side. I have no loops in loops or potentially infinite while-loops or anything like that. I really do believe this is an error on the GAE-side. Also, I have now tried to once again fill the datastore with ~40 entries, and the CPU-time never goes over 250ms. This feels somewhat scary, since we are planning on releasing our product quite soon, and then the datastore will be much larger. On Nov 12, 8:28 pm, Ikai L (Google) ika...@google.com wrote: What is the exact error? Also, can you host a static asset and try to retrieve it with the Android application? I want to rule out that it's a connectivity issue. 2009/11/11 Per Sandström pg.sandst...@gmail.com I am doing an app that is supposed to be used from my android phone. Some communication with my GAE-app works, but when I try to send a post-message tohttp://pg-sandstrom-guestbook.appspot.com/checklistsI get a server error. This only holds true from my android phones. Nothing is shown in the Admin panel log, so I dont think the error is due to my GAE-code. Any help would be greatly appriciated. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com google-appengine%2bunsubscr...@googlegroups.comgoogle-appengine%252bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~--- -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=. -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Error: Server Error from Android phone, no log
Hi! This is the exact message I retrieve when my servlet is the way I want it to be: htmlhead meta http-equiv=content-type content=text/html;charset=utf-8 title500 Server Error/title /head body text=#00 bgcolor=#ff h1Error: Server Error/h1 h2The server encountered an error and could not complete your request.pIf the problem persists, please A HREF=http:// code.google.com/appengine/community.htmlreport/A your problem and mention this error message and the query that caused it./h2 h2/h2 /body/html But still, there are no entries in my admin panel-log under errors. However, I just noticed that they appear under requests like this: 11-23 03:04AM 23.201 /checklists 500 41499ms 175cpu_ms 175api_cpu_ms 0kb Apache-HttpClient/UNAVAILABLE (java 1.4),gzip(gfe) See details 81.224.65.211 - - [23/Nov/2009:03:05:04 -0800] POST /checklists HTTP/1.1 500 0 - Apache-HttpClient/UNAVAILABLE (java 1.4),gzip(gfe) They take up 4ms CPU! I tried decreasing the amount of code that I executed to find the problem, but it only helped when I didnt access the permanent storage. So I tried deleting everything that I had stored, and after that it worked fine even after putting in new entries. However, my datastore only held ~40 entries. There are absolutely no reason for this kind of delay on my side. I have no loops in loops or potentially infinite while-loops or anything like that. I really do believe this is an error on the GAE-side. Also, I have now tried to once again fill the datastore with ~40 entries, and the CPU-time never goes over 250ms. This feels somewhat scary, since we are planning on releasing our product quite soon, and then the datastore will be much larger. On Nov 12, 8:28 pm, Ikai L (Google) ika...@google.com wrote: What is the exact error? Also, can you host a static asset and try to retrieve it with the Android application? I want to rule out that it's a connectivity issue. 2009/11/11 Per Sandström pg.sandst...@gmail.com I am doing an app that is supposed to be used from my android phone. Some communication with my GAE-app works, but when I try to send a post-message tohttp://pg-sandstrom-guestbook.appspot.com/checklistsI get a server error. This only holds true from my android phones. Nothing is shown in the Admin panel log, so I dont think the error is due to my GAE-code. Any help would be greatly appriciated. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~--- -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=.
[google-appengine] Error: Server Error from Android phone, no log
I am doing an app that is supposed to be used from my android phone. Some communication with my GAE-app works, but when I try to send a post-message to http://pg-sandstrom-guestbook.appspot.com/checklists I get a server error. This only holds true from my android phones. Nothing is shown in the Admin panel log, so I dont think the error is due to my GAE-code. Any help would be greatly appriciated. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] one servlet stopped working
So I just noticed that one of my servlets has stopped working. I get att 500 error message. In the admin console, instead of a crash as I expected, I am getting this message below. What is it? 40 seconds and no CPU used. Is this temporary problems with google apps engine? It has been like this all evening. # 1. 10-12 09:55AM 03.698 /checklists 500 41115ms 120cpu_ms 120api_cpu_ms 0kb Apache-HttpClient/UNAVAILABLE (java 1.4),gzip(gfe) See details 83.248.35.150 - - [12/Oct/2009:09:55:44 -0700] POST /checklists HTTP/1.1 500 0 - Apache-HttpClient/UNAVAILABLE (java 1.4),gzip(gfe) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] DatastoreNeedIndexException, but index present and serving
Im trying to perform this query: Query query = pm.newQuery(ListStore.class); query.setFilter(user == userParam); query.setOrdering(listPosition ASC); query.declareParameters(long userParam); ListListStore listStoreLists = (ListListStore) query.execute (user); My index looks like this: ListStore listPosition ▲ , user ▲ Serving Still, I get a DatastoreNeedIndexException. Any suggestions? Full stacktrace following: /checklists com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found. at com.google.appengine.api.datastore.DatastoreApiHelper.translateError (DatastoreApiHelper.java:35) at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall (DatastoreApiHelper.java:56) at com.google.appengine.api.datastore.DatastoreServiceImpl $PreparedQueryImpl.runQuery(DatastoreServiceImpl.java:342) at com.google.appengine.api.datastore.DatastoreServiceImpl $PreparedQueryImpl.access$100(DatastoreServiceImpl.java:269) at com.google.appengine.api.datastore.DatastoreServiceImpl $PreparedQueryImpl$1.iterator(DatastoreServiceImpl.java:303) at org.datanucleus.store.appengine.query.RuntimeExceptionWrappingIterable.iterator (RuntimeExceptionWrappingIterable.java:42) at org.datanucleus.store.appengine.query.StreamingQueryResult.init (StreamingQueryResult.java:77) at org.datanucleus.store.appengine.query.DatastoreQuery.newStreamingQueryResultForEntities (DatastoreQuery.java:324) at org.datanucleus.store.appengine.query.DatastoreQuery.fulfillEntityQuery (DatastoreQuery.java:310) at org.datanucleus.store.appengine.query.DatastoreQuery.performExecute (DatastoreQuery.java:242) at org.datanucleus.store.appengine.query.JDOQLQuery.performExecute (JDOQLQuery.java:84) at org.datanucleus.store.query.Query.executeQuery(Query.java:1489) at org.datanucleus.store.query.Query.executeWithArray(Query.java: 1371) at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:243) at guestbook.CheckLists.doPost(CheckLists.java:88) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 487) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1093) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter (SaveSessionFilter.java:35) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter (TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle (ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle (SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle (SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle (ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 405) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle (AppVersionHandlerMap.java:237) at org.mortbay.jetty.handler.HandlerWrapper.handle (HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:313) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 506) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete (HttpConnection.java:830) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable (RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest (JettyServletEngineAdapter.java:139) at com.google.apphosting.runtime.JavaRuntime.handleRequest (JavaRuntime.java:235) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:4950) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:4948) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest (BlockingApplicationHandler.java:24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java: 359) at com.google.net.rpc.impl.Server$2.run(Server.java:823) at com.google.tracing.LocalTraceSpanRunnable.run (LocalTraceSpanRunnable.java:56) at com.google.tracing.LocalTraceSpanBuilder.internalContinueSpan (LocalTraceSpanBuilder.java:516) at com.google.net.rpc.impl.Server.startRpc(Server.java:778) at com.google.net.rpc.impl.Server.processRequest(Server.java:351) at com.google.net.rpc.impl.ServerConnection.messageReceived (ServerConnection.java:437)
[google-appengine] how to delete indexes with java SDK
The indexes required by my app was not automatically added. So I added them in the datastore-indexes-auto.xml and deployed my app. However this has only led to me having three different entities on the same table, each holding different indexes. What I want to do is to remove the two entities that has the wrong columns indexed. Right now my app isn't working, and I guess this comes from it using the wrong entity, which doesnt hold the necessary indexes. When running it I get: com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
