[google-appengine] Re: GAE Latency & Instance issues

[Trevor]

I simply followed the advice kindly offered in this thread and experimented 
with lowering the max concurrent requests. Our best settings ended up being 2 
max concurrent requests with 2 max idle instances. 

I think an important part is that before this tweaking, we separated our 
front-end and task modules (recently renamed services) such that their 
instances could be completely independent.

By taking those two steps, our front-end average latency fell from an average 
of 600ms to 180ms, and our daily instance costs went from $100 to $30. So 
performance improved while cutting costs by >60%. We're pretty happy with the 
results.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6aaae667-d62f-4b31-adf9-db47073d1902%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: GAE Latency & Instance issues

[Trevor]

Well, I have to say thank you very, very much. Thanks to your advice we 
have our lowest latency in 3 years! Sub 300ms average.  As expected though, 
we are now sitting on 21 billed f4 instances, which will potentially cost 
us in the order of 3x our current ($30-40 -> $100+), but we will tweak that 
from tomorrow onwards. Peak hour is about to hit so we are going to see if 
the system can keep sub-300ms at the current "automatic" setting for 
scaling. But yes, once again, thank you for solving in 5 minutes what I 
have been working on doing for 2 weeks (my tears are from joy and sadness 
all at once)






On Thursday, June 30, 2016 at 6:03:23 PM UTC+9, troberti wrote:
>
> Right, you should definitely test and see what the results are. My first 
> inclination was also to increase max_concurrent_requests, but because then 
> all those requests have increased latency, the actual QPS per instance 
> decreased! Lowering max_concurrent_requests decreased request latency, so 
> each instance could process more requests/second.
>
> We use F1 instances, because we do not need the additional memory, and our 
> requests perform mostly RPCs. In our testing, faster instance classes do 
> process requests faster, but also cost significantly more.  F1s provide the 
> best performance/cost ratio for us. This could be a Python thing, not sure. 
> Again, you should really test and figure out what is the best for your 
> application+runtime.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/286855e9-52ac-4ded-bed5-adfc3abebc8f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: GAE Latency & Instance issues

[Trevor Chinn]

Thanks for the advice! I hadn't considered setting it that low because we 
are on F4 instances and I would really, *really* hope that they could 
handle at least the default 8 concurrent requests. That being said, I will 
throw the front-end on those settings this evening and monitor until noon 
tomorrow. The only thing I worry about is the amount of instances 
increasing rapidly and blowing out our monthly costs. We have 10 PV per sec 
during low times, and up to 30/35 per sec during peak hours. That means to 
maintain our current costs (3-5 billed f4 instances) we would need a 
sub-300ms request-completion time during peak, if my muddled math is 
correct. I suppose if there are less requests going to each instance, we 
could drop down to a lower class, perhaps?




What instance class do you run, if I may ask?

On Thursday, June 30, 2016 at 4:53:53 PM UTC+9, troberti wrote:
>
> I recommend trying to set max_concurrent_requests to 2. As you said, 
> higher values only makes latency (much) worse. In our experience, a value 
> of 2 gets the best latency results without an increase in cost.
>
> We still have some of those pauses mid-request in a trace, and I really 
> would like to know where they come from (and get rid of them), but they 
> seem much shorter with a lower max_concurrent_requests value.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e22233db-bcfe-4a69-ba58-69b5b8cf0bce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] GAE Latency & Instance issues

[Trevor Chinn]

Hello ladies and gentlemen, I am here to hopefully draw on some collective 
knowledge about App Engine and its intricacies. 

For the last two weeks our (my company) site has been experiencing very odd 
latency issues, and having now tried about 7 different methods of solving 
it, we are left at exactly where we began: Rising costs with performance 
that is much lower than previously. 




Essentially what happens is that say 50-60% of our requests are served 
normally, however the remainder have these extremely long "pauses" in the 
middle of the trace which is basically during the "processing" phase of the 
backend handling (after the datastore & memcache data has been retrieved). 
Here is an example of a single page that in the space of an hour had wildly 
different loading times for users. The vast majority were the same thing, 
grab 3 things from memcache and spit out the html retrieved from memcache. 
That's it... 



And some individual traces to see what is happening












So essentially the troubleshooting steps we took to figure out what was 
going wrong. 

   - Checked all deployed code over the week preceding and following the 
   latency spike to ensure we hadn't let some truly horrendous, heavy code 
   slip through the review process. Everything deployed around that period was 
   rather light, backend/cms based updates, hardly anything touching 
   customer-facing requests. 
   - Appstats, obviously. On the development server (and even unloaded test 
   versions on the production server) such behavior is not seen. Didn't help. 
   - Reducing unnecessary requests (figure 1) - We noticed some of our 
   ajax-loaded content was creating 2-3 additional, separate requests per 
   user-page-load, and as such refactored the code to only call those things 
   when absolutely necessary, and eliminated one altogether. For the most 
   part, a page load now equals one request. This had no effect on the latency 
   spikes
   - Created a separate system that meant that our backend task-based 
   processing was cut down by 90%, and thus the instance average load was 
   reduced significantly. This had the opposite effect and average latency 
   actually climbed, I suspect because of the extensive memcache use with 
   large chunks of data (tracking what things should be updated by the backend 
   tasks)
   - Separated the front end and tasks-back-end into modules/services so 
   that frontend requests could have 100% instance attention. This had a small 
   effect, but the spikes are still regularly happening (as seen in the above 
   traces). 
   - Played with max_idle_instances  - This had a wonderful effect of 
   *halving* our daily instance costs, with almost no effect on latency. 
   When this is set to automatic, we get charged for a huge amount of unused 
   instances, it actually borders on ludicrous (figure 2) 
   - Played with max_concurrent_requests (8->16->10) which only served to 
   make the latency issues worse. 
   - Hours and hours pouring over logs, traces, dashboard graphs. 


* Figure 1 (Since the latency spike on June 5th, we have worked to reduce 
meaningless requests through API calls or task queuing) *



*Figure 2 (14:40 is when the auto-scaling setting was deployed)*



What I have noticed is when the CPU cycles spike, *so does the latency*. So 
it would lead in the direction that our requests are starved for CPU time, 
however now that we have deployed the instance auto scaling (and are paying 
for an average of around 8 instances vs 4-5 previously), it has not 
improved the latency, which confuses me considerably. 

If it were all requests that had slowed down, our code would clearly need 
optimization. If the rise in latency coincided with a change in our 
frontend processing, it would make sense, but there were only very light 
backend changes deployed within +/- 2 days of the first latency spike 

[google-appengine] Re: Elevated Error Levels: "Request was aborted after waiting too long to attempt to service your request."

[Trevor Chinn]

Hi Nick,

I did have a look at the issue tracker, however it wasn't absolutely 
certain that it was a platform issue, and thus the frantic search for 
recent posts relating to that issue. I will post there from now on when it 
is more likely to be on Google's end than ours. Thanks for your support. 
We'd love to subscribe to a support package to be able to get these things 
sorted out right away but the cost:benefit ratio ($150-400+p/m) just 
doesn't work for a small startup like ours (these kinds of issues only pop 
up once every 6 months or so). In any case, have a nice day.

Regards,

Trevor

On Thursday, February 4, 2016 at 8:51:16 AM UTC+9, Nick (Cloud Platform 
Support) wrote:
>
> Hey Trevor,
>
> Thanks for updating the thread. It's possible that you were affected by a 
> transient issue in production. In future, keep in mind that the best place 
> to post details of an issue which might be related to the platform and not 
> your own configuration / code would be the Public Issue Tracker 
> <https://code.google.com/p/googleappengine/issues/list>. This is a more 
> formal means of tracking issues and while we monitor both Google Groups and 
> Public Issue Trackers, we will invariably direct such important information 
> to be posted in a Public Issue Tracker thread.
>
> Regards,
>
> Nick
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9b58972f-49a1-4213-ac27-9aaf97f1c4b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Elevated Error Levels: "Request was aborted after waiting too long to attempt to service your request."

[Trevor Chinn]

Hi Nick,

Just as a follow up, the errors suddenly stopped at 12pm JST, so I suppose 
something was happening on Google's backend that was fixed around noon. 

<https://lh3.googleusercontent.com/-ngu6r0wRRuc/VrGh36S7fNI/NoE/drIHbdwpa6M/s1600/Screen%2BShot%2B2016-02-03%2Bat%2B15.42.45.png>

Thanks.

-Trevor 
On Tuesday, January 26, 2016 at 9:52:55 AM UTC+9, Nick (Cloud Platform 
Support) wrote:
>
> Hey Danny,
>
> Is this issue still occurring for you?
>
> Thanks,
>
> Nick
>
> On Thursday, January 21, 2016 at 2:10:15 PM UTC-5, Danny Leshem wrote:
>>
>> We are experiencing the same issue, right now.
>>
>> App ID = spice-prod
>>
>> On Thursday, January 21, 2016 at 8:20:30 PM UTC+2, David Fischer wrote:
>>>
>>> As of about 4PM PST yesterday, this is resolved.
>>>
>>> On Wednesday, January 20, 2016 at 2:20:04 PM UTC-8, David Fischer wrote:
>>>>
>>>> Starting at about 8AM PST today, I'm seeing elevated error levels on my 
>>>> app on Appengine. I see batches of requests where my code on Appengine is 
>>>> not even executed and I simply receive a generic server error (not my 
>>>> normal custom 500 page) that says "Error: Server Error - The server 
>>>> encountered an error and could not complete your request - Please try 
>>>> again 
>>>> in 30 seconds". When I view the logs, I see 500 errors taking 10-15 
>>>> seconds 
>>>> with 0 byte responses and the only log message is "Request was aborted 
>>>> after waiting too long to attempt to service your request.". Since I log 
>>>> quite verbosely and there's no logs I wrote in there, I can safely assume 
>>>> Appengine didn't even get to my code in its execution. The overall number 
>>>> of requests my app is getting is normal relative to past loads.
>>>>
>>>> Nothing has changed on my end and there were no deploys to my app or 
>>>> changes to my settings at or around that time.
>>>>
>>>> Is anybody else seeing these types of problems? Any advice?
>>>>
>>>> Relevant parts of my app.yaml:
>>>>
>>>>> version: 37
>>>>
>>>> runtime: python27
>>>>
>>>> api_version: 1
>>>>
>>>> threadsafe: false
>>>>
>>>>
>>>>> module: default
>>>>
>>>> instance_class: F4
>>>>> automatic_scaling:
>>>>>   max_idle_instances: 3
>>>>>   min_pending_latency: 5.0s
>>>>
>>>>
>>>> I figured this could be an issue with insufficient idle instances. I 
>>>> changed my automatic scaling preferences to have a min_idle_instances and 
>>>> that has not helped. 
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/852fbea4-ba84-4fa0-b480-d39eb3bf021b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Password failure

[trevor . price1]

 

I get the following error even though I use the correct password (repeated 
deploy 10 times, turned firewall off etc). I am using windows 8. Anyone 
have any ideas?

2013-01-04 11:46:01 Running command: "['C:\Python27\pythonw.exe', '-u', 
'C:\Program Files (x86)\Google\google_appengine\appcfg.py', '--no_cookies', 
u'--email=trevor.pri...@ntlworld.com', '--passin', 'update', 
'C:\Data\DOCS\Python\hello']"
11:46 AM Host: appengine.google.com
11:46 AM Application: cliveprice11; version: 1
11:46 AM 
Starting update of app: cliveprice11, version: 1
11:46 AM Getting current resource limits.
Password for trevor.pri...@ntlworld.com: Invalid username or password.
2013-01-04 11:46:08,801 ERROR appcfg.py:2203 An error occurred processing 
file '': HTTP Error 401: Unauthorized. Aborting. 
Error 401: --- begin server output ---
Must authenticate first.
--- end server output ---
2013-01-04 11:46:08 (Process exited with code 1)

You can close this window now.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/ZynsyGf6f9oJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] App deployment stuck and then failed with "version not ready"

[Trevor Reid]

Hi,

Did you have any luck with the sample app?

I'm currently having the same problem on multiple apps. I've had one 
successful deploy in the last 24 hours. 

Currently running GAE SDK 1.7.1

The problem apps are running python 2.5 with the deprecated master/slave 
datastore.

Any help would be much appreciated.

thanks,

trevor



On Thursday, September 13, 2012 11:47:38 AM UTC-7, gks wrote:
>
> Thanks for the confirmation on the deprecated API. It makes my migration 
> work easier. :)
>
> For the deployment issue, it becomes intermittent for me now. For about 
> 30% time, I am able to finish the deployment. I will try to create a new 
> environment and try to deploy a sample app there.
>
> On Thursday, September 13, 2012 11:35:21 AM UTC-7, Alfred Fuller wrote:
>>
>>
>>
>> On Thu, Sep 13, 2012 at 9:19 AM, gks  wrote:
>>
>>> With the introduction of OR query, GAE SDK 1.7+ marks Query.setFilter as 
>>> deprecated so that upgrading to 1.7 will lead to a lot of warnings in my 
>>> project. Is backward compatibility guaranteed in 1.7.1 with those warning?
>>
>>
>> The warnings are safe to ignore. The deprecated functions will never be 
>> removed (at least not from the runtime). They are marked deprecated mainly 
>> to clarify the interface (i.e. there should only be one API to do something 
>> and it should be clear what that API is).
>>  
>>
>>>
>>> In addition, can you confirm the deployment issue I see is a known issue 
>>> for GAE and everyone is forced to upgrade to 1.7.1 to work around it?
>>>
>>>
>> This seems strange to me. Can you deploy one of the sample/demo java apps 
>> with the 1.6.6 SDK?
>>  
>>
>>> Thanks!
>>>
>>>
>>> On Thursday, September 13, 2012 4:16:50 AM UTC-7, Shilendra Sharma wrote:
>>>
>>>> Yeah !  You upgrade your GAE SDK 1.7.1 then deploy. its not take the 
>>>> time to upgrade only take 20-30 min for upgrading process.
>>>>
>>>> Regards & Thanks
>>>> Shilendra Sharma
>>>> +919891343808
>>>> shilen...@googlemail.com
>>>>
>>>>
>>>>
>>>> On Thu, Sep 13, 2012 at 4:13 PM, gks  wrote:
>>>>
>>>>> Thanks for the response. I am using GAE SDK 1.6.6 right now. We are in 
>>>>> the final release stage and cannot take such big change. Do I have to 
>>>>> upgrade to 1.7.1 in order to deploy?
>>>>>
>>>>>
>>>>> On Thursday, September 13, 2012 2:00:24 AM UTC-7, Shilendra Sharma 
>>>>> wrote:
>>>>>
>>>>>> Hi, 
>>>>>>
>>>>>> you check your GAE SDK 1.7.1 version you update that then deploy 
>>>>>> again 
>>>>>>
>>>>>>
>>>>>> Regards & Thanks
>>>>>> Shilendra Sharma
>>>>>> +919891343808
>>>>>> shilen...@googlemail.com
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 13, 2012 at 8:35 AM, gks  wrote:
>>>>>>
>>>>>>>  I am unable to deploy to my apps in the past 20 hrs. All 
>>>>>>> deployments stuck at the stage of "Verifying availability of frontend" 
>>>>>>> and 
>>>>>>> eventually failed with "Version not ready".
>>>>>>>
>>>>>>> My AppIDs are aos-prod-1 & aos-prod-2.
>>>>>>>
>>>>>>> Any help will be greatly appreciated!
>>>>>>>
>>>>>>> -- 
>>>>>>> You received this message because you are subscribed to the Google 
>>>>>>> Groups "Google App Engine" group.
>>>>>>> To view this discussion on the web visit 
>>>>>>> https://groups.google.com/d/**ms**g/google-appengine/-/**7Hs6RaPSM**
>>>>>>> 74J<https://groups.google.com/d/msg/google-appengine/-/7Hs6RaPSM74J>
>>>>>>> .
>>>>>>>  To post to this group, send email to google-a...@googlegroups.**com
>>>>>>> .
>>>>>>> To unsubscribe from this group, send email to google-appengi...@**
>>>>>>> googlegroups**.com.
>>>>>>>
>>>>>>> For more options, visit this group at http://groups.google.com/**
>>>>>>> group**/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>
>>

[google-appengine] App Engine using jQuery Mobile setup for Manifest

[Trevor]

I have a web app  and now created a jQuery Mobile app to offer the same and 
it works

I want to make the jQuery Mobile app "manifest" ie work offline but it will 
not work

1) I have added the line to the html template
 

2) have down loaded the jquery files and place them and the images in a 
folder

3) I created the manifest file offline.manifest

4) I have added the lines to my app.yaml  file

- url: /mobile/(.*\.manifest)
  static_files: mobile/\1
  mime_type: text/cache-manifest
  upload: mobile/(.*\.manifest)

  
But when I try the Android 2.2 browser in Aeroplane mode it says it needs to 
be online

Anyone have success with this and if so how?


 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/F_spHAp3IqEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Up loading jpg files to google docs area

[trevor]

I am trying to write application to upload a jpeg file or png file
into google docs folder and set the share parameter to publish on web.

I am triing to do this to allow my users to email documents fromn
within app engine such the documents have the logo from the joeg or
png file that has been uploaded.

I note that uploading a file in google app engine seems to allow for a
document type only ( Word, Spreadsheet...) I can find nothing where I
set the permisions or share status.


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.