Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[wesley chun]

Reviving this for a bit: @Joshua: are you able to try out Python 3.8 (or
3.9) vs. 3.7 on App Engine? Several performance enhancements went into 3.8
, and I'm
curious whether that helps with what you've been experiencing. Also any
updates from your side?

Cheers,
--Wesley
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"A computer never does what you want... only what you tell it."
wesley chun :: @wescpy  :: Software
Architect & Engineer
Developer Advocate at Google
 by day; at
night: Core Python 



On Sat, Nov 6, 2021 at 7:12 AM Joshua Smith 
wrote:

> I'll put it on my to-do list to make a mock application that just does
> this hammer-on-startup thing. That seems pretty easy. (And it'll be
> interesting to see if a trivial mock app takes 5s to start up, or if
> there's some import I'm doing that's weighing things down.)
>
> -Joshua
>
> On Nov 5, 2021, at 6:39 PM, Jason Collins 
> wrote:
>
> Yes, it is the same scheduler (this is all on GAE Standard, right?) Of
> course, that scheduler has lots of knobs.
>
> If you can build a repro, can you work with support? If anything, the
> concurrency of GAE Python 3 should be better than GAE Python 2.
>
> On Fri, Nov 5, 2021 at 3:36 PM Joshua Smith 
> wrote:
>
>> I don't have threadsafe in my 3.7 app's app.yaml. Just runtime and
>> instance class. (I do have it in my 2.7 app, of course.)
>>
>> I'll try playing with max concurrent threads, but if I bump it from 10
>> (the default) to 16, that really just changes it from 6 request to start an
>> instance, to 9. Hard to see how that would make any difference.
>>
>> As I said, the 2.7 only starts one more instance when I hammer it, and it
>> doesn't end up giving it any work. That's very different behavior from the
>> 3.7 app which starts a bunch of instances and spreads the work to all of
>> them (resulting in all the requests taking at least 5 seconds to fulfill,
>> since the instance takes 5s to start up). This is clearly visible in the
>> logs I included in the first message.
>>
>> You're positive the scheduler is the same?
>>
>> -Joshua
>>
>> On Nov 5, 2021, at 5:08 PM, Jason Collins 
>> wrote:
>>
>> The scheduler is the same.
>>
>> This might have to do with `threadsafe` in app.yaml, which is complicated
>> and confusing. I'd suggest removing `threadsafe` if it's present, then try
>> setting the max_concurrent_requests to something higher (like 2x your
>> number of workers):
>> https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests
>>
>>
>> On Fri, Nov 5, 2021 at 1:48 PM Joshua Smith 
>> wrote:
>>
>>> I figured out from the logs (since it prints a message for each worker
>>> it starts) that 3.7 is starting 8 workers by default. So adding the
>>> entrypoint with defaults has no effect.
>>>
>>> However, I did some experiments and I think I can characterize the
>>> difference between 2.7 and 3.7 when it comes to scaling.
>>>
>>> When first landing on my app's page it shows a list of thumbnails. These
>>> come from the datastore, with each thumbnail making a HTTP request (GET
>>> /image?which=1, GET /image?which=2, etc.). If we:
>>>
>>> 1. Query for all of these at once so the server gets hammered with
>>> simultaneous requests:
>>> - 2.7 starts one extra instance, and while that's happening the images
>>> are all served by the instance that already existed; the new instance never
>>> actually serves any of those image requests.
>>> - 3.7 starts a lot of instances; it dumps a bunch of requests to each of
>>> them (one per worker thread, I presume), all of which are going to take 5
>>> seconds because that's how long it takes each instance to start.
>>>
>>> The net result is the user of the 2.7 app sees all the images load right
>>> away, while the 3.7 app gets them over the course of 15 seconds or so, and
>>> sometimes there are timeout errors.
>>>
>>> 2. Query for these with a delay that's long enough to avoid overlapping
>>> requests:
>>> - Neither starts extra instances.
>>>
>>> The user sees all the images load pretty quick, but not quite as quick
>>> as 2.7 with the first approach.
>>>
>>> My conclusion is that the change is that the old scheduler wouldn't use
>>> an instance until it was running, whereas the new scheduler queues up
>>> requests for instances that are still starting up. The old scheduler
>>> doesn't start a new instance until the one it started most recently is
>>> running, whereas the new scheduler starts a new instance every time the
>>> request queue fills up for existing instances.
>>>
>>> If you can serve a burst of requests in the time it take an instance to
>>> start, the old scheduler will get all those served. But the new scheduler
>>> will choke and perform very badly.
>>>
>>> I'm not saying that the old scheduler was better. It just handled this
>>> 

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Sajal Gupta]

Hi Matt,

Yes, earlier I was trying to make dev_appserver work (along with this flag) 
from within PyCharm. That did not work if the base python is set up to be 
Python 3 in PyCharm project.

It worked through command line.

It also started to work when I changed the base python to python 2 in pycharm 
project setup.

Now the only thing that does not work is PyCharm debugger along with 
dev_appserver with this runtime_python_path flag. I am assuming it is due to 
some conflict between python 2 and python 3 libs somehow getting mixed up.

Even if I run dev_appserver thru console/command line and then try to attach 
PyCharm debugger with this task, it doesn’t work.

It will help if there is a way to make it work and also is well documented.

Regards

Sajal

> On 01-Dec-2021, at 11:11 PM, 'Matt Larkin' via Google App Engine 
>  wrote:
> 
> >dev_appserver doesn't seem to work anyway - gives an assertion error 
> >(expects python 2, I believe) while my project is Python 3 - in fact trying 
> >to migrate from 2 to 3.
> Sorry, missed this.  Without any more details, my guess is you need to 
> provide the `--runtime_python_path` flag (docs 
> ).
>   If you do not, dev_appserver looks for a Py2 interpreter in order to run 
> (that's the default to remain consistent with legacy behavior).  If that 
> doesn't work, please provide more details (e.g. full error message).
> 
> On Wed, Dec 1, 2021 at 9:34 AM Matt Larkin  > wrote:
> Thanks.  I'm following-up with the engineering team to determine if this is 
> expected (meaning the docs need to be updated) or a bug.
> 
> On Wed, Dec 1, 2021 at 1:38 AM Sajal Gupta  > wrote:
> Am able to make this work now using dev_appserver. It doesn't work with flask 
> server (at least I cold not make it work using flask server as above).
> 
> If that is indeed the case, may be the GAE getting started guide for python 3 
> page needs to change to reflect this fact that memcache etc. will work only 
> thru dev_appserver - at least as of now.
> 
> Regards
> On Wednesday, December 1, 2021 at 11:43:09 AM UTC+5:30 Sajal Gupta wrote:
> 
> I am using Flask's server as follows:
> 
> if __name__ == '__main__':
> # # This is used when running locally only. When deploying to Google App
> # # Engine, a webserver process such as Gunicorn will serve the app. You
> # # can configure startup instructions by adding `entrypoint` to app.yaml.
> app.run(host='127.0.0.1', port=8080, debug=True)
> 
> And in Pycharm, have set up environment parameter as follows:
> FLASK_ENV=development
> 
> Also tried by not having this variable but end result is still the same.
> 
> dev_appserver doesn't seem to work anyway - gives an assertion error (expects 
> python 2, I believe) while my project is Python 3 - in fact trying to migrate 
> from 2 to 3.
> 
> Happy to share any additional information which can help debug this.
> 
> Regards
> On Tuesday, November 30, 2021 at 9:51:42 PM UTC+5:30 matthe...@google.com <> 
> wrote:
> Hi Sajal,
> Are you using dev_appserver to run locally, or are you using some other type 
> of toolset (e.g., the Flask dev server)?
> 
> Best,
> Matt
> 
> On Tue, Nov 30, 2021 at 6:05 AM Sajal Gupta > wrote:
> Hi,
> 
> I was trying to run the now bundled memcache service on localhost 
> (python3/flask) but got an error: Attempted RPC call without active security 
> ticket.
> 
> Could you shed some light on this and if localhost is not supported as of now 
> and if so, by when will this support be available.
> 
> Thanks
> 
> Sajal
> 
> 
> On Thursday, November 4, 2021 at 8:17:52 PM UTC+5:30 Wesley C (Google) wrote:
> App Engine and Google Cloud serverless users:
> 
> The App Engine  team at Google Cloud 
>  had a flurry of announcements and product updates 
> over the past few months. In case you missed them, we've summarized them here 
> with relevant links.
> 
> New features to better secure your Google App Engine apps 
> 
>  (Aug 2021) —  Introducing a pair of networking and security features:
> Egress Controls for Serverless VPC Access 
> 
>  — consistent app (static) outbound IP address
> User-managed service accounts 
> 
>  — custom service accounts to restrict apps to only the permissions 
> necessary, following the best practice of "least privileges"
> Exploring serverless with a nebulous app 
> 
>  

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Joshua Smith]

I'll put it on my to-do list to make a mock application that just does this 
hammer-on-startup thing. That seems pretty easy. (And it'll be interesting to 
see if a trivial mock app takes 5s to start up, or if there's some import I'm 
doing that's weighing things down.)

-Joshua

> On Nov 5, 2021, at 6:39 PM, Jason Collins  wrote:
> 
> Yes, it is the same scheduler (this is all on GAE Standard, right?) Of 
> course, that scheduler has lots of knobs.
> 
> If you can build a repro, can you work with support? If anything, the 
> concurrency of GAE Python 3 should be better than GAE Python 2.
> 
> On Fri, Nov 5, 2021 at 3:36 PM Joshua Smith  > wrote:
> I don't have threadsafe in my 3.7 app's app.yaml. Just runtime and instance 
> class. (I do have it in my 2.7 app, of course.)
> 
> I'll try playing with max concurrent threads, but if I bump it from 10 (the 
> default) to 16, that really just changes it from 6 request to start an 
> instance, to 9. Hard to see how that would make any difference.
> 
> As I said, the 2.7 only starts one more instance when I hammer it, and it 
> doesn't end up giving it any work. That's very different behavior from the 
> 3.7 app which starts a bunch of instances and spreads the work to all of them 
> (resulting in all the requests taking at least 5 seconds to fulfill, since 
> the instance takes 5s to start up). This is clearly visible in the logs I 
> included in the first message.
> 
> You're positive the scheduler is the same?
> 
> -Joshua
> 
>> On Nov 5, 2021, at 5:08 PM, Jason Collins > > wrote:
>> 
>> The scheduler is the same.
>> 
>> This might have to do with `threadsafe` in app.yaml, which is complicated 
>> and confusing. I'd suggest removing `threadsafe` if it's present, then try 
>> setting the max_concurrent_requests to something higher (like 2x your number 
>> of workers): 
>> https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests
>>  
>> 
>> 
>> 
>> On Fri, Nov 5, 2021 at 1:48 PM Joshua Smith > > wrote:
>> I figured out from the logs (since it prints a message for each worker it 
>> starts) that 3.7 is starting 8 workers by default. So adding the entrypoint 
>> with defaults has no effect.
>> 
>> However, I did some experiments and I think I can characterize the 
>> difference between 2.7 and 3.7 when it comes to scaling.
>> 
>> When first landing on my app's page it shows a list of thumbnails. These 
>> come from the datastore, with each thumbnail making a HTTP request (GET 
>> /image?which=1, GET /image?which=2, etc.). If we:
>> 
>> 1. Query for all of these at once so the server gets hammered with 
>> simultaneous requests:
>>  - 2.7 starts one extra instance, and while that's happening the images 
>> are all served by the instance that already existed; the new instance never 
>> actually serves any of those image requests.
>>  - 3.7 starts a lot of instances; it dumps a bunch of requests to each 
>> of them (one per worker thread, I presume), all of which are going to take 5 
>> seconds because that's how long it takes each instance to start.
>> 
>> The net result is the user of the 2.7 app sees all the images load right 
>> away, while the 3.7 app gets them over the course of 15 seconds or so, and 
>> sometimes there are timeout errors.
>> 
>> 2. Query for these with a delay that's long enough to avoid overlapping 
>> requests:
>>  - Neither starts extra instances.
>> 
>> The user sees all the images load pretty quick, but not quite as quick as 
>> 2.7 with the first approach.
>> 
>> My conclusion is that the change is that the old scheduler wouldn't use an 
>> instance until it was running, whereas the new scheduler queues up requests 
>> for instances that are still starting up. The old scheduler doesn't start a 
>> new instance until the one it started most recently is running, whereas the 
>> new scheduler starts a new instance every time the request queue fills up 
>> for existing instances.
>> 
>> If you can serve a burst of requests in the time it take an instance to 
>> start, the old scheduler will get all those served. But the new scheduler 
>> will choke and perform very badly.
>> 
>> I'm not saying that the old scheduler was better. It just handled this 
>> particular case (cold start with a firehose) a lot better. Once the two 
>> systems are running at scale, I wouldn't expect there to be much difference.
>> 
>> -Joshua
>> 
>> 
>>> On Nov 5, 2021, at 3:29 PM, Joshua Smith >> > wrote:
>>> 
>>> I think it's telling me that since my P3.7 app has a F4_1G instance, I 
>>> should be configuring it to run gunicorn with 8 workers. I don't see any 
>>> information about the default number of workers. I guess I can do an 
>>> experiment to see what explicitly setting that does...

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Jason Collins]

Yes, it is the same scheduler (this is all on GAE Standard, right?) Of
course, that scheduler has lots of knobs.

If you can build a repro, can you work with support? If anything, the
concurrency of GAE Python 3 should be better than GAE Python 2.

On Fri, Nov 5, 2021 at 3:36 PM Joshua Smith 
wrote:

> I don't have threadsafe in my 3.7 app's app.yaml. Just runtime and
> instance class. (I do have it in my 2.7 app, of course.)
>
> I'll try playing with max concurrent threads, but if I bump it from 10
> (the default) to 16, that really just changes it from 6 request to start an
> instance, to 9. Hard to see how that would make any difference.
>
> As I said, the 2.7 only starts one more instance when I hammer it, and it
> doesn't end up giving it any work. That's very different behavior from the
> 3.7 app which starts a bunch of instances and spreads the work to all of
> them (resulting in all the requests taking at least 5 seconds to fulfill,
> since the instance takes 5s to start up). This is clearly visible in the
> logs I included in the first message.
>
> You're positive the scheduler is the same?
>
> -Joshua
>
> On Nov 5, 2021, at 5:08 PM, Jason Collins 
> wrote:
>
> The scheduler is the same.
>
> This might have to do with `threadsafe` in app.yaml, which is complicated
> and confusing. I'd suggest removing `threadsafe` if it's present, then try
> setting the max_concurrent_requests to something higher (like 2x your
> number of workers):
> https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests
>
>
> On Fri, Nov 5, 2021 at 1:48 PM Joshua Smith 
> wrote:
>
>> I figured out from the logs (since it prints a message for each worker it
>> starts) that 3.7 is starting 8 workers by default. So adding the entrypoint
>> with defaults has no effect.
>>
>> However, I did some experiments and I think I can characterize the
>> difference between 2.7 and 3.7 when it comes to scaling.
>>
>> When first landing on my app's page it shows a list of thumbnails. These
>> come from the datastore, with each thumbnail making a HTTP request (GET
>> /image?which=1, GET /image?which=2, etc.). If we:
>>
>> 1. Query for all of these at once so the server gets hammered with
>> simultaneous requests:
>> - 2.7 starts one extra instance, and while that's happening the images
>> are all served by the instance that already existed; the new instance never
>> actually serves any of those image requests.
>> - 3.7 starts a lot of instances; it dumps a bunch of requests to each of
>> them (one per worker thread, I presume), all of which are going to take 5
>> seconds because that's how long it takes each instance to start.
>>
>> The net result is the user of the 2.7 app sees all the images load right
>> away, while the 3.7 app gets them over the course of 15 seconds or so, and
>> sometimes there are timeout errors.
>>
>> 2. Query for these with a delay that's long enough to avoid overlapping
>> requests:
>> - Neither starts extra instances.
>>
>> The user sees all the images load pretty quick, but not quite as quick as
>> 2.7 with the first approach.
>>
>> My conclusion is that the change is that the old scheduler wouldn't use
>> an instance until it was running, whereas the new scheduler queues up
>> requests for instances that are still starting up. The old scheduler
>> doesn't start a new instance until the one it started most recently is
>> running, whereas the new scheduler starts a new instance every time the
>> request queue fills up for existing instances.
>>
>> If you can serve a burst of requests in the time it take an instance to
>> start, the old scheduler will get all those served. But the new scheduler
>> will choke and perform very badly.
>>
>> I'm not saying that the old scheduler was better. It just handled this
>> particular case (cold start with a firehose) a lot better. Once the two
>> systems are running at scale, I wouldn't expect there to be much difference.
>>
>> -Joshua
>>
>>
>> On Nov 5, 2021, at 3:29 PM, Joshua Smith 
>> wrote:
>>
>> I think it's telling me that since my P3.7 app has a F4_1G instance, I
>> should be configuring it to run gunicorn with 8 workers. I don't see any
>> information about the default number of workers. I guess I can do an
>> experiment to see what explicitly setting that does...
>>
>> On Nov 5, 2021, at 3:17 PM, Jason Collins 
>> wrote:
>>
>> It sounds like there might be some differences due to concurrency between
>> 2.7 and 3.7/8/9. There are some notes on how to tweak the number of worker
>> processes started for Python 3 here:
>> https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
>>
>>
>> On Friday, 5 November 2021 at 11:50:52 UTC-7 George (Cloud Platform
>> Support) wrote:
>>
>>> There is no official description, somewhere, of "how the scaling has
>>> changed" or "how it has been implemented differently", as there is no
>>> change in scaling behavior, and no different implementation. This is an
>>> 

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Joshua Smith]

I don't have threadsafe in my 3.7 app's app.yaml. Just runtime and instance 
class. (I do have it in my 2.7 app, of course.)

I'll try playing with max concurrent threads, but if I bump it from 10 (the 
default) to 16, that really just changes it from 6 request to start an 
instance, to 9. Hard to see how that would make any difference.

As I said, the 2.7 only starts one more instance when I hammer it, and it 
doesn't end up giving it any work. That's very different behavior from the 3.7 
app which starts a bunch of instances and spreads the work to all of them 
(resulting in all the requests taking at least 5 seconds to fulfill, since the 
instance takes 5s to start up). This is clearly visible in the logs I included 
in the first message.

You're positive the scheduler is the same?

-Joshua

> On Nov 5, 2021, at 5:08 PM, Jason Collins  wrote:
> 
> The scheduler is the same.
> 
> This might have to do with `threadsafe` in app.yaml, which is complicated and 
> confusing. I'd suggest removing `threadsafe` if it's present, then try 
> setting the max_concurrent_requests to something higher (like 2x your number 
> of workers): 
> https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests
>  
> 
> 
> 
> On Fri, Nov 5, 2021 at 1:48 PM Joshua Smith  > wrote:
> I figured out from the logs (since it prints a message for each worker it 
> starts) that 3.7 is starting 8 workers by default. So adding the entrypoint 
> with defaults has no effect.
> 
> However, I did some experiments and I think I can characterize the difference 
> between 2.7 and 3.7 when it comes to scaling.
> 
> When first landing on my app's page it shows a list of thumbnails. These come 
> from the datastore, with each thumbnail making a HTTP request (GET 
> /image?which=1, GET /image?which=2, etc.). If we:
> 
> 1. Query for all of these at once so the server gets hammered with 
> simultaneous requests:
>   - 2.7 starts one extra instance, and while that's happening the images 
> are all served by the instance that already existed; the new instance never 
> actually serves any of those image requests.
>   - 3.7 starts a lot of instances; it dumps a bunch of requests to each 
> of them (one per worker thread, I presume), all of which are going to take 5 
> seconds because that's how long it takes each instance to start.
> 
> The net result is the user of the 2.7 app sees all the images load right 
> away, while the 3.7 app gets them over the course of 15 seconds or so, and 
> sometimes there are timeout errors.
> 
> 2. Query for these with a delay that's long enough to avoid overlapping 
> requests:
>   - Neither starts extra instances.
> 
> The user sees all the images load pretty quick, but not quite as quick as 2.7 
> with the first approach.
> 
> My conclusion is that the change is that the old scheduler wouldn't use an 
> instance until it was running, whereas the new scheduler queues up requests 
> for instances that are still starting up. The old scheduler doesn't start a 
> new instance until the one it started most recently is running, whereas the 
> new scheduler starts a new instance every time the request queue fills up for 
> existing instances.
> 
> If you can serve a burst of requests in the time it take an instance to 
> start, the old scheduler will get all those served. But the new scheduler 
> will choke and perform very badly.
> 
> I'm not saying that the old scheduler was better. It just handled this 
> particular case (cold start with a firehose) a lot better. Once the two 
> systems are running at scale, I wouldn't expect there to be much difference.
> 
> -Joshua
> 
> 
>> On Nov 5, 2021, at 3:29 PM, Joshua Smith > > wrote:
>> 
>> I think it's telling me that since my P3.7 app has a F4_1G instance, I 
>> should be configuring it to run gunicorn with 8 workers. I don't see any 
>> information about the default number of workers. I guess I can do an 
>> experiment to see what explicitly setting that does...
>> 
>>> On Nov 5, 2021, at 3:17 PM, Jason Collins >> > wrote:
>>> 
>>> It sounds like there might be some differences due to concurrency between 
>>> 2.7 and 3.7/8/9. There are some notes on how to tweak the number of worker 
>>> processes started for Python 3 here: 
>>> https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
>>>  
>>> 
>>> 
>>> 
>>> On Friday, 5 November 2021 at 11:50:52 UTC-7 George (Cloud Platform 
>>> Support) wrote:
>>> There is no official description, somewhere, of "how the scaling has 
>>> changed" or "how it has been implemented differently", as there is no 
>>> change in scaling behavior, and no different implementation. This is an 

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Jason Collins]

The scheduler is the same.

This might have to do with `threadsafe` in app.yaml, which is complicated
and confusing. I'd suggest removing `threadsafe` if it's present, then try
setting the max_concurrent_requests to something higher (like 2x your
number of workers):
https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests


On Fri, Nov 5, 2021 at 1:48 PM Joshua Smith 
wrote:

> I figured out from the logs (since it prints a message for each worker it
> starts) that 3.7 is starting 8 workers by default. So adding the entrypoint
> with defaults has no effect.
>
> However, I did some experiments and I think I can characterize the
> difference between 2.7 and 3.7 when it comes to scaling.
>
> When first landing on my app's page it shows a list of thumbnails. These
> come from the datastore, with each thumbnail making a HTTP request (GET
> /image?which=1, GET /image?which=2, etc.). If we:
>
> 1. Query for all of these at once so the server gets hammered with
> simultaneous requests:
> - 2.7 starts one extra instance, and while that's happening the images are
> all served by the instance that already existed; the new instance never
> actually serves any of those image requests.
> - 3.7 starts a lot of instances; it dumps a bunch of requests to each of
> them (one per worker thread, I presume), all of which are going to take 5
> seconds because that's how long it takes each instance to start.
>
> The net result is the user of the 2.7 app sees all the images load right
> away, while the 3.7 app gets them over the course of 15 seconds or so, and
> sometimes there are timeout errors.
>
> 2. Query for these with a delay that's long enough to avoid overlapping
> requests:
> - Neither starts extra instances.
>
> The user sees all the images load pretty quick, but not quite as quick as
> 2.7 with the first approach.
>
> My conclusion is that the change is that the old scheduler wouldn't use an
> instance until it was running, whereas the new scheduler queues up requests
> for instances that are still starting up. The old scheduler doesn't start a
> new instance until the one it started most recently is running, whereas the
> new scheduler starts a new instance every time the request queue fills up
> for existing instances.
>
> If you can serve a burst of requests in the time it take an instance to
> start, the old scheduler will get all those served. But the new scheduler
> will choke and perform very badly.
>
> I'm not saying that the old scheduler was better. It just handled this
> particular case (cold start with a firehose) a lot better. Once the two
> systems are running at scale, I wouldn't expect there to be much difference.
>
> -Joshua
>
>
> On Nov 5, 2021, at 3:29 PM, Joshua Smith  wrote:
>
> I think it's telling me that since my P3.7 app has a F4_1G instance, I
> should be configuring it to run gunicorn with 8 workers. I don't see any
> information about the default number of workers. I guess I can do an
> experiment to see what explicitly setting that does...
>
> On Nov 5, 2021, at 3:17 PM, Jason Collins 
> wrote:
>
> It sounds like there might be some differences due to concurrency between
> 2.7 and 3.7/8/9. There are some notes on how to tweak the number of worker
> processes started for Python 3 here:
> https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
>
>
> On Friday, 5 November 2021 at 11:50:52 UTC-7 George (Cloud Platform
> Support) wrote:
>
>> There is no official description, somewhere, of "how the scaling has
>> changed" or "how it has been implemented differently", as there is no
>> change in scaling behavior, and no different implementation. This is an
>> assumption while we investigate the situation described by Joshua above. I
>> could not find a similar issue in the Public Issue Tracker
>> , so the difference in scaling between
>> Python 2 and 3 is likely a one-time occurrence, and caused by idiosyncratic
>> settings in the project, or application-specific factors. Such issue are
>> difficult to approach in a publicly-accessible thread as this one, as
>> investigation needs particulars such as project ID, and access to other
>> private data. To have this issue properly addressed, it's better to open
>> a support case
>>  or
>> an issue in the Public Issue Tracker .
>>
>> On Friday, 05 November 2021 at 05:50:42 UTC-4 Nicolas Fonrose (Teevity)
>> wrote:
>>
>>> Hello David,
>>>
>>> >There’s not an official way of forcing the scaling to behave like it
>>> used to be on python 2.7.
>>> Is there an official description, somewhere, of "how the scaling has
>>> changed" or "how it has been implemented differently" ?
>>> Or this change just a side effect of technical choices that were made in
>>> the new runtimes?
>>>
>>> Thanks,
>>>
>>> On Thursday, November 4, 2021 at 9:56:43 PM UTC+1 David 

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Joshua Smith]

I figured out from the logs (since it prints a message for each worker it 
starts) that 3.7 is starting 8 workers by default. So adding the entrypoint 
with defaults has no effect.

However, I did some experiments and I think I can characterize the difference 
between 2.7 and 3.7 when it comes to scaling.

When first landing on my app's page it shows a list of thumbnails. These come 
from the datastore, with each thumbnail making a HTTP request (GET 
/image?which=1, GET /image?which=2, etc.). If we:

1. Query for all of these at once so the server gets hammered with simultaneous 
requests:
- 2.7 starts one extra instance, and while that's happening the images 
are all served by the instance that already existed; the new instance never 
actually serves any of those image requests.
- 3.7 starts a lot of instances; it dumps a bunch of requests to each 
of them (one per worker thread, I presume), all of which are going to take 5 
seconds because that's how long it takes each instance to start.

The net result is the user of the 2.7 app sees all the images load right away, 
while the 3.7 app gets them over the course of 15 seconds or so, and sometimes 
there are timeout errors.

2. Query for these with a delay that's long enough to avoid overlapping 
requests:
- Neither starts extra instances.

The user sees all the images load pretty quick, but not quite as quick as 2.7 
with the first approach.

My conclusion is that the change is that the old scheduler wouldn't use an 
instance until it was running, whereas the new scheduler queues up requests for 
instances that are still starting up. The old scheduler doesn't start a new 
instance until the one it started most recently is running, whereas the new 
scheduler starts a new instance every time the request queue fills up for 
existing instances.

If you can serve a burst of requests in the time it take an instance to start, 
the old scheduler will get all those served. But the new scheduler will choke 
and perform very badly.

I'm not saying that the old scheduler was better. It just handled this 
particular case (cold start with a firehose) a lot better. Once the two systems 
are running at scale, I wouldn't expect there to be much difference.

-Joshua


> On Nov 5, 2021, at 3:29 PM, Joshua Smith  wrote:
> 
> I think it's telling me that since my P3.7 app has a F4_1G instance, I should 
> be configuring it to run gunicorn with 8 workers. I don't see any information 
> about the default number of workers. I guess I can do an experiment to see 
> what explicitly setting that does...
> 
>> On Nov 5, 2021, at 3:17 PM, Jason Collins > > wrote:
>> 
>> It sounds like there might be some differences due to concurrency between 
>> 2.7 and 3.7/8/9. There are some notes on how to tweak the number of worker 
>> processes started for Python 3 here: 
>> https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
>>  
>> 
>> 
>> 
>> On Friday, 5 November 2021 at 11:50:52 UTC-7 George (Cloud Platform Support) 
>> wrote:
>> There is no official description, somewhere, of "how the scaling has 
>> changed" or "how it has been implemented differently", as there is no change 
>> in scaling behavior, and no different implementation. This is an assumption 
>> while we investigate the situation described by Joshua above. I could not 
>> find a similar issue in the Public Issue Tracker 
>> , so the difference in scaling between 
>> Python 2 and 3 is likely a one-time occurrence, and caused by idiosyncratic 
>> settings in the project, or application-specific factors. Such issue are 
>> difficult to approach in a publicly-accessible thread as this one, as 
>> investigation needs particulars such as project ID, and access to other 
>> private data. To have this issue properly addressed, it's better to open a 
>> support case 
>>  or an 
>> issue in the Public Issue Tracker . 
>> 
>> On Friday, 05 November 2021 at 05:50:42 UTC-4 Nicolas Fonrose (Teevity) 
>> wrote:
>> Hello David,
>> 
>> >There’s not an official way of forcing the scaling to behave like it used 
>> >to be on python 2.7.
>> Is there an official description, somewhere, of "how the scaling has 
>> changed" or "how it has been implemented differently" ?
>> Or this change just a side effect of technical choices that were made in the 
>> new runtimes?
>> 
>> Thanks,
>> 
>> On Thursday, November 4, 2021 at 9:56:43 PM UTC+1 David (Cloud Platform 
>> Support) wrote:
>> Hello,
>> 
>> Other than applying changes within your own application so it will go easy 
>> at startup, you can try using warmup requests 
>> 
>>  along with tweaking 

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Jason Collins]

We'll work to open source the launcher source. In the meantime, the default 
logic looks like this:

workerCount := 8
if memoryMb <= 128 {
workerCount = 1
} else if memoryMb <= 256 {
workerCount = 2
} else if memoryMb <= 512 {
workerCount = 4
}

The number of workers to use is highly dependent on your workload, so these 
are only reasonable general starting points.


On Friday, 5 November 2021 at 12:29:43 UTC-7 Joshua Smith wrote:

> I think it's telling me that since my P3.7 app has a F4_1G instance, I 
> should be configuring it to run gunicorn with 8 workers. I don't see any 
> information about the default number of workers. I guess I can do an 
> experiment to see what explicitly setting that does...
>
> On Nov 5, 2021, at 3:17 PM, Jason Collins  wrote:
>
> It sounds like there might be some differences due to concurrency between 
> 2.7 and 3.7/8/9. There are some notes on how to tweak the number of worker 
> processes started for Python 3 here: 
> https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
>
>
> On Friday, 5 November 2021 at 11:50:52 UTC-7 George (Cloud Platform 
> Support) wrote:
>
>> There is no official description, somewhere, of "how the scaling has 
>> changed" or "how it has been implemented differently", as there is no 
>> change in scaling behavior, and no different implementation. This is an 
>> assumption while we investigate the situation described by Joshua above. I 
>> could not find a similar issue in the Public Issue Tracker 
>> , so the difference in scaling between 
>> Python 2 and 3 is likely a one-time occurrence, and caused by idiosyncratic 
>> settings in the project, or application-specific factors. Such issue are 
>> difficult to approach in a publicly-accessible thread as this one, as 
>> investigation needs particulars such as project ID, and access to other 
>> private data. To have this issue properly addressed, it's better to open 
>> a support case 
>>  or 
>> an issue in the Public Issue Tracker . 
>>
>> On Friday, 05 November 2021 at 05:50:42 UTC-4 Nicolas Fonrose (Teevity) 
>> wrote:
>>
>>> Hello David,
>>>
>>> >There’s not an official way of forcing the scaling to behave like it 
>>> used to be on python 2.7.
>>> Is there an official description, somewhere, of "how the scaling has 
>>> changed" or "how it has been implemented differently" ?
>>> Or this change just a side effect of technical choices that were made in 
>>> the new runtimes?
>>>
>>> Thanks,
>>>
>>> On Thursday, November 4, 2021 at 9:56:43 PM UTC+1 David (Cloud Platform 
>>> Support) wrote:
>>>
 Hello,

 Other than applying changes within your own application so it will go 
 easy at startup, you can try using warmup requests 
 
  
 along with tweaking the min_idle_instances element 
 
  
 in your app.yaml, in order to reduce request and response latency during 
 the time when your app's code is being loaded to a newly created instance.
 There’s not an official way of forcing the scaling to behave like it 
 used to be on python 2.7. However, this type of feedback can be passed to 
 the App Engine engineering team in the form of a feature request. You can 
 create such a feature request here 
 .
  
 The App Engine engineering team would then evaluate it and decide whether 
 it could be implemented or not.
 On Thursday, November 4, 2021 at 11:38:52 AM UTC-4 Joshua Smith wrote:

> Thanks for sending out this update. I did, indeed, miss most of this 
> news.
>
> This item, in particular, is *awesome*:
>
>
>
>- 
>
>Extending support for App Engine bundled services 
>
> 
>  
>(Sep 2021) 
>
>
> This sounds like it will make it so much easier to migrate to Python 
> 3.7.
>
> One thing I've noticed is that my older apps on 2.7 seem to handle 
> peak scaling a lot better than my newer apps on 3.7. For example, if I 
> have 
> a web page that hits a 2.7 app with 100 REST calls at startup (bad 
> design, 
> but it happens), the old app serves them all eventually. But if I do the 
> same thing in a 3.7 app, it's likely to choke and fail a bunch of those 
> requests with these:
>
>
> The specific pattern is that after the first couple requests, it spins 
> up a new instance. That takes 5 seconds to serve its first request 
> (simple 
> app, so I guess 

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Joshua Smith]

I think it's telling me that since my P3.7 app has a F4_1G instance, I should 
be configuring it to run gunicorn with 8 workers. I don't see any information 
about the default number of workers. I guess I can do an experiment to see what 
explicitly setting that does...

> On Nov 5, 2021, at 3:17 PM, Jason Collins  wrote:
> 
> It sounds like there might be some differences due to concurrency between 2.7 
> and 3.7/8/9. There are some notes on how to tweak the number of worker 
> processes started for Python 3 here: 
> https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
>  
> 
> 
> 
> On Friday, 5 November 2021 at 11:50:52 UTC-7 George (Cloud Platform Support) 
> wrote:
> There is no official description, somewhere, of "how the scaling has changed" 
> or "how it has been implemented differently", as there is no change in 
> scaling behavior, and no different implementation. This is an assumption 
> while we investigate the situation described by Joshua above. I could not 
> find a similar issue in the Public Issue Tracker 
> , so the difference in scaling between 
> Python 2 and 3 is likely a one-time occurrence, and caused by idiosyncratic 
> settings in the project, or application-specific factors. Such issue are 
> difficult to approach in a publicly-accessible thread as this one, as 
> investigation needs particulars such as project ID, and access to other 
> private data. To have this issue properly addressed, it's better to open a 
> support case 
>  or an 
> issue in the Public Issue Tracker . 
> 
> On Friday, 05 November 2021 at 05:50:42 UTC-4 Nicolas Fonrose (Teevity) wrote:
> Hello David,
> 
> >There’s not an official way of forcing the scaling to behave like it used to 
> >be on python 2.7.
> Is there an official description, somewhere, of "how the scaling has changed" 
> or "how it has been implemented differently" ?
> Or this change just a side effect of technical choices that were made in the 
> new runtimes?
> 
> Thanks,
> 
> On Thursday, November 4, 2021 at 9:56:43 PM UTC+1 David (Cloud Platform 
> Support) wrote:
> Hello,
> 
> Other than applying changes within your own application so it will go easy at 
> startup, you can try using warmup requests 
> 
>  along with tweaking the min_idle_instances element 
> 
>  in your app.yaml, in order to reduce request and response latency during the 
> time when your app's code is being loaded to a newly created instance.
> 
> There’s not an official way of forcing the scaling to behave like it used to 
> be on python 2.7. However, this type of feedback can be passed to the App 
> Engine engineering team in the form of a feature request. You can create such 
> a feature request here 
> .
>  The App Engine engineering team would then evaluate it and decide whether it 
> could be implemented or not.
> On Thursday, November 4, 2021 at 11:38:52 AM UTC-4 Joshua Smith wrote:
> Thanks for sending out this update. I did, indeed, miss most of this news.
> 
> This item, in particular, is awesome:
> 
> 
>> Extending support for App Engine bundled services 
>> 
>>  (Sep 2021) 
> 
> This sounds like it will make it so much easier to migrate to Python 3.7.
> 
> One thing I've noticed is that my older apps on 2.7 seem to handle peak 
> scaling a lot better than my newer apps on 3.7. For example, if I have a web 
> page that hits a 2.7 app with 100 REST calls at startup (bad design, but it 
> happens), the old app serves them all eventually. But if I do the same thing 
> in a 3.7 app, it's likely to choke and fail a bunch of those requests with 
> these:
> 
> 
> 
> The specific pattern is that after the first couple requests, it spins up a 
> new instance. That takes 5 seconds to serve its first request (simple app, so 
> I guess that's just GAE overhead). Then it spins up a couple more. Then I 
> start getting those errors on some of the requests, because 15s have passed. 
> In this capture the blue ones are new instances spinning up, and the orange 
> ones are timeout errors (note the 15s time on those).
> 
> 
> 
> I've been designing around the issue by making sure my web apps go easy on 
> the server on startup. But it is yet another concern about migrating.
> 
> Is there a way to tune the auto-scaling for 3.7 to behave like the 2.7? 
> Here's a similar set of requests to my 2.7 app, which only spins up one extra 
> instance, and never throws timeout 

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Jason Collins]

It sounds like there might be some differences due to concurrency between 
2.7 and 3.7/8/9. There are some notes on how to tweak the number of worker 
processes started for Python 3 
here: 
https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices


On Friday, 5 November 2021 at 11:50:52 UTC-7 George (Cloud Platform 
Support) wrote:

> There is no official description, somewhere, of "how the scaling has 
> changed" or "how it has been implemented differently", as there is no 
> change in scaling behavior, and no different implementation. This is an 
> assumption while we investigate the situation described by Joshua above. I 
> could not find a similar issue in the Public Issue Tracker 
> , so the difference in scaling between 
> Python 2 and 3 is likely a one-time occurrence, and caused by idiosyncratic 
> settings in the project, or application-specific factors. Such issue are 
> difficult to approach in a publicly-accessible thread as this one, as 
> investigation needs particulars such as project ID, and access to other 
> private data. To have this issue properly addressed, it's better to open 
> a support case 
>  or an 
> issue in the Public Issue Tracker . 
>
> On Friday, 05 November 2021 at 05:50:42 UTC-4 Nicolas Fonrose (Teevity) 
> wrote:
>
>> Hello David,
>>
>> >There’s not an official way of forcing the scaling to behave like it 
>> used to be on python 2.7.
>> Is there an official description, somewhere, of "how the scaling has 
>> changed" or "how it has been implemented differently" ?
>> Or this change just a side effect of technical choices that were made in 
>> the new runtimes?
>>
>> Thanks,
>>
>> On Thursday, November 4, 2021 at 9:56:43 PM UTC+1 David (Cloud Platform 
>> Support) wrote:
>>
>>> Hello,
>>>
>>> Other than applying changes within your own application so it will go 
>>> easy at startup, you can try using warmup requests 
>>> 
>>>  
>>> along with tweaking the min_idle_instances element 
>>> 
>>>  
>>> in your app.yaml, in order to reduce request and response latency during 
>>> the time when your app's code is being loaded to a newly created instance.
>>> There’s not an official way of forcing the scaling to behave like it 
>>> used to be on python 2.7. However, this type of feedback can be passed to 
>>> the App Engine engineering team in the form of a feature request. You can 
>>> create such a feature request here 
>>> .
>>>  
>>> The App Engine engineering team would then evaluate it and decide whether 
>>> it could be implemented or not.
>>> On Thursday, November 4, 2021 at 11:38:52 AM UTC-4 Joshua Smith wrote:
>>>
 Thanks for sending out this update. I did, indeed, miss most of this 
 news.

 This item, in particular, is *awesome*:



- 

Extending support for App Engine bundled services 

 
  
(Sep 2021) 


 This sounds like it will make it so much easier to migrate to Python 
 3.7.

 One thing I've noticed is that my older apps on 2.7 seem to handle peak 
 scaling a lot better than my newer apps on 3.7. For example, if I have a 
 web page that hits a 2.7 app with 100 REST calls at startup (bad design, 
 but it happens), the old app serves them all eventually. But if I do the 
 same thing in a 3.7 app, it's likely to choke and fail a bunch of those 
 requests with these:


 The specific pattern is that after the first couple requests, it spins 
 up a new instance. That takes 5 seconds to serve its first request (simple 
 app, so I guess that's just GAE overhead). Then it spins up a couple more. 
 Then I start getting those errors on some of the requests, because 15s 
 have 
 passed. In this capture the blue ones are new instances spinning up, and 
 the orange ones are timeout errors (note the 15s time on those).


 I've been designing around the issue by making sure my web apps go easy 
 on the server on startup. But it is yet another concern about migrating.

 Is there a way to tune the auto-scaling for 3.7 to behave like the 2.7? 
 Here's a similar set of requests to my 2.7 app, which only spins up one 
 extra instance, and never throws timeout errors, ever:


 -Joshua




-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from 

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

['George (Cloud Platform Support)' via Google App Engine]

There is no official description, somewhere, of "how the scaling has 
changed" or "how it has been implemented differently", as there is no 
change in scaling behavior, and no different implementation. This is an 
assumption while we investigate the situation described by Joshua above. I 
could not find a similar issue in the Public Issue Tracker 
, so the difference in scaling between 
Python 2 and 3 is likely a one-time occurrence, and caused by idiosyncratic 
settings in the project, or application-specific factors. Such issue are 
difficult to approach in a publicly-accessible thread as this one, as 
investigation needs particulars such as project ID, and access to other 
private data. To have this issue properly addressed, it's better to open a 
support case 
 or an 
issue in the Public Issue Tracker . 

On Friday, 05 November 2021 at 05:50:42 UTC-4 Nicolas Fonrose (Teevity) 
wrote:

> Hello David,
>
> >There’s not an official way of forcing the scaling to behave like it used 
> to be on python 2.7.
> Is there an official description, somewhere, of "how the scaling has 
> changed" or "how it has been implemented differently" ?
> Or this change just a side effect of technical choices that were made in 
> the new runtimes?
>
> Thanks,
>
> On Thursday, November 4, 2021 at 9:56:43 PM UTC+1 David (Cloud Platform 
> Support) wrote:
>
>> Hello,
>>
>> Other than applying changes within your own application so it will go 
>> easy at startup, you can try using warmup requests 
>> 
>>  
>> along with tweaking the min_idle_instances element 
>> 
>>  
>> in your app.yaml, in order to reduce request and response latency during 
>> the time when your app's code is being loaded to a newly created instance.
>> There’s not an official way of forcing the scaling to behave like it used 
>> to be on python 2.7. However, this type of feedback can be passed to the 
>> App Engine engineering team in the form of a feature request. You can 
>> create such a feature request here 
>> .
>>  
>> The App Engine engineering team would then evaluate it and decide whether 
>> it could be implemented or not.
>> On Thursday, November 4, 2021 at 11:38:52 AM UTC-4 Joshua Smith wrote:
>>
>>> Thanks for sending out this update. I did, indeed, miss most of this 
>>> news.
>>>
>>> This item, in particular, is *awesome*:
>>>
>>>
>>>
>>>- 
>>>
>>>Extending support for App Engine bundled services 
>>>
>>> 
>>>  
>>>(Sep 2021) 
>>>
>>>
>>> This sounds like it will make it so much easier to migrate to Python 3.7.
>>>
>>> One thing I've noticed is that my older apps on 2.7 seem to handle peak 
>>> scaling a lot better than my newer apps on 3.7. For example, if I have a 
>>> web page that hits a 2.7 app with 100 REST calls at startup (bad design, 
>>> but it happens), the old app serves them all eventually. But if I do the 
>>> same thing in a 3.7 app, it's likely to choke and fail a bunch of those 
>>> requests with these:
>>>
>>>
>>> The specific pattern is that after the first couple requests, it spins 
>>> up a new instance. That takes 5 seconds to serve its first request (simple 
>>> app, so I guess that's just GAE overhead). Then it spins up a couple more. 
>>> Then I start getting those errors on some of the requests, because 15s have 
>>> passed. In this capture the blue ones are new instances spinning up, and 
>>> the orange ones are timeout errors (note the 15s time on those).
>>>
>>>
>>> I've been designing around the issue by making sure my web apps go easy 
>>> on the server on startup. But it is yet another concern about migrating.
>>>
>>> Is there a way to tune the auto-scaling for 3.7 to behave like the 2.7? 
>>> Here's a similar set of requests to my 2.7 app, which only spins up one 
>>> extra instance, and never throws timeout errors, ever:
>>>
>>>
>>> -Joshua
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/321e134a-09d4-4536-9426-1a65170ca498n%40googlegroups.com.

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Nicolas Fonrose (Teevity)]

Hello David,

>There’s not an official way of forcing the scaling to behave like it used 
to be on python 2.7.
Is there an official description, somewhere, of "how the scaling has 
changed" or "how it has been implemented differently" ?
Or this change just a side effect of technical choices that were made in 
the new runtimes?

Thanks,

On Thursday, November 4, 2021 at 9:56:43 PM UTC+1 David (Cloud Platform 
Support) wrote:

> Hello,
>
> Other than applying changes within your own application so it will go easy 
> at startup, you can try using warmup requests 
> 
>  
> along with tweaking the min_idle_instances element 
> 
>  
> in your app.yaml, in order to reduce request and response latency during 
> the time when your app's code is being loaded to a newly created instance.
> There’s not an official way of forcing the scaling to behave like it used 
> to be on python 2.7. However, this type of feedback can be passed to the 
> App Engine engineering team in the form of a feature request. You can 
> create such a feature request here 
> .
>  
> The App Engine engineering team would then evaluate it and decide whether 
> it could be implemented or not.
> On Thursday, November 4, 2021 at 11:38:52 AM UTC-4 Joshua Smith wrote:
>
>> Thanks for sending out this update. I did, indeed, miss most of this news.
>>
>> This item, in particular, is *awesome*:
>>
>>
>>
>>- 
>>
>>Extending support for App Engine bundled services 
>>
>> 
>>  
>>(Sep 2021) 
>>
>>
>> This sounds like it will make it so much easier to migrate to Python 3.7.
>>
>> One thing I've noticed is that my older apps on 2.7 seem to handle peak 
>> scaling a lot better than my newer apps on 3.7. For example, if I have a 
>> web page that hits a 2.7 app with 100 REST calls at startup (bad design, 
>> but it happens), the old app serves them all eventually. But if I do the 
>> same thing in a 3.7 app, it's likely to choke and fail a bunch of those 
>> requests with these:
>>
>>
>> The specific pattern is that after the first couple requests, it spins up 
>> a new instance. That takes 5 seconds to serve its first request (simple 
>> app, so I guess that's just GAE overhead). Then it spins up a couple more. 
>> Then I start getting those errors on some of the requests, because 15s have 
>> passed. In this capture the blue ones are new instances spinning up, and 
>> the orange ones are timeout errors (note the 15s time on those).
>>
>>
>> I've been designing around the issue by making sure my web apps go easy 
>> on the server on startup. But it is yet another concern about migrating.
>>
>> Is there a way to tune the auto-scaling for 3.7 to behave like the 2.7? 
>> Here's a similar set of requests to my 2.7 app, which only spins up one 
>> extra instance, and never throws timeout errors, ever:
>>
>>
>> -Joshua
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/8b77544e-2755-42ae-8b36-04897cea41c1n%40googlegroups.com.

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

['David (Cloud Platform Support)' via Google App Engine]

Hello,

Other than applying changes within your own application so it will go easy 
at startup, you can try using warmup requests 

 
along with tweaking the min_idle_instances element 

 
in your app.yaml, in order to reduce request and response latency during 
the time when your app's code is being loaded to a newly created instance.
There’s not an official way of forcing the scaling to behave like it used 
to be on python 2.7. However, this type of feedback can be passed to the 
App Engine engineering team in the form of a feature request. You can 
create such a feature request here 
. 
The App Engine engineering team would then evaluate it and decide whether 
it could be implemented or not.
On Thursday, November 4, 2021 at 11:38:52 AM UTC-4 Joshua Smith wrote:

> Thanks for sending out this update. I did, indeed, miss most of this news.
>
> This item, in particular, is *awesome*:
>
>
>
>- 
>
>Extending support for App Engine bundled services 
>
> 
>  
>(Sep 2021) 
>
>
> This sounds like it will make it so much easier to migrate to Python 3.7.
>
> One thing I've noticed is that my older apps on 2.7 seem to handle peak 
> scaling a lot better than my newer apps on 3.7. For example, if I have a 
> web page that hits a 2.7 app with 100 REST calls at startup (bad design, 
> but it happens), the old app serves them all eventually. But if I do the 
> same thing in a 3.7 app, it's likely to choke and fail a bunch of those 
> requests with these:
>
>
> The specific pattern is that after the first couple requests, it spins up 
> a new instance. That takes 5 seconds to serve its first request (simple 
> app, so I guess that's just GAE overhead). Then it spins up a couple more. 
> Then I start getting those errors on some of the requests, because 15s have 
> passed. In this capture the blue ones are new instances spinning up, and 
> the orange ones are timeout errors (note the 15s time on those).
>
>
> I've been designing around the issue by making sure my web apps go easy on 
> the server on startup. But it is yet another concern about migrating.
>
> Is there a way to tune the auto-scaling for 3.7 to behave like the 2.7? 
> Here's a similar set of requests to my 2.7 app, which only spins up one 
> extra instance, and never throws timeout errors, ever:
>
>
> -Joshua
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/be0758b0-a47c-4d4e-bdee-6b27ec308e3fn%40googlegroups.com.

[google-appengine] Updates from the Google App Engine team (Fall 2021)

[wesley chun]

App Engine and Google Cloud serverless users:

The App Engine  team at Google Cloud
 had a flurry of announcements and product updates
over the past few months. In case you missed them, we've summarized them
here with relevant links.


   -

   New features to better secure your Google App Engine apps
   

   (Aug 2021) —  Introducing a pair of networking and security features:
   -

  Egress Controls for Serverless VPC Access
  

  — consistent app (static) outbound IP address
  -

  User-managed service accounts
  

  — custom service accounts to restrict apps to only the permissions
  necessary, following the best practice of "least privileges"
  -

   Exploring serverless with a nebulous app
   

   (Sep 2021) — Deploy/shift the same app to App Engine, Cloud Functions,
   or Cloud Run without any code changes, and show how to access Cloud APIs
   from serverless
   -

   Extending support for App Engine bundled services
   

   (Sep 2021) — Legacy App Engine APIs/bundled services, e.g., Memcache
   ,
   Datastore
   , etc.,
   were left out of the next-generation service (to fight against "vendor
   lock-in" not because they were deprecated). Many matured into standalone
   products, e.g., Cloud Memorystore
, Cloud
   Datastore , etc., respectively
   anyway. To help with updating language versions (e.g., Python 2 to 3, Java
   8 to 11, etc.), we've added many of these bundled services "back" to the
   latest App Engine runtimes.
   -

   Modernizing your serverless applications
   

   (Sep 2021) — Introducing codelabs (hands-on tutorials) and new video
   series  to help customers migrate to standalone
   services to modernize & make their apps more portable (second-generation
   App Engine, Cloud Functions, Cloud Run; Kubernetes Engine (GKE), Compute
   Engine VMs; other cloud, multi-cloud, hybrid cloud, on-prem, etc.):
   -

  App Engine webapp2 to Flask
  

  -

  App Engine taskqueue to Cloud Tasks
  

  (push queues)
  -

  App Engine ndb (Datastore) to Cloud NDB
  


  -

  Cloud NDB to Cloud Datastore
  

  -

  App Engine to Cloud Run (with Docker
  

  or without Docker using Cloud Buildpacks
  

  )
  -

  More are on the way…


While we are working hard to improve your App Engine experience, recognize
App Engine is no longer the only serverless option available from Google
Cloud:



   -

   If you don't have an entire app, have single-function utilities or
   microservices, Cloud Functions  is a
   great alternative; it also supports event-driven workloads
   -

   For those leveraging containerization as part of their app modernization
   or software development workflows or wish to avoid some of the constraints
   from App Engine or Cloud Functions, consider Cloud Run