Re: [google-appengine] Re: When will GAE Python 2.7 be shutdown

[Steven Prosser]

I have been using GAE since it was in beta. It has been nearly perfect for 
my company's use model. Recently I have been very concerned about Python2 
getting closed down. Just wanted to say that the information provided by 
Google in this thread has been really great. Most importantly is the "one 
year notice" which will give us time to migrate to Python3. For now, 
though, Python2 is as solid as it has ever been. Hope it continues on 
forever (I keep my "Google AppEngineLauncher" app ready for any code 
testing we need -- yeah its that old...). Again, my thanks to Google for 
being really transparent. -Steve

On Saturday, January 8, 2022 at 12:24:40 PM UTC-8 Wesley C (Google) wrote:

> Hi again,
>
>1. *dev_appserver*: I replied in my other email that the tool 
>currently is available for Python 3 but still requires some components 
> from 
>2.7 to function properly and suggested you can file an external bug to 
> make 
>it fully functional in 3.x.
>2. *bundled services*: Google has made available as many of the 
>bundled services as possible, but unfortunately there's currently no plan 
>on making search available in the next generation runtimes. It is 
> currently 
>recommended to migrate to Elastic Search using a Compute Engine VM 
>
> 
>  as 
>recommended here 
>
> .
>  
>The goal is not "to move people away from App Engine," but to make 
>developers' apps more portable, so they can take their apps elsewhere if 
>App Engine isn't the right fit for their needs (removing the notion of 
>"vendor lock-in") by encouraging developers to migrate away from those 
>legacy bundled services. They are strickly a migration tool to ease 
>upgrading to Python 3 and will not receive any new features... all of the 
>innovation is going into any standalone Cloud equivalent services.
>3. *2.x support*: As Alejandro alluded to, there are no current plans 
>to shutdown the Python 2 runtime, even with the sunset of Python 2 
>, older Java, PHP 5, and Go 
>versions older than 1.11, by their respective communities. Google Cloud 
> has 
>assured users by expressing continued long-term support of these 
>legacy runtimes 
>, 
>including maintaining the Python 2 runtime 
>. So while there 
>is no requirement for users to migrate to Python 3 at this time, 
> developers 
>like you are expressing interest in updating their applications to the 
>latest language releases.
>
> Best regards,
> --Wesley
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> "A computer never does what you want... only what you tell it."
> wesley chun :: @wescpy  :: Software 
> Architect & Engineer
> Developer Advocate at Google 
>  by day; at 
> night: Core Python 
>
>
>
> On Fri, Jan 7, 2022 at 11:38 PM Sajal Gupta  wrote:
>
>> Thanks Alejandro.
>>
>> *dev_appserver*: The challenge seems to be that it needs python 2 
>> installation for it to support python 3.x. So, looks like dev_appserver is 
>> still not "native" to python 3.x.And possibly due to this, I am not able to 
>> run dev_appserver in debugger mode for python 3.x applications.  That is 
>> what I meant about dev_appserver not being available fully for python 3.x. 
>>
>> *Bundled services: *Yes, I looks like now memcache and task queue is 
>> being made available for python 3.x and these seem to be in beta as of now. 
>> However, from what I understand, *search * is not available as yet 
>> unless there is a concrete plan to make it available for 3.x. The current 
>> suggestion for search seems to be host our own on Kubernetes, which defeats 
>> the whole purpose of someone using Google App Engine - which hides all the 
>> infra from the developers.
>>
>> So, unless, the unstated intent of Google team is to move people away 
>> from App Engine eventually, current, proposed migration path is not helping.
>>
>> *Notice of termination of python 2.x*: Thank you. This helps.
>>
>> Regards
>>
>> Sajal
>> On Friday, January 7, 2022 at 9:24:48 PM UTC+5:30 Alejandro L wrote:
>>
>>> Hello,
>>>
>>> Thank you for your message.
>>>
>>> As you mention, Python has ended support for versions 2.x, and we do 
>>> encourage our users to use a 3.x version. However, I would like to answer 
>>> all your questions, point by point:
>>>
>>> 0 - Dev_appserver does indeed have support for Python 3, which you can 
>>> find here 
>>> 

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[wesley chun]

Reviving this for a bit: @Joshua: are you able to try out Python 3.8 (or
3.9) vs. 3.7 on App Engine? Several performance enhancements went into 3.8
, and I'm
curious whether that helps with what you've been experiencing. Also any
updates from your side?

Cheers,
--Wesley
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"A computer never does what you want... only what you tell it."
wesley chun :: @wescpy  :: Software
Architect & Engineer
Developer Advocate at Google
 by day; at
night: Core Python 



On Sat, Nov 6, 2021 at 7:12 AM Joshua Smith 
wrote:

> I'll put it on my to-do list to make a mock application that just does
> this hammer-on-startup thing. That seems pretty easy. (And it'll be
> interesting to see if a trivial mock app takes 5s to start up, or if
> there's some import I'm doing that's weighing things down.)
>
> -Joshua
>
> On Nov 5, 2021, at 6:39 PM, Jason Collins 
> wrote:
>
> Yes, it is the same scheduler (this is all on GAE Standard, right?) Of
> course, that scheduler has lots of knobs.
>
> If you can build a repro, can you work with support? If anything, the
> concurrency of GAE Python 3 should be better than GAE Python 2.
>
> On Fri, Nov 5, 2021 at 3:36 PM Joshua Smith 
> wrote:
>
>> I don't have threadsafe in my 3.7 app's app.yaml. Just runtime and
>> instance class. (I do have it in my 2.7 app, of course.)
>>
>> I'll try playing with max concurrent threads, but if I bump it from 10
>> (the default) to 16, that really just changes it from 6 request to start an
>> instance, to 9. Hard to see how that would make any difference.
>>
>> As I said, the 2.7 only starts one more instance when I hammer it, and it
>> doesn't end up giving it any work. That's very different behavior from the
>> 3.7 app which starts a bunch of instances and spreads the work to all of
>> them (resulting in all the requests taking at least 5 seconds to fulfill,
>> since the instance takes 5s to start up). This is clearly visible in the
>> logs I included in the first message.
>>
>> You're positive the scheduler is the same?
>>
>> -Joshua
>>
>> On Nov 5, 2021, at 5:08 PM, Jason Collins 
>> wrote:
>>
>> The scheduler is the same.
>>
>> This might have to do with `threadsafe` in app.yaml, which is complicated
>> and confusing. I'd suggest removing `threadsafe` if it's present, then try
>> setting the max_concurrent_requests to something higher (like 2x your
>> number of workers):
>> https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests
>>
>>
>> On Fri, Nov 5, 2021 at 1:48 PM Joshua Smith 
>> wrote:
>>
>>> I figured out from the logs (since it prints a message for each worker
>>> it starts) that 3.7 is starting 8 workers by default. So adding the
>>> entrypoint with defaults has no effect.
>>>
>>> However, I did some experiments and I think I can characterize the
>>> difference between 2.7 and 3.7 when it comes to scaling.
>>>
>>> When first landing on my app's page it shows a list of thumbnails. These
>>> come from the datastore, with each thumbnail making a HTTP request (GET
>>> /image?which=1, GET /image?which=2, etc.). If we:
>>>
>>> 1. Query for all of these at once so the server gets hammered with
>>> simultaneous requests:
>>> - 2.7 starts one extra instance, and while that's happening the images
>>> are all served by the instance that already existed; the new instance never
>>> actually serves any of those image requests.
>>> - 3.7 starts a lot of instances; it dumps a bunch of requests to each of
>>> them (one per worker thread, I presume), all of which are going to take 5
>>> seconds because that's how long it takes each instance to start.
>>>
>>> The net result is the user of the 2.7 app sees all the images load right
>>> away, while the 3.7 app gets them over the course of 15 seconds or so, and
>>> sometimes there are timeout errors.
>>>
>>> 2. Query for these with a delay that's long enough to avoid overlapping
>>> requests:
>>> - Neither starts extra instances.
>>>
>>> The user sees all the images load pretty quick, but not quite as quick
>>> as 2.7 with the first approach.
>>>
>>> My conclusion is that the change is that the old scheduler wouldn't use
>>> an instance until it was running, whereas the new scheduler queues up
>>> requests for instances that are still starting up. The old scheduler
>>> doesn't start a new instance until the one it started most recently is
>>> running, whereas the new scheduler starts a new instance every time the
>>> request queue fills up for existing instances.
>>>
>>> If you can serve a burst of requests in the time it take an instance to
>>> start, the old scheduler will get all those served. But the new scheduler
>>> will choke and perform very badly.
>>>
>>> I'm not saying that the old scheduler was better. It just handled this
>>> 

Re: [google-appengine] Re: When will GAE Python 2.7 be shutdown

[wesley chun]

Hi again,

   1. *dev_appserver*: I replied in my other email that the tool currently
   is available for Python 3 but still requires some components from 2.7 to
   function properly and suggested you can file an external bug to make it
   fully functional in 3.x.
   2. *bundled services*: Google has made available as many of the bundled
   services as possible, but unfortunately there's currently no plan on making
   search available in the next generation runtimes. It is currently
   recommended to migrate to Elastic Search using a Compute Engine VM
   

as
   recommended here
   
.
   The goal is not "to move people away from App Engine," but to make
   developers' apps more portable, so they can take their apps elsewhere if
   App Engine isn't the right fit for their needs (removing the notion of
   "vendor lock-in") by encouraging developers to migrate away from those
   legacy bundled services. They are strickly a migration tool to ease
   upgrading to Python 3 and will not receive any new features... all of the
   innovation is going into any standalone Cloud equivalent services.
   3. *2.x support*: As Alejandro alluded to, there are no current plans to
   shutdown the Python 2 runtime, even with the sunset of Python 2
   , older Java, PHP 5, and Go
   versions older than 1.11, by their respective communities. Google Cloud has
   assured users by expressing continued long-term support of these legacy
   runtimes
   ,
   including maintaining the Python 2 runtime
   . So while there is
   no requirement for users to migrate to Python 3 at this time, developers
   like you are expressing interest in updating their applications to the
   latest language releases.

Best regards,
--Wesley
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"A computer never does what you want... only what you tell it."
wesley chun :: @wescpy  :: Software
Architect & Engineer
Developer Advocate at Google
 by day; at
night: Core Python 



On Fri, Jan 7, 2022 at 11:38 PM Sajal Gupta  wrote:

> Thanks Alejandro.
>
> *dev_appserver*: The challenge seems to be that it needs python 2
> installation for it to support python 3.x. So, looks like dev_appserver is
> still not "native" to python 3.x.And possibly due to this, I am not able to
> run dev_appserver in debugger mode for python 3.x applications.  That is
> what I meant about dev_appserver not being available fully for python 3.x.
>
> *Bundled services: *Yes, I looks like now memcache and task queue is
> being made available for python 3.x and these seem to be in beta as of now.
> However, from what I understand, *search * is not available as yet unless
> there is a concrete plan to make it available for 3.x. The current
> suggestion for search seems to be host our own on Kubernetes, which defeats
> the whole purpose of someone using Google App Engine - which hides all the
> infra from the developers.
>
> So, unless, the unstated intent of Google team is to move people away from
> App Engine eventually, current, proposed migration path is not helping.
>
> *Notice of termination of python 2.x*: Thank you. This helps.
>
> Regards
>
> Sajal
> On Friday, January 7, 2022 at 9:24:48 PM UTC+5:30 Alejandro L wrote:
>
>> Hello,
>>
>> Thank you for your message.
>>
>> As you mention, Python has ended support for versions 2.x, and we do
>> encourage our users to use a 3.x version. However, I would like to answer
>> all your questions, point by point:
>>
>> 0 - Dev_appserver does indeed have support for Python 3, which you can
>> find here
>> 
>> [1].
>>
>> 1 - GAE still supports many 2.x bundled services in the Python 3 runtime,
>> including memcache and task queue. You can find more information about this
>> in this paragraph
>> 
>> [2] from the migrate to Python 3 page, and in this link
>> 
>> [3]. However, currently, the recommendation is to consider migrating to 
>> unbundled
>> Google Cloud products
>> 
>> [4]. There is a feature request about this which you can follow in this
>> link  [5].
>>
>> 2 - Currently we do not have any information about ending Python 2.x
>> support.
>>
>> 3 - As mentioned in the Google 

Re: [google-appengine] Debugger in Python 3

[wesley chun]

Hi, there are several issues at play here:

1. *missing pdb section in the 3.x docs*: this section may be missing
possibly because it doesn't work in Python 3 or hasn't been tested or
validated. The best thing to do here is to go to the Python 3 page you
linked to, scroll down to the bottom, and click the blue "Send feedback"
button, then specify the Python 2 page PDB section link like you did in
your email and ask why this section is missing from *this* (Python 3) page.

2. *dev_appserver.py running on Python 3 still requires the presence of a
2.7 interpreter*: Look specifically at the blue sidebar on this page:
https://cloud.google.com/appengine/docs/standard/python3/tools/using-local-server#running_the_local_development_server
which tells you how to specify the location of your Python 3 interpreter.
While there is an internal bug created against removing final 2.x
dependencies (for a purely 3.x version), I don't see an equivalent external
bug for an App Engine dev_appserver issue
 yet,
so if you want there to be a public record tracking this work, please file
a new bug in that same component

.

Depending on how you feel, you can take either (or both) of the above
actions.

Best regards,
--Wesley

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"A computer never does what you want... only what you tell it."
wesley chun :: @wescpy  :: Software
Architect & Engineer
Developer Advocate at Google
 by day; at
night: Core Python 



On Sat, Jan 8, 2022 at 1:17 AM Sajal Gupta  wrote:

> Hello,
>
> I am trying to use debugger in python 3 in localhost - using
> dev_appserver. While I find documentation for python debugger usage in
> python 2.7 documentation, there is no similar section for python 3.
>
> See the url below for python 2:
> https://cloud.google.com/appengine/docs/standard/python/tools/using-local-server#debugging_with_pdb
>
> But corresponding section for python 3 is missing:
> https://cloud.google.com/appengine/docs/standard/python3/tools/using-local-server
>
> IDE's own debugger doesn't seem to work - I am using pycharm. Trying to
> use it gives the following error:
> ===
> /usr/bin/python2.7 "/Applications/PyCharm
> CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --multiproc
> --qt-support=auto --client 127.0.0.1 --port 57002 --file
> /Users/sajal/google-cloud-sdk/bin/dev_appserver.py
> --runtime_python_path=/usr/bin/python3 --support_datastore_emulator=yes
> Connected to pydev debugger (build 213.6461.77)
>
> *ERROR: (dev_appserver) python2: command not found*
> Process finished with exit code 1
> 
>
> Could you please advise.
>
> Thank you.
>
> Sajal
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/17f597c9-ce21-4a28-a6d7-bf8e26b5d935n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAB6eaA53uPMNzgTLBFxN7f%3DRtQcB8yU4UCztjxhHd2EqyMNFzQ%40mail.gmail.com.

[google-appengine] Debugger in Python 3

[Sajal Gupta]

Hello,

I am trying to use debugger in python 3 in localhost - using dev_appserver. 
While I find documentation for python debugger usage in python 2.7 
documentation, there is no similar section for python 3.

See the url below for python 2: 
https://cloud.google.com/appengine/docs/standard/python/tools/using-local-server#debugging_with_pdb

But corresponding section for python 3 is missing: 
https://cloud.google.com/appengine/docs/standard/python3/tools/using-local-server

IDE's own debugger doesn't seem to work - I am using pycharm. Trying to use 
it gives the following error:
===
/usr/bin/python2.7 "/Applications/PyCharm 
CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --multiproc 
--qt-support=auto --client 127.0.0.1 --port 57002 --file 
/Users/sajal/google-cloud-sdk/bin/dev_appserver.py 
--runtime_python_path=/usr/bin/python3 --support_datastore_emulator=yes 
Connected to pydev debugger (build 213.6461.77)

*ERROR: (dev_appserver) python2: command not found*
Process finished with exit code 1


Could you please advise.

Thank you.

Sajal

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/17f597c9-ce21-4a28-a6d7-bf8e26b5d935n%40googlegroups.com.