Yes, multiprocessing is not supported in the Google App Engine standard environment. The information I'm giving you pertains to the App Engine flexible environment. I'm sorry if there was any confusion, however in your second response you specifically stated "i can't use multiprocess in Python flexible appengine."
On Tuesday, September 6, 2016 at 2:14:25 PM UTC-4, Jungho Ahn wrote: > > Yes, there are no core restrictions, but multiprocessing.Process() is a > fake thread. > > I received an answer from Google like > 'For security reasons, multiprocessing is not supported in App Engine > Standard Environment. More specifically, the multiprocess module is > replaced with a stub that provides limited compatibility (hence the "fake" > PIDs you are seeing), but does not actually spawn processes.' > > AFAIK, python threads cannot utilize all CPU cores due to GIL. So, I'm > trying to fork processes for computationally heavy jobs. > > On Tue, Sep 6, 2016 at 10:58 AM, 'Adam (Cloud Platform Support)' via > Google App Engine <google-appengine@googlegroups.com> wrote: > >> There's no special 'App Engine Python' on the 'python' runtime; it's >> just standard Python running in a Docker container. There are no core >> restrictions imposed by App Engine or by Docker. >> >> In your app, are you checking the number of cores available using >> 'multiprocessing.cpu_count()'? When you say 'multiprocessing.Process() >> was not a real process' can you clarify what you mean, and describe how are >> you determining this (eg. pool._processes)? >> >> You can also test the number of cores available to your deployed app from >> the Cloud Console by going to 'Compute' -> 'VM instances' -> 'SSH' to SSH >> to your instance in Debug mode, then from the shell type this to connect to >> your app's Docker container: >> >> $ sudo docker exec -it gaeapp /bin/bash >> >> Once there you can run 'nproc' to see the number of compute units or >> 'top' to see the cpu usage of each process eg. >> >> root@d0f61e0950fe:/home/vmagent/app# nproc >> 2 >> >> root@d0f61e0950fe:/home/vmagent/app# TERM=xterm top >> ... >> >> In my case I have 2 cores available, and my app.yaml looks like the >> following: >> >> runtime: python >> vm: true >> entrypoint: gunicorn -w 4 -b :$PORT main:app >> >> resources: >> cpu: 2.0 >> memory_gb: 2.6 >> disk_size_gb: 10 >> >> On Tuesday, September 6, 2016 at 1:14:48 PM UTC-4, Jungho Ahn wrote: >>> >>> You mean the python in App Engine supports multiprocessing? >>> multiprocessing.Process() was not a real process. >>> >>> >>> On Tue, Sep 6, 2016 at 9:57 AM, 'Adam (Cloud Platform Support)' via >>> Google App Engine <google-appengine@googlegroups.com> wrote: >>> >>>> Could you post an example of the code you're using to test, and how >>>> you're checking the CPU utilization on the target instance? >>>> >>>> >>>> On Friday, September 2, 2016 at 6:01:03 PM UTC-4, Jungho Ahn wrote: >>>>> >>>>> Yes, right. More than 2 cores. >>>>> >>>>> On Fri, Sep 2, 2016 at 2:48 PM, 'Adam (Cloud Platform Support)' via >>>>> Google App Engine <google-appengine@googlegroups.com> wrote: >>>>> >>>>>> Possibly a silly question, but are you specifying at least 2 cores in >>>>>> your 'resources' configuration? The normal 'python' runtime should not >>>>>> be >>>>>> sandboxed and allow you to utilize more than one core. >>>>>> >>>>>> >>>>>> On Monday, August 29, 2016 at 3:52:33 PM UTC-4, Jungho Ahn wrote: >>>>>>> >>>>>>> I'm using python: >>>>>>> >>>>>>> runtime: python >>>>>>> runtime_config: >>>>>>> python_version: 2 >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 29, 2016 at 12:32 PM, 'Adam (Cloud Platform Support)' >>>>>>> via Google App Engine <google-appengine@googlegroups.com> wrote: >>>>>>> >>>>>>>> Which Python runtime are you using? (the 'runtime >>>>>>>> <https://cloud.google.com/appengine/docs/flexible/python/configuring-your-app-with-app-yaml#general>' >>>>>>>> >>>>>>>> line in your app.yaml) >>>>>>>> >>>>>>>> On Saturday, August 27, 2016 at 7:36:31 PM UTC-4, Jungho Ahn wrote: >>>>>>>>> >>>>>>>>> On Aug 27, 2016 4:16 PM, "'Adam (Cloud Platform Support)' via >>>>>>>>> Google App Engine" <google-appengine@googlegroups.com> wrote: >>>>>>>>> > >>>>>>>>> > Multiple cores are not available to App Engine standard runtime >>>>>>>>> instances. If you need access to multiple cores on the same instance, >>>>>>>>> consider using the App Engine flexible runtime, which uses Compute >>>>>>>>> Engine >>>>>>>>> as the underlying VM and allows you to configure the number of cores >>>>>>>>> available. >>>>>>>>> > >>>>>>>>> >>>>>>>>> Yes, but i can't use multiprocess in Python flexible appengine. >>>>>>>>> When I created a process, it was a fake process which is a thread >>>>>>>>> like in >>>>>>>>> appengine. >>>>>>>>> >>>>>>>>> > >>>>>>>>> > On Wednesday, August 24, 2016 at 4:27:17 PM UTC-4, Jungho Ahn >>>>>>>>> wrote: >>>>>>>>> >> >>>>>>>>> >> Hello, >>>>>>>>> >> >>>>>>>>> >> Is it allowed to use python multiprocessing in appengine? >>>>>>>>> >> I tried, but it looks like only one core is utilized. >>>>>>>>> >> >>>>>>>>> >> Thanks, >>>>>>>>> > >>>>>>>>> > -- >>>>>>>>> > You received this message because you are subscribed to a topic >>>>>>>>> in the Google Groups "Google App Engine" group. >>>>>>>>> > To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/google-appengine/D6z_WoLsBH8/unsubscribe >>>>>>>>> . >>>>>>>>> > To unsubscribe from this group and all its topics, send an email >>>>>>>>> to google-appengine+unsubscr...@googlegroups.com. >>>>>>>>> > To post to this group, send email to >>>>>>>>> google-appengine@googlegroups.com. >>>>>>>>> > Visit this group at >>>>>>>>> https://groups.google.com/group/google-appengine. >>>>>>>>> > To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/google-appengine/88dafadf-a328-423e-8ddc-7c4f585f88a9%40googlegroups.com >>>>>>>>> . >>>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "Google App Engine" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/google-appengine/D6z_WoLsBH8/unsubscribe >>>>>>>> . >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> google-appengine+unsubscr...@googlegroups.com. >>>>>>>> To post to this group, send email to >>>>>>>> google-appengine@googlegroups.com. >>>>>>>> Visit this group at >>>>>>>> https://groups.google.com/group/google-appengine. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/google-appengine/2fd63c21-f33f-48a2-932d-bd7db89cb56c%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/google-appengine/2fd63c21-f33f-48a2-932d-bd7db89cb56c%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Jungho Ahn >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "Google App Engine" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/google-appengine/D6z_WoLsBH8/unsubscribe >>>>>> . >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> google-appengine+unsubscr...@googlegroups.com. >>>>>> To post to this group, send email to >>>>>> google-appengine@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/google-appengine. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/google-appengine/cc1bc1bd-188d-4490-a4dc-5c5138a68f2e%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/google-appengine/cc1bc1bd-188d-4490-a4dc-5c5138a68f2e%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Jungho Ahn >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Google App Engine" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/google-appengine/D6z_WoLsBH8/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> google-appengine+unsubscr...@googlegroups.com. >>>> To post to this group, send email to google-appengine@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/google-appengine. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/google-appengine/46012019-4c23-4e0f-a854-12449e34c1e7%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/google-appengine/46012019-4c23-4e0f-a854-12449e34c1e7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Jungho Ahn >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Google App Engine" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/google-appengine/D6z_WoLsBH8/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> google-appengine+unsubscr...@googlegroups.com. >> To post to this group, send email to google-appengine@googlegroups.com. >> Visit this group at https://groups.google.com/group/google-appengine. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/8d4d650e-79b2-43ca-843b-02320c64dd4b%40googlegroups.com >> >> <https://groups.google.com/d/msgid/google-appengine/8d4d650e-79b2-43ca-843b-02320c64dd4b%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Jungho Ahn > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/e6f21a22-b0fa-4b55-917a-bf40c881fd24%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
