Thanks for the suggestions, Phillip.
Yes the memory is as expected. This was never in doubt, the out-of-memory
message says both usage and the quota. The quota matches GAE_MEMORY_MB. The
problem is why the usage is so high, not that the quota isn't changing.
Thank you for the command to use. Google's docs didn't make it clear that
this is how you control parallelism. After adding gunicorn to
requirements.txt, the following now helps:
entrypoint: gunicorn main:app --threads 4 --workers 1 -c /config/gunicorn.py
It only solves half the problem. This now no longer crashes an F4. It does
still crash an F2, with 270 MB usage. Locally it's still only 115MB.
On Monday, September 17, 2018 at 1:25:55 PM UTC-7, Phillip Pearson wrote:
>
> Hi Adam,
>
> Here's a few ideas for diagnosis:
>
> 1. Can you try printing or logging repr(os.environ) and checking for the
> GAE_MEMORY_MB environment variable? I just tried deploying a test app with
> instance_class: F4 and service: xyz, and print(repr(os.environ)) gives me
> something like this:
>
> environ({'GAE_MEMORY_MB': '512', 'GAE_INSTANCE': ...etc etc...})
>
> The python37 runtime *should* start one worker if GAE_MEMORY_MB <= 128,
> two if <= 256, four if <= 512, otherwise eight.
>
> If you see 'GAE_MEMORY_MB': '512' in there but anything other than four
> workers starting up, I'm extremely interested in figuring out why :)
> Likewise if you see anything other than 512 for an F4 instance.
>
> 2. Check that you're looking at the logs for the correct service -- make
> sure you see "GAE Application, xyz" and not "GAE Application, Default
> Service" or just "GAE Application" (which combines logs from all services)
> in the logs viewer.
>
> 3. Once you've done those two, if nothing sticks out, you can manually
> configure the worker count by adding a line like this to app.yaml:
>
> entrypoint: gunicorn main:app --workers 2 -c /config/gunicorn.py
>
> This will start up two workers. You can also configure the number of
> threads per worker:
>
> entrypoint: gunicorn main:app --threads 8 --workers 2 -c
> /config/gunicorn.py
>
> (The default if you just using 'gunicorn main:app -c /config/gunicorn.py'
> is four workers with four threads each.)
>
> Cheers,
> Phil
>
> On Monday, September 17, 2018 at 10:30:06 AM UTC-7, Adam Lugowski wrote:
>>
>> Hi Rahul,
>>
>> I'm not manually configuring workers. Happy to learn of how to do that.
>> My app.yaml is simple:
>>
>> runtime: python37
>> service : xyz
>>
>> inbound_services:
>> - warmup
>>
>> instance_class: F4
>>
>> handlers:
>> ....
>>
>> On Sunday, September 16, 2018 at 6:59:38 AM UTC-7, rah...@google.com
>> wrote:
>>>
>>> Hello Adam,
>>> Could you paste your app.yaml? Are you explicitly configuring the
>>> number of workers?
>>> Thanks,
>>> ~Rahul.
>>>
>>>
>>> On Friday, September 14, 2018 at 3:47:41 PM UTC-7, Adam Lugowski wrote:
>>>>
>>>> The only theory I have is informed by these log messages on startup:
>>>>
>>>> A [2018-09-14 19:28:38 +0000] [1] [INFO] Starting gunicorn 19.9.0
>>>> A [2018-09-14 19:28:38 +0000] [1] [INFO] Listening at:
>>>> http://0.0.0.0:8080 (1)
>>>> A [2018-09-14 19:28:38 +0000] [1] [INFO] Using worker: threads
>>>> A [2018-09-14 19:28:38 +0000] [8] [INFO] Booting worker with pid: 8
>>>> A [2018-09-14 19:28:38 +0000] [9] [INFO] Booting worker with pid: 9
>>>> A [2018-09-14 19:28:38 +0000] [10] [INFO] Booting worker with pid: 10
>>>> A [2018-09-14 19:28:38 +0000] [11] [INFO] Booting worker with pid: 11
>>>> A [2018-09-14 19:28:38 +0000] [12] [INFO] Booting worker with pid: 12
>>>> A [2018-09-14 19:28:38 +0000] [13] [INFO] Booting worker with pid: 13
>>>> A [2018-09-14 19:28:39 +0000] [14] [INFO] Booting worker with pid: 14
>>>> A [2018-09-14 19:28:39 +0000] [15] [INFO] Booting worker with pid: 15
>>>>
>>>>
>>>> It looks like multiple workers are started, so maybe the fixed memory
>>>> overhead is just duplicated by each worker.
>>>> This is also supported by the fact that some of my requests load
>>>> matplotlib. Locally this makes a slow first request then fast on
>>>> subsequent
>>>> ones. On GAE F4 the first few are slow, then it stays fast, as if multiple
>>>> processes have to be warmed up.
>>>> If this is true, then going to a larger instance is poor advice;
>>>> doubling memory size also doubles CPU count which doubles fixed overhead
>>>> and we're back at square one.
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/1c80dd99-9b67-41c2-aec2-f1c3492b6a7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
