[google-appengine] AppEngine standard memory error

[Paolo Conte]

Hello everyone,
in my GAE standard java8 project, I sporadically see in the logs the 
following error message:

While handling this request, the process that handled this request was 
found to be using too much memory and was terminated. This is likely to 
cause a new process to be used for the next request to your application. If 
you see this message frequently, you may have a memory leak in your 
application.

Being a Java project, the heap space is capped to 128MB and, as confirmed 
by tests, a Java OutOfMemoryError would be logged, so it doesn't seem true 
that this error may be caused by a memory leak in my application.
Also, even if the heap size is 128MB, in the instances dashboard I see that 
each instance consumes from 250MB up to 600MB, but what is this memory used 
for? and if that is causing the error, do I have any control over it?

Thanks for any insight

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/f46ff33d-4c1f-45ed-87dc-45fedcb0e77e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] AppEngine standard memory error

[Paolo Conte]

Hello everyone,
in my GAE standard java8 project, I sporadically see the error:

While handling this request, the process that handled this request was 
found to be using too much memory and was terminated. This is likely to 
cause a new process to be used for the next request to your application. If 
you see this message frequently, you may have a memory leak in your 
application.

Being a java project, the heap space is limited at 128mb and, as confirmed 
by tests, an OutOfMemoryError would be logged, so it doesn't seem to be 
true that the cause may be memory leak in the application.
Also, despite the heap size being 128mb, the instances dashboard shows a 
memory usage that goes from 250m up to 600m, but what is that memory used 
for? and if that's the cause of the memory issue, do I have any control on 
it or it's a platform bug?

Thanks for any insight

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9d98e147-9bbd-4544-91f2-2850e0ed2338%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: AppEngine standard memory error

[Paolo Conte]

Hi Kenworth,
thanks for your reply, I've submitted a new 
issue: https://issuetracker.google.com/issues/80443146

Best regards

Il giorno martedì 29 maggio 2018 23:24:57 UTC+2, Kenworth (Google Cloud 
Platform) ha scritto:
>
> Hi Paolo, a thread like this is off-topic for Google Groups. As it may 
> potentially be a defect on the platform and most likely lead to a specific 
> situation which may require some exchange of project information to 
> reproduce the behavior you described, I highly encourage you to submit a 
> defect report  as 
> described in this article 
>  so that proper 
> attention and weight will be given to it. We monitor that issue tracker 
> closely.
>
> We look forward to this issue report.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9624a2a7-165e-4d60-b729-5d374c95972a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] AppEngie OUTAGE

[Paolo Conte]

Hello,
suddently most requests return error 500, log shows "The request failed 
because the instance could not start successfully"
Project is in europe-west

Anyone else?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/b72c9e22-e43b-4891-a551-2bb3db946288%40googlegroups.com.

[google-appengine] Re: AppEngie OUTAGE

[Paolo Conte]

Thanks Mark. I'm trying to use manual scaling to see if it improves

On Tuesday, October 8, 2019 at 10:02:41 AM UTC+2, mark negus wrote:
>
> https://status.cloud.google.com/?_ga=2.187839338.-379351283.1560761695
>
> On Tuesday, October 8, 2019 at 8:57:06 AM UTC+1, mark negus wrote:
>>
>> Yes, we are having problems for the last hour in europe-west. Mainly 
>> 500s. Looks like the auto-scaling is having issues.
>>
>> On Tuesday, October 8, 2019 at 8:48:59 AM UTC+1, Paolo Conte wrote:
>>>
>>> Hello,
>>> suddently most requests return error 500, log shows "The request failed 
>>> because the instance could not start successfully"
>>> Project is in europe-west
>>>
>>> Anyone else?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/63bc0117-1ac1-4fb4-852c-c39685f95ff8%40googlegroups.com.

[google-appengine] Re: AppEngie OUTAGE

[Paolo Conte]

Manual scaling seems to also have trouble starting instances. For now I 
kept autoscaling with a high minimum instances number and the error rate is 
almost zero at the moment 

On Tuesday, October 8, 2019 at 10:16:57 AM UTC+2, mark negus wrote:
>
> Hi Paolo,
> Let me know how you get on. We were manual scaling for ages as we had 
> terrible issues with auto-scaling a while back. Once it randomly launched 
> 40 instances and burnt through our daily budget in a few hours. Its been ok 
> though this last year.
>
> On Tuesday, October 8, 2019 at 9:10:06 AM UTC+1, Paolo Conte wrote:
>>
>> Thanks Mark. I'm trying to use manual scaling to see if it improves
>>
>> On Tuesday, October 8, 2019 at 10:02:41 AM UTC+2, mark negus wrote:
>>>
>>> https://status.cloud.google.com/?_ga=2.187839338.-379351283.1560761695
>>>
>>> On Tuesday, October 8, 2019 at 8:57:06 AM UTC+1, mark negus wrote:
>>>>
>>>> Yes, we are having problems for the last hour in europe-west. Mainly 
>>>> 500s. Looks like the auto-scaling is having issues.
>>>>
>>>> On Tuesday, October 8, 2019 at 8:48:59 AM UTC+1, Paolo Conte wrote:
>>>>>
>>>>> Hello,
>>>>> suddently most requests return error 500, log shows "The request 
>>>>> failed because the instance could not start successfully"
>>>>> Project is in europe-west
>>>>>
>>>>> Anyone else?
>>>>>
>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/31272cb1-c766-4d4c-a1f2-cd0fc1f5787a%40googlegroups.com.

[google-appengine] Re: AppEngie OUTAGE

[Paolo Conte]

My settings are below. By looking at the logs of /_ah/warmup to me it seems 
that some instances start and some don't, so you need some luck to get a 
good amount of working instances


F1



35
35

3
5
500ms
5 
0.9
0.8



On Tuesday, October 8, 2019 at 11:05:13 AM UTC+2, mark negus wrote:
>
> This app version won't even start at all. What else did you do?
> java8, F4
>
> automatic_scaling:
>   min_idle_instances: 3
>   max_idle_instances: automatic
>   min_pending_latency: automatic
>   max_pending_latency: automatic
>   max_instances: 4
>
> On Tuesday, October 8, 2019 at 9:30:26 AM UTC+1, Paolo Conte wrote:
>>
>> Manual scaling seems to also have trouble starting instances. For now I 
>> kept autoscaling with a high minimum instances number and the error rate is 
>> almost zero at the moment 
>>
>> On Tuesday, October 8, 2019 at 10:16:57 AM UTC+2, mark negus wrote:
>>>
>>> Hi Paolo,
>>> Let me know how you get on. We were manual scaling for ages as we had 
>>> terrible issues with auto-scaling a while back. Once it randomly launched 
>>> 40 instances and burnt through our daily budget in a few hours. Its been ok 
>>> though this last year.
>>>
>>> On Tuesday, October 8, 2019 at 9:10:06 AM UTC+1, Paolo Conte wrote:
>>>>
>>>> Thanks Mark. I'm trying to use manual scaling to see if it improves
>>>>
>>>> On Tuesday, October 8, 2019 at 10:02:41 AM UTC+2, mark negus wrote:
>>>>>
>>>>> https://status.cloud.google.com/?_ga=2.187839338.-379351283.1560761695
>>>>>
>>>>> On Tuesday, October 8, 2019 at 8:57:06 AM UTC+1, mark negus wrote:
>>>>>>
>>>>>> Yes, we are having problems for the last hour in europe-west. Mainly 
>>>>>> 500s. Looks like the auto-scaling is having issues.
>>>>>>
>>>>>> On Tuesday, October 8, 2019 at 8:48:59 AM UTC+1, Paolo Conte wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>> suddently most requests return error 500, log shows "The request 
>>>>>>> failed because the instance could not start successfully"
>>>>>>> Project is in europe-west
>>>>>>>
>>>>>>> Anyone else?
>>>>>>>
>>>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/eaa9c19d-db3a-478f-a38b-3d4c105055cd%40googlegroups.com.

[google-appengine] Re: AppEngie OUTAGE

[Paolo Conte]

Looks normal now

On Tuesday, October 8, 2019 at 11:36:11 AM UTC+2, mark negus wrote:
>
> Thanks, thats useful.
> Yes I see a few of these  A problem was encountered with the process that 
> handled this request, causing it to exit. This is likely to cause a new 
> process to be used for the next request to your application. (Error code 
> 121)  
> during warmup calls but just can't get enough started.
>
> Just had 15mins without an error and now they are back again. Sigh.
>
> On Tuesday, October 8, 2019 at 10:17:42 AM UTC+1, Paolo Conte wrote:
>>
>> My settings are below. By looking at the logs of /_ah/warmup to me it 
>> seems that some instances start and some don't, so you need some luck to 
>> get a good amount of working instances
>>
>>
>> F1
>>
>> 
>>
>> 35
>> 35
>>
>> 3
>> 5
>> 500ms
>> 5 
>> 0.9
>> 0.8
>> 
>>
>>
>> On Tuesday, October 8, 2019 at 11:05:13 AM UTC+2, mark negus wrote:
>>>
>>> This app version won't even start at all. What else did you do?
>>> java8, F4
>>>
>>> automatic_scaling:
>>>   min_idle_instances: 3
>>>   max_idle_instances: automatic
>>>   min_pending_latency: automatic
>>>   max_pending_latency: automatic
>>>   max_instances: 4
>>>
>>> On Tuesday, October 8, 2019 at 9:30:26 AM UTC+1, Paolo Conte wrote:
>>>>
>>>> Manual scaling seems to also have trouble starting instances. For now I 
>>>> kept autoscaling with a high minimum instances number and the error rate 
>>>> is 
>>>> almost zero at the moment 
>>>>
>>>> On Tuesday, October 8, 2019 at 10:16:57 AM UTC+2, mark negus wrote:
>>>>>
>>>>> Hi Paolo,
>>>>> Let me know how you get on. We were manual scaling for ages as we had 
>>>>> terrible issues with auto-scaling a while back. Once it randomly launched 
>>>>> 40 instances and burnt through our daily budget in a few hours. Its been 
>>>>> ok 
>>>>> though this last year.
>>>>>
>>>>> On Tuesday, October 8, 2019 at 9:10:06 AM UTC+1, Paolo Conte wrote:
>>>>>>
>>>>>> Thanks Mark. I'm trying to use manual scaling to see if it improves
>>>>>>
>>>>>> On Tuesday, October 8, 2019 at 10:02:41 AM UTC+2, mark negus wrote:
>>>>>>>
>>>>>>>
>>>>>>> https://status.cloud.google.com/?_ga=2.187839338.-379351283.1560761695
>>>>>>>
>>>>>>> On Tuesday, October 8, 2019 at 8:57:06 AM UTC+1, mark negus wrote:
>>>>>>>>
>>>>>>>> Yes, we are having problems for the last hour in europe-west. 
>>>>>>>> Mainly 500s. Looks like the auto-scaling is having issues.
>>>>>>>>
>>>>>>>> On Tuesday, October 8, 2019 at 8:48:59 AM UTC+1, Paolo Conte wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>> suddently most requests return error 500, log shows "The request 
>>>>>>>>> failed because the instance could not start successfully"
>>>>>>>>> Project is in europe-west
>>>>>>>>>
>>>>>>>>> Anyone else?
>>>>>>>>>
>>>>>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/1a75efa0-0e27-4579-b1a5-3c3bfbaea427%40googlegroups.com.

[google-appengine] Correlating Logs with Requests on Java 11 AppEngine

[Paolo Conte]

Hello,
trying to correlate logs with requests by following the documentation at 
https://cloud.google.com/appengine/docs/standard/java11/writing-application-logs
Here it says: *The highest severity from the "child" log entries does not 
automatically apply to the top-level entry. If that behavior is desired, 
manually set the highest severity in the top-level entry.*
But how to "manually set the highest severity in the top-level entry" it 
they are automatically created? Is this even possible? 

Also, correlated logs don't seems to show up on the first request of the 
instance, on the web console. Is it a bug of the console?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/cf57c0d8-289b-4b4d-82b5-031cc423dd2a%40googlegroups.com.

[google-appengine] Re: Correlating Logs with Requests on Java 11 AppEngine

[Paolo Conte]

Hi Victor, 
what I'm asking is not really a how-to, but pointing out 1. what seems to 
be an error in the documentation, but not clear, and 2. what seems to be a 
bug in the the log viewer.

Thanks

On Friday, 28 February 2020 13:57:22 UTC+1, Victor (Cloud Support) wrote:
>
> Hi there,
>
> I can see this thread is more of a technical/how-to question. 
>
>
> Since it is not getting much attention here, I suggest you try to post it 
> on Stackoverflow <https://stackoverflow.com/>, as these questions usually 
> have greater visibility there.
>
> If you decide to post it there, please remember to tag the question with 
> at least the following tags:
>
> *google-app-engine*
>
> *stackdriver*
>
>
> Regards,
>
> On Wednesday, February 26, 2020 at 6:42:36 PM UTC+1, Paolo Conte wrote:
>>
>> Hello,
>> trying to correlate logs with requests by following the documentation at 
>> https://cloud.google.com/appengine/docs/standard/java11/writing-application-logs
>> Here it says: *The highest severity from the "child" log entries does 
>> not automatically apply to the top-level entry. If that behavior is 
>> desired, manually set the highest severity in the top-level entry.*
>> But how to "manually set the highest severity in the top-level entry" it 
>> they are automatically created? Is this even possible? 
>>
>> Also, correlated logs don't seems to show up on the first request of the 
>> instance, on the web console. Is it a bug of the console?
>>
>> Thanks
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/86a0f4a3-6b60-487a-99d5-87cd1e10b596%40googlegroups.com.

[google-appengine] Re: Correlating Logs with Requests on Java 11 AppEngine

[Paolo Conte]

Thanks Jason

On Friday, 28 February 2020 20:24:23 UTC+1, Jason Collins wrote:
>
> "If that behavior is desired, manually set the highest severity in the 
> top-level entry."
>
> You're correct, this isn't possible. I'll have the docs updated.
>
>
> "correlated logs don't seems to show up on the first request of the 
> instance"
>
> This is a known issue that we are working on fixing (it's due to the 
> "embedded" log with the cold start notification; this defeats the 
> correlation logic).
>
>
> On Friday, 28 February 2020 05:31:06 UTC-8, Paolo Conte wrote:
>>
>> Hi Victor, 
>> what I'm asking is not really a how-to, but pointing out 1. what seems to 
>> be an error in the documentation, but not clear, and 2. what seems to be a 
>> bug in the the log viewer.
>>
>> Thanks
>>
>> On Friday, 28 February 2020 13:57:22 UTC+1, Victor (Cloud Support) wrote:
>>>
>>> Hi there,
>>>
>>> I can see this thread is more of a technical/how-to question. 
>>>
>>>
>>> Since it is not getting much attention here, I suggest you try to post 
>>> it on Stackoverflow <https://stackoverflow.com/>, as these questions 
>>> usually have greater visibility there.
>>>
>>> If you decide to post it there, please remember to tag the question with 
>>> at least the following tags:
>>>
>>> *google-app-engine*
>>>
>>> *stackdriver*
>>>
>>>
>>> Regards,
>>>
>>> On Wednesday, February 26, 2020 at 6:42:36 PM UTC+1, Paolo Conte wrote:
>>>>
>>>> Hello,
>>>> trying to correlate logs with requests by following the documentation 
>>>> at 
>>>> https://cloud.google.com/appengine/docs/standard/java11/writing-application-logs
>>>> Here it says: *The highest severity from the "child" log entries does 
>>>> not automatically apply to the top-level entry. If that behavior is 
>>>> desired, manually set the highest severity in the top-level entry.*
>>>> But how to "manually set the highest severity in the top-level entry" 
>>>> it they are automatically created? Is this even possible? 
>>>>
>>>> Also, correlated logs don't seems to show up on the first request of 
>>>> the instance, on the web console. Is it a bug of the console?
>>>>
>>>> Thanks
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/c277ad48-f38f-43b2-97f0-50d150151c74%40googlegroups.com.

[google-appengine] ghs.googlehosted.com DNS failing?!

[Paolo Conte]

Hello,
I have a custom domain on App Engine pointing of course to 
ghs.googlehosted.com 

Today a lot of requests are failing every few minutes, from what I see it 
looks like the ghs.googlehosted.com 

 DNS 
is not working correctly (the TTL on that is 5m).

Pointing directly to xx.appspot.com doesn't seem to present issues.

The app is in europe-west and most clients are in Italy

Anyone else?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/34f521a4-5525-4f8f-8c84-61cb0b986197n%40googlegroups.com.

Re: [google-appengine] ghs.googlehosted.com DNS failing?!

[Paolo Conte]

Glad I'm not alone, Stefano 🤞🏻
Hopefully we can get some feedback on this

On Saturday, 31 October 2020 at 08:09:26 UTC+1 Stefano Ciccarelli wrote:

> Here too. Our app is on us-central1 and we have problems with .it domain 
> only, we have .software domains that are working correctly.
>
>
> Il giorno sab 31 ott 2020 alle ore 08:05 Paolo Conte  
> ha scritto:
>
>> Hello,
>> I have a custom domain on App Engine pointing of course to 
>> ghs.googlehosted.com 
>> <https://mxtoolbox.com/SuperTool.aspx?action=dns%3abackend.orariotreni.app&run=toolpage#>
>> Today a lot of requests are failing every few minutes, from what I see it 
>> looks like the ghs.googlehosted.com 
>> <https://mxtoolbox.com/SuperTool.aspx?action=dns%3abackend.orariotreni.app&run=toolpage#>
>>  DNS 
>> is not working correctly (the TTL on that is 5m).
>>
>> Pointing directly to xx.appspot.com doesn't seem to present issues.
>>
>> The app is in europe-west and most clients are in Italy
>>
>> Anyone else?
>>
>> Thanks
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/34f521a4-5525-4f8f-8c84-61cb0b986197n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/google-appengine/34f521a4-5525-4f8f-8c84-61cb0b986197n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> -- 
>
> *Stefano Ciccarelli*
> YAP Division
> / Director
> +39.0546.637765 <+39%200546%20637765>
> stefano.c...@mmbsoftware.it
>
> *M.M.B. s.r.l.*
> via Granarolo, 177/7 - 48018 Faenza (RA) - Italy
> tel. +39.0546.637711 <+39%200546%20637711> - fax +39.0546.46077 
> <+39%200546%2046077>
> www.mmbsoftware.it - in...@mmbsoftware.it
>
> Le informazioni contenute in questa comunicazione sono riservate e 
> destinate esclusivamente alla/e persona/e o all'ente sopra indicati. E' 
> vietato ai soggetti diversi dai destinatari qualsiasi uso, copia, 
> diffusione di quanto in esso contenuto sia ai sensi dell'art. 616 c.p., sia 
> ai sensi del Regolamento UE 679/2016. Se questa comunicazione Vi e' 
> pervenuta per errore, Vi preghiamo di rispondere a questa e-mail e 
> successivamente cancellarla dal Vostro sistema.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/575b603c-8578-4d44-be47-88529ccd5eebn%40googlegroups.com.