Re: [google-appengine] Re: Google App Engine TCP Connections

[Bradley Barrows]

Hi,
I found the issue and the solution.

I thought that GCP was killing off the VM/container running the App Engine 
server as the logging and request handling would just STOP right when this 
larger query was made. I was sure it had something to do with the limit on 
the amount of data that an App Engine service could request from another 
service though. However, this is a TCP connection to the MongoDB. And as 
HTTP has a 32MB limit, TCP supposedly as no limit. So for a while we just 
modified the query to return a much smaller result while I continued to 
search for an answer.

The weirdest part was that this HTTP route/App Engine service would work 
most of the time, handling very large queries without issues. The issue 
though is that it needs to run on a cron/Cloud Scheduler at midnight. With 
consistency, working every time. But it was failing some nights for some 
unknown reason. And I couldn't find any exceptions in the logs from the 
code. The issue is that I had somehow missed an error message, probably due 
to our over zealous logging, from Google App Engine itself letting me know 
about memory constraints.

So yesterday we had the same issue. 

Searching through the logs I find that the HTTP request that we were making 
to this App Engine service which does this large query was failing. It was 
scheduled in Scheduler so I could see there in the logs that it errored out.

Looking through the logs I found this gem of a log message:

"Exceeded soft memory limit of 256 MB with 269 MB after servicing 0 
requests total. Consider setting a larger instance class in app.yaml."

It turns out that the VM/container (whatever is being used to host an 
AppEngine service) has a maximum memory limit of 256MB by default. And some 
nights our site would not get many requests or a fresh AppEngine server 
would be spun up and handle the expensive query request on it's own. Free 
to use all 256 MB of memory. Otherwise it was sharing memory with all the 
other "normal" requests that were coming through our service.

So to solve this I just created a Dockerfile, basically the template 
provided by the documentation, created the image, uploaded to the Container 
Registry and created a Cloud Run instance. So now we have routing to a 
service dedicated to handling requests for these expensive requests (memory 
wise) and we can even configure easily what the memory limit is on the 
different Cloud Run services.

So the issue it turns out was that we ran out of RAM in a shared 
environment and the solution was to move the service off to it's own 
hosting solution, Cloud Run. So we could route these expensive requests and 
only these requests to these new Cloud Run instances dedicated to being 
able to handle the load of the query.

Thank you so much for you time, any one who tried to help!

Thanks and hope this is helpful for anyone else,
Brad Barrows

On Wednesday, June 9, 2021 at 8:59:38 AM UTC-7 Bradley Barrows wrote:

> I thought that the original post did not actually send. My browser quit 
> right after hitting send so I re wrote it.
>
> I also found no acknowledgement that the original was received so had no 
> way of verifying if I needed to re type it or not.
>
> Sorry about that.
>
> Thanks
> Brad
>
> On Wed, Jun 9, 2021 at 3:01 AM 'Angel (Google Cloud Platform Support)' via 
> Google App Engine  wrote:
>
>> Hello Brad,
>>
>> I believe this is a similar, if not the same question that has already 
>> been answered here [1]. Please let me know if this is the case or if I 
>> misunderstood your question.
>>
>> Kind regards.
>> __
>> [1] - 
>> https://groups.google.com/g/google-appengine/c/Lj6qaT2lC90/m/TvhGSteRAwAJ
>>
>> On Tuesday, June 8, 2021 at 9:05:27 AM UTC+2 bradeba...@gmail.com wrote:
>>
>>> Hi
>>>
>>> I have a NodeJS App Engine server which is awaiting a MongoDB query 
>>> which, when reaching a certain size, never returns. There is no exception 
>>> thrown from App Engine either. It is as if the process is just killed 
>>> because execution flow and therefore logging from the process making the 
>>> MongoDB query stops.
>>>
>>> If this were a request to a HTTP server I could look at the limits and 
>>> quotas page and see that a maximum of 32MB requests and response size is 
>>> allowed.
>>>
>>> I am unable to find any limits or quotas for TCP connections from an App 
>>> Engine server.
>>>
>>> We temporarily solved the issue by using a limit or projection to 
>>> decrease the amount of data returned by the MongoDB query. When less data 
>>> is sent back to the App Engine server execution continues and the server 
>>> runs as expected.
>&g

Re: [google-appengine] Re: Google App Engine TCP Connections

[Bradley Barrows]

I thought that the original post did not actually send. My browser quit
right after hitting send so I re wrote it.

I also found no acknowledgement that the original was received so had no
way of verifying if I needed to re type it or not.

Sorry about that.

Thanks
Brad

On Wed, Jun 9, 2021 at 3:01 AM 'Angel (Google Cloud Platform Support)' via
Google App Engine  wrote:

> Hello Brad,
>
> I believe this is a similar, if not the same question that has already
> been answered here [1]. Please let me know if this is the case or if I
> misunderstood your question.
>
> Kind regards.
> __
> [1] -
> https://groups.google.com/g/google-appengine/c/Lj6qaT2lC90/m/TvhGSteRAwAJ
>
> On Tuesday, June 8, 2021 at 9:05:27 AM UTC+2 bradeba...@gmail.com wrote:
>
>> Hi
>>
>> I have a NodeJS App Engine server which is awaiting a MongoDB query
>> which, when reaching a certain size, never returns. There is no exception
>> thrown from App Engine either. It is as if the process is just killed
>> because execution flow and therefore logging from the process making the
>> MongoDB query stops.
>>
>> If this were a request to a HTTP server I could look at the limits and
>> quotas page and see that a maximum of 32MB requests and response size is
>> allowed.
>>
>> I am unable to find any limits or quotas for TCP connections from an App
>> Engine server.
>>
>> We temporarily solved the issue by using a limit or projection to
>> decrease the amount of data returned by the MongoDB query. When less data
>> is sent back to the App Engine server execution continues and the server
>> runs as expected.
>>
>> I was wondering if it was possible if at some point if TCP data usage
>> reaches a limit will the App Engine process be killed? Without any logging?
>> Why would the execution of the process handling the response that fires off
>> the MongoDB query stop and no exception or anything further be logged?
>>
>> Thank you
>> Brad
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/qQSy-QQLZjM/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/7e3c5f5b-cc3b-4460-965f-00a20896feb0n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAGJGpkrX_X8dfT77JxgvH6jE2G-3efx-pcPObc%2Bgc3cDsTD%3Dkw%40mail.gmail.com.

Re: [google-appengine] Re: Larger MongoDB Query from App Engine neither succeeding nor failing nor throwing exception

[Bradley Barrows]

This is exactly the article I have been looking for. Thank you so much!

I will respond if I can find any specific information related to this issue
in case anyone else runs into a similar problem in the future as well.

Thank you so much,
Brad

On Tue, Jun 8, 2021 at 4:45 AM 'Angel (Google Cloud Platform Support)' via
Google App Engine  wrote:

> Hello Brad,
>
> to my knowledge there are no direct data limits in App Engine when used
> over TCP. There are, however, some architectural considerations that are
> worth being aware of for Google Cloud Platform. You can find the related
> article in [1]. It details some use cases in which a final user might
> experience latency issues or network inefficiencies. I believe it might
> give you some insight as to what might be happening in your case.
>
> Kind regards.
> __
> [1] -
> https://cloud.google.com/architecture/tcp-optimization-for-network-performance-in-gcp-and-hybrid
>
> On Tuesday, June 8, 2021 at 9:05:34 AM UTC+2 bradeba...@gmail.com wrote:
>
>> Hi
>>
>> This weekend we ran into an issue. A function, using Mongoose (a NodeJS
>> MongoDB library) was making a pretty simple query, which when run locally
>> either by running the query directly or when running the entire Node server
>> directly ran fine.
>>
>> However, when deployed to Google App Engine we noticed that logging was
>> logged out up until we awaited the query to MongoDB and then never
>> continued. No exception was ever logged as well.
>>
>> It is as if the Mongo query was taking too long and App Engine just
>> killed the Node process all together.
>>
>> The server did return a 500 error. I believe it was a 503 specifically.
>>
>> We ended up realizing by using limit or projection we could decrease the
>> response size of the query and get a result and the execution of the NodeJS
>> application would continue.
>>
>> Now if Mongo was an HTTP server I would see that there is a 32MB maximum
>> request size and be able to find that there are only so many records I can
>> query at once. However, since this is a TCP connection to the Mongo server
>> I cannot find any quotas or limitations on how much data a TCP connection
>> from an App Engine server can send or receive.
>>
>> As we know that decreasing the size of the response from MongoDB fixed
>> our issue, I believe it is safe to say the issue is related to the amount
>> of data sent/received or more specifically received over TCP when
>> communicating with an App Engine server.
>>
>> Does anyone have any details on known limitations of App Engine
>> communicating with other services over TCP?
>>
>> Thank you so much!
>> Brad
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/Lj6qaT2lC90/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/6d0562a1-a9e7-4baf-ba03-0335ffa9568dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAGJGpkoNKVS2Oyyh%2B6PEiaCaY63O27WtFX21jATFmZqPYpcYjA%40mail.gmail.com.

[google-appengine] Larger MongoDB Query from App Engine neither succeeding nor failing nor throwing exception

[Bradley Barrows]

Hi

This weekend we ran into an issue. A function, using Mongoose (a NodeJS 
MongoDB library) was making a pretty simple query, which when run locally 
either by running the query directly or when running the entire Node server 
directly ran fine.

However, when deployed to Google App Engine we noticed that logging was 
logged out up until we awaited the query to MongoDB and then never 
continued. No exception was ever logged as well.

It is as if the Mongo query was taking too long and App Engine just killed 
the Node process all together.

The server did return a 500 error. I believe it was a 503 specifically.

We ended up realizing by using limit or projection we could decrease the 
response size of the query and get a result and the execution of the NodeJS 
application would continue.

Now if Mongo was an HTTP server I would see that there is a 32MB maximum 
request size and be able to find that there are only so many records I can 
query at once. However, since this is a TCP connection to the Mongo server 
I cannot find any quotas or limitations on how much data a TCP connection 
from an App Engine server can send or receive.

As we know that decreasing the size of the response from MongoDB fixed our 
issue, I believe it is safe to say the issue is related to the amount of 
data sent/received or more specifically received over TCP when 
communicating with an App Engine server.

Does anyone have any details on known limitations of App Engine 
communicating with other services over TCP?

Thank you so much!
Brad

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/4085b9f4-a849-459f-aab6-43f2b636d253n%40googlegroups.com.

[google-appengine] Google App Engine TCP Connections

[Bradley Barrows]

Hi

I have a NodeJS App Engine server which is awaiting a MongoDB query which, 
when reaching a certain size, never returns. There is no exception thrown 
from App Engine either. It is as if the process is just killed because 
execution flow and therefore logging from the process making the MongoDB 
query stops.

If this were a request to a HTTP server I could look at the limits and 
quotas page and see that a maximum of 32MB requests and response size is 
allowed.

I am unable to find any limits or quotas for TCP connections from an App 
Engine server.

We temporarily solved the issue by using a limit or projection to decrease 
the amount of data returned by the MongoDB query. When less data is sent 
back to the App Engine server execution continues and the server runs as 
expected.

I was wondering if it was possible if at some point if TCP data usage 
reaches a limit will the App Engine process be killed? Without any logging? 
Why would the execution of the process handling the response that fires off 
the MongoDB query stop and no exception or anything further be logged?

Thank you
Brad

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/2ace1818-2ac8-49d4-bcec-0980c30eeb78n%40googlegroups.com.

[google-appengine] Java Runtime Acceptance

[Bradley Barrows]

I am also waiting to be accepted for the GAE java runtime. Any help
would be greatly appreciated.
Thanks

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---

[google-appengine] Re: Activating GAE (Java) account issues

[Bradley Barrows]

Hi Jason,
I was wondering how long it takes to be approved for the Java runtime
on average. I have been waiting for about a week.
Thanks,
Brad

On May 15, 11:37 am, "Jason (Google)"  wrote:
> You should receive an email shortly.
>
> - Jason
>
> On Thu, May 14, 2009 at 1:47 PM, David Levy
> wrote:
>
>
>
> > Jason,
> > I am waiting for the email response as well.  Has the quota limit been
> > surpassed?
>
> > thanks
> > David

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---