Re: [google-appengine] Re: Exceeded soft memory limit

[Robert Kluin]

Are you using global variables to cache stuff at the module level or
something similar?



Robert






On Tue, Oct 5, 2010 at 13:25, sahid  wrote:
> up
>
> On Oct 3, 10:06 pm, sahid  wrote:
>> Hello,
>>
>> I have a little problem with my application,
>> i have a lot of "Exceeded soft memory limit" exception.
>>
>> But i know why.
>> My application is a daiting website. All users has a possibility to
>> make a search with many criteria
>> So i have a model with a lot of criteria, i use the filter system of
>> the datastore to get a part of the result (like gender, and
>> has_picture, country)
>> but with the result (limited has 200 and ordered by last_presence) i
>> use python to filter it.
>>
>> I have many exceptions, but i dont know how i can do better...
>> Already, the result is limited to 200 and it's really small, i need
>> about 500.
>>
>> Exceeded soft memory limit with 299.676 MB after servicing 8724
>> requests total
>> Exceeded soft memory limit with 299.145 MB after servicing 1182
>> requests total
>> Exceeded soft memory limit with 206.145 MB after servicing 636
>> requests total
>> Exceeded soft memory limit with 299.145 MB after servicing 371
>> requests total
>> Exceeded soft memory limit with 287.895 MB after servicing 2843
>> requests total
>> Exceeded soft memory limit with 299.594 MB after servicing 7138
>> requests total
>>
>> APPID: devel-inchallah
>>
>> Cordialy,
>> Sahid
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Exceeded soft memory limit

[Robert Kluin]

The message just means your app has used "too much" memory.  I just
re-read your first email, you indicate that you need about 500MB of
memory... so I guess you are aware of what you are doing that uses the
memory.

You probably have three choices:
1) Try submitting a production issue request to Google.
2) Change your design so that it requires less memory.
3) Just ignore the "error."


Robert








On Wed, Oct 6, 2010 at 08:02, sahid  wrote:
> just an extract of my log http://goo.gl/226V
>
> On Oct 6, 10:48 am, sahid  wrote:
>> No, i dont use global variables or just variables with a constants
>> values
>>
>> You know why exactely signifie this message?
>>
>> On Oct 5, 8:47 pm, Robert Kluin  wrote:
>>
>>
>>
>>
>>
>>
>>
>> > Are you using global variables to cache stuff at the module level or
>> > something similar?
>>
>> > Robert
>>
>> > On Tue, Oct 5, 2010 at 13:25, sahid  wrote:
>> > > up
>>
>> > > On Oct 3, 10:06 pm, sahid  wrote:
>> > >> Hello,
>>
>> > >> I have a little problem with my application,
>> > >> i have a lot of "Exceeded soft memory limit" exception.
>>
>> > >> But i know why.
>> > >> My application is a daiting website. All users has a possibility to
>> > >> make a search with many criteria
>> > >> So i have a model with a lot of criteria, i use the filter system of
>> > >> the datastore to get a part of the result (like gender, and
>> > >> has_picture, country)
>> > >> but with the result (limited has 200 and ordered by last_presence) i
>> > >> use python to filter it.
>>
>> > >> I have many exceptions, but i dont know how i can do better...
>> > >> Already, the result is limited to 200 and it's really small, i need
>> > >> about 500.
>>
>> > >> Exceeded soft memory limit with 299.676 MB after servicing 8724
>> > >> requests total
>> > >> Exceeded soft memory limit with 299.145 MB after servicing 1182
>> > >> requests total
>> > >> Exceeded soft memory limit with 206.145 MB after servicing 636
>> > >> requests total
>> > >> Exceeded soft memory limit with 299.145 MB after servicing 371
>> > >> requests total
>> > >> Exceeded soft memory limit with 287.895 MB after servicing 2843
>> > >> requests total
>> > >> Exceeded soft memory limit with 299.594 MB after servicing 7138
>> > >> requests total
>>
>> > >> APPID: devel-inchallah
>>
>> > >> Cordialy,
>> > >> Sahid
>>
>> > > --
>> > > You received this message because you are subscribed to the Google 
>> > > Groups "Google App Engine" group.
>> > > To post to this group, send email to google-appeng...@googlegroups.com.
>> > > To unsubscribe from this group, send email to 
>> > > google-appengine+unsubscr...@googlegroups.com.
>> > > For more options, visit this group 
>> > > athttp://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Exceeded soft memory limit - how to configure app.yaml for java 11

[Raphael André Bauer]

Thanks Ludo!
I have zero idea how this is implemented internally
- but might it be the case that the VPN connector uses a substantial
amount of memory of the instance?


On Thu, Mar 25, 2021 at 10:20 PM Ludovic Champenois
 wrote:
>
>
> Hi,
> I am not sure JAVA_TOOL_OPTIONS is supported (nor documented) for GAE Java11..
> You have a few options:
> 1/ either move to
> instance_class: F4
> or
> instance_class: F2
>
> to get bigger instances.
> Or manually control your entrypoint... I assume you deployed a jar, let's 
> call it myjar.jar
>
> the default entrypoint generated is:
> java -jar myjar.jar
>
> You can change it by adding this to your app.yaml:
>
> entrypoint: java -XX:MaxRAM=200m -Xmx200m -jar myjar.jar
>
> Some more doc at 
> https://cloud.google.com/appengine/docs/standard/java11/runtime#application_startup
>
> Hope this helps,
> Ludo
>
> On Wednesday, March 24, 2021 at 10:31:54 AM UTC-7 ra wrote:
>>
>> Hi,
>>
>> I am running in the issue "Exceeded soft memory limit of 256 MB with
>> 283 MB after servicing 3 requests total.
>> Consider setting a larger instance class in app.yaml.".
>>
>> My app.yaml looks like this:
>>
>> 
>> runtime: java11
>> instance_class: F1
>>
>> env_variables:
>> JAVA_TOOL_OPTIONS: "-XX:MaxRAM=200m -Xmx200m"
>> inbound_services:
>> - warmup
>>
>> handlers:
>> - url: /.*
>> script: auto
>> secure: always
>> redirect_http_response_code: 301
>>
>> automatic_scaling:
>> min_instances: 1
>> max_instances: 2
>>
>> # Connecting to the cloud sql instance via app engine standard needs a
>> vpc access connector:
>> # 
>> https://stackoverflow.com/questions/51328359/how-to-connect-to-cloud-sql-mysql-database-from-app-engine
>> vpc_access_connector:
>> name: "projects/locations/europe-west3/connectors/serverless-connector"
>> ##
>>
>> My question now is: How can I tell the java runtime to NOT use more
>> than the available RAM of the instance?
>>
>> Strangely: Anything I set in terms of -Xmx gets discarded and my app
>> always has a max heap of 127.729664MB.
>> So it seems my JAVA_TOOL_OPTIONS do not get picked up by the JVM.
>>
>> So where does the rest of the memory go?
>> Or is this "something else" like the serverless connector in the
>> container that consumes all the memory?
>>
>>
>> Thanks!
>>
>>
>> Raphael
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/55cf6419-9a2b-4fb5-a51d-086d084ade78n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CACRkZ6J4gtG5pcyOsNUtve73wWc8E2q1Qc8%2B5h-hyXFPN%3DD87Q%40mail.gmail.com.

Re: [google-appengine] Re: Exceeded soft memory limit - how to configure app.yaml for java 11

['Andrew Cooper' via Google App Engine]

Do you know how much memory it uses when you run it locally? 

Increasing the size of the instance is not generally the best approach. For 
example, if you have a memory leak, then the instance size might need to 
grow exponentially!!

You can use Cloud Code  to help you debug 
 these types of 
situations. You can also use the Cloud Profiler for Java 
 to understand 
better what your code is consuming.

I am more familiar with Python than Java, but I wonder how you are 
implementing the Java code? Looping in Python is notoriously problematic as 
it has a habit of leaving behind and accumulating "garbage" in memory. You 
can implement List Comprehensions, Generators, and Yield returns in Python 
to consume memory more efficiently. I have added a link that explains my 
thinking better and provides a helpful comparison 
.
 
I believe that you can do something similar in Java 
, but I might 
be wrong. You can also try chunking your data, but you might be limited in 
how you can split your data, so it might not be the best fit.
On Saturday, March 27, 2021 at 8:33:52 AM UTC+1 ra wrote:

> Thanks Ludo!
> I have zero idea how this is implemented internally
> - but might it be the case that the VPN connector uses a substantial
> amount of memory of the instance?
>
>
> On Thu, Mar 25, 2021 at 10:20 PM Ludovic Champenois
>  wrote:
> >
> >
> > Hi,
> > I am not sure JAVA_TOOL_OPTIONS is supported (nor documented) for GAE 
> Java11..
> > You have a few options:
> > 1/ either move to
> > instance_class: F4
> > or
> > instance_class: F2
> >
> > to get bigger instances.
> > Or manually control your entrypoint... I assume you deployed a jar, 
> let's call it myjar.jar
> >
> > the default entrypoint generated is:
> > java -jar myjar.jar
> >
> > You can change it by adding this to your app.yaml:
> >
> > entrypoint: java -XX:MaxRAM=200m -Xmx200m -jar myjar.jar
> >
> > Some more doc at 
> https://cloud.google.com/appengine/docs/standard/java11/runtime#application_startup
> >
> > Hope this helps,
> > Ludo
> >
> > On Wednesday, March 24, 2021 at 10:31:54 AM UTC-7 ra wrote:
> >>
> >> Hi,
> >>
> >> I am running in the issue "Exceeded soft memory limit of 256 MB with
> >> 283 MB after servicing 3 requests total.
> >> Consider setting a larger instance class in app.yaml.".
> >>
> >> My app.yaml looks like this:
> >>
> >> 
> >> runtime: java11
> >> instance_class: F1
> >>
> >> env_variables:
> >> JAVA_TOOL_OPTIONS: "-XX:MaxRAM=200m -Xmx200m"
> >> inbound_services:
> >> - warmup
> >>
> >> handlers:
> >> - url: /.*
> >> script: auto
> >> secure: always
> >> redirect_http_response_code: 301
> >>
> >> automatic_scaling:
> >> min_instances: 1
> >> max_instances: 2
> >>
> >> # Connecting to the cloud sql instance via app engine standard needs a
> >> vpc access connector:
> >> # 
> https://stackoverflow.com/questions/51328359/how-to-connect-to-cloud-sql-mysql-database-from-app-engine
> >> vpc_access_connector:
> >> name: 
> "projects/locations/europe-west3/connectors/serverless-connector"
> >> 
> ##
> >>
> >> My question now is: How can I tell the java runtime to NOT use more
> >> than the available RAM of the instance?
> >>
> >> Strangely: Anything I set in terms of -Xmx gets discarded and my app
> >> always has a max heap of 127.729664MB.
> >> So it seems my JAVA_TOOL_OPTIONS do not get picked up by the JVM.
> >>
> >> So where does the rest of the memory go?
> >> Or is this "something else" like the serverless connector in the
> >> container that consumes all the memory?
> >>
> >>
> >> Thanks!
> >>
> >>
> >> Raphael
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "Google App Engine" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to google-appengi...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/55cf6419-9a2b-4fb5-a51d-086d084ade78n%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/54285a1f-2ece-4841-b14d-77fa975e6c2an%40googlegroups.com.