Re: [google-appengine] Re: Java Backends are killed and started

[Vinny P]

On Tuesday, March 19, 2013 7:41:59 AM UTC-5, aswath wrote:
>
> It is better while using B8 instances.  The memory usage keeps shooting up 
> as I run long scheme migrations and finally the instance dies at 1024MB.
> I believe, I have to investigate my code for possible memory leaks. 
>
> Since, my program is continuously looping and doing some entity updates, I 
> think the java garbage collector is not getting a chance to do the garbage 
> collection.
> Is there any way to forcebly perform the garbage collection or, better, I 
> want to give a chance for the JVM to perform it garbage collection process.
>
 

Glad to hear it's better now. I'm not sure if there's a way to access GC 
processes; that might be outside of AppEngine's whitelist requirement. Can 
you post some code, and we can see if there's some optimizations we can do?


On Tuesday, March 19, 2013 12:24:34 PM UTC-5, Rafael Sanches wrote:
>
> If you see that screenshot you will see that my dynamic instances are ok, 
> but the dedicated ones keep dying and re-starting, but they die after some 
> seconds.
>  
>

Just as a temporary band aid (and to save you a bit of money) try lowering 
the resident instances to just 1, and let the dynamic instances carry the 
load. A lot of people are having issues today, seems like it's one of those 
days.

-Vinny P


 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Java Backends are killed and started

[Aswath Satrasala]

It is better while using B8 instances.  The memory usage keeps shooting up
as I run long scheme migrations and finally the instance dies at 1024MB.
I believe, I have to investigate my code for possible memory leaks.

Since, my program is continuously looping and doing some entity updates, I
think the java garbage collector is not getting a chance to do the garbage
collection.
Is there any way to forcebly perform the garbage collection or, better, I
want to give a chance for the JVM to perform it garbage collection process.

-Aswath




On Fri, Mar 15, 2013 at 5:13 PM, Aswath Satrasala <
aswath.satras...@gmail.com> wrote:

> I have increased the Backend class to B4 also.   The backend is just
> getting killled after 3-4 minutes.
> I look at the Memory Usage chart for my instance.  It shows above 512MB.
> Now, I am trying with B8 instance.
>
>
> -Aswath
>
>
>
>
> On Fri, Mar 15, 2013 at 2:01 AM, Vinny P  wrote:
>
>> Wow. OK, that's not a code problem, that's AppEngine with a full blown
>> case of bird flu.
>>
>> Time to feed Starburst to Drake's magical unicorns?
>>
>> -Vinny P
>>
>>
>> On Thursday, March 14, 2013 2:04:42 PM UTC-5, Rafael Sanches wrote:
>>
>>> Hi Vinni,
>>>
>>> I believe it's much worst than that. Look of what just happened to my
>>> servers, 5 minutes ago: 
>>> http://cl.ly/image/**2t1y2O2S2U1h
>>>
>>> This is making me to pay $40k a year for a service that I could handle
>>> with $3.
>>>
>>> I hope someone fixes this soon. Even if it's happening, on a constant
>>> basis, for a week already.
>>>
>>> thanks
>>> rafa
>>>
>>>
>>> On Thu, Mar 14, 2013 at 7:25 AM, Vinny P  wrote:
>>>
 The backends are getting killed because something or someone is sending
 _ah/stop requests to them. You can read more about shutdown requests here:
 https://developers.**google.com/appengine/docs/**
 java/backends/overview#**Shutdown

 The first thing that comes to mind is that you may be exceeding the
 memory limit, especially considering that AppEngine is complaining about
 chunk size. See here for more details: http://stackoverflow.**
 com/questions/7198738/for-**google-app-engine-java-how-do-**
 i-set-and-use-chunk-size-in-**fetchoptions
  but
 the TLDR is that you're requesting a lot of entities in one operation.
 Depending on your code and what you're doing, you may be exceeding the
 memory limit for the backend. Try increasing the backend size and see if it
 solves the problem. If not, can you post some code?

 -Vinny


 On Thursday, March 14, 2013 7:55:17 AM UTC-5, aswath wrote:

>
> Hello,
> I am running a tasks in the Java Backends.  Within 7 min, the Java
> Backends is started 3 times.
> The bad part is , the backends are getting killed in the middle of a
> running task.
>
> Please advise
>
> -Aswath
>
>
>
>1.
>   1. 2013-03-14 17:45:46.214 /admin/**addGlAcctO**
>   rgPartyDateMonthYear**ToEntry 50**0 99494ms 0kb instan**ce=0
>   AppEn**gine-Google; 
> (+http://code.google.com/**appen**gine
>   )
>   2. W2013-03-14 17:44:19.341 com.google.**appeng**
>   ine.api.datastore.**QueryResults**SourceImpl
>   logChunkSizeWarning: This query does not have a chunk size set in
>   FetchOptions and has retur
>   3. W2013-03-14 17:44:19.608 
> com.veersoft.**AddD**ateMonthYearEntities
>   doProcess: procing mailId=26apr...@gmail.com
>   4. W2013-03-14 17:44:19.609 
> com.veersoft.**AddD**ateMonthYearEntities
>   createGlAcctOrgPartyDateMonthYear: crnt nameSpace=c-p-r-p-l-
>   **26april197**3-gmail.com
>   5. W2013-03-14 17:44:20.386 
> com.veersoft.**AddD**ateMonthYearEntities
>   createGlAcctOrgPartyDateMonthYear: No.of AcctgTrans= 73
>   6. W2013-03-14 17:44:22.537 
> com.veersoft.**AddD**ateMonthYearEntities
>   createGlAcctOrgPartyDateMonthYear: AcctgTran count: 0
>   7. W2013-03-14 17:45:24.938 
> com.veersoft.**AddD**ateMonthYearEntities
>   createGlAcctOrgPartyDateMonthYear: AcctgTran count: 25
>   8. E2013-03-14 17:45:46.213 Process terminated because the
>   backend took too long to shutdown.
>   2.
>   1. 2013-03-14 17:45:35.268 /_ah/stop 200 119ms 0kb
>   instance=0
>3.
>   1. 2013-03-14 17:44:18.297 /_ah/start 404 10478ms 0kb
>   instance=0
>   2. W2013-03-14 17:44:10.419 
> com.veersoft.**list**ener.**VsServletContextListener
>   contextInitialized: 2013-03-14 18:14:10 IST
>   

Re: [google-appengine] Re: Java Backends are killed and started

[Aswath Satrasala]

I have increased the Backend class to B4 also.   The backend is just
getting killled after 3-4 minutes.
I look at the Memory Usage chart for my instance.  It shows above 512MB.
Now, I am trying with B8 instance.


-Aswath




On Fri, Mar 15, 2013 at 2:01 AM, Vinny P  wrote:

> Wow. OK, that's not a code problem, that's AppEngine with a full blown
> case of bird flu.
>
> Time to feed Starburst to Drake's magical unicorns?
>
> -Vinny P
>
>
> On Thursday, March 14, 2013 2:04:42 PM UTC-5, Rafael Sanches wrote:
>
>> Hi Vinni,
>>
>> I believe it's much worst than that. Look of what just happened to my
>> servers, 5 minutes ago: 
>> http://cl.ly/image/**2t1y2O2S2U1h
>>
>> This is making me to pay $40k a year for a service that I could handle
>> with $3.
>>
>> I hope someone fixes this soon. Even if it's happening, on a constant
>> basis, for a week already.
>>
>> thanks
>> rafa
>>
>>
>> On Thu, Mar 14, 2013 at 7:25 AM, Vinny P  wrote:
>>
>>> The backends are getting killed because something or someone is sending
>>> _ah/stop requests to them. You can read more about shutdown requests here:
>>> https://developers.**google.com/appengine/docs/**java/backends/overview#
>>> **Shutdown
>>>
>>> The first thing that comes to mind is that you may be exceeding the
>>> memory limit, especially considering that AppEngine is complaining about
>>> chunk size. See here for more details: http://stackoverflow.**
>>> com/questions/7198738/for-**google-app-engine-java-how-do-**
>>> i-set-and-use-chunk-size-in-**fetchoptions
>>>  but
>>> the TLDR is that you're requesting a lot of entities in one operation.
>>> Depending on your code and what you're doing, you may be exceeding the
>>> memory limit for the backend. Try increasing the backend size and see if it
>>> solves the problem. If not, can you post some code?
>>>
>>> -Vinny
>>>
>>>
>>> On Thursday, March 14, 2013 7:55:17 AM UTC-5, aswath wrote:
>>>

 Hello,
 I am running a tasks in the Java Backends.  Within 7 min, the Java
 Backends is started 3 times.
 The bad part is , the backends are getting killed in the middle of a
 running task.

 Please advise

 -Aswath



1.
   1. 2013-03-14 17:45:46.214 /admin/**addGlAcctO**
   rgPartyDateMonthYear**ToEntry 50**0 99494ms 0kb instan**ce=0
   AppEn**gine-Google; 
 (+http://code.google.com/**appen**gine
   )
   2. W2013-03-14 17:44:19.341 com.google.**appeng**
   ine.api.datastore.**QueryResults**SourceImpl
   logChunkSizeWarning: This query does not have a chunk size set in
   FetchOptions and has retur
   3. W2013-03-14 17:44:19.608 com.veersoft.**AddD**ateMonthYearEntities
   doProcess: procing mailId=26apr...@gmail.com
   4. W2013-03-14 17:44:19.609 com.veersoft.**AddD**ateMonthYearEntities
   createGlAcctOrgPartyDateMonthYear: crnt nameSpace=c-p-r-p-l-*
   *26april197**3-gmail.com
   5. W2013-03-14 17:44:20.386 com.veersoft.**AddD**ateMonthYearEntities
   createGlAcctOrgPartyDateMonthYear: No.of AcctgTrans= 73
   6. W2013-03-14 17:44:22.537 com.veersoft.**AddD**ateMonthYearEntities
   createGlAcctOrgPartyDateMonthYear: AcctgTran count: 0
   7. W2013-03-14 17:45:24.938 com.veersoft.**AddD**ateMonthYearEntities
   createGlAcctOrgPartyDateMonthYear: AcctgTran count: 25
   8. E2013-03-14 17:45:46.213 Process terminated because the
   backend took too long to shutdown.
   2.
   1. 2013-03-14 17:45:35.268 /_ah/stop 200 119ms 0kb instance=0
3.
   1. 2013-03-14 17:44:18.297 /_ah/start 404 10478ms 0kb
   instance=0
   2. W2013-03-14 17:44:10.419 
 com.veersoft.**list**ener.**VsServletContextListener
   contextInitialized: 2013-03-14 18:14:10 IST
   3. I2013-03-14 17:44:13.889 com.google.**inject**.internal.util.$
   **FinalizableRef**erenceQueue$**SystemLoader loadFinalizer: Not
   allowed to access system class loader.
   4. I2013-03-14 17:44:13.895 com.google.**inject**.internal.util.$
   **FinalizableRef**erenceQueue : Failed to start reference
   finalizer thread. Reference cleanup will only occur when
   5. I2013-03-14 17:44:18.297 This request caused a new process to
   be started for your application, and thus caused your application 
 code to
   be loaded for the first time. This requ
4.
   1. 2013-03-14 17:43:46.513 /admin/**addGlAcctO**
   rgPartyDateMonthYear**ToEntry 50**0 380429ms 0kb insta**nce=0
   AppE**ngine-

Re: [google-appengine] Re: Java Backends are killed and started

[Vinny P]

Wow. OK, that's not a code problem, that's AppEngine with a full blown case 
of bird flu.

Time to feed Starburst to Drake's magical unicorns?

-Vinny P

On Thursday, March 14, 2013 2:04:42 PM UTC-5, Rafael Sanches wrote:
>
> Hi Vinni, 
>
> I believe it's much worst than that. Look of what just happened to my 
> servers, 5 minutes ago: http://cl.ly/image/2t1y2O2S2U1h
>
> This is making me to pay $40k a year for a service that I could handle 
> with $3. 
>
> I hope someone fixes this soon. Even if it's happening, on a constant 
> basis, for a week already.
>
> thanks
> rafa
>
>
> On Thu, Mar 14, 2013 at 7:25 AM, Vinny P  >wrote:
>
>> The backends are getting killed because something or someone is sending 
>> _ah/stop requests to them. You can read more about shutdown requests here: 
>> https://developers.google.com/appengine/docs/java/backends/overview#Shutdown
>>
>> The first thing that comes to mind is that you may be exceeding the 
>> memory limit, especially considering that AppEngine is complaining about 
>> chunk size. See here for more details: 
>> http://stackoverflow.com/questions/7198738/for-google-app-engine-java-how-do-i-set-and-use-chunk-size-in-fetchoptions
>>  but 
>> the TLDR is that you're requesting a lot of entities in one operation. 
>> Depending on your code and what you're doing, you may be exceeding the 
>> memory limit for the backend. Try increasing the backend size and see if it 
>> solves the problem. If not, can you post some code?
>>
>> -Vinny
>>
>>
>> On Thursday, March 14, 2013 7:55:17 AM UTC-5, aswath wrote:
>>
>>>
>>> Hello,
>>> I am running a tasks in the Java Backends.  Within 7 min, the Java 
>>> Backends is started 3 times.  
>>> The bad part is , the backends are getting killed in the middle of a 
>>> running task. 
>>>
>>> Please advise
>>>
>>> -Aswath
>>>
>>>
>>>
>>>1. 
>>>   1. 2013-03-14 17:45:46.214 /admin/**addGlAcctOrgPartyDateMonthYear
>>>   **ToEntry 500 99494ms 0kb instan**ce=0 AppEngine-Google; (+
>>>   http://code.google.com/**appengine
>>>   ) 
>>>   2. W2013-03-14 17:44:19.341 com.google.**appengine.api.datastore.*
>>>   *QueryResultsSourceImpl logChunkSizeWarning: This query does not 
>>>   have a chunk size set in FetchOptions and has retur 
>>>   3. W2013-03-14 17:44:19.608 com.veersoft.**AddDateMonthYearEntities 
>>>   doProcess: procing mailId=26apr...@gmail.com 
>>>   4. W2013-03-14 17:44:19.609 com.veersoft.**AddDateMonthYearEntities 
>>>   createGlAcctOrgPartyDateMonthY**ear: crnt nameSpace=c-p-r-p-l-**
>>>   26april1973-gmail.com  
>>>   5. W2013-03-14 17:44:20.386 com.veersoft.**AddDateMonthYearEntities 
>>>   createGlAcctOrgPartyDateMonthY**ear: No.of AcctgTrans= 73 
>>>   6. W2013-03-14 17:44:22.537 com.veersoft.**AddDateMonthYearEntities 
>>>   createGlAcctOrgPartyDateMonthY**ear: AcctgTran count: 0 
>>>   7. W2013-03-14 17:45:24.938 com.veersoft.**AddDateMonthYearEntities 
>>>   createGlAcctOrgPartyDateMonthY**ear: AcctgTran count: 25 
>>>   8. E2013-03-14 17:45:46.213 Process terminated because the 
>>>   backend took too long to shutdown. 
>>>   2. 
>>>   1. 2013-03-14 17:45:35.268 /_ah/stop 200 119**ms 0kb instance=0 
>>>3. 
>>>   1. 2013-03-14 17:44:18.297 /_ah/start 404 10**478ms 0kb instance=0 
>>>   2. W2013-03-14 17:44:10.419 
>>> com.veersoft.**listener.**VsServletContextListener 
>>>   contextInitialized: 2013-03-14 18:14:10 IST 
>>>   3. I2013-03-14 17:44:13.889 com.google.**inject.internal.util.$**
>>>   FinalizableReferenceQueue$**SystemLoader loadFinalizer: Not 
>>>   allowed to access system class loader. 
>>>   4. I2013-03-14 17:44:13.895 
>>> com.google.**inject.internal.util.$**FinalizableReferenceQueue 
>>>   : Failed to start reference finalizer thread. Reference cleanup 
>>> will 
>>>   only occur when 
>>>   5. I2013-03-14 17:44:18.297 This request caused a new process to 
>>>   be started for your application, and thus caused your application 
>>> code to 
>>>   be loaded for the first time. This requ 
>>>4. 
>>>   1. 2013-03-14 17:43:46.513 /admin/**addGlAcctOrgPartyDateMonthYear
>>>   **ToEntry 500 380429ms 0kb insta**nce=0 AppEngine-Google; (+
>>>   http://code.google.com/**appengine
>>>   ) 
>>>   2. W2013-03-14 17:37:46.188 com.google.**appengine.api.datastore.*
>>>   *QueryResultsSourceImpl logChunkSizeWarning: This query does not 
>>>   have a chunk size set in FetchOptions and has retur 
>>>   3. W2013-03-14 17:37:46.651 com.veersoft.**AddDateMonthYearEntities 
>>>   doProcess: procing mailId=26apr...@gmail.com 
>>>   4. W2013-03-14 17:37:46.653 com.veersoft.**AddDateMonthYearEntities 
>>>   createGlAcctOrgPartyDateMonthY**ear: crnt nameSpace=c-p-r-p-l-**
>>>   26april1973-gmail.com 

[google-appengine] Re: Java Backends are killed and started

[Vinny P]

The backends are getting killed because something or someone is sending 
_ah/stop requests to them. You can read more about shutdown requests here: 
https://developers.google.com/appengine/docs/java/backends/overview#Shutdown

The first thing that comes to mind is that you may be exceeding the memory 
limit, especially considering that AppEngine is complaining about chunk 
size. See here for more details: 
http://stackoverflow.com/questions/7198738/for-google-app-engine-java-how-do-i-set-and-use-chunk-size-in-fetchoptions
 but 
the TLDR is that you're requesting a lot of entities in one operation. 
Depending on your code and what you're doing, you may be exceeding the 
memory limit for the backend. Try increasing the backend size and see if it 
solves the problem. If not, can you post some code?

-Vinny

On Thursday, March 14, 2013 7:55:17 AM UTC-5, aswath wrote:
>
>
> Hello,
> I am running a tasks in the Java Backends.  Within 7 min, the Java 
> Backends is started 3 times.  
> The bad part is , the backends are getting killed in the middle of a 
> running task. 
>
> Please advise
>
> -Aswath
>
>
>
>1. 
>   1. 2013-03-14 17:45:46.214 
>   /admin/addGlAcctOrgPartyDateMonthYearToEntry 500 99494ms 0kb 
>   instance=0 AppEngine-Google; (+http://code.google.com/appengine)
>   2. W2013-03-14 17:44:19.341 
> com.google.appengine.api.datastore.QueryResultsSourceImpl 
>   logChunkSizeWarning: This query does not have a chunk size set in 
>   FetchOptions and has retur
>   3. W2013-03-14 17:44:19.608 com.veersoft.AddDateMonthYearEntities 
>   doProcess: procing mailId=26apr...@gmail.com
>   4. W2013-03-14 17:44:19.609 com.veersoft.AddDateMonthYearEntities 
>   createGlAcctOrgPartyDateMonthYear: crnt nameSpace=
>   c-p-r-p-l-26april1973-gmail.com
>   5. W2013-03-14 17:44:20.386 com.veersoft.AddDateMonthYearEntities 
>   createGlAcctOrgPartyDateMonthYear: No.of AcctgTrans= 73
>   6. W2013-03-14 17:44:22.537 com.veersoft.AddDateMonthYearEntities 
>   createGlAcctOrgPartyDateMonthYear: AcctgTran count: 0
>   7. W2013-03-14 17:45:24.938 com.veersoft.AddDateMonthYearEntities 
>   createGlAcctOrgPartyDateMonthYear: AcctgTran count: 25
>   8. E2013-03-14 17:45:46.213 Process terminated because the backend 
>   took too long to shutdown.
>2. 
>   1. 2013-03-14 17:45:35.268 /_ah/stop 200 119ms 0kb instance=0
>3. 
>   1. 2013-03-14 17:44:18.297 /_ah/start 404 10478ms 0kb instance=0
>   2. W2013-03-14 17:44:10.419 
> com.veersoft.listener.VsServletContextListener 
>   contextInitialized: 2013-03-14 18:14:10 IST
>   3. I2013-03-14 17:44:13.889 
> com.google.inject.internal.util.$FinalizableReferenceQueue$SystemLoader 
>   loadFinalizer: Not allowed to access system class loader.
>   4. I2013-03-14 17:44:13.895 
> com.google.inject.internal.util.$FinalizableReferenceQueue 
>   : Failed to start reference finalizer thread. Reference cleanup 
> will 
>   only occur when
>   5. I2013-03-14 17:44:18.297 This request caused a new process to be 
>   started for your application, and thus caused your application code to 
> be 
>   loaded for the first time. This requ
>4. 
>   1. 2013-03-14 17:43:46.513 
>   /admin/addGlAcctOrgPartyDateMonthYearToEntry 500 380429ms 0kb 
>   instance=0 AppEngine-Google; (+http://code.google.com/appengine)
>   2. W2013-03-14 17:37:46.188 
> com.google.appengine.api.datastore.QueryResultsSourceImpl 
>   logChunkSizeWarning: This query does not have a chunk size set in 
>   FetchOptions and has retur
>   3. W2013-03-14 17:37:46.651 com.veersoft.AddDateMonthYearEntities 
>   doProcess: procing mailId=26apr...@gmail.com
>   4. W2013-03-14 17:37:46.653 com.veersoft.AddDateMonthYearEntities 
>   createGlAcctOrgPartyDateMonthYear: crnt nameSpace=
>   c-p-r-p-l-26april1973-gmail.com
>   5. W2013-03-14 17:37:47.343 com.veersoft.AddDateMonthYearEntities 
>   createGlAcctOrgPartyDateMonthYear: No.of AcctgTrans= 73
>   6. W2013-03-14 17:37:50.271 com.veersoft.AddDateMonthYearEntities 
>   createGlAcctOrgPartyDateMonthYear: AcctgTran count: 0
>   7. W2013-03-14 17:39:04.011 com.veersoft.AddDateMonthYearEntities 
>   createGlAcctOrgPartyDateMonthYear: AcctgTran count: 25
>   8. E2013-03-14 17:43:46.513 Process terminated because the backend 
>   took too long to shutdown.
>5. 
>   1. 2013-03-14 17:43:46.512 /_ah/stop 500 10199ms 0kb instance=0
>   2. E2013-03-14 17:43:46.512 Process terminated because the backend 
>   took too long to shutdown.
>6. 
>   1. 2013-03-14 17:37:43.668 /_ah/start 404 17718ms 0kb instance=0
>   2. W2013-03-14 17:37:29.765 
> com.veersoft.listener.VsServletContextListener 
>   contextInitialized: 2013-03-14 18:07:29 IST
>   3. I2013-03-14 17:37:36.186 
> com.google.inject.internal.util.$FinalizableReferenceQueue$SystemLoader 
>   loadFinalize