Re: [appengine-java] Re: how to do initialization at startup?
A reliable, high available, scalable and failover tolerant application requires a distributed environment where various application server instances run in different Java VMs on different machines with some kind of loadbalancing in front. So many things that you have to rethink arise from this change from a single server / single Java VM approach to a distributed environment with different Java VMs that do not share the same memory - no matter if it is GAE, JBoss, Tomcat or whatever application server that supports clustering and distributed webapps. 2010/2/18 tsp...@green20now.com tsp...@green20now.com AJ, With GAE, many fundamental assumptions should be questioned. For example, why do you need a counter? Most developers use a counter for the key, jusst because they always have. With GAE you need to often challange the fundamental requirements and rethink the approach. Overall all I think it is good for many developers since thy get in a rut and aply the same solution to all problems (over using a design pattern cause it always worked before) Sent from my Verizon Wireless Phone - Reply message - From: AJ Chen ajc...@web2express.org Date: Wed, Feb 17, 2010 10:52 PM Subject: [appengine-java] Re: how to do initialization at startup? To: google-appengine-java@googlegroups.com There is significant difference between dealing with infrequent crash event and dealing with frequent shutdown by GAE. The difference is huge when you want to have some intermediate data in memory for performance reason. When there is a system crash, you just start over, which is tolerable in most cases. Restarting app by GAE is a total different story because it makes storing data in memory no longer a valid approach. For my google app, I see it is restarted by GAE even after a few minutes. My usual singleton object become useless because the data objects it holds are recreated every few minutes. This is why I think GAE's frequent restart behavior is a constraint forcing me (probably other developers) to change design pattern, which may be good or bad. In this case, the change is not good, I'm afraid. Let's look at the counter example. Normally it's trivial to keep a counter of some sort on the server. But in GAE, it's non-trivial. You can't get a total count from datastore easily and storing a counter in memory is not reliable. SO, you have to do some creative work-around as proposed in GAE documentation. Am I making sense? In any case, I'm hoping someone has an easy/reliable way to keep tracking a counter in memory within GAE. I'll appreciate any suggestion. -aj On Wed, Feb 17, 2010 at 1:56 PM, Stephan Hartmann hartm...@metamesh.dewrote: Hi AJ, Your consideration is not specific to GAE. You always have the potential risk that a server could crash and then all your unstored data changes will get lost. So for critical data you should use a write-through cache. However, in a distributed environment like GAE (but not specific to GAE, every standard conform servlet container will support this) you have to take special care to keep the caches of all nodes in sync, or just use a distributed cache like Memcache in GAE. Regards, Stephan 2010/2/17 AJ Chen ajc...@web2express.org yes, the new console is a good addition. however, because the app can be shutdown/restarted by GAE at any time, you would still need to put the initialization code in context listener (or similar place) so that it will be called automatically when the app is restarted. I just realize a potential major issue in GAE environment, which may require some paradigm shift in server programming (at least to me). Usually, one the server side, you may have a singleton class to keep some data objects closeby as well as updating the data at run time. The data may change so fast that they are conveniently kept in memory for some time before put into permanent storage. This is safe because the web server does not kill the app at will. Now that GAE may kill the app and restart it at any time, keeping data in memory becomes a big potential problem because the data will be gone after the app starting. This means you would have to save the new data into datastore immediately. If you have lots of intermediate data or temporary data, you have to save them to datastore immediately as well. This always-using-datastore situation created by GAE may slow down some applications in addition to a lot more coding for storage. Without knowing what exactly happens when GAE automatically shutdowns the app, my worry may be wrong. Does anyone see the similar issue? Any suggestion for safely keeping data in memory? -aj On Wed, Feb 17, 2010 at 10:00 AM, Vlad Skarzhevskyy skarzhevs...@gmail.com wrote: There is an Custom Admin Console pages in new SDK 1.3.1. I think the best place to preload the data to your application is using servlet/page exposed in this Console. -- You received
Re: [appengine-java] Re: how to do initialization at startup?
Addressing the stable situation where the VM is shutdown by GAE, similarly to the Servlet context initialization you can call the Servlet context destroyed listener to take care of any persistence you want to do there. Of course, this doesn't take care of any exceptional circumstances such as a crash etc ... cowper On Wed, Feb 17, 2010 at 8:58 PM, AJ Chen ajc...@web2express.org wrote: yes, the new console is a good addition. however, because the app can be shutdown/restarted by GAE at any time, you would still need to put the initialization code in context listener (or similar place) so that it will be called automatically when the app is restarted. I just realize a potential major issue in GAE environment, which may require some paradigm shift in server programming (at least to me). Usually, one the server side, you may have a singleton class to keep some data objects closeby as well as updating the data at run time. The data may change so fast that they are conveniently kept in memory for some time before put into permanent storage. This is safe because the web server does not kill the app at will. Now that GAE may kill the app and restart it at any time, keeping data in memory becomes a big potential problem because the data will be gone after the app starting. This means you would have to save the new data into datastore immediately. If you have lots of intermediate data or temporary data, you have to save them to datastore immediately as well. This always-using-datastore situation created by GAE may slow down some applications in addition to a lot more coding for storage. Without knowing what exactly happens when GAE automatically shutdowns the app, my worry may be wrong. Does anyone see the similar issue? Any suggestion for safely keeping data in memory? -aj On Wed, Feb 17, 2010 at 10:00 AM, Vlad Skarzhevskyy skarzhevs...@gmail.com wrote: There is an Custom Admin Console pages in new SDK 1.3.1. I think the best place to preload the data to your application is using servlet/page exposed in this Console. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- AJ Chen, PhD Chair, Semantic Web SIG, sdforum.org http://web2express.org @web2express on twitter Palo Alto, CA, USA 650-283-4091 *Monitoring social media in real time* -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: how to do initialization at startup?
There is an Custom Admin Console pages in new SDK 1.3.1. I think the best place to preload the data to your application is using servlet/page exposed in this Console. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: how to do initialization at startup?
yes, the new console is a good addition. however, because the app can be shutdown/restarted by GAE at any time, you would still need to put the initialization code in context listener (or similar place) so that it will be called automatically when the app is restarted. I just realize a potential major issue in GAE environment, which may require some paradigm shift in server programming (at least to me). Usually, one the server side, you may have a singleton class to keep some data objects closeby as well as updating the data at run time. The data may change so fast that they are conveniently kept in memory for some time before put into permanent storage. This is safe because the web server does not kill the app at will. Now that GAE may kill the app and restart it at any time, keeping data in memory becomes a big potential problem because the data will be gone after the app starting. This means you would have to save the new data into datastore immediately. If you have lots of intermediate data or temporary data, you have to save them to datastore immediately as well. This always-using-datastore situation created by GAE may slow down some applications in addition to a lot more coding for storage. Without knowing what exactly happens when GAE automatically shutdowns the app, my worry may be wrong. Does anyone see the similar issue? Any suggestion for safely keeping data in memory? -aj On Wed, Feb 17, 2010 at 10:00 AM, Vlad Skarzhevskyy skarzhevs...@gmail.comwrote: There is an Custom Admin Console pages in new SDK 1.3.1. I think the best place to preload the data to your application is using servlet/page exposed in this Console. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- AJ Chen, PhD Chair, Semantic Web SIG, sdforum.org http://web2express.org @web2express on twitter Palo Alto, CA, USA 650-283-4091 *Monitoring social media in real time* -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: how to do initialization at startup?
Hi AJ, Your consideration is not specific to GAE. You always have the potential risk that a server could crash and then all your unstored data changes will get lost. So for critical data you should use a write-through cache. However, in a distributed environment like GAE (but not specific to GAE, every standard conform servlet container will support this) you have to take special care to keep the caches of all nodes in sync, or just use a distributed cache like Memcache in GAE. Regards, Stephan 2010/2/17 AJ Chen ajc...@web2express.org yes, the new console is a good addition. however, because the app can be shutdown/restarted by GAE at any time, you would still need to put the initialization code in context listener (or similar place) so that it will be called automatically when the app is restarted. I just realize a potential major issue in GAE environment, which may require some paradigm shift in server programming (at least to me). Usually, one the server side, you may have a singleton class to keep some data objects closeby as well as updating the data at run time. The data may change so fast that they are conveniently kept in memory for some time before put into permanent storage. This is safe because the web server does not kill the app at will. Now that GAE may kill the app and restart it at any time, keeping data in memory becomes a big potential problem because the data will be gone after the app starting. This means you would have to save the new data into datastore immediately. If you have lots of intermediate data or temporary data, you have to save them to datastore immediately as well. This always-using-datastore situation created by GAE may slow down some applications in addition to a lot more coding for storage. Without knowing what exactly happens when GAE automatically shutdowns the app, my worry may be wrong. Does anyone see the similar issue? Any suggestion for safely keeping data in memory? -aj On Wed, Feb 17, 2010 at 10:00 AM, Vlad Skarzhevskyy skarzhevs...@gmail.com wrote: There is an Custom Admin Console pages in new SDK 1.3.1. I think the best place to preload the data to your application is using servlet/page exposed in this Console. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- AJ Chen, PhD Chair, Semantic Web SIG, sdforum.org http://web2express.org @web2express on twitter Palo Alto, CA, USA 650-283-4091 *Monitoring social media in real time* -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: how to do initialization at startup?
There is significant difference between dealing with infrequent crash event and dealing with frequent shutdown by GAE. The difference is huge when you want to have some intermediate data in memory for performance reason. When there is a system crash, you just start over, which is tolerable in most cases. Restarting app by GAE is a total different story because it makes storing data in memory no longer a valid approach. For my google app, I see it is restarted by GAE even after a few minutes. My usual singleton object become useless because the data objects it holds are recreated every few minutes. This is why I think GAE's frequent restart behavior is a constraint forcing me (probably other developers) to change design pattern, which may be good or bad. In this case, the change is not good, I'm afraid. Let's look at the counter example. Normally it's trivial to keep a counter of some sort on the server. But in GAE, it's non-trivial. You can't get a total count from datastore easily and storing a counter in memory is not reliable. SO, you have to do some creative work-around as proposed in GAE documentation. Am I making sense? In any case, I'm hoping someone has an easy/reliable way to keep tracking a counter in memory within GAE. I'll appreciate any suggestion. -aj On Wed, Feb 17, 2010 at 1:56 PM, Stephan Hartmann hartm...@metamesh.dewrote: Hi AJ, Your consideration is not specific to GAE. You always have the potential risk that a server could crash and then all your unstored data changes will get lost. So for critical data you should use a write-through cache. However, in a distributed environment like GAE (but not specific to GAE, every standard conform servlet container will support this) you have to take special care to keep the caches of all nodes in sync, or just use a distributed cache like Memcache in GAE. Regards, Stephan 2010/2/17 AJ Chen ajc...@web2express.org yes, the new console is a good addition. however, because the app can be shutdown/restarted by GAE at any time, you would still need to put the initialization code in context listener (or similar place) so that it will be called automatically when the app is restarted. I just realize a potential major issue in GAE environment, which may require some paradigm shift in server programming (at least to me). Usually, one the server side, you may have a singleton class to keep some data objects closeby as well as updating the data at run time. The data may change so fast that they are conveniently kept in memory for some time before put into permanent storage. This is safe because the web server does not kill the app at will. Now that GAE may kill the app and restart it at any time, keeping data in memory becomes a big potential problem because the data will be gone after the app starting. This means you would have to save the new data into datastore immediately. If you have lots of intermediate data or temporary data, you have to save them to datastore immediately as well. This always-using-datastore situation created by GAE may slow down some applications in addition to a lot more coding for storage. Without knowing what exactly happens when GAE automatically shutdowns the app, my worry may be wrong. Does anyone see the similar issue? Any suggestion for safely keeping data in memory? -aj On Wed, Feb 17, 2010 at 10:00 AM, Vlad Skarzhevskyy skarzhevs...@gmail.com wrote: There is an Custom Admin Console pages in new SDK 1.3.1. I think the best place to preload the data to your application is using servlet/page exposed in this Console. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- AJ Chen, PhD Chair, Semantic Web SIG, sdforum.org http://web2express.org @web2express on twitter Palo Alto, CA, USA 650-283-4091 *Monitoring social media in real time* -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to
Re: [appengine-java] Re: how to do initialization at startup?
A reliable counter needs to be stored in the datastore and sharded - this has always been the only place to put data that your app depends on. Static variables are only useful for caching a small amount of data for speed - never to be depended on. Also keep in mind that your app is not just initialised after being quiet for some time. I have found that new instances are constantly being spun up and down. Perhaps one per concurrent request even? You are absolutely right that many things are much harder to code on App Engine. But I expect that extra time to pay off in the long run with increased reliability, scalability and having Google Engineers maintain the servers rather than me. On 18 Feb 2010, at 10:52, AJ Chen wrote: There is significant difference between dealing with infrequent crash event and dealing with frequent shutdown by GAE. The difference is huge when you want to have some intermediate data in memory for performance reason. When there is a system crash, you just start over, which is tolerable in most cases. Restarting app by GAE is a total different story because it makes storing data in memory no longer a valid approach. For my google app, I see it is restarted by GAE even after a few minutes. My usual singleton object become useless because the data objects it holds are recreated every few minutes. This is why I think GAE's frequent restart behavior is a constraint forcing me (probably other developers) to change design pattern, which may be good or bad. In this case, the change is not good, I'm afraid. Let's look at the counter example. Normally it's trivial to keep a counter of some sort on the server. But in GAE, it's non-trivial. You can't get a total count from datastore easily and storing a counter in memory is not reliable. SO, you have to do some creative work-around as proposed in GAE documentation. Am I making sense? In any case, I'm hoping someone has an easy/ reliable way to keep tracking a counter in memory within GAE. I'll appreciate any suggestion. -aj On Wed, Feb 17, 2010 at 1:56 PM, Stephan Hartmann hartm...@metamesh.de wrote: Hi AJ, Your consideration is not specific to GAE. You always have the potential risk that a server could crash and then all your unstored data changes will get lost. So for critical data you should use a write-through cache. However, in a distributed environment like GAE (but not specific to GAE, every standard conform servlet container will support this) you have to take special care to keep the caches of all nodes in sync, or just use a distributed cache like Memcache in GAE. Regards, Stephan 2010/2/17 AJ Chen ajc...@web2express.org yes, the new console is a good addition. however, because the app can be shutdown/restarted by GAE at any time, you would still need to put the initialization code in context listener (or similar place) so that it will be called automatically when the app is restarted. I just realize a potential major issue in GAE environment, which may require some paradigm shift in server programming (at least to me). Usually, one the server side, you may have a singleton class to keep some data objects closeby as well as updating the data at run time. The data may change so fast that they are conveniently kept in memory for some time before put into permanent storage. This is safe because the web server does not kill the app at will. Now that GAE may kill the app and restart it at any time, keeping data in memory becomes a big potential problem because the data will be gone after the app starting. This means you would have to save the new data into datastore immediately. If you have lots of intermediate data or temporary data, you have to save them to datastore immediately as well. This always-using-datastore situation created by GAE may slow down some applications in addition to a lot more coding for storage. Without knowing what exactly happens when GAE automatically shutdowns the app, my worry may be wrong. Does anyone see the similar issue? Any suggestion for safely keeping data in memory? -aj On Wed, Feb 17, 2010 at 10:00 AM, Vlad Skarzhevskyy skarzhevs...@gmail.com wrote: There is an Custom Admin Console pages in new SDK 1.3.1. I think the best place to preload the data to your application is using servlet/page exposed in this Console. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en . -- AJ Chen, PhD Chair, Semantic Web SIG, sdforum.org http://web2express.org @web2express on twitter Palo Alto, CA, USA 650-283-4091 *Monitoring social media in
Re: [appengine-java] Re: how to do initialization at startup?
AJ, With GAE, many fundamental assumptions should be questioned. For example, why do you need a counter? Most developers use a counter for the key, jusst because they always have. With GAE you need to often challange the fundamental requirements and rethink the approach. Overall all I think it is good for many developers since thy get in a rut and aply the same solution to all problems (over using a design pattern cause it always worked before) Sent from my Verizon Wireless Phone - Reply message - From: AJ Chen ajc...@web2express.org Date: Wed, Feb 17, 2010 10:52 PM Subject: [appengine-java] Re: how to do initialization at startup? To: google-appengine-java@googlegroups.com -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
