[appengine-java] Re: Cold starts
We're also experiencing frequent application loading. It seems to have gotten more frequent since yesterday's scheduled maintenance. When a new user logs in to our app, there is a burst of requests which come in and this now almost always seems to trigger an application load. I don't know if a new instance is being started up to handle the increased load but if that is the case that instance and the original instance should hang around for some reasonable amount of time. In our case when another user logs in say a minute after, then yet another app load occurs. The frequent app loading causes problems but it is compounded for us because: 1) app load normally takes a little less than 20 seconds but since app loading time (and actually processing time in general) on appengine seems to be highly variable, it can easily wind up taking greater than 30 seconds and appengine cuts off the processing and app is not initialized properly. 2) when load time exceeds 30 seconds and appengine cuts off the load (or more precisely cuts off the request which triggered the load), appengine will often continue to use the loaded app. This app is no good in our case because initialization was not completed. To be safe, appengine should not use the app when it had to cut off the request which triggered the load after 30 seconds. Len On Apr 29, 6:58 am, Jake wrote: > I feel paying for a warm JVM instance doesn't really fall in with what > I want. When I first started GAE, someone from Google hinted that a > single JVM should stay warm for quite some time (perhaps up to an > hour?) from a single hit. In my application, users can expect to sit > on a page for 30+ seconds. As a result, I would say at least 30-40% > of my requests are loading requests. > > Jake > > On Apr 28, 2:01 am, Richard wrote: > > > > > > > Done, thanks. I had also starred the pay-for-instances but it was far > > too noisy a topic! > > > On Apr 27, 7:16 pm, Jake wrote: > > > > Star this? :) > > > >http://code.google.com/p/googleappengine/issues/detail?id=2931 > > > > On Apr 27, 4:33 am, Richard wrote: > > > > > My app has a *lot* of cold starts. Sometimes after 2 or 3 minutes, > > > > which seems very aggressive. I've let it go, tried to be good and > > > > waited for it to improve. Today I tried Pingdom and set it to a > > > > minute. Instant improvement. > > > > > Googleistas, I really, really don't like doing that. I'll likely > > > > switch it off because I'm trying to be a good citizen, but the system > > > > is really driving bad behaviour - I imagine anyone who vaguely depends > > > > on GAE for money is pinging the heck out of their apps. Any chance > > > > we're going to get it sorted soon? > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "Google App Engine for Java" group. > > > > To post to this group, send email to > > > > google-appengine-j...@googlegroups.com. > > > > To unsubscribe from this group, send email to > > > > google-appengine-java+unsubscr...@googlegroups.com. > > > > For more options, visit this group > > > > athttp://groups.google.com/group/google-appengine-java?hl=en. > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine for Java" group. > > > To post to this group, send email to > > > google-appengine-j...@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine-java+unsubscr...@googlegroups.com. > > > For more options, visit this group > > > athttp://groups.google.com/group/google-appengine-java?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to google-appengine-j...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine-java+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine-java?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine-java?hl=en.- Hide quoted > text - > > - Show quoted text - -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Sporadic problems with very high response times
Ikai, it has been a week since this response from you. Do you have an ETA of when this new document will go live? Regards, Len On Mar 8, 1:03 pm, "Ikai L (Google)" wrote: > Michael, > > I have an answer, but it's not authoritative and it also may not be > 100% correct. If you don't mind waiting a few days, we're working on > some documentation about this which should go live soon. It's going > through peer review right now for correctness. I'll go on the main > site in the section about concurrent requests. > > > > > > On Thu, Mar 4, 2010 at 2:25 PM, Michael wrote: > > Hi Ikai, > > > Yes, that was me who turned up at office hours last night. Thanks for > > the response, both then and now. I've gone ahead and starred the > > issue you noted; it's an interesting proposal. > > > I understand the need forloadingtimes, but I have a question as to > > how this scales. Presumably, theloadingtimes occur as new machines > > which serve requests are added; normally, I would think that scaling > > up by only one or two instances at a time is sufficient to meet > > request per second growth. What if, however, there is a (temporary) > > exponential growth in the number of requests per second; will that > > impact the response time of all of those requests? I was going to run > > some simulations to test this, but if you already have intuition as to > > what will happen, that would be helpful. > > > To be more clear, if your application can normally be easily serviced > > by one or two instances (1 to 10 qps), but then shoots up over a very > > short span of time (less than ten seconds) to 300-500 qps, will all of > > those requests lag behind while new instances are loaded to service > > the traffic spike? That is, will the service time for each request be > > increased by theloadingtime except for those requests that are able > > to be served by the original instances? > > > Right now I'm not using any frameworks or any dependencies outside of > > XML parsing (I had to manually add XML jar files to my war's lib > > folder in order for XML parsing not to generate an error when > > deployed, even though it worked fine locally), so I would expect my > > load times to be modest. The user-facing functionality of the site > > actually does use a static page with a "loading" message while it > > queries the server, which I find very useful, but the concern over > >loadingtimes is generated by the functionality that is invisible to > > the user. > > > Thanks again for the help. If I find anything interesting out in > > testing, I'll be sure to post the results online. > > > Regards, > > - Michael > > > On Mar 4, 11:43 am, "Ikai L (Google)" wrote: > >> Michael, > > >> (Molson from the IRC office hours?) > > >> Some small percentage of your application's requests will always be > >>loadingrequests, as this is us spinning up a new instance of your > >> application to either grow for capacity or tearing down your instance > >> and putting it back up as resource allocation demands. We can't > >> predict when this will happen. You may want to star this issue: > > >>http://code.google.com/p/googleappengine/issues/detail?id=2456 > > >> Startup time is generally a function of several different things: > > >> - Spinning up the JVM (Relatively cheap, but on the order of magnitude > >> of 500ms - 1s) > >> - How many dependencies are youloading? (Relatively cheap compared to > >> JVM spinup) > >> - Framework init (Can be VERY expensive -loadingup a dynamic > >> language runtime will always take a few seconds. Some frameworks will > >> also scan every class in your classpath. Spring, for instance, does > >> this to look for annotations eagerly on init time) > > >> Strategies to counteract these factors include optimizing for lazy > >>loading, which spreads the total load time across acess to several > >> different resources. Not many existing frameworks do this. > > >> As your application grows,loadingrequests should account for a > >> smaller and smaller percentage of your total requests. I've seen > >> solutions with rich applications that show a static pageloading > >> dynamic resources as a general landing page. This doesn't solve the > >> load time solution, but it meets the user halfway by making a web app > >> appear to load faster as opposed to causing a user's brower window to > >> be blank while waiting for a request to be handled. > > >> On Tue, Mar 2, 2010 at 4:32 PM, Michael wrote: > >> > Looking at my App Engine logs, I see troubling results when viewing > >> > the response times for requests. In my current log set, the first 80 > >> > requests all complete in under 100 ms with less than 100 ms of cpu or > >> > api time. Then, oddly, the 83rd request, from the exact same client > >> > with the exact same request parameters, takes 7,192 ms to respond with > >> > 10,123 cpu ms (and 12 api ms). > > >> > These kinds of spikes are dotted throughout my logs. They occur in > >> > less than 1% of cases, as far as I
[appengine-java] Re: cron job - too many continues
I figured out what the problem is. The problem started when we turned on HTTPS for our app. I saw a posting in the Google App Engine group that "too many continues" and 302 is due to scheduled jobs will not follow redirects and the scheduled jobs use HTTP and gets redirected to HTTPS and it fails. http://groups.google.com/group/google-appengine/browse_frm/thread/8ad1d04a50deff53/ba7db7e75ec5e81a?lnk=gst&q=cron+job+302#ba7db7e75ec5e81a Len On Feb 5, 9:38 am, lent wrote: > Hello, > > The cron job that I have scheduled to run displays status: > on time Too many continues > > And in the log all the cron jobs requests show a status of 302. > > What does "Too many continues" mean and how can I get the cron jobs > back working again? > > Regards, > Len -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] cron job - too many continues
Hello, The cron job that I have scheduled to run displays status: on time Too many continues And in the log all the cron jobs requests show a status of 302. What does "Too many continues" mean and how can I get the cron jobs back working again? Regards, Len -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: JPA Entities Queried Outside Transaction Carried Into Next Transaction
Thanks Andy for your quick response. I'm using JPA (as mentioned in the title) so I used the equivalent of pm.evictAll in JPA which I think is entityManager.clear and it did the trick. The entities were removed from the level 1 cache and the following transaction had no problems. Thanks, Len On Jan 26, 7:55 am, datanucleus wrote: > When running in non-tx mode objects will be put in the L1 cache, yes, > since you are in non-tx mode. All other DN supported datastores don't > have any issue with objects being reused, so that is a problem for GAE/ > J. If you wanted to remove them from the L1 cache (to attempt to get > around their problem) then call pm.evictAll(...). -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] JPA Entities Queried Outside Transaction Carried Into Next Transaction
Hello, I perform a query for some entities without a transaction on purpose since the query returns ids of entities which are not in the same entity group. Then a transaction is started to update one of the entities whose id is returned before the transaction was started. When the transaction goes to commit, then I get an error saying that I cannot operate on multiple entity groups in a single transaction. I turned on datanucleus debug logging and what seems to be happening is that the query done with no transaction is resulting in entities being put in the level 1 cache and somehow these entities in the cache are being carried into the next transaction and causing the problem. Is my analysis is correct and if so is this the expected behavior? I have attached the debug log output below. Any insights into this would be appreciated. Regards, Len 01-25 09:58PM 12.169 org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id "com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzch0LEgRVc2VyGIP6AQwLEgtDb250cmlidXRvchgBDA" not found in Level 1 cache [cache size = 0] D 01-25 09:58PM 12.169 org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id "com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzch0LEgRVc2VyGIP6AQwLEgtDb250cmlidXRvchgBDA" not found in Level 1 cache [cache size = 0] D 01-25 09:58PM 12.170 org.datanucleus.ObjectManagerImpl putObjectIntoCache: Object "com.jostleme.jostle.server.domain.contribu...@1b18f22" (id="com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzch0LEgRVc2VyGIP6AQwLEgtDb250cmlidXRvchgBDA") added to Level 1 cache (loadedFlags="[YNNN]") D 01-25 09:58PM 12.171 org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id "com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzch0LEgRVc2VyGIT6AQwLEgtDb250cmlidXRvchgBDA" not found in Level 1 cache [cache size = 1] D 01-25 09:58PM 12.171 org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id "com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzch0LEgRVc2VyGIT6AQwLEgtDb250cmlidXRvchgBDA" not found in Level 1 cache [cache size = 1] D 01-25 09:58PM 12.171 org.datanucleus.ObjectManagerImpl putObjectIntoCache: Object "com.jostleme.jostle.server.domain.contribu...@2c189d" (id="com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzch0LEgRVc2VyGIT6AQwLEgtDb250cmlidXRvchgBDA") added to Level 1 cache (loadedFlags="[YNNN]") D 01-25 09:58PM 12.172 org.datanucleus.transaction.Transaction : Transaction created [DataNucleus Transaction, ID=Xid=D 01-25 09:58PM 12.172 org.datanucleus.TransactionImpl internalBegin: Transaction begun for ObjectManager org.datanucleus.objectmanageri...@b3529b (optimistic=true) D 01-25 09:58PM 12.173 org.datanucleus.ObjectManagerImpl getObjectFromCache: Object "com.jostleme.jostle.server.domain.contribu...@1b18f22" (id="com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzch0LEgRVc2VyGIP6AQwLEgtDb250cmlidXRvchgBDA") taken from Level 1 cache (loadedFlags="[YNNN]") [cache size = 2] D 01-25 09:58PM 12.174 org.datanucleus.transaction.Transaction enlistResource: Running enlist operation on resource: org.datanucleus.store.appengine.datastorexaresou...@181e8ba, error code TMNOFLAGS and transaction: [DataNucleus Transaction, ID=Xid=D 01-25 09:58PM 12.190 org.datanucleus.store.appengine.DatastoreXAResource start: Started new datastore transaction: -5667809142913818324 D 01-25 09:58PM 12.191 org.datanucleus.store.appengine.DatastorePersistenceHandler get: Getting entity of kind Contributor with key User(32003)/Contributor(1) D 01-25 09:58PM 12.211 org.datanucleus.ObjectManagerImpl enlistInTransaction: Object "com.jostleme.jostle.server.domain.contribu...@1b18f22" (id="agpzaGFxYmFtYnUzch0LEgRVc2VyGIP6AQwLEgtDb250cmlidXRvchgBDA") enlisted in transactional cache D 01-25 09:58PM 12.212 org.datanucleus.ObjectManagerImpl persistObjectInternal: Making object persistent : "com.jostleme.jostle.server.domain.contribu...@1b18f22" D 01-25 09:58PM 12.213 org.datanucleus.TransactionImpl internalPreCommit: Transaction committing for ObjectManager org.datanucleus.objectmanageri...@b3529b D 01-25 09:58PM 12.213 org.datanucleus.ObjectManagerImpl flushInternal: ObjectManager internalFlush() process started - 1 dirty objects D 01-25 09:58PM 12.214 org.datanucleus.store.appengine.DatastorePersistenceHandler put: Putting entity of kind Contributor with key User(32003)/Contributor(1) D 01-25 09:58PM 12.214 org.datanucleus.store.appengine.DatastorePersistenceHandler put: firstName : Bob D 01-25 09:58PM 12.215 org.datanucleus.store.appengine.DatastorePersistenceHandler put: lastName : Smith ... D 01-25 09:58PM 12.244 org.datanucleus.ObjectManagerImpl flushInternal: ObjectManager internalFlush() process finished D 01-25 09:58PM 12.245 org.datanucleus.store.appengine.DatastorePersistenceHandler get: Getting entity of kind Contributor with
[appengine-java] Re: database transaction problem
Max, > "attempted to deregister a transaction that is not currently registered" is > an error thrown by the datastore api when a call is made to commit a txn and > there is no txn to commit. Is it possible you have some path through your > code that would result in calling commit() more than once or calling > commit() after a rollback()? On the first occurrence of this exception, the request triggered application to be loaded and the request processing time was getting to 30 seconds (according to the log total time for the request was 30.417s). Is it possible that appengine's rollback logic for the 30 second limit started kicking in just about the time the app tried to call commit? > I can't explain the partial data you're seeing - transactional writes are > atomic from your app down to the datastore down to the storage layer that > the datastore is built on. Since you had 2 cron jobs executing concurrently > is it possible that the writes interacted with one another in some > previously unforeseen way? It is possible. The request processing logic runs within a spring transaction context. There is a third party library which is using low level api to do updates so maybe these are not behaving in the way that I'm expecting. I will take a more closer look into if the requests are properly being processed within a transaction. Thanks, Len -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Application Memory Usage
I just did some tests and it looks like limit is around 100MB as I start getting OutOfMemory exceptions in the java heap space around that point. Len On Dec 7, 5:09 pm, lent wrote: > Hi, > > Jason (Google) has mentioned that there is a limit on the amount of > memory an application can consume in another thread: > > "There is an upper limit to the amount of memory your app can consume, > but > it's more than reasonable for the type of applications that App Engine > is > built to serve. If your application surpasses this limit, an exception > will > be thrown. " > > What is the limit (if not an exact value an approximate value)? I > need to make some decisions about what to store in memcache and what > to store in the application (in static variables, servlet instance > variables, etc) and this decision depends to some degree on how much > data I can reasonably store in the application. > > Regards, > Len -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: database transaction problem
I took a look at the log entries more closely and it looks like the cron job which is to run daily got kicked off twice for some reason. Instead of one chain of 500 sequential tasks being started up two chains of 500 were running in parallel. This may explain the errors I was getting though I don't understand how there seems to have been a partial update. Len On Dec 8, 12:35 am, lent wrote: > Hello, > > I have a daily cron job which kicks off a sequential 500 tasks > (chained). These usually run successfuly but a few days ago, on the > first task, I got the following error: > > [xxx/1.338222621935386721].: > java.lang.IllegalStateException: Attempted to deregister a transaction > that is not currently registered. > > On the next task, I got the error again: > > [xxx/1.338222621935386721].: > java.lang.IllegalStateException: Attempted to deregister a transaction > that is not currently registered. > > Then on the remaining 498 tasks I got the error: > > com.jostleme.jostle.ui.web.controller.DriverController > processRefreshContributorTagsTask: failure while updating refresh tags > info. Cause: Transaction rolled back due to failure during commit; > nested exception is javax.persistence.PersistenceException: > Transaction rolled back due to failure during commit > > After this data seems to have gotten corrupted. By this I mean that > something that should have run in a transaction (all or nothing) seems > to have done a partial update. > > Does anyone (especially google folks) have any insights on this > behavior? > > Regards, > Len -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] database transaction problem
Hello, I have a daily cron job which kicks off a sequential 500 tasks (chained). These usually run successfuly but a few days ago, on the first task, I got the following error: [xxx/1.338222621935386721].: java.lang.IllegalStateException: Attempted to deregister a transaction that is not currently registered. On the next task, I got the error again: [xxx/1.338222621935386721].: java.lang.IllegalStateException: Attempted to deregister a transaction that is not currently registered. Then on the remaining 498 tasks I got the error: com.jostleme.jostle.ui.web.controller.DriverController processRefreshContributorTagsTask: failure while updating refresh tags info. Cause: Transaction rolled back due to failure during commit; nested exception is javax.persistence.PersistenceException: Transaction rolled back due to failure during commit After this data seems to have gotten corrupted. By this I mean that something that should have run in a transaction (all or nothing) seems to have done a partial update. Does anyone (especially google folks) have any insights on this behavior? Regards, Len -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Application Memory Usage
Hi, Jason (Google) has mentioned that there is a limit on the amount of memory an application can consume in another thread: "There is an upper limit to the amount of memory your app can consume, but it's more than reasonable for the type of applications that App Engine is built to serve. If your application surpasses this limit, an exception will be thrown. " What is the limit (if not an exact value an approximate value)? I need to make some decisions about what to store in memcache and what to store in the application (in static variables, servlet instance variables, etc) and this decision depends to some degree on how much data I can reasonably store in the application. Regards, Len -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Text Search Support for Java
Hi, This question is targeted at Google guys. Is there any plans to have limited support for full text search for Java, i.e. something like the equivalent of SearchableModel in Python, and if so when can this be expected? When is FULL support for full text search be expected for appengine (Python and Java)? Regards, Len -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: bulkloader dump and restore
Thanks Ikai. The approach you suggested for exporting the key field
worked for me.
Len
On Nov 19, 5:36 pm, "Ikai L (Google)" wrote:
> Len,
>
> I assume you are doing this by having different application versions for a
> Python application and Java application as documented here:
>
> http://googleappengine.blogspot.com/2009/06/10-things-you-probably-di...
>
> Then redefining your Java data class in Python in your Exporter class and
> passing a string similar to this to appcfg.py:
>
> appcfg.py --server=python.latest.YOURAPPID.appspot.com download_data APPDIR
> --filename=output.csv --kind=Thing --config_file=APPDIR/thing_exporter.py
>
> There's a trick you'll need to export the key. Here's my Thing.java
> (http://pastie.org/706983):
>
> import com.google.appengine.api.datastore.Key;
> import javax.jdo.annotations.*;
>
> @PersistenceCapable(identityType = IdentityType.APPLICATION)
> public class Thing {
>
> @PrimaryKey
> @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
> private Key key;
>
> @Persistent
> private String name;
>
> @Persistent
> private int number;
>
> public Thing(String name, int number) {
> this.name = name;
> this.number = number;
> }
>
> public Key getKey() {
> return key;
> }
>
> public String getName() {
> return name;
> }
>
> public void setName(String name) {
> this.name = name;
> }
>
> public int getNumber() {
> return number;
> }
>
> public void setNumber(int number) {
> this.number = number;
> }
>
> }
>
> Here's the corresponding Python exporter code (http://pastie.org/706981):
>
> from google.appengine.ext import db
> from google.appengine.tools import bulkloader
>
> class Thing(db.Model):
> name = db.StringProperty()
> number = db.IntegerProperty()
>
> def AddKeys(entity_generator):
> for entity in entity_generator:
> entity['key'] = entity.key()
> yield entity
>
> class ThingExporter(bulkloader.Exporter):
> def __init__(self):
> bulkloader.Exporter.__init__(self, 'Thing',
> [('key', str, None),
> ('name', str, None),
> ('number', str, None)
> ])
>
> def output_entities(self, entity_generator):
> bulkloader.Exporter.output_entities(self, AddKeys(entity_generator))
>
> exporters = [ThingExporter]
>
> The magic here is in the AddKey that you define. You add the "key" field to
> the objects in each iteration of your loop exporting an entity.
>
>
>
>
>
> On Fri, Nov 13, 2009 at 4:15 PM, lent wrote:
> > Hello,
>
> > I got python bulkloader upload data and download data working with
> > loader and exporter for my java app. Do the bulkloader dump and
> > restore options work when used with java app? They don't seem to work
> > for me. I get an error like:
> > BadRequestError: app xxx cannot access app bulkload.latest.xxx's data
>
> > When doing download data, how can I get the id field exported, i.e.
> > what do you need to specify in the exporter to export the id field? I
> > tried field names such as key and id (of type db.Key) but they didn't
> > work.
>
> > Regards,
> > Len
>
> > --
>
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine for Java" group.
> > To post to this group, send email to
> > google-appengine-j...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine-java+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine-java?hl=.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine- Hide quoted text -
>
> - Show quoted text -
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=.
[appengine-java] bulkloader dump and restore
Hello, I got python bulkloader upload data and download data working with loader and exporter for my java app. Do the bulkloader dump and restore options work when used with java app? They don't seem to work for me. I get an error like: BadRequestError: app xxx cannot access app bulkload.latest.xxx's data When doing download data, how can I get the id field exported, i.e. what do you need to specify in the exporter to export the id field? I tried field names such as key and id (of type db.Key) but they didn't work. Regards, Len -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=.
[appengine-java] Re: Maven
1.2.6 jars have still not been put in mvnsearch repository: http://www.mvnsearch.org/maven2/com/google/appengine/ Are there plans to do this? Or is it already available in a different public maven repository? Please make putting the appengine-jdk on a public maven repository part of your release process. Regards, Len On Oct 19, 1:06 am, jettro wrote: > +1, would be really helpful > > On Oct 16, 8:53 pm, Philippe Marschall > wrote: > > > > > On Oct 16, 8:30 pm, lent wrote: > > > > please include 1.2.6 jars > > > tohttp://www.mvnsearch.org/maven2/com/google/appengine/ > > > +1 > > > Cheers > > Philippe- Hide quoted text - > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: JPA Problem with Multiple Owned One to One relationships to same type
Max, thanks for letting me know. I'm going to work around it for now
by turning it into a OneToMany and differentiating the values myself.
Len
On Oct 29, 4:17 pm, "Max Ross (Google)"
wrote:
> This is a current limitation that I hope to get rid of fairly soon. The
> problem is that the parent does not have the key of the child as a field.
> Instead, in order to find the child, the parent issues an ancestor query to
> find the single object of the appropriate type that is a direct descendant.
> I describe the reason for this odd implementation in detail in my Google IO
> talk <http://www.youtube.com/watch?v=2jW2iSKDipY> from last summer. An
> annoying side effect of this implementation is that you can't have two
> relationship fields of the same type because we have no way of knowing which
> child is associated with which field. We'll get it straightened out in the
> next few releases, I promise.
>
> Max
>
>
>
> On Wed, Oct 28, 2009 at 9:49 PM, lent wrote:
>
> > Hello,
>
> > I'm using JPA with SDK 1.2.5 and I'm having a problem with multiple
> > owned One To One relationships where the relationships are to the same
> > type. The code is as below and Contributor class has three owned One
> > to One relationships to ContentValue.
>
> > What I'm finding is that when I set one of the One to One properties
> > in one transaction (e.g. picutre) in one request and then later on in
> > a different request and transaction I retrieve another of the One to
> > One properties (e.g. tagData), I get back the ContentValue I set for
> > the first property (picture).
>
> > I'm doing the following to set the property:
>
> > Contributor contributor = ;
> > ContentValue picture = contributor.getPicture();
> > if ( picture == null ) {
> > picture = new ContentValue();
> > contributor.setPicture( picture );
> > pictureContent.setContentValueType( ContentValueType.PICTURE );
> > }
> > ...
>
> > If anyone knows why I'm having this problem, please let me know.
>
> > Regards,
> > Len
>
> > Contributor.java
> >
>
> > package abc;
>
> > //import com.google.appengine.api.datastore.Key;
> > //import com.google.appengine.api.datastore.KeyFactory;
> > import org.datanucleus.jpa.annotations.Extension;
>
> > import javax.persistence.AttributeOverrides;
> > import javax.persistence.CascadeType;
> > import javax.persistence.Embedded;
> > import javax.persistence.FetchType;
> > import javax.persistence.Entity;
> > import javax.persistence.Enumerated;
> > import javax.persistence.GeneratedValue;
> > import javax.persistence.GenerationType;
> > import javax.persistence.Id;
> > import javax.persistence.OneToMany;
> > import javax.persistence.OneToOne;
> > import javax.persistence.Transient;
>
> > import java.util.ArrayList;
> > import java.util.Date;
> > import java.util.List;
> > import java.util.Date;
>
> > @Entity
> > public class Contributor {
> > �...@id
> > �...@generatedvalue(strategy=GenerationType.IDENTITY)
> > �...@extension(vendorName="datanucleus", key="gae.encoded-pk",
> > value="true")
> > private String id;
>
> > �...@enumerated
> > private String firstName;
>
> > �...@enumerated
> > private String lastName;
>
> > �...@onetoone(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
> > private ContentValue picture;
>
> > �...@onetoone(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
> > private ContentValue thumbnailPicture;
>
> > �...@onetoone(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
> > private ContentValue tagData;
>
> > public void setId(String id) {
> > this.id = id;
> > }
>
> > public String getId() {
> > return id;
> > }
>
> > public void setFirstName(String firstName) {
> > this.firstName = firstName;
> > }
>
> > public String getFirstName() {
> > return firstName;
> > }
>
> > public void setLastName(String lastName) {
> > this.lastName = lastName;
> > }
>
> > public String getLastName() {
> > return lastName;
> > }
>
> > public void setPicture(ContentValue picture) {
> >
[appengine-java] JPA Problem with Multiple Owned One to One relationships to same type
Hello,
I'm using JPA with SDK 1.2.5 and I'm having a problem with multiple
owned One To One relationships where the relationships are to the same
type. The code is as below and Contributor class has three owned One
to One relationships to ContentValue.
What I'm finding is that when I set one of the One to One properties
in one transaction (e.g. picutre) in one request and then later on in
a different request and transaction I retrieve another of the One to
One properties (e.g. tagData), I get back the ContentValue I set for
the first property (picture).
I'm doing the following to set the property:
Contributor contributor = ;
ContentValue picture = contributor.getPicture();
if ( picture == null ) {
picture = new ContentValue();
contributor.setPicture( picture );
pictureContent.setContentValueType( ContentValueType.PICTURE );
}
...
If anyone knows why I'm having this problem, please let me know.
Regards,
Len
Contributor.java
package abc;
//import com.google.appengine.api.datastore.Key;
//import com.google.appengine.api.datastore.KeyFactory;
import org.datanucleus.jpa.annotations.Extension;
import javax.persistence.AttributeOverrides;
import javax.persistence.CascadeType;
import javax.persistence.Embedded;
import javax.persistence.FetchType;
import javax.persistence.Entity;
import javax.persistence.Enumerated;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.OneToMany;
import javax.persistence.OneToOne;
import javax.persistence.Transient;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Date;
@Entity
public class Contributor {
@Id
@GeneratedValue(strategy=GenerationType.IDENTITY)
@Extension(vendorName="datanucleus", key="gae.encoded-pk",
value="true")
private String id;
@Enumerated
private String firstName;
@Enumerated
private String lastName;
@OneToOne(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
private ContentValue picture;
@OneToOne(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
private ContentValue thumbnailPicture;
@OneToOne(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
private ContentValue tagData;
public void setId(String id) {
this.id = id;
}
public String getId() {
return id;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getFirstName() {
return firstName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
public String getLastName() {
return lastName;
}
public void setPicture(ContentValue picture) {
this.picture = picture;
}
public ContentValue getPicture() {
return picture;
}
public void setThumbnailPicture(ContentValue thumbnailPicture) {
this.thumbnailPicture = thumbnailPicture;
}
public ContentValue getThumbnailPicture() {
return thumbnailPicture;
}
public void setTagData(ContentValue tagData) {
this.tagData = tagData;
}
public ContentValue getTagData() {
return tagData;
}
}
ContentValue.java
-
package abc;
import org.datanucleus.jpa.annotations.Extension;
import com.google.appengine.api.datastore.Blob;
import javax.persistence.AttributeOverrides;
import javax.persistence.Entity;
import javax.persistence.CascadeType;
import javax.persistence.Enumerated;
import javax.persistence.Embedded;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.NamedQuery;
import javax.persistence.OneToOne;
@Entity
public class ContentValue {
@Id
@GeneratedValue(strategy=GenerationType.IDENTITY)
@Extension(vendorName="datanucleus", key="gae.encoded-pk",
value="true")
private String id;
@Enumerated
@Extension(vendorName="datanucleus", key="gae.parent-pk",
value="true")
private String parentId;
@Enumerated
private ContentValueType contentValueType;
@Enumerated
private Blob content;
@Enumerated
private String contentType;
public void setId(String id) {
this.id = id;
}
public String getId() {
return id;
}
public void setParentId(String parentId) {
this.parentId = parentId;
}
public String getParentId() {
return parentId;
}
public void setContentValueType(ContentValueType contentValueType) {
this.contentValueType = conte
[appengine-java] Re: Possible to Add Child to Owned One to Many Relationship Without Loading All Children?
Thanks Nacho. I asked the same question in google appengine chat irc and got the same answer. Will have to use the allocate id feature in low level api. Len On Oct 21, 7:41 am, Nacho Coloma wrote: > Maybe what you are looking for is DatastoreService.allocateIds(). > > On Oct 21, 9:35 am, leszek wrote: > > > > > You hit the nail on the head. As far as I know there is nothing like > > "sequence" in Google App Engine. Either you need to have another > > entity with a counter and increase it in transactional way. Or you can > > use memcache. > > >http://code.google.com/intl/pl/appengine/docs/java/javadoc/com/google... > > > There is a method: > > > > java.lang.Long increment(java.lang.Object key, long delta) > > Atomically fetches, increments, and stores a given integral > > value. > > --- > > > Pay attention to "atomically". May be this can be used for generating > > sequential and avoid bottleneck in the case of datastore entity and > > additional transaction. But because of the risk of memcache expiring > > this method needs some more elaboration.- Hide quoted text - > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: Possible to Add Child to Owned One to Many Relationship Without Loading All Children?
Thanks leszek. We are going do what you suggest (the JPA
equivalent).
In order to generate the unique part of the child's key, I'm wondering
if there is a way to use sequence in JPA. In JDO, I think you can do
something like:
Sequence seq = pm.getSequence("child.sequence");
long id = seq.nextValue();
Does anyone know if there is an equivalent to this in JPA, i.e. to
get direct access to a sequence?
Thanks,
Len
On Oct 20, 4:04 am, leszek wrote:
> Look at the code snippet below:
>
> // entity class
>
> @PersistenceCapable(identityType = IdentityType.APPLICATION)
> public class ChildEnt {
>
> @PrimaryKey
> @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
> // �...@extension(vendorName="datanucleus", key="gae.encoded-
> pk",value="true")
> private Key encodedKey;
>
> private String info;
>
> public Key getEncodedKey() {
> return encodedKey;
> }
>
> public void setEncodedKey(Key encodedKey) {
> this.encodedKey = encodedKey;
> }
>
> public String getInfo() {
> return info;
> }
>
> public void setInfo(String info) {
> this.info = info;
> }
>
> }
>
> // transaction code
>
> PersistenceManager pm = PMF.get().getPersistenceManager();
>
> Key parkey = KeyFactory.createKey("Parent","P01");
>
> ChildEnt e1 = new ChildEnt();
> ChildEnt e2 = new ChildEnt();
> ChildEnt e3 = new ChildEnt();
> Key k1 = KeyFactory.createKey(parkey,ChildEnt.class.getSimpleName(),
> 1);
> Key k2 = KeyFactory.createKey(parkey,ChildEnt.class.getSimpleName(),
> 2);
> Key k3 = KeyFactory.createKey(parkey,ChildEnt.class.getSimpleName(),
> 3);
> e1.setEncodedKey(k1);
> e1.setInfo("info1");
> e2.setEncodedKey(k2);
> e2.setInfo("info2");
> e3.setEncodedKey(k3);
> e3.setInfo("info3");
> pm.currentTransaction().begin();
> pm.makePersistent(e1);
> pm.makePersistent(e2);
> pm.makePersistent(e3);
> pm.currentTransaction().commit();
> pm.close();
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Possible to Add Child to Owned One to Many Relationship Without Loading All Children?
Hi We're using JPA and we have an owned one to many relationship that has many children (we create this owned relationship so that any combination of children can be updated in a transaction). When we add an additional child, we add it to the parent's children list. This seems to cause a loading of all children and hence takes a long time due to the large number of children. Is there any way to add a child without causing this loading to happen (some other way to set the parent without getting the parent's children list and adding to it)? Thanks, Len --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: query performance problem
We upgraded to appengine-java-sdk 1.2.5 today (from 1.2.2) and the
performance seems to
be back to what we were experiencing earlier which was well under a
second for 100 entities.
We're not sure why but the performance when we were using 1.2.2 was
similar to not specifying
the one to one owned fields as lazy loading.
Len
On Sep 22, 10:42 am, lent wrote:
> Hello,
>
> We are using JPA and the retrieval of one of our kinds seems to be
> quite slow at about 3-4 seconds per 100 in production. The entities
> of this kind are not in the same entity group. We have made some
> adjustments to the kind recently which may have caused the slow down.
> The persistent class definition is as below. We used to have a blob
> field for content which we have moved to a separate class
> (ContentValue) and created OneToOne owned relationship with (we used
> to have one blob for content but now we have 2 ContentValue fields).
> Two of our fields used to be String but now has been converted to
> List fields. We have added a OneToMany owned relationship to
> another class. We seemed to recall performance used to be better
> prior to some of these adjustments. Is the performance we're seeing
> reasonable?
>
> The query we issue is as follows with the number of results capped at
> 100 using setMaxResult on the query in JPA:
> select c from Contributor c where id > someId
>
> From the retrieved query result, we do not access the two ContentValue
> fields or the ContactCollection list. Some datanucleus debug output
> is also at the bottom.
>
> Any insights into the performance would be appreciated.
>
> Regards,
> Len
>
> @Entity
> public class Contributor {
> @Id
> @GeneratedValue(strategy=GenerationType.IDENTITY)
> @Extension(vendorName="datanucleus", key="gae.encoded-pk",
> value="true")
> private String id;
>
> @Enumerated
> @Extension(vendorName="datanucleus", key="gae.parent-pk",
> value="true")
> private String userId;
>
> @Enumerated
> private String firstName;
>
> @Enumerated
> private String lastName;
>
> @Enumerated
> private String title;
>
> @Enumerated
> private String division;
>
> @OneToOne(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
> private ContentValue picture;
>
> @OneToOne(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
> private ContentValue thumbnailPicture;
>
> @OneToMany(cascade = CascadeType.ALL)
> private List collections = new
> ArrayList();
>
> @Enumerated
> private String bio;
>
> @Enumerated
> private List skills;
>
> @Enumerated
> private List qualifications;
>
> @Enumerated
> private Date birthDate;
>
> @Enumerated
> private String workEmail;
>
> @Enumerated
> private String workOfficePhone;
>
> @Enumerated
> private String workMobilePhone;
>
> @Enumerated
> private String personalEmail;
>
> @Enumerated
> private String homePhone;
>
> @Enumerated
> private String personalMobilePhone;
>
> // accessors
> ...
>
> }
>
> DataNucleus Debug Output:
>
> D 09-21 04:36PM 34.665
> org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id
> "com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzchwLEgRVc2VyGN87DAsSC0NvbnRyaWJ1dG9yGAEM"
> not found in Level 1 cache [cache size = 0]
> D 09-21 04:36PM 34.667
> org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id
> "com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzchwLEgRVc2VyGN87DAsSC0NvbnRyaWJ1dG9yGAEM"
> not found in Level 1 cache [cache size = 0]
> D 09-21 04:36PM 34.667
> org.datanucleus.ObjectManagerImpl putObjectIntoCache: Object
> "com.jostleme.jostle.server.domain.contribu...@12c4d93"
> (id="com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzchwLEgRVc2VyGN87DAsSC0NvbnRyaWJ1dG9yGAEM")
> added to Level 1 cache (loadedFlags="[NNNYNN]")
> D 09-21 04:36PM 34.766
> org.datanucleus.sco.backed.List : Object
> "com.jostleme.jostle.server.domain.contribu...@12c4d93" field
> "collections" is replaced by a SCO wrapper of type
> "org.datanucleus.sco.backed.List" [cache-values=true, lazy-
> loading=true, queued-operations=true, allow-nulls=false]
> D 09-2
[appengine-java] query performance problem
Hello,
We are using JPA and the retrieval of one of our kinds seems to be
quite slow at about 3-4 seconds per 100 in production. The entities
of this kind are not in the same entity group. We have made some
adjustments to the kind recently which may have caused the slow down.
The persistent class definition is as below. We used to have a blob
field for content which we have moved to a separate class
(ContentValue) and created OneToOne owned relationship with (we used
to have one blob for content but now we have 2 ContentValue fields).
Two of our fields used to be String but now has been converted to
List fields. We have added a OneToMany owned relationship to
another class. We seemed to recall performance used to be better
prior to some of these adjustments. Is the performance we're seeing
reasonable?
The query we issue is as follows with the number of results capped at
100 using setMaxResult on the query in JPA:
select c from Contributor c where id > someId
>From the retrieved query result, we do not access the two ContentValue
fields or the ContactCollection list. Some datanucleus debug output
is also at the bottom.
Any insights into the performance would be appreciated.
Regards,
Len
@Entity
public class Contributor {
@Id
@GeneratedValue(strategy=GenerationType.IDENTITY)
@Extension(vendorName="datanucleus", key="gae.encoded-pk",
value="true")
private String id;
@Enumerated
@Extension(vendorName="datanucleus", key="gae.parent-pk",
value="true")
private String userId;
@Enumerated
private String firstName;
@Enumerated
private String lastName;
@Enumerated
private String title;
@Enumerated
private String division;
@OneToOne(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
private ContentValue picture;
@OneToOne(cascade = CascadeType.ALL, fetch = FetchType.LAZY )
private ContentValue thumbnailPicture;
@OneToMany(cascade = CascadeType.ALL)
private List collections = new
ArrayList();
@Enumerated
private String bio;
@Enumerated
private List skills;
@Enumerated
private List qualifications;
@Enumerated
private Date birthDate;
@Enumerated
private String workEmail;
@Enumerated
private String workOfficePhone;
@Enumerated
private String workMobilePhone;
@Enumerated
private String personalEmail;
@Enumerated
private String homePhone;
@Enumerated
private String personalMobilePhone;
// accessors
...
}
DataNucleus Debug Output:
D 09-21 04:36PM 34.665
org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id
"com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzchwLEgRVc2VyGN87DAsSC0NvbnRyaWJ1dG9yGAEM"
not found in Level 1 cache [cache size = 0]
D 09-21 04:36PM 34.667
org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id
"com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzchwLEgRVc2VyGN87DAsSC0NvbnRyaWJ1dG9yGAEM"
not found in Level 1 cache [cache size = 0]
D 09-21 04:36PM 34.667
org.datanucleus.ObjectManagerImpl putObjectIntoCache: Object
"com.jostleme.jostle.server.domain.contribu...@12c4d93"
(id="com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzchwLEgRVc2VyGN87DAsSC0NvbnRyaWJ1dG9yGAEM")
added to Level 1 cache (loadedFlags="[NNNYNN]")
D 09-21 04:36PM 34.766
org.datanucleus.sco.backed.List : Object
"com.jostleme.jostle.server.domain.contribu...@12c4d93" field
"collections" is replaced by a SCO wrapper of type
"org.datanucleus.sco.backed.List" [cache-values=true, lazy-
loading=true, queued-operations=true, allow-nulls=false]
D 09-21 04:36PM 34.773
org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id
"com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzchwLEgRVc2VyGOA7DAsSC0NvbnRyaWJ1dG9yGAEM"
not found in Level 1 cache [cache size = 1]
D 09-21 04:36PM 34.773
org.datanucleus.ObjectManagerImpl getObjectFromCache: Object with id
"com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzchwLEgRVc2VyGOA7DAsSC0NvbnRyaWJ1dG9yGAEM"
not found in Level 1 cache [cache size = 1]
D 09-21 04:36PM 34.774
org.datanucleus.ObjectManagerImpl putObjectIntoCache: Object
"com.jostleme.jostle.server.domain.contribu...@1845cce"
(id="com.jostleme.jostle.server.domain.Contributor:agpzaGFxYmFtYnUzchwLEgRVc2VyGOA7DAsSC0NvbnRyaWJ1dG9yGAEM")
added to Level 1 cache (loadedFlags="[NNNYNN]")
D 09-21 04:36PM 34.805
org.datanucleus.sco.backed.List : Object
"com.jostleme.jostle.server.domain.contribu...@1845cce" field
"collections" is replaced by a SCO wrapper of type
"org.datanucleus.sco.backed.List" [cache-values=true, lazy-
loading=true, queued-operations=true, allow-nulls=false]
D 09-21 04:36PM 34.806
org.datanu
[appengine-java] Re: Getting 'Request was aborted after waiting too long to attempt to service your request.' after application idle
I found one thing that was slowing down creating entity manager
factory. I was not specifying the persistent classes in
persistence.xml so the persistence framework (jpa) code was loading
all the classes in my jar. I listed my persistent classes and now the
entity manager creation is now usually less than 4 seconds.
Len
On Sep 14, 2:04 pm, lent wrote:
> Don, I have sent you my application id through email (by doing reply
> to author).
>
> A few of things I should mention:
> 1) we're using JPA
> 2) we are not creating EMF by doing:
> Persistence.createEntityManagerFactory("transactions-optional");
> instead it is created through spring's LocalEntityManagerFactoryBean
> (which we subclass to ensure only one instance ofentitymanagerfactory)
>
> The EMF initialization is done during spring intialization so it is in
> the servlet initialization path and not request processing path. It
> is just that if GAE chooses to start a new application instance to
> service a request, then the application startup (servlet
> initialization) occurs during request processing.
>
> Len
>
> On Sep 14, 10:37 am, Don Schwarz wrote:
>
>
>
> > Len, what is your application ID?
>
> > On Mon, Sep 14, 2009 at 12:33 PM, Larry Cable wrote:
>
> > > can you move the EMF creation out of the request path into the
> > > initialization path (Servlet.init(ServletConfig) or a static
> > > initializer
> > > in the Servlet class at startup ...
>
> > > BTW I also am using Spring IoC to create and inject the JDO EMF
> > > and I dont think I am seeing the same performance issues?
>
> > > Do you see similar startup costs on the local dev server as well as
> > > production?
>
> > > good luck
>
> > > - Larry
>
> > > On Sep 13, 5:27 pm, lent wrote:
> > > > I have been doing some profiling on application startup and it seems
> > > > that there is a large amount of overhead:
> > > > - it takes about 5 seconds between the time the request is received to
> > > > where our spring initialization code is invoked
> > > > - creating anentitymanagerfactoryis taking on average 8+ seconds
> > > > Sometimes these numbers can be much bigger than this, for example we
> > > > have seen creatingentitymanagerfactoryas much as 14 seconds.
>
> > > > Given that the application needs to its own initialization (which we
> > > > are trying to minimize), we find that the application initialization
> > > > becomes very long and when the server is busy and the above number are
> > > > even higher, the application initialization fails to complete in 30
> > > > seconds. Note that this is just the application initialization part
> > > > and then the actual request has to be processed which will take some
> > > > amount of time.
>
> > > > Given that applications can be started on any given request, it makes
> > > > things difficult with application startup time and request processing
> > > > time with the 30 second limit.
>
> > > > Len
>
> > > > On Sep 8, 10:04 am, lent wrote:
>
> > > > > Hello,
>
> > > > > We have an application which uses GWT (using GWT RPC) on the client
> > > > > side. When the application is left idle for a while (~15 minutes) and
> > > > > the application sends a request, we get this reported on the client
> > > > > side:
>
> > > > > "The server encountered an error and could not complete your request.
> > > > > If the problem persists, please report your problem and mention this
> > > > > error message and the query that caused it."
>
> > > > > In the log, the following is logged:
>
> > > > > "Request was aborted after waiting too long to attempt to service your
> > > > > request. Most likely, this indicates that you have reached your
> > > > > simultaneous active request limit. This is almost always due to
> > > > > excessively high latency in your app. Please seehttp://
> > > code.google.com/appengine/docs/quotas.htmlformore details."
>
> > > > > I believe that our application has been swapped out and and when the
> > > > > request comes in the application is being loaded again as we see some
> > > > > log entries we make when the application (servlet) gets initialized.
> > > > > It does take some time to get the application initialized but the
> > > > > error is reported way before 30 seconds has elapsed. Is there any way
> > > > > to avoid this problem?
>
> > > > > Len- Hide quoted text -
>
> > > > - Show quoted text -- Hide quoted text -
>
> > - Show quoted text -- Hide quoted text -
>
> - Show quoted text -
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: Getting 'Request was aborted after waiting too long to attempt to service your request.' after application idle
Don, I have sent you my application id through email (by doing reply
to author).
A few of things I should mention:
1) we're using JPA
2) we are not creating EMF by doing:
Persistence.createEntityManagerFactory("transactions-optional");
instead it is created through spring's LocalEntityManagerFactoryBean
(which we subclass to ensure only one instance of entity manager
factory)
The EMF initialization is done during spring intialization so it is in
the servlet initialization path and not request processing path. It
is just that if GAE chooses to start a new application instance to
service a request, then the application startup (servlet
initialization) occurs during request processing.
Len
On Sep 14, 10:37 am, Don Schwarz wrote:
> Len, what is your application ID?
>
>
>
> On Mon, Sep 14, 2009 at 12:33 PM, Larry Cable wrote:
>
> > can you move the EMF creation out of the request path into the
> > initialization path (Servlet.init(ServletConfig) or a static
> > initializer
> > in the Servlet class at startup ...
>
> > BTW I also am using Spring IoC to create and inject the JDO EMF
> > and I dont think I am seeing the same performance issues?
>
> > Do you see similar startup costs on the local dev server as well as
> > production?
>
> > good luck
>
> > - Larry
>
> > On Sep 13, 5:27 pm, lent wrote:
> > > I have been doing some profiling on application startup and it seems
> > > that there is a large amount of overhead:
> > > - it takes about 5 seconds between the time the request is received to
> > > where our spring initialization code is invoked
> > > - creating an entity manager factory is taking on average 8+ seconds
> > > Sometimes these numbers can be much bigger than this, for example we
> > > have seen creating entity manager factory as much as 14 seconds.
>
> > > Given that the application needs to its own initialization (which we
> > > are trying to minimize), we find that the application initialization
> > > becomes very long and when the server is busy and the above number are
> > > even higher, the application initialization fails to complete in 30
> > > seconds. Note that this is just the application initialization part
> > > and then the actual request has to be processed which will take some
> > > amount of time.
>
> > > Given that applications can be started on any given request, it makes
> > > things difficult with application startup time and request processing
> > > time with the 30 second limit.
>
> > > Len
>
> > > On Sep 8, 10:04 am, lent wrote:
>
> > > > Hello,
>
> > > > We have an application which uses GWT (using GWT RPC) on the client
> > > > side. When the application is left idle for a while (~15 minutes) and
> > > > the application sends a request, we get this reported on the client
> > > > side:
>
> > > > "The server encountered an error and could not complete your request.
> > > > If the problem persists, please report your problem and mention this
> > > > error message and the query that caused it."
>
> > > > In the log, the following is logged:
>
> > > > "Request was aborted after waiting too long to attempt to service your
> > > > request. Most likely, this indicates that you have reached your
> > > > simultaneous active request limit. This is almost always due to
> > > > excessively high latency in your app. Please seehttp://
> > code.google.com/appengine/docs/quotas.htmlformore details."
>
> > > > I believe that our application has been swapped out and and when the
> > > > request comes in the application is being loaded again as we see some
> > > > log entries we make when the application (servlet) gets initialized.
> > > > It does take some time to get the application initialized but the
> > > > error is reported way before 30 seconds has elapsed. Is there any way
> > > > to avoid this problem?
>
> > > > Len- Hide quoted text -
>
> > > - Show quoted text -- Hide quoted text -
>
> - Show quoted text -
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: Getting 'Request was aborted after waiting too long to attempt to service your request.' after application idle
I have been doing some profiling on application startup and it seems that there is a large amount of overhead: - it takes about 5 seconds between the time the request is received to where our spring initialization code is invoked - creating an entity manager factory is taking on average 8+ seconds Sometimes these numbers can be much bigger than this, for example we have seen creating entity manager factory as much as 14 seconds. Given that the application needs to its own initialization (which we are trying to minimize), we find that the application initialization becomes very long and when the server is busy and the above number are even higher, the application initialization fails to complete in 30 seconds. Note that this is just the application initialization part and then the actual request has to be processed which will take some amount of time. Given that applications can be started on any given request, it makes things difficult with application startup time and request processing time with the 30 second limit. Len On Sep 8, 10:04 am, lent wrote: > Hello, > > We have an application which uses GWT (using GWT RPC) on the client > side. When the application is left idle for a while (~15 minutes) and > the application sends a request, we get this reported on the client > side: > > "The server encountered an error and could not complete your request. > If the problem persists, please report your problem and mention this > error message and the query that caused it." > > In the log, the following is logged: > > "Request was aborted after waiting too long to attempt to service your > request. Most likely, this indicates that you have reached your > simultaneous active request limit. This is almost always due to > excessively high latency in your app. Please > seehttp://code.google.com/appengine/docs/quotas.htmlfor more details." > > I believe that our application has been swapped out and and when the > request comes in the application is being loaded again as we see some > log entries we make when the application (servlet) gets initialized. > It does take some time to get the application initialized but the > error is reported way before 30 seconds has elapsed. Is there any way > to avoid this problem? > > Len --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: Maven
On Sep 4, 8:36 am, Nikolay Gorylenko wrote: > please include 1.2.5 jars > tohttp://www.mvnsearch.org/maven2/com/google/appengine/ +1 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Getting 'Request was aborted after waiting too long to attempt to service your request.' after application idle
Hello, We have an application which uses GWT (using GWT RPC) on the client side. When the application is left idle for a while (~15 minutes) and the application sends a request, we get this reported on the client side: "The server encountered an error and could not complete your request. If the problem persists, please report your problem and mention this error message and the query that caused it." In the log, the following is logged: "Request was aborted after waiting too long to attempt to service your request. Most likely, this indicates that you have reached your simultaneous active request limit. This is almost always due to excessively high latency in your app. Please see http://code.google.com/appengine/docs/quotas.html for more details." I believe that our application has been swapped out and and when the request comes in the application is being loaded again as we see some log entries we make when the application (servlet) gets initialized. It does take some time to get the application initialized but the error is reported way before 30 seconds has elapsed. Is there any way to avoid this problem? Len --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
