The development server datastore stub is an in memory Map that is persisted to disk. Performance in the local server is not generally indicative of performance in the live system. I'd suggest using a smaller number of entities or doing this type of testing against a live application running on appspot.com.
Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Sat, Mar 19, 2011 at 2:10 AM, Bernhard P. <bernhard.pxx...@gmail.com>wrote: > I have made many attempts to import a large set ot entities > (1.000.000) from a .csv file (14 MB) into the local datastore of the > development server. > > All entities are of one very simple kind which has just one string > property. The values are rather small too: > > key name values: 1 - 1000000, e.g. "1000" > property values: strings with exactly 6 characters, e.g. "qttrqg" > > After the bulkloader has worked for some time info(/error) messages > appear on the command line: > [INFO ] [Thread-9] Backing off due to errors: 1.0 seconds > > There are no further messages in the bulkloader log explaining those > "errors" nor does the development server output any error message. > > While searching for a clue I realized that the JVM of the development > server was using over 512 MB of memory with memory consumption > increasing rapidly while the bulkloader is running. After increasing > the limit (VM argument -Xmx) to 1024 MB and after that to 2 GB the > same errors still appear albeit later. > > In my last attempt the errors started when the JVM was using about 1,4 > GB of memory. > Some bulkloader numbers after stopping the server: > [INFO ] 406400 entities (37193549 bytes) transferred in 1291.9 > seconds > The local datastore file (local_db.bin) has a size of 117 MB. > > Can somebody confirm this behaviour? Is the development server not > able to handle such amounts of data? > > Google App Engine Java SDK 1.4.2.v201102111811, running the server > inside Eclipse. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
