[appengine-java] Re: Looking for Bulk Loader beta testers
Thanks Mathew -- The --dump --restore is exactly what I've been looking for. However... While I was able to export some of my entities (mostly those with only a handful of items), for entities with a LOT of items I'm seeing the following: on the command line from bulkloader.py: HTTPError: HTTP Error 500: Internal Server Error [INFO] Backing off due to errors: 256.0 seconds and in my error logs: /remote_api java.nio.BufferOverflowException at java.nio.HeapByteBuffer.put(Unknown Source) at java.nio.ByteBuffer.put(Unknown Source) at com.google.apphosting.runtime.jetty.RpcResponseGenerator.addContent (RpcResponseGenerator.java:68) at org.mortbay.jetty.AbstractGenerator$Output.write (AbstractGenerator.java:624) at org.mortbay.jetty.AbstractGenerator$Output.write (AbstractGenerator.java:585) at com.google.apphosting.utils.remoteapi.RemoteApiServlet.doPost (RemoteApiServlet.java:139) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 487) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1093) at org.purplex.lbas.servlet.filter.ResponseHeaderFilter.doFilter (ResponseHeaderFilter.java:49) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter (ParseBlobUploadFilter.java:97) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter (SaveSessionFilter.java:35) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter (TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle (ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle (SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle (SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle (ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 405) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle (AppVersionHandlerMap.java:238) at org.mortbay.jetty.handler.HandlerWrapper.handle (HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:313) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 506) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete (HttpConnection.java:830) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable (RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest (JettyServletEngineAdapter.java:135) at com.google.apphosting.runtime.JavaRuntime.handleRequest (JavaRuntime.java:235) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:5235) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:5233) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest (BlockingApplicationHandler.java:24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java: 363) at com.google.net.rpc.impl.Server$2.run(Server.java:838) at com.google.tracing.LocalTraceSpanRunnable.run (LocalTraceSpanRunnable.java:56) at com.google.tracing.LocalTraceSpanBuilder.internalContinueSpan (LocalTraceSpanBuilder.java:536) at com.google.net.rpc.impl.Server.startRpc(Server.java:793) at com.google.net.rpc.impl.Server.processRequest(Server.java:368) at com.google.net.rpc.impl.ServerConnection.messageReceived (ServerConnection.java:448) at com.google.net.rpc.impl.RpcConnection.parseMessages (RpcConnection.java:319) at com.google.net.rpc.impl.RpcConnection.dataReceived (RpcConnection.java:290) at com.google.net.async.Connection.handleReadEvent(Connection.java: 466) at com.google.net.async.EventDispatcher.processNetworkEvents (EventDispatcher.java:759) at com.google.net.async.EventDispatcher.internalLoop (EventDispatcher.java:205) at com.google.net.async.EventDispatcher.loop(EventDispatcher.java: 101) at com.google.net.rpc.RpcService.runUntilServerShutdown (RpcService.java:251) at com.google.apphosting.runtime.JavaRuntime$RpcRunnable.run (JavaRuntime.java:394) at
[appengine-java] Re: Looking for Bulk Loader beta testers
Nevermind... Playing around with the options, I got it to work setting: --batch_size=5 On Dec 15, 9:09 pm, stephan smor...@gmail.com wrote: Thanks Mathew -- The --dump --restoreis exactly what I've been looking for. However... While I was able to export some of my entities (mostly those with only a handful of items), for entities with a LOT of items I'm seeing the following: on the command line from bulkloader.py: HTTPError: HTTP Error 500: Internal Server Error [INFO ] Backing off due to errors: 256.0 seconds and in my error logs: /remote_api java.nio.BufferOverflowException at java.nio.HeapByteBuffer.put(Unknown Source) at java.nio.ByteBuffer.put(Unknown Source) at com.google.apphosting.runtime.jetty.RpcResponseGenerator.addContent (RpcResponseGenerator.java:68) at org.mortbay.jetty.AbstractGenerator$Output.write (AbstractGenerator.java:624) at org.mortbay.jetty.AbstractGenerator$Output.write (AbstractGenerator.java:585) at com.google.apphosting.utils.remoteapi.RemoteApiServlet.doPost (RemoteApiServlet.java:139) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 487) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1093) at org.purplex.lbas.servlet.filter.ResponseHeaderFilter.doFilter (ResponseHeaderFilter.java:49) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter (ParseBlobUploadFilter.java:97) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter (SaveSessionFilter.java:35) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter (TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle (ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle (SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle (SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle (ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 405) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle (AppVersionHandlerMap.java:238) at org.mortbay.jetty.handler.HandlerWrapper.handle (HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:313) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 506) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete (HttpConnection.java:830) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable (RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest (JettyServletEngineAdapter.java:135) at com.google.apphosting.runtime.JavaRuntime.handleRequest (JavaRuntime.java:235) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:5235) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:5233) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest (BlockingApplicationHandler.java:24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java: 363) at com.google.net.rpc.impl.Server$2.run(Server.java:838) at com.google.tracing.LocalTraceSpanRunnable.run (LocalTraceSpanRunnable.java:56) at com.google.tracing.LocalTraceSpanBuilder.internalContinueSpan (LocalTraceSpanBuilder.java:536) at com.google.net.rpc.impl.Server.startRpc(Server.java:793) at com.google.net.rpc.impl.Server.processRequest(Server.java:368) at com.google.net.rpc.impl.ServerConnection.messageReceived (ServerConnection.java:448) at com.google.net.rpc.impl.RpcConnection.parseMessages (RpcConnection.java:319) at com.google.net.rpc.impl.RpcConnection.dataReceived (RpcConnection.java:290) at com.google.net.async.Connection.handleReadEvent(Connection.java: 466) at com.google.net.async.EventDispatcher.processNetworkEvents (EventDispatcher.java:759) at com.google.net.async.EventDispatcher.internalLoop (EventDispatcher.java:205) at
[appengine-java] Re: Looking for Bulk Loader beta testers
Hi everyone, Thanks for the response so far--we'll be getting back to you over the next few days. I'd like to point out an existing feature which will be helpful to many of you: --dump and --restore. This will download or upload all of the entities for a particular Kind and save them in a local sqlite database. It's useful for general backup (e.g. a weekly backup), and also useful for moving data across applications, such as between your app running on the dev_appserver and on App Engine, or to load a staging instance with a known set of test data. You can find more information here: http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Downloading_and_Uploading_All_Data Also, for Java developers, there is a remote API handler available; adding following to your web xml file should work (disclaimer: I have not yet personally tested this.) servlet servlet-nameremoteapi/servlet-name servlet-classcom.google.apphosting.utils.remoteapi.RemoteApiServlet/servlet-class /servlet servlet-mapping servlet-nameremoteapi/servlet-name url-pattern/remote_api/url-pattern /servlet-mapping security-constraint web-resource-collection web-resource-nameremoteapi/web-resource-name url-pattern/remote_api/url-pattern /web-resource-collection auth-constraint role-nameadmin/role-name /auth-constraint /security-constraint --Matthew On Tue, Nov 17, 2009 at 12:53 PM, Matthew Blain matthew.bl...@google.comwrote: Hi App Engine developers, We're working on some improvements and additions to the bulk loader to make it easier to move data between the App Engine Datastore and other data files you may have. If you're doing this right now, we're looking for some beta testers. Two specific issues we're addressing at this time are: * More language-neutral: Java developers, this will help you! * More format-neutral: If you use some format other than CSV, this will help you! If you're interested, fill out the form at https://spreadsheets.google.com/viewform?formkey=dC15V2hwczhpZ1VVWFhPZGhXR1dydUE6MQ to get started. --Matthew -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=.
