[appengine-java] Advice for looping through 100K records and updating multiple entities on each iteration
Hello, I have a non-user triggered process that queries the datastore for a dataset of about 100k records, then loops through each record to update two other entities in the datastore. What would be a good way to go about doing this? I'm concerned about timing out and/or consuming more resources than I need to. Task queue? Cheers. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: java.net.DatagramPacket supported ?
There is no way to query a server using UDP packets on GAE - the only supported transport protocol is HTTP. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/JIKR5yW8M3UJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Task queue locking up
I switched back to true , upgraded to SDK 1.5.2 and for a week the number of deadlocks seems to be greatly reduced. I do still get deadlocks but now the error reporting is MUCH better. They normally occur in some Google generated method com.google.appengine.runtime.Request.process-XXX. Often they are due to the class loader accessing the file system concurrently. There doesn't seem to be anything my code can do to protect against these. They usually occur while an instance is spinning up - loading classes etc. In fact, last night for a period, almost every loading request caused a dead lock. Not good. Normal loading requests complete in 6 seconds. Here are a few example stack traces: com.google.apphosting.runtime.HardDeadlineExceededError: This request (fa381c78a3815eac) started at 2011/08/24 23:52:33.115 UTC and was still executing at 2011/08/24 23:53:04.127 UTC. at com.google.appengine.runtime.Request.process-fa381c78a3815eac(Request.java) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.(FileInputStream.java:139) at java.lang.ClassLoader.loadClass(ClassLoader.java:266) Caused by: com.google.apphosting.runtime.HardDeadlineExceededError: This request (fd3fb35c7fd41f00) started at 2011/08/25 00:00:41.072 UTC and was still executing at 2011/08/25 00:01:14.736 UTC. at com.google.appengine.runtime.Request.process-fd3fb35c7fd41f00(Request.java) at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:243) at java.io.File.exists(File.java:807) com.google.apphosting.runtime.HardDeadlineExceededError: This request (c7de12e8c6c87de3) started at 2011/08/25 00:01:17.557 UTC and was still executing at 2011/08/25 00:01:48.517 UTC. at com.google.appengine.runtime.Request.process-c7de12e8c6c87de3(Request.java) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.(FileInputStream.java:139) at java.lang.Class.getGenericSuperclass(Class.java:386) at com.google.inject.TypeLiteral.getSuperclassTypeParameter(TypeLiteral.java:97) at com.google.inject.TypeLiteral.(TypeLiteral.java:77) Caused by: com.google.apphosting.api.DeadlineExceededException: This request (5cb100e4cb6e2818) started at 2011/08/25 00:13:55.275 UTC and was still executing at 2011/08/25 00:14:26.967 UTC. at com.google.appengine.runtime.Request.process-5cb100e4cb6e2818(Request.java) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.(FileInputStream.java:139) at java.lang.Class.forName(Class.java:110) -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/TDMIg2gMovwJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: 500 error updating index definitions
Hi, I am having similar issue. My app deployment is failing always with 500 server error when "uploading index definitions". LAst 4 hrs it's been like this. My old app with same index file works file. I created a new version of app wit hsame index file and it fails consistently. I put an empty index file and the deployment is successful. But my app fails with a missing index. When I include it, the deployment hangs. I think there is some problem with appengine today with new apps and index files. Could someone in GAE team comment on this? Thanks Neeli -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/VqAucCAR4cQJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] deployment issues
This is old post but I exactly have same problem. The deployment stalls whiel "Uploading index definitions" and can't get past that step for last 4 hrs now. . Any thoughts? When I use my app, it complains about hte missing index and when I include it, the deployment hangs during that step. If I have empty index file, teh deployment is successful Initiating update. Cloning 5 static files. Cloning 68 application files. Uploading 1 files. Uploaded 1 files. Initializing precompilation... Deploying new version. Will check again in 1 seconds. Will check again in 2 seconds. Closing update: new version is ready to start serving. Uploading index definitions. java.io.IOException: Error posting to URL: https://appengine.google.com/api/datastore/index/add?app_id=&version=1& 500 Internal Server Error -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/Tf7ZMCSLTAAJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: 500 error updating index definitions
Sorry, I should have included version info. I've gotten the error with 1.5.2 and 1.5.3 of the Java SDK. Here's some output: $ ~/bin/appengine-java-sdk-1.5.3/bin/appcfg.sh update war Reading application configuration data... Aug 24, 2011 5:50:41 PM com.google.apphosting.utils.config.AppEngineWebXmlReader readAppEngineWebXml INFO: Successfully processed war/WEB-INF/appengine-web.xml Aug 24, 2011 5:50:42 PM com.google.apphosting.utils.config.AbstractConfigXmlReader readConfigXml INFO: Successfully processed war/WEB-INF/web.xml Aug 24, 2011 5:50:42 PM com.google.apphosting.utils.config.IndexesXmlReader readConfigXml INFO: Successfully processed war/WEB-INF/appengine-generated/datastore-indexes-auto.xml Beginning server interaction for clj-contact... 0% Created staging directory at: '/var/folders/N8/N8ijLJCfGp4EcySYUwGd5k+++TI/-Tmp-/appcfg4075726641025123142.tmp' 5% Scanning for jsp files. 20% Scanning files on local disk. 25% Initiating update. 28% Cloning 48 application files. 40% Uploading 3 files. 52% Uploaded 1 files. 61% Uploaded 2 files. 68% Uploaded 3 files. 73% Initializing precompilation... 90% Deploying new version. 95% Will check again in 1 seconds. 98% Will check again in 2 seconds. 99% Will check again in 4 seconds. 99% Will check again in 8 seconds. 99% Closing update: new version is ready to start serving. 99% Uploading index definitions. java.io.IOException: Error posting to URL: https://appengine.google.com/api/datastore/index/add?app_id=clj-contact&version=2&; 500 Internal Server Error -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/UK_Q89dPDqcJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] 500 error updating index definitions
Hi, I'm writing a proof of concept with appengine, and I'm getting this error consistently: java.io.IOException: Error posting to URL: https://appengine.google.com/api/datastore/index/add?app_id=clj-contact&version=2&; 500 Internal Server Error The app deploys, but since my datastore-indexes.xml file wasn't processed, my queries fail. Index file is trivial: This error has occurred at least 10 times today for me with zero successes, so it doesn't seem very transient. Any thoughts? -- James -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/AF3Usj_UmvkJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Failing to deploy my webapp to appengine
Hi, I disabled my 2 old apps on appengine and created a new application. My deploy to appengine consistently fails with an error message 500 server error. The failure occurs during the step "uploading index definitions". Anyone has thoughts on how to correct the error? Closing update: new version is ready to start serving. Uploading index definitions. java.io.IOException: Error posting to URL: https://appengine.google.com/api/datastore/index/add?app_id=&version=1& 500 Internal Server Error Thanks Neeli -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/6kvKuAhrEucJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] ClassNotFoundException for java.util.HashMap$Entry
It's hard to say based on this information alone. I hate to say it, but if you've got a good version and a bad version, it's time to binary search on what has changed between versions to zone in on the problem. If you use "git" "git bisect" is a lifesaver. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Wed, Aug 24, 2011 at 2:33 AM, Jeremy Brown wrote: > Does anyone know why I would be seeing this error message: > > java.lang.ClassNotFoundException: java.util.HashMap$Entry > > I had some code that was working fine. I'm upgrading a couple of jar > libraries to newer versions and testing on a version other than the > default. I've gotten quite a few weird errors and this is one of them. > > Any ideas? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Upload attachments Google Sites
There's AppEngineFile in Java. It's not 100% File compatible, but it's something you can use. It writes to the blobstore. Alternative, you can use a datastore blob property. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Tue, Aug 23, 2011 at 10:28 PM, Thomas Wiradikusuma < wiradikus...@gmail.com> wrote: > Hi Ikai, > > Just wondering, is it possible if instances are given limited space > "temp" dir where programs can write to it? > It will greatly help 3rd party frameworks which use file system for > temp data. > > On Aug 24, 5:53 am, "Ikai Lan (Google)" wrote: > > Hah, I should have looked at the code. Yeah, my example writes the thing > to > > disk. Thanks for the catch, Stephen! > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-java@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] java.net.DatagramPacket supported ?
I have a code that uses java.net.DatagramPacket; java.net.DatagramSocket; java.net.InetAddress; on server side in GWT App. Are they Supported by Google App engine . Or is there any other way to query a server (can only be queried through UDP Packets) in google app engine ? -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/R1XoCIASK-4J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: "ApplicationError: 10: Unknown" when writing to blobstore using FileService
Good call - I added BufferedInputStream and BufferedOutputStream and
it solved the problem.
i.e. ObjectOutputStream out = new ObjectOutputStream( new
BufferedOutputStream( Channels.newOutputStream( writeChannel ),
65536 ) );
Thanks!
- Mike
On Aug 19, 10:48 am, "Ikai Lan (Google)" wrote:
> We're surfacing a bad error here. Basically, you get this error message if
> you hold the file open too long; we recommend around 30 seconds.
>
> I'll file a bug internally to get a better error message. Can you break this
> up into multiple writes?
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai
>
>
>
>
>
>
>
> On Wed, Aug 17, 2011 at 5:32 PM, Mike Prince wrote:
> > I'm trying to write a Java serialized HashMap to the
> > blobstore using the FileService and getting an "Unknown" error 10.
> > I'm guessing there's 10,000+ items in the map.
>
> > Below is the code I'm using, followed by the stack trace.
>
> > Can someone clue me in on what I'm doing wrong? And yes, it works
> > fine in the development environment ;)
>
> > Thanks,
>
> > Mike
>
> > public class Filer
> > {
> > private static final FileService fileService =
> > FileServiceFactory.getFileService();
> > private static final BlobstoreService blobstoreService =
> > BlobstoreServiceFactory.getBlobstoreService();
>
> > public static String save( Object value )
> > throws IOException
> > {
> > if( value == null ) return null;
>
> > AppEngineFile file = fileService.createNewBlobFile(
> > "text/plain" );
> > return save( file, value );
> > }
>
> > private static String save( AppEngineFile file, Object value )
> > throws IOException
> > {
> > // Open a channel and write out object
> > FileWriteChannel writeChannel =
> > fileService.openWriteChannel( file,
> > true );
> > ObjectOutputStream out = new
> > ObjectOutputStream( Channels.newOutputStream( writeChannel ) );
> > out.writeObject( value );
>
> > // Close it up
> > out.close();
> > writeChannel.closeFinally();
>
> > return file.getFullPath();
> > }
>
> > And here's the exception...
>
> > org.apache.jsp.tasks.create_002duser_002dsegment_jsp _jspService:
> > Failed to create user segment
> > java.io.IOException
> > at
>
> > com.google.appengine.api.files.FileServiceImpl.translateException(FileServi
> > ceImpl.java:
> > 501)
> > at
>
> > com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl
> > .java:
> > 479)
> > at
> > com.google.appengine.api.files.FileServiceImpl.append(FileServiceImpl.java:
> > 426)
> > at
> > com.google.appengine.api.files.FileServiceImpl.append(FileServiceImpl.java:
> > 171)
> > at
>
> > com.google.appengine.api.files.FileWriteChannelImpl.write(FileWriteChannelI
> > mpl.java:
> > 49)
> > at
>
> > com.google.appengine.api.files.FileWriteChannelImpl.write(FileWriteChannelI
> > mpl.java:
> > 41)
> > at java.nio.channels.Channels.write(Channels.java:80)
> > at java.nio.channels.Channels.access$000(Channels.java:64)
> > at java.nio.channels.Channels$1.write(Channels.java:151)
> > at java.io.ObjectOutputStream
> > $BlockDataOutputStream.drain(ObjectOutputStream.java:1855)
> > at java.io.ObjectOutputStream
> > $BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1764)
> > at
> > java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:
> > 1558)
> > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:
> > 346)
> > at
> > com.yumyumlabs.foundation.appengine.util.Filer.save(Filer.java:42)
> > at
> > com.yumyumlabs.foundation.appengine.util.Filer.save(Filer.java:33)
> > at
> > com.yumyumlabs.server.recipe.model.UserSegment.saveUsers(UserSegment.java:
> > 51)
> > at
>
> > org.apache.jsp.tasks.create_002duser_002dsegment_jsp._jspService(create_002
> > duser_002dsegment_jsp.java:
> > 104)
> > at
> > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> > at
> > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
> > 511)
> > at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
> > 390)
> > at
> > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
> > 216)
> > at
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
> > 182)
> > at
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
> > 765)
> > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:
> > 418)
> > at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
> > at org.mortbay.jetty.se
Re: [appengine-java] Android authentication broken?
Do you need more information about this? -- Erlend On Tue, Aug 23, 2011 at 8:17 AM, Erlend Hamnaberg wrote: > Hello. > > Sorry about the lack of error message. > > Here is an excerpt from the log of the phone: > > 08-23 08:12:38.895: DEBUG/GetCookieAsyncTask(19335): Will authenticate by > getting the following uri: http://.appspot.com/_ah/login?auth= > > 08-23 08:12:39.085: WARN/GetCookieAsyncTask(19335): The HttpResponse has > wrong status. Expected 302, got 500. > 08-23 08:12:39.085: DEBUG/GetCookieAsyncTask(19335): Status code: 500 > 08-23 08:12:39.085: DEBUG/GetCookieAsyncTask(19335): Entity content: content notifying me to go to the appengine forums> > 08-23 08:12:39.085: INFO/GetCookieAsyncTask(19335): Failed to obtain > authentication cookie! > > -- > Erlend > > On Mon, Aug 22, 2011 at 8:12 PM, Ikai Lan (Google) wrote: > >> Can you describe the errors you're seeing? Exceptions that are thrown, >> etc? "It doesn't work" isn't a particularly helpful error report. >> >> -- >> Ikai Lan >> Developer Programs Engineer, Google App Engine >> plus.ikailan.com | twitter.com/ikai >> >> >> >> On Mon, Aug 22, 2011 at 7:00 AM, Erlend Hamnaberg wrote: >> >>> Hi all. >>> >>> I'm using Google appengine from Android, and have been using this [1] >>> method for auth. >>> Is there something that has changed recently? It used to work a couple of >>> weeks ago. >>> Coming back from vacation it failed to work. >>> >>> Any pointers? >>> >>> >>> 1: >>> http://blog.notdot.net/2010/05/Authenticating-against-App-Engine-from-an-Android-app >>> >>> Best regards >>> >>> Erlend >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine for Java" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-appengine-java/-/fS-61cGsFCsJ. >>> To post to this group, send email to >>> google-appengine-java@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine-java+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine-java?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> google-appengine-java@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine-java+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] ClassNotFoundException for java.util.HashMap$Entry
Does anyone know why I would be seeing this error message: java.lang.ClassNotFoundException: java.util.HashMap$Entry I had some code that was working fine. I'm upgrading a couple of jar libraries to newer versions and testing on a version other than the default. I've gotten quite a few weird errors and this is one of them. Any ideas? -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: GWT will serialize java.util.Hashet and won't serialize org.datanucleus.sco.backed.HashSet from JDO. Why?
> Not sure what all the low-level difference between the two is > either One of them returns objects in a "detached" state (i.e representations of persisted objects, with an "identity", but disconnected from the datastore, and if persisted would update the existing object in the datastore) The other returns objects in a "transient" state (i.e objects with no "identity", not representing anything in the datastore, and if persisted would result in a new entry in the datastore) -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Problem Adding a User Library [.Jar file]
I try adding an user .jar file to my project. 1. Using: Project -> Properties -> Java Build Path -> Libraries -> [ Add External JARS...] - This fix the compilation errors 2. Using Project -> Goole -> App Engine Settings... -> Web Application Under "Suppress warnings about these build path entries being ouside of WEB-INF/lib": - Select [Add] button to add the path of the .jar file. 3. Click on Project -> Google -> GWT Compile [ERROR] The import com..yyy cannot be resolved [ERROR] Hint: Check the inheritance chain from your module; it may not be inheriting a required module or a module may not be adding its source path entries properly Did you experienced a similar problem? I am doing something wrong? Please help! -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/qfSwe9F8p4gJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
