[appengine-java] Diagnosing problems with backends that are freezing
Are are using backends to parallelize large batches of work for our users. This involves adding 100's of tasks to a dedicated queue which spreads the tasks across 5 dynamic backends. Many of these tasks do a url fetch to external web services. When we run this for one user it seems to work fine but as soon as we run it for many users (e.g. in a nightly batch run) we see most of the backends eventually stopping processing new tasks. We've tried rate limiting it but that has not helped either. I can manually stop the backends that are frozen and new ones will fire up and start processing until they too freeze. Eventually it all completes after a few iterations of stopping frozen servers. Obviously this is severely limiting our ability to scale so I'm wondering how to diagnose this problem. We cannot reproduce this on localhost since its not really multi-threaded and doesn't truly replicate the deployed data patterns. We could possibly handle this by using pull queues but that's more code to write and I'd rather let the queue/backend scheduler do that work for us. What I'd like to do is attach a profiler to a backend - that would immediately tell me where the freezes are coming from. Is that even possible? If not, does anybody have any other tricks that we could use to diagnose frozen backends? fyi : the frozen instances have varied levels of memory from 150mb to 250mb, consume zero cpu when frozen and process no new tasks. Thanks for your attention and help, Steve -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] playframework-crudsiena-2.0.2-gae-1.6.0_b1
On Thu, Jan 19, 2012 at 5:19 AM, Priyatham Suresh priyat...@ariveguru.com wrote: hi this is priyatham suresh i have devoloped a simple playframework-crudsiena-2.0.2-gae-1.6.0_b1 application. it runs successfully in my local db but when i run this in app engine it displays the following error message The server encountered an error and could not complete your request. Can you give us more information like a stack trace? Best, Raphael please help me .. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/noGzIVAGwooJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- inc: http://ars-machina.raphaelbauer.com tech: http://ars-codia.raphaelbauer.com web: http://raphaelbauer.com -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Problem Aborting MapReduce Job
Hi there, I have been following Ikai's blog (which I have found to be most useful, so thanks): http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine/ It seems I have set things up properly with web.xml / mapreduce.xml my mapper class. However, there is a problem. When I run my job locally (for testing purposes in Intellij environment) over a kind with 6 entities, the job doesn't seem to stop running, nor can I abort the job (A loading dialog will appear and nothing other than that seems to happen). I have done some research and the only thing I see that was a solution for people was that they had the purge the queue. I have tried this to no success. This is what my logs in console looks like (and that is the only debugging output I see related to MapReduce): INFO: Getting input splits for: LEADERBOARDENTRYTEMP Jan 19, 2012 1:32:13 PM com.google.appengine.tools.mapreduce.DatastoreInputFormat getSplits INFO: Requested 128 scatter entities. Got 3 so using oversample factor 1 Jan 19, 2012 1:32:13 PM com.google.appengine.tools.mapreduce.DatastoreInputFormat getSplitsFromSplitPoints INFO: Added DatastoreInputSplit com.google.appengine.tools.mapreduce.DatastoreInputSplit@7920afc0 LEADERBOARDENTRYTEMP(114) LEADERBOARDENTRYTEMP(117) Jan 19, 2012 1:32:13 PM com.google.appengine.tools.mapreduce.DatastoreInputFormat getSplitsFromSplitPoints INFO: Added DatastoreInputSplit com.google.appengine.tools.mapreduce.DatastoreInputSplit@50f2a460 LEADERBOARDENTRYTEMP(117) LEADERBOARDENTRYTEMP(125) Jan 19, 2012 1:32:16 PM com.google.appengine.tools.mapreduce.DatastoreInputSplit write INFO: Writing DatastoreInputSplit com.google.appengine.tools.mapreduce.DatastoreInputSplit@7920afc0 LEADERBOARDENTRYTEMP(114) LEADERBOARDENTRYTEMP(117) Jan 19, 2012 1:32:16 PM com.google.appengine.tools.mapreduce.DatastoreInputSplit write INFO: Writing DatastoreInputSplit com.google.appengine.tools.mapreduce.DatastoreInputSplit@50f2a460 LEADERBOARDENTRYTEMP(117) LEADERBOARDENTRYTEMP(125) Jan 19, 2012 1:32:16 PM com.google.appengine.tools.mapreduce.DatastoreInputSplit write INFO: Writing DatastoreInputSplit com.google.appengine.tools.mapreduce.DatastoreInputSplit@7b6626e3 LEADERBOARDENTRYTEMP(125) null Also, it seems that I can never hit the breakpoints in my mapper class, is this because it doesn't ever manage to get that far? Thank you for any help with can be provided, I hope I have given an adequate amount of information for you to aid me :) Mark -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/3VtYXwFH1FIJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Problem Aborting MapReduce Job
Oh, I also see: Status ViewID NameActivityStart time Time elapsedControl RunningDetailhttp://192.168.0.56:8080/mapreduce/detail?mapreduce_id=job_1326969729540e8154184868d4b80b1fc0b7ae289f8c0_0001 job_1326969729540e8154184868d4b80b1fc0b7ae289f8c0_0001MapReduce for Leaderboard Entries0 / 0 shardsThu Jan 19 2012 10:42:09 GMT+ (GMT) 13:17:50Abort http://192.168.0.56:8080/mapreduce/status -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/vHhmUKU5ERYJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Problem Aborting MapReduce Job
... and my task queue for mapreduce: mapreduce http://192.168.0.56:8080/_ah/admin/taskqueue?queueName=mapreduce 5.0/s100 -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/PYQmfrfWS5sJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Problem Aborting MapReduce Job
I can remove the Jobs from the datastore and get rid of them that way, but this does not explain why my Jobs get stuck in RUNNING state and it never hits the break points in my Mapper class. I must have set something up slightly wrong, but I can't think what. Here is my xml: configurations configuration name=MapReduce for Leaderboard Entries property namemapreduce.map.class/name valuecom.mapreduce.LeaderboardEntryMapper/value /property property namemapreduce.inputformat.class/name valuecom.google.appengine.tools.mapreduce.DatastoreInputFormat/value /property property name human=Entity Kind to Map Overmapreduce.mapper.inputformat.datastoreinputformat.entitykind/name value template=optionalLEADERBOARDENTRYTEMP/value /property /configuration /configurations -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/KWpRiU1X6JoJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Authorization options does'nt limit access for non-domain users
Hi everyone, this is my first post here :) I recently started developing GAE apps, currently following as many tutorials as I can and implementing my own apps... I'm having a problem with the OAuth tutorial: not with the app itself, which works just fine, but with the Authentication Options I set when I created the it on the GAE platform. I'm trying to limit the access only to my Google Apps domain users: I set the domain in the app options, and I added the app as a service in my domain panel. However, when I try to log in to the app with a gmail account, I can access it without problem. Am i missing something? Thanks in advance, I hope to be useful to the community someday, and not only ask questions here :) -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/zSj1QlSNKAAJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Use of com.google.appengine.repackaged
Hint: You want to have this class on your classpath: org.json.JSONObject -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/W9YJOmnaycUJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Sending Authorization header in a SOAP call
Hi, I've followed this (http://code.google.com/appengine/articles/soap.html) and created a SOAP client for a webservice (ASP.NET). The client is used by a G/A/E Backend. I've verified that backed+client works correctly against a development server that (1) uses HTTP and (2) requires no authorization. It fails authorization when connecting to a production server that requires Basic auth over an HTTPS connection. Here's the code: URL url = new URL(https://.../FOOService.asmx?WSDL;); HttpURLConnection con = (HttpURLConnection) url.openConnection(); con.setRequestProperty(Authorization, Basic +CREDENTIALS); // CREDENTIALS is base64 of User:Pass FOOService fs = new FOOService(url, new QName(http://.../;, FOOService)); FOOServiceSoap ws = fs.getFOOServiceSoap(); // etc... I've set logging to FINEST, but can't see the request headers being sent. Also the WWW-Authenticate response coming back indicates Authorization header is not being sent. It finally fails like this: Jan 19, 2012 10:57:28 PM com.google.appengine.repackaged.org.apache.http.impl.client.DefaultRequestDirector updateAuthState FINE: Credentials not found What's the trick? (I'm somewhat new to AppEngine and have a lot of contempt for SOAP) Thanks, -Skip -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/un0fMRCIVS4J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Eclipse unable to compile after updating
I updated the google plug-in for eclipse and now I get the following error: Error: Could not find or load main class com.google.appengine.tools.development.DevAppServerMain Since I can't load main class, I can't even compile my projects. Please suggest me what to do. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Eclipse unable to compile after updating
This is the problem description for the same: Description ResourcePathLocationType The project 'Wooosbee' does not have any App Engine SDKs on its build pathWooosbeeUnknown Google App Engine Problem The project cannot be built until build path errors are resolved WooosbeeUnknown Java Problem Unbound classpath container: 'App Engine SDK [invalid]' in project 'Wooosbee' WooosbeeBuild path Build Path Problem In google Web Toolkit, its now not showing any SDKs. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
