[appengine-java] Re: A MapReduce job isn't going anywhere
Solved. Rebuilt the appengine-mapper.jar from the latest code and removed CONFIDENTIAL for the mapped URL from web.xml. Kind regards, Borys Burnayev actioncomplete.com GTD for Android and Web On Jan 17, 11:35 pm, burnayev wrote: > Update: > > In fact there is some interesting info in the log: > > 0.1.0.2 - - [17/Jan/2011:20:16:38 -0800] "POST .../mapperCallback > HTTP/1.1" 403 234 ".../command/start_job" "AppEngine-Google; (+http:// > code.google.com/appengine)" "appspot.com" ms=6 cpu_ms=23 > api_cpu_ms=0 cpm_usd=0.000728 queue_name=default task_name=worker- > attempt-1295324073192-0001-m-03-1--0 > > 0.1.0.2 - - [17/Jan/2011:20:16:40 -0800] "POST .../controllerCallback > HTTP/1.1" 403 234 ".../command/start_job" "AppEngine-Google; (+http:// > code.google.com/appengine)" "appspot.com" ms=8 cpu_ms=23 > api_cpu_ms=0 cpm_usd=0.000723 queue_name=default task_name=controller- > job-1295324073192-0001--0 > > It seems like I'm getting 403 errors. > > Any clues why? Can it be related to the AppEngine 1.4 upgrade? > > Kind regards, > Borys Burnayev > actioncomplete.com > GTD for Android and Web > > On Jan 17, 11:41 am, burnayev wrote: > > > I've got a few MapReduce jobs based on the AppEngine implementation > > that used to run just fine. Today I figured that one of the jobs is > > not good anymore. Namely it sort of hangs every time I run it. The > > MapReduce Overview console shows "0 / 0 shards" all the time and > > there's no chart in the details screen. The status is "running" and > > the console is indifferent about my clicking the Abort button. The > > number of retries for the job in the default task queue keep > > increasing though. The log, through INFO level, is silent about the > > MapReduce jobs. Over the past few hours I cleaned up the jobs manually > > and restarted with the same result. > > > I'm on AppEngine for Java. > > > Any ideas on what's going on? > > > Kind regards, > > Borys Burnayev > > actioncomplete.com > > GTD for Android and Web -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: A MapReduce job isn't going anywhere
Update: In fact there is some interesting info in the log: 0.1.0.2 - - [17/Jan/2011:20:16:38 -0800] "POST .../mapperCallback HTTP/1.1" 403 234 ".../command/start_job" "AppEngine-Google; (+http:// code.google.com/appengine)" "appspot.com" ms=6 cpu_ms=23 api_cpu_ms=0 cpm_usd=0.000728 queue_name=default task_name=worker- attempt-1295324073192-0001-m-03-1--0 0.1.0.2 - - [17/Jan/2011:20:16:40 -0800] "POST .../controllerCallback HTTP/1.1" 403 234 ".../command/start_job" "AppEngine-Google; (+http:// code.google.com/appengine)" "appspot.com" ms=8 cpu_ms=23 api_cpu_ms=0 cpm_usd=0.000723 queue_name=default task_name=controller- job-1295324073192-0001--0 It seems like I'm getting 403 errors. Any clues why? Can it be related to the AppEngine 1.4 upgrade? Kind regards, Borys Burnayev actioncomplete.com GTD for Android and Web On Jan 17, 11:41 am, burnayev wrote: > I've got a few MapReduce jobs based on the AppEngine implementation > that used to run just fine. Today I figured that one of the jobs is > not good anymore. Namely it sort of hangs every time I run it. The > MapReduce Overview console shows "0 / 0 shards" all the time and > there's no chart in the details screen. The status is "running" and > the console is indifferent about my clicking the Abort button. The > number of retries for the job in the default task queue keep > increasing though. The log, through INFO level, is silent about the > MapReduce jobs. Over the past few hours I cleaned up the jobs manually > and restarted with the same result. > > I'm on AppEngine for Java. > > Any ideas on what's going on? > > Kind regards, > Borys Burnayev > actioncomplete.com > GTD for Android and Web -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] A MapReduce job isn't going anywhere
I've got a few MapReduce jobs based on the AppEngine implementation that used to run just fine. Today I figured that one of the jobs is not good anymore. Namely it sort of hangs every time I run it. The MapReduce Overview console shows "0 / 0 shards" all the time and there's no chart in the details screen. The status is "running" and the console is indifferent about my clicking the Abort button. The number of retries for the job in the default task queue keep increasing though. The log, through INFO level, is silent about the MapReduce jobs. Over the past few hours I cleaned up the jobs manually and restarted with the same result. I'm on AppEngine for Java. Any ideas on what's going on? Kind regards, Borys Burnayev actioncomplete.com GTD for Android and Web -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: How to get rid of a hanging mapreduce job?
That was it! There were three guys sitting in there for the past few weeks. Doh. Thank you for the pointer! On Oct 18, 1:22 pm, Peter Liu wrote: > Did you clear the Default queue as well? > > It's possible that the task queue backed off due to failure and retry > at later time. But then it if it happens long time ago the queue > should have given up already. > > On Oct 17, 5:37 am, burnayev wrote: > > > I deleted the mapreduce-related entities long time ago and it was > > inconsequential. > > > On Oct 17, 2:50 am, Peter Liu wrote: > > > > I had the same problem. > > > > I fix it by deleting the entries of the 2 map reduce table. Go to to > > > the datastore viewer and there's 2 "Kind" used by map reduce. > > > > By the way, how to mark the task as completed? When my task is done it > > > always end with the "Unknown" state. > > > I asked this question before but there was no answer. > > > > On Oct 16, 9:30 am, burnayev wrote: > > > > > Mapper API for Java. > > > > Usedhttp://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-ap... > > > > as a starting point. > > > > > On Oct 15, 5:45 pm, Guillermo Schwarz > > > > wrote: > > > > > > Which map reduce library are you using? > > > > > > Saludos, > > > > > Guillermo Schwarz. > > > > > > El 15-10-2010, a las 18:44, burnayev escribió: > > > > > > > Here's the scoop... > > > > > > > One of my first mapreduce jobs didn't want to complete by itself. It > > > > > > did not want to abort either. To get rid of the sucker I deployed a > > > > > > new application version and deleted the one the job was running > > > > > > against. I also manually deleted all the residual state mapreduce > > > > > > created in the datastore. That seemed to kill most of it. However > > > > > > now, > > > > > > 5 days later, there are still two artifacts - mapperCallback and > > > > > > controllerCallback - that disturb my serenity (and keep sucking the > > > > > > juice) by popping up every hour or so. Obviously they are looking > > > > > > for > > > > > > a job that exists no more and fail miserably with a stack trace > > > > > > similar to below. > > > > > > > Is there a way to make them go away? > > > > > > > java.lang.RuntimeException: Couldn't find MR with job ID: > > > > > > job_1286643750234_0001 > > > > > > at > > > > > > com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest( > > > > > > AppEngineJobContext.java: > > > > > > 157) > > > > > > at > > > > > > com.google.appengine.tools.mapreduce.AppEngineJobContext. > > > > > > (AppEngineJobContext.java: > > > > > > 110) > > > > > > at > > > > > > com.google.appengine.tools.mapreduce.MapReduceServlet.handleController( > > > > > > MapReduceServlet.java: > > > > > > 507) > > > > > > at > > > > > > com.google.appengine.tools.mapreduce.MapReduceServlet.doPost > > > > > > (MapReduceServlet.java: > > > > > > 222) > > > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) > > > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) > > > > > > at org.mortbay.jetty.servlet.ServletHolder.handle > > > > > > (ServletHolder.java: > > > > > > 511) > > > > > > at org.mortbay.jetty.servlet.ServletHandler > > > > > > $CachedChain.doFilter(ServletHandler.java:1166) > > > > > > at > > > > > > com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter > > > > > > (ParseBlobUploadFilter.java: > > > > > > 97) > > > > > > at org.mortbay.jetty.servlet.ServletHandler > > > > > > $CachedChain.doFilter(ServletHandler.java:1157) > > > > > > at > > > > > > com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter > > > > > > (SaveSessionFilter.java: > > > > > > 35) > > > > > > at org.mortbay.jetty.servlet.ServletHandler >
[appengine-java] Re: How to get rid of a hanging mapreduce job?
I deleted the mapreduce-related entities long time ago and it was inconsequential. On Oct 17, 2:50 am, Peter Liu wrote: > I had the same problem. > > I fix it by deleting the entries of the 2 map reduce table. Go to to > the datastore viewer and there's 2 "Kind" used by map reduce. > > By the way, how to mark the task as completed? When my task is done it > always end with the "Unknown" state. > I asked this question before but there was no answer. > > On Oct 16, 9:30 am, burnayev wrote: > > > Mapper API for Java. > > Usedhttp://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-ap... > > as a starting point. > > > On Oct 15, 5:45 pm, Guillermo Schwarz > > wrote: > > > > Which map reduce library are you using? > > > > Saludos, > > > Guillermo Schwarz. > > > > El 15-10-2010, a las 18:44, burnayev escribió: > > > > > Here's the scoop... > > > > > One of my first mapreduce jobs didn't want to complete by itself. It > > > > did not want to abort either. To get rid of the sucker I deployed a > > > > new application version and deleted the one the job was running > > > > against. I also manually deleted all the residual state mapreduce > > > > created in the datastore. That seemed to kill most of it. However now, > > > > 5 days later, there are still two artifacts - mapperCallback and > > > > controllerCallback - that disturb my serenity (and keep sucking the > > > > juice) by popping up every hour or so. Obviously they are looking for > > > > a job that exists no more and fail miserably with a stack trace > > > > similar to below. > > > > > Is there a way to make them go away? > > > > > java.lang.RuntimeException: Couldn't find MR with job ID: > > > > job_1286643750234_0001 > > > > at > > > > com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest( > > > > AppEngineJobContext.java: > > > > 157) > > > > at > > > > com.google.appengine.tools.mapreduce.AppEngineJobContext. > > > > (AppEngineJobContext.java: > > > > 110) > > > > at > > > > com.google.appengine.tools.mapreduce.MapReduceServlet.handleController( > > > > MapReduceServlet.java: > > > > 507) > > > > at > > > > com.google.appengine.tools.mapreduce.MapReduceServlet.doPost > > > > (MapReduceServlet.java: > > > > 222) > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) > > > > at org.mortbay.jetty.servlet.ServletHolder.handle > > > > (ServletHolder.java: > > > > 511) > > > > at org.mortbay.jetty.servlet.ServletHandler > > > > $CachedChain.doFilter(ServletHandler.java:1166) > > > > at > > > > com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter > > > > (ParseBlobUploadFilter.java: > > > > 97) > > > > at org.mortbay.jetty.servlet.ServletHandler > > > > $CachedChain.doFilter(ServletHandler.java:1157) > > > > at > > > > com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter > > > > (SaveSessionFilter.java: > > > > 35) > > > > at org.mortbay.jetty.servlet.ServletHandler > > > > $CachedChain.doFilter(ServletHandler.java:1157) > > > > at > > > > com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter > > > > (TransactionCleanupFilter.java: > > > > 43) > > > > at org.mortbay.jetty.servlet.ServletHandler > > > > $CachedChain.doFilter(ServletHandler.java:1157) > > > > at > > > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: > > > > 388) > > > > at > > > > org.mortbay.jetty.security.SecurityHandler.handle > > > > (SecurityHandler.java: > > > > 216) > > > > at > > > > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: > > > > 182) > > > > at > > > > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: > > > > 765) > > > > at org.mortbay.jetty.webapp.WebAppContext.handle > > > > (WebAppContext.java: > > > > 418) > > > > at > > > > com.google.apphosti
[appengine-java] Re: How to get rid of a hanging mapreduce job?
Mapper API for Java. Used http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine/ as a starting point. On Oct 15, 5:45 pm, Guillermo Schwarz wrote: > Which map reduce library are you using? > > Saludos, > Guillermo Schwarz. > > El 15-10-2010, a las 18:44, burnayev escribió: > > > Here's the scoop... > > > One of my first mapreduce jobs didn't want to complete by itself. It > > did not want to abort either. To get rid of the sucker I deployed a > > new application version and deleted the one the job was running > > against. I also manually deleted all the residual state mapreduce > > created in the datastore. That seemed to kill most of it. However now, > > 5 days later, there are still two artifacts - mapperCallback and > > controllerCallback - that disturb my serenity (and keep sucking the > > juice) by popping up every hour or so. Obviously they are looking for > > a job that exists no more and fail miserably with a stack trace > > similar to below. > > > Is there a way to make them go away? > > > java.lang.RuntimeException: Couldn't find MR with job ID: > > job_1286643750234_0001 > > at > > com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest( > > AppEngineJobContext.java: > > 157) > > at > > com.google.appengine.tools.mapreduce.AppEngineJobContext. > > (AppEngineJobContext.java: > > 110) > > at > > com.google.appengine.tools.mapreduce.MapReduceServlet.handleController( > > MapReduceServlet.java: > > 507) > > at > > com.google.appengine.tools.mapreduce.MapReduceServlet.doPost > > (MapReduceServlet.java: > > 222) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) > > at org.mortbay.jetty.servlet.ServletHolder.handle > > (ServletHolder.java: > > 511) > > at org.mortbay.jetty.servlet.ServletHandler > > $CachedChain.doFilter(ServletHandler.java:1166) > > at > > com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter > > (ParseBlobUploadFilter.java: > > 97) > > at org.mortbay.jetty.servlet.ServletHandler > > $CachedChain.doFilter(ServletHandler.java:1157) > > at > > com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter > > (SaveSessionFilter.java: > > 35) > > at org.mortbay.jetty.servlet.ServletHandler > > $CachedChain.doFilter(ServletHandler.java:1157) > > at > > com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter > > (TransactionCleanupFilter.java: > > 43) > > at org.mortbay.jetty.servlet.ServletHandler > > $CachedChain.doFilter(ServletHandler.java:1157) > > at > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: > > 388) > > at > > org.mortbay.jetty.security.SecurityHandler.handle > > (SecurityHandler.java: > > 216) > > at > > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: > > 182) > > at > > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: > > 765) > > at org.mortbay.jetty.webapp.WebAppContext.handle > > (WebAppContext.java: > > 418) > > at > > com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle > > (AppVersionHandlerMap.java: > > 238) > > at > > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: > > 152) > > at org.mortbay.jetty.Server.handle(Server.java:326) > > at org.mortbay.jetty.HttpConnection.handleRequest > > (HttpConnection.java: > > 542) > > at org.mortbay.jetty.HttpConnection > > $RequestHandler.headerComplete(HttpConnection.java:923) > > at > > com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable > > (RpcRequestParser.java: > > 76) > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > > at > > com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest( > > JettyServletEngineAdapter.java: > > 135) > > at > > com.google.apphosting.runtime.JavaRuntime.handleRequest > > (JavaRuntime.java: > > 261) > > at com.google.apphosting.base.RuntimePb$EvaluationRuntime > > $6.handleBlockingRequest(RuntimePb.java:8483) > > at com.google.apphosting.base.RuntimePb$EvaluationRuntime > > $6.handleBlockingRequest(RuntimePb.java:8481) > > at > > com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest > > (BlockingApplicationHandler.java
[appengine-java] How to get rid of a hanging mapreduce job?
Here's the scoop... One of my first mapreduce jobs didn't want to complete by itself. It did not want to abort either. To get rid of the sucker I deployed a new application version and deleted the one the job was running against. I also manually deleted all the residual state mapreduce created in the datastore. That seemed to kill most of it. However now, 5 days later, there are still two artifacts - mapperCallback and controllerCallback - that disturb my serenity (and keep sucking the juice) by popping up every hour or so. Obviously they are looking for a job that exists no more and fail miserably with a stack trace similar to below. Is there a way to make them go away? java.lang.RuntimeException: Couldn't find MR with job ID: job_1286643750234_0001 at com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest(AppEngineJobContext.java: 157) at com.google.appengine.tools.mapreduce.AppEngineJobContext.(AppEngineJobContext.java: 110) at com.google.appengine.tools.mapreduce.MapReduceServlet.handleController(MapReduceServlet.java: 507) at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost(MapReduceServlet.java: 222) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1166) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java: 97) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java: 238) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java: 135) at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java: 261) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:8483) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:8481) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest(BlockingApplicationHandler.java: 24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java: 418) at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java: 572) at com.google.tracing.TraceContext$TraceContextRunnable $1.run(TraceContext.java:448) at com.google.tracing.TraceContext.runInContext(TraceContext.java: 688) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java: 326) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContext(TraceContext.java: 318) at com.google.tracing.TraceContext $TraceContextRunnable.run(TraceContext.java:446) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1110) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Uncaught exception from servlet java.lang.RuntimeException: Couldn't find MR with job ID: job_1286643750234_0001 at com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest(AppEngineJobContext.java: 157) at com.google.appengine.tools.mapreduce.AppEngineJobContext.(AppEngineJobContext.java: 110
