[appengine-java] Re: How to get rid of a hanging mapreduce job?
That was it! There were three guys sitting in there for the past few weeks. Doh. Thank you for the pointer! On Oct 18, 1:22 pm, Peter Liu tinyee...@gmail.com wrote: Did you clear the Default queue as well? It's possible that the task queue backed off due to failure and retry at later time. But then it if it happens long time ago the queue should have given up already. On Oct 17, 5:37 am, burnayev burna...@gmail.com wrote: I deleted the mapreduce-related entities long time ago and it was inconsequential. On Oct 17, 2:50 am, Peter Liu tinyee...@gmail.com wrote: I had the same problem. I fix it by deleting the entries of the 2 map reduce table. Go to to the datastore viewer and there's 2 Kind used by map reduce. By the way, how to mark the task as completed? When my task is done it always end with the Unknown state. I asked this question before but there was no answer. On Oct 16, 9:30 am, burnayev burna...@gmail.com wrote: Mapper API for Java. Usedhttp://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-ap... as a starting point. On Oct 15, 5:45 pm, Guillermo Schwarz guillermo.schw...@gmail.com wrote: Which map reduce library are you using? Saludos, Guillermo Schwarz. El 15-10-2010, a las 18:44, burnayev burna...@gmail.com escribió: Here's the scoop... One of my first mapreduce jobs didn't want to complete by itself. It did not want to abort either. To get rid of the sucker I deployed a new application version and deleted the one the job was running against. I also manually deleted all the residual state mapreduce created in the datastore. That seemed to kill most of it. However now, 5 days later, there are still two artifacts - mapperCallback and controllerCallback - that disturb my serenity (and keep sucking the juice) by popping up every hour or so. Obviously they are looking for a job that exists no more and fail miserably with a stack trace similar to below. Is there a way to make them go away? java.lang.RuntimeException: Couldn't find MR with job ID: job_1286643750234_0001 at com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest( AppEngineJobContext.java: 157) at com.google.appengine.tools.mapreduce.AppEngineJobContext.init (AppEngineJobContext.java: 110) at com.google.appengine.tools.mapreduce.MapReduceServlet.handleController( MapReduceServlet.java: 507) at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost (MapReduceServlet.java: 222) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle (ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1166) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter (ParseBlobUploadFilter.java: 97) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter (SaveSessionFilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter (TransactionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle (SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle (WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle (AppVersionHandlerMap.java: 238) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest (HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable (RpcRequestParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
[appengine-java] Re: How to get rid of a hanging mapreduce job?
Did you clear the Default queue as well? It's possible that the task queue backed off due to failure and retry at later time. But then it if it happens long time ago the queue should have given up already. On Oct 17, 5:37 am, burnayev burna...@gmail.com wrote: I deleted the mapreduce-related entities long time ago and it was inconsequential. On Oct 17, 2:50 am, Peter Liu tinyee...@gmail.com wrote: I had the same problem. I fix it by deleting the entries of the 2 map reduce table. Go to to the datastore viewer and there's 2 Kind used by map reduce. By the way, how to mark the task as completed? When my task is done it always end with the Unknown state. I asked this question before but there was no answer. On Oct 16, 9:30 am, burnayev burna...@gmail.com wrote: Mapper API for Java. Usedhttp://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-ap... as a starting point. On Oct 15, 5:45 pm, Guillermo Schwarz guillermo.schw...@gmail.com wrote: Which map reduce library are you using? Saludos, Guillermo Schwarz. El 15-10-2010, a las 18:44, burnayev burna...@gmail.com escribió: Here's the scoop... One of my first mapreduce jobs didn't want to complete by itself. It did not want to abort either. To get rid of the sucker I deployed a new application version and deleted the one the job was running against. I also manually deleted all the residual state mapreduce created in the datastore. That seemed to kill most of it. However now, 5 days later, there are still two artifacts - mapperCallback and controllerCallback - that disturb my serenity (and keep sucking the juice) by popping up every hour or so. Obviously they are looking for a job that exists no more and fail miserably with a stack trace similar to below. Is there a way to make them go away? java.lang.RuntimeException: Couldn't find MR with job ID: job_1286643750234_0001 at com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest( AppEngineJobContext.java: 157) at com.google.appengine.tools.mapreduce.AppEngineJobContext.init (AppEngineJobContext.java: 110) at com.google.appengine.tools.mapreduce.MapReduceServlet.handleController( MapReduceServlet.java: 507) at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost (MapReduceServlet.java: 222) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle (ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1166) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter (ParseBlobUploadFilter.java: 97) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter (SaveSessionFilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter (TransactionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle (SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle (WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle (AppVersionHandlerMap.java: 238) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest (HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable (RpcRequestParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest( JettyServletEngineAdapter.java: 135) at com.google.apphosting.runtime.JavaRuntime.handleRequest (JavaRuntime.java: 261) at
[appengine-java] Re: How to get rid of a hanging mapreduce job?
I had the same problem. I fix it by deleting the entries of the 2 map reduce table. Go to to the datastore viewer and there's 2 Kind used by map reduce. By the way, how to mark the task as completed? When my task is done it always end with the Unknown state. I asked this question before but there was no answer. On Oct 16, 9:30 am, burnayev burna...@gmail.com wrote: Mapper API for Java. Usedhttp://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-ap... as a starting point. On Oct 15, 5:45 pm, Guillermo Schwarz guillermo.schw...@gmail.com wrote: Which map reduce library are you using? Saludos, Guillermo Schwarz. El 15-10-2010, a las 18:44, burnayev burna...@gmail.com escribió: Here's the scoop... One of my first mapreduce jobs didn't want to complete by itself. It did not want to abort either. To get rid of the sucker I deployed a new application version and deleted the one the job was running against. I also manually deleted all the residual state mapreduce created in the datastore. That seemed to kill most of it. However now, 5 days later, there are still two artifacts - mapperCallback and controllerCallback - that disturb my serenity (and keep sucking the juice) by popping up every hour or so. Obviously they are looking for a job that exists no more and fail miserably with a stack trace similar to below. Is there a way to make them go away? java.lang.RuntimeException: Couldn't find MR with job ID: job_1286643750234_0001 at com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest( AppEngineJobContext.java: 157) at com.google.appengine.tools.mapreduce.AppEngineJobContext.init (AppEngineJobContext.java: 110) at com.google.appengine.tools.mapreduce.MapReduceServlet.handleController( MapReduceServlet.java: 507) at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost (MapReduceServlet.java: 222) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle (ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1166) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter (ParseBlobUploadFilter.java: 97) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter (SaveSessionFilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter (TransactionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle (SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle (WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle (AppVersionHandlerMap.java: 238) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest (HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable (RpcRequestParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest( JettyServletEngineAdapter.java: 135) at com.google.apphosting.runtime.JavaRuntime.handleRequest (JavaRuntime.java: 261) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:8483) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:8481) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest (BlockingApplicationHandler.java: 24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication (RpcUtil.java: 418) at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java: 572) at com.google.tracing.TraceContext$TraceContextRunnable $1.run(TraceContext.java:448) at com.google.tracing.TraceContext.runInContext(TraceContext.java: 688)
[appengine-java] Re: How to get rid of a hanging mapreduce job?
I deleted the mapreduce-related entities long time ago and it was inconsequential. On Oct 17, 2:50 am, Peter Liu tinyee...@gmail.com wrote: I had the same problem. I fix it by deleting the entries of the 2 map reduce table. Go to to the datastore viewer and there's 2 Kind used by map reduce. By the way, how to mark the task as completed? When my task is done it always end with the Unknown state. I asked this question before but there was no answer. On Oct 16, 9:30 am, burnayev burna...@gmail.com wrote: Mapper API for Java. Usedhttp://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-ap... as a starting point. On Oct 15, 5:45 pm, Guillermo Schwarz guillermo.schw...@gmail.com wrote: Which map reduce library are you using? Saludos, Guillermo Schwarz. El 15-10-2010, a las 18:44, burnayev burna...@gmail.com escribió: Here's the scoop... One of my first mapreduce jobs didn't want to complete by itself. It did not want to abort either. To get rid of the sucker I deployed a new application version and deleted the one the job was running against. I also manually deleted all the residual state mapreduce created in the datastore. That seemed to kill most of it. However now, 5 days later, there are still two artifacts - mapperCallback and controllerCallback - that disturb my serenity (and keep sucking the juice) by popping up every hour or so. Obviously they are looking for a job that exists no more and fail miserably with a stack trace similar to below. Is there a way to make them go away? java.lang.RuntimeException: Couldn't find MR with job ID: job_1286643750234_0001 at com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest( AppEngineJobContext.java: 157) at com.google.appengine.tools.mapreduce.AppEngineJobContext.init (AppEngineJobContext.java: 110) at com.google.appengine.tools.mapreduce.MapReduceServlet.handleController( MapReduceServlet.java: 507) at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost (MapReduceServlet.java: 222) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle (ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1166) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter (ParseBlobUploadFilter.java: 97) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter (SaveSessionFilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter (TransactionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle (SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle (WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle (AppVersionHandlerMap.java: 238) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest (HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable (RpcRequestParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest( JettyServletEngineAdapter.java: 135) at com.google.apphosting.runtime.JavaRuntime.handleRequest (JavaRuntime.java: 261) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:8483) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:8481) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest (BlockingApplicationHandler.java: 24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication (RpcUtil.java:
[appengine-java] Re: How to get rid of a hanging mapreduce job?
Mapper API for Java. Used http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine/ as a starting point. On Oct 15, 5:45 pm, Guillermo Schwarz guillermo.schw...@gmail.com wrote: Which map reduce library are you using? Saludos, Guillermo Schwarz. El 15-10-2010, a las 18:44, burnayev burna...@gmail.com escribió: Here's the scoop... One of my first mapreduce jobs didn't want to complete by itself. It did not want to abort either. To get rid of the sucker I deployed a new application version and deleted the one the job was running against. I also manually deleted all the residual state mapreduce created in the datastore. That seemed to kill most of it. However now, 5 days later, there are still two artifacts - mapperCallback and controllerCallback - that disturb my serenity (and keep sucking the juice) by popping up every hour or so. Obviously they are looking for a job that exists no more and fail miserably with a stack trace similar to below. Is there a way to make them go away? java.lang.RuntimeException: Couldn't find MR with job ID: job_1286643750234_0001 at com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest( AppEngineJobContext.java: 157) at com.google.appengine.tools.mapreduce.AppEngineJobContext.init (AppEngineJobContext.java: 110) at com.google.appengine.tools.mapreduce.MapReduceServlet.handleController( MapReduceServlet.java: 507) at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost (MapReduceServlet.java: 222) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle (ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1166) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter (ParseBlobUploadFilter.java: 97) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter (SaveSessionFilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter (TransactionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle (SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle (WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle (AppVersionHandlerMap.java: 238) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest (HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable (RpcRequestParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest( JettyServletEngineAdapter.java: 135) at com.google.apphosting.runtime.JavaRuntime.handleRequest (JavaRuntime.java: 261) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:8483) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:8481) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest (BlockingApplicationHandler.java: 24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication (RpcUtil.java: 418) at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java: 572) at com.google.tracing.TraceContext$TraceContextRunnable $1.run(TraceContext.java:448) at com.google.tracing.TraceContext.runInContext(TraceContext.java: 688) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContextNoUnref (TraceContext.java: 326) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContext(TraceContext.java: 318) at com.google.tracing.TraceContext $TraceContextRunnable.run(TraceContext.java:446) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java: 1110) at java.util.concurrent.ThreadPoolExecutor
