Re: [appengine-java] How to get rid of a hanging mapreduce job?

2010-10-15 Thread Guillermo Schwarz

Which map reduce library are you using?

Saludos,
Guillermo Schwarz.

El 15-10-2010, a las 18:44, burnayev  escribió:


Here's the scoop...

One of my first mapreduce jobs didn't want to complete by itself. It
did not want to abort either. To get rid of the sucker I deployed a
new application version and deleted the one the job was running
against. I also manually deleted all the residual state mapreduce
created in the datastore. That seemed to kill most of it. However now,
5 days later, there are still two artifacts - mapperCallback and
controllerCallback - that disturb my serenity (and keep sucking the
juice) by popping up every hour or so. Obviously they are looking for
a job that exists no more and fail miserably with a stack trace
similar to below.

Is there a way to make them go away?



java.lang.RuntimeException: Couldn't find MR with job ID:
job_1286643750234_0001
   at
com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest(
AppEngineJobContext.java:
157)
   at
com.google.appengine.tools.mapreduce.AppEngineJobContext. 
(AppEngineJobContext.java:

110)
   at
com.google.appengine.tools.mapreduce.MapReduceServlet.handleController( 
MapReduceServlet.java:

507)
   at
com.google.appengine.tools.mapreduce.MapReduceServlet.doPost 
(MapReduceServlet.java:

222)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
   at org.mortbay.jetty.servlet.ServletHolder.handle 
(ServletHolder.java:

511)
   at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1166)
   at
com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter 
(ParseBlobUploadFilter.java:

97)
   at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
   at
com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter 
(SaveSessionFilter.java:

35)
   at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
   at
com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter 
(TransactionCleanupFilter.java:

43)
   at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
   at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
388)
   at
org.mortbay.jetty.security.SecurityHandler.handle 
(SecurityHandler.java:

216)
   at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
182)
   at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
765)
   at org.mortbay.jetty.webapp.WebAppContext.handle 
(WebAppContext.java:

418)
   at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle 
(AppVersionHandlerMap.java:

238)
   at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
   at org.mortbay.jetty.Server.handle(Server.java:326)
   at org.mortbay.jetty.HttpConnection.handleRequest 
(HttpConnection.java:

542)
   at org.mortbay.jetty.HttpConnection
$RequestHandler.headerComplete(HttpConnection.java:923)
   at
com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable 
(RpcRequestParser.java:

76)
   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
   at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest( 
JettyServletEngineAdapter.java:

135)
   at
com.google.apphosting.runtime.JavaRuntime.handleRequest 
(JavaRuntime.java:

261)
   at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$6.handleBlockingRequest(RuntimePb.java:8483)
   at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$6.handleBlockingRequest(RuntimePb.java:8481)
   at
com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest 
(BlockingApplicationHandler.java:

24)
   at com.google.net.rpc.impl.RpcUtil.runRpcInApplication 
(RpcUtil.java:

418)
   at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:
572)
   at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:448)
   at com.google.tracing.TraceContext.runInContext(TraceContext.java:
688)
   at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref 
(TraceContext.java:

326)
   at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
318)
   at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:446)
   at
java.util.concurrent.ThreadPoolExecutor.runWorker 
(ThreadPoolExecutor.java:

1110)
   at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:636)

Uncaught exception from servlet
java.lang.RuntimeException: Couldn't find MR with job ID:
job_1286643750234_0001
   at
com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest( 
AppEngineJobContext.java:

157)
   at
com.google.appengine.tools.mapreduce.AppEngineJobContext. 
(AppEngineJobContext.java:

110)
   at
com.google.appengine.tool

[appengine-java] How to get rid of a hanging mapreduce job?

2010-10-15 Thread burnayev
Here's the scoop...

One of my first mapreduce jobs didn't want to complete by itself. It
did not want to abort either. To get rid of the sucker I deployed a
new application version and deleted the one the job was running
against. I also manually deleted all the residual state mapreduce
created in the datastore. That seemed to kill most of it. However now,
5 days later, there are still two artifacts - mapperCallback and
controllerCallback - that disturb my serenity (and keep sucking the
juice) by popping up every hour or so. Obviously they are looking for
a job that exists no more and fail miserably with a stack trace
similar to below.

Is there a way to make them go away?



java.lang.RuntimeException: Couldn't find MR with job ID:
job_1286643750234_0001
at
com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest(AppEngineJobContext.java:
157)
at
com.google.appengine.tools.mapreduce.AppEngineJobContext.(AppEngineJobContext.java:
110)
at
com.google.appengine.tools.mapreduce.MapReduceServlet.handleController(MapReduceServlet.java:
507)
at
com.google.appengine.tools.mapreduce.MapReduceServlet.doPost(MapReduceServlet.java:
222)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
511)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1166)
at
com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:
97)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:
35)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:
43)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
388)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:
418)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:
238)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
542)
at org.mortbay.jetty.HttpConnection
$RequestHandler.headerComplete(HttpConnection.java:923)
at
com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:
76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:
135)
at
com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:
261)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$6.handleBlockingRequest(RuntimePb.java:8483)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$6.handleBlockingRequest(RuntimePb.java:8481)
at
com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest(BlockingApplicationHandler.java:
24)
at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:
418)
at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:
572)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:448)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
688)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
326)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
318)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:446)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

Uncaught exception from servlet
java.lang.RuntimeException: Couldn't find MR with job ID:
job_1286643750234_0001
at
com.google.appengine.tools.mapreduce.AppEngineJobContext.getConfigurationFromRequest(AppEngineJobContext.java:
157)
at
com.google.appengine.tools.mapreduce.AppEngineJobContext.(AppEngineJobContext.java:
110