We've been using the V2 client from 0.6.1 to do updates with the beta_updater. The only custom-y thing we do is use the add_metadata() function on the AuroraConfig object before issuing our updates.
Since we have no task history that we care about in this cluster right now I tried setting all -history_prune_threshold=1mins but the problem still persists. Do you have any other suggestions? Thanks for the help! Josh On Tue, Mar 31, 2015 at 1:50 PM, Maxim Khutornenko <[email protected]> wrote: > Interesting. If your client is built off of master it points towards > this line: > https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/client/cli/jobs.py#L570 > > Somehow you have tasks without status? No sure what could cause that. > Do you have any client or scheduler customizations that may interfere > with thrift layer? > > Anyway, the only sane way forward from here is to wait or force the > task history cleanup I described earlier. > > On Tue, Mar 31, 2015 at 1:34 PM, Josh Adams <[email protected]> wrote: > > Ah ok. So for those that exist in the history I see events like this for > > each: > > > > 2015-03-20 22:43:33 PENDING: None > > 2015-03-20 22:43:33 ASSIGNED: None > > 2015-03-20 22:43:36 STARTING: Initializing sandbox. > > 2015-03-20 22:43:39 RUNNING: None > > 2015-03-31 00:54:46 KILLING: Killed by UNSECURE > > 2015-03-31 00:55:06 KILLED: Instructed to kill task. > > > > There are also some that I expected to exist but threw the exception: > > > > Traceback (most recent call last): > > File > > > "/home/josh/.loko/pkg/infrastructure-aurora-base/staged/d786fd7/bin/aurora.pex/.bootstrap/_pex/pex.py", > > line 272, in execute > > self.execute_entry(entry_point, args) > > File > > > "/home/josh/.loko/pkg/infrastructure-aurora-base/staged/d786fd7/bin/aurora.pex/.bootstrap/_pex/pex.py", > > line 320, in execute_entry > > runner(entry_point) > > File > > > "/home/josh/.loko/pkg/infrastructure-aurora-base/staged/d786fd7/bin/aurora.pex/.bootstrap/_pex/pex.py", > > line 343, in execute_pkg_resources > > runner() > > File "apache/aurora/client/cli/client.py", line 95, in proxy_main > > File "apache/aurora/client/cli/__init__.py", line 329, in execute > > File "apache/aurora/client/cli/__init__.py", line 306, in _execute > > File "apache/aurora/client/cli/__init__.py", line 382, in execute > > File "apache/aurora/client/cli/jobs.py", line 659, in execute > > File "apache/aurora/client/cli/jobs.py", line 626, in > get_status_for_jobs > > File "apache/aurora/client/cli/jobs.py", line 598, in > render_tasks_pretty > > File "apache/aurora/client/cli/jobs.py", line 570, in > render_task_pretty > > KeyError: None > > > > Josh > > > > On Tue, Mar 31, 2015 at 1:27 PM, Maxim Khutornenko <[email protected]> > wrote: > > > >> If you remember your job names you should still be able to run "aurora > >> job status" though for as long as there is task history. Would be > >> great to know what caused that problem in the first place. > >> > >> In case you want to unblock, you can temporarily change > >> -history_prune_threshold and let scheduler purge your task history. Be > >> careful as that will purge history for all jobs in your cluster, so > >> perhaps it's better wait for the scheduler to clean by itself (2 days > >> after your kill). > >> > >> On Tue, Mar 31, 2015 at 1:03 PM, Josh Adams <[email protected]> wrote: > >> > Hey Maxim, > >> > > >> > I forgot to mention that yesterday I killed all the running jobs in an > >> > attempt to get it working again. When I run aurora.pex job list > >> > <cluster>/<role>/<env> it just prints this out before handing me back > a > >> > terminal: > >> > > >> > INFO] Retrieving jobs for role produser > >> > > >> > Cheers, > >> > Josh > >> > > >> > On Tue, Mar 31, 2015 at 12:54 PM, Maxim Khutornenko <[email protected] > > > >> > wrote: > >> > > >> >> Hi Josh, > >> >> > >> >> How many jobs do you have under that role? Any chance you could run > >> >> "aurora job status" for every one of them to see if you have any > tasks > >> >> with abnormal event sequences? Every task must have at least one > event > >> >> with a timestamp. Here is a reference example: > >> >> > >> >> events: > >> >> 2015-03-23 01:01:28 PENDING: Rescheduled > >> >> 2015-03-23 01:01:28 ASSIGNED: None > >> >> 2015-03-23 01:01:30 STARTING: Initializing sandbox. > >> >> 2015-03-23 01:01:31 RUNNING: None > >> >> > >> >> On Tue, Mar 31, 2015 at 12:17 PM, Josh Adams <[email protected]> wrote: > >> >> > Hi there, > >> >> > > >> >> > I'm getting this error when clicking on a Role on the /scheduler > >> page. We > >> >> > only have a single role and this is our staging cluster. It seems > like > >> >> > other operations like `aurora.pex job list ...` and similar > commands > >> work > >> >> > fine, but I can't seem to figure out how to get the scheduler out > of > >> this > >> >> > state so I can view the Job summary page. > >> >> > > >> >> > I've upgraded from Mesos/0.20.1 & Aurora/0.6.1 to Mesos/0.21.1 and > >> >> > Aurora/0.7.1@3ab8a9e and the problem persists. Below is the stack > >> trace > >> >> > from the scheduler process. > >> >> > > >> >> > Thank you! > >> >> > Josh > >> >> > > >> >> > D0331 19:14:31.229 THREAD2827 > org.eclipse.jetty.server.Server.handle: > >> >> > REQUEST /api on AsyncHttpConnection@6fa15c80 > >> >> > > ,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=362,c=43},r=4 > >> >> > W0331 19:14:31.229 THREAD2828 > >> >> > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke: > >> >> Uncaught > >> >> > exception while handling getJobSummary(produser) > >> >> > com.google.common.collect.Ordering$IncomparableValueException: > Cannot > >> >> > compare value: null > >> >> > at > >> >> > > >> > com.google.common.collect.ExplicitOrdering.rank(ExplicitOrdering.java:46) > >> >> > at > >> >> > > >> >> > >> > com.google.common.collect.ExplicitOrdering.compare(ExplicitOrdering.java:40) > >> >> > at > >> >> > > >> >> > >> > com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46) > >> >> > at > >> >> > > >> >> > >> > com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43) > >> >> > at > com.google.common.collect.Ordering.max(Ordering.java:572) > >> >> > at > com.google.common.collect.Ordering.max(Ordering.java:539) > >> >> > at > com.google.common.collect.Ordering.max(Ordering.java:555) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.base.Tasks.getLatestActiveTask(Tasks.java:215) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:382) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl$8.transformEntry(ReadOnlySchedulerImpl.java:374) > >> >> > at > com.google.common.collect.Maps$10.getValue(Maps.java:1855) > >> >> > at java.util.HashMap.putAll(HashMap.java:647) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobs(ReadOnlySchedulerImpl.java:373) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobSummary(ReadOnlySchedulerImpl.java:257) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobSummary(SchedulerThriftInterface.java:461) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:906) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobSummary.getResult(ReadOnlyScheduler.java:891) > >> >> > at > >> >> > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > >> >> > at > >> >> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > >> >> > at > org.apache.thrift.server.TServlet.doPost(TServlet.java:83) > >> >> > at > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > >> >> > at > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69) > >> >> > at > >> >> > > >> >> > >> > org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) > >> >> > at > >> >> > org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > >> >> > at > >> >> > > >> >> > >> > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > >> >> > at > >> >> > > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) > >> >> > at > >> >> > > >> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > >> >> > at org.eclipse.jetty.server.Server.handle(Server.java:365) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005) > >> >> > at > >> >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) > >> >> > at > >> >> > > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > >> >> > at > >> >> > > >> >> > >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > >> >> > at java.lang.Thread.run(Thread.java:744) > >> >> > >> >
