Ah, sorry, that's definitely what Shixiong mentioned. The patch I mentioned did not make it into 1.3...
On Thu, May 7, 2015 at 11:48 AM, Koert Kuipers <ko...@tresata.com> wrote: > seems i got one thread spinning 100% for a while now, in > FsHistoryProvider.initialize(). maybe something wrong with my logs on hdfs > that its reading? or could it simply really take 30 mins to read all the > history on dhfs? > > jstack: > > Deadlock Detection: > > No deadlocks found. > > Thread 2272: (state = BLOCKED) > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, > boolean, long) @bci=174, line=460 (Compiled frame) > - > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=359 (Interpreted frame) > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=942 (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1068 > (Interpreted frame) > - > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=26, line=1130 (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) > > > Thread 1986: (state = BLOCKED) > - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) > - org.apache.hadoop.hdfs.PeerCache.run() @bci=41, line=250 (Interpreted > frame) > - > org.apache.hadoop.hdfs.PeerCache.access$000(org.apache.hadoop.hdfs.PeerCache) > @bci=1, line=41 (Interpreted frame) > - org.apache.hadoop.hdfs.PeerCache$1.run() @bci=4, line=119 (Interpreted > frame) > - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) > > > Thread 1970: (state = BLOCKED) > > > Thread 1969: (state = BLOCKED) > - java.lang.Object.wait(long) @bci=0 (Interpreted frame) > - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 > (Interpreted frame) > - java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Interpreted > frame) > - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 > (Interpreted frame) > > > Thread 1968: (state = BLOCKED) > - java.lang.Object.wait(long) @bci=0 (Interpreted frame) > - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame) > - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 > (Interpreted frame) > > > Thread 1958: (state = IN_VM) > - java.lang.Throwable.fillInStackTrace(int) @bci=0 (Compiled frame; > information may be imprecise) > - java.lang.Throwable.fillInStackTrace() @bci=16, line=783 (Compiled > frame) > - java.lang.Throwable.<init>(java.lang.String, java.lang.Throwable) > @bci=24, line=287 (Compiled frame) > - java.lang.Exception.<init>(java.lang.String, java.lang.Throwable) > @bci=3, line=84 (Compiled frame) > - org.json4s.package$MappingException.<init>(java.lang.String, > java.lang.Exception) @bci=13, line=56 (Compiled frame) > - org.json4s.reflect.package$.fail(java.lang.String, java.lang.Exception) > @bci=6, line=96 (Compiled frame) > - org.json4s.Extraction$.convert(org.json4s.JsonAST$JValue, > org.json4s.reflect.ScalaType, org.json4s.Formats, scala.Option) @bci=2447, > line=554 (Compiled frame) > - org.json4s.Extraction$.extract(org.json4s.JsonAST$JValue, > org.json4s.reflect.ScalaType, org.json4s.Formats) @bci=796, line=331 > (Compiled frame) > - org.json4s.Extraction$.extract(org.json4s.JsonAST$JValue, > org.json4s.Formats, scala.reflect.Manifest) @bci=10, line=42 (Compiled > frame) > - org.json4s.Extraction$.extractOpt(org.json4s.JsonAST$JValue, > org.json4s.Formats, scala.reflect.Manifest) @bci=7, line=54 (Compiled frame) > - org.json4s.ExtractableJsonAstNode.extractOpt(org.json4s.Formats, > scala.reflect.Manifest) @bci=9, line=40 (Compiled frame) > - > org.apache.spark.util.JsonProtocol$.shuffleWriteMetricsFromJson(org.json4s.JsonAST$JValue) > @bci=116, line=702 (Compiled frame) > - > org.apache.spark.util.JsonProtocol$$anonfun$taskMetricsFromJson$2.apply(org.json4s.JsonAST$JValue) > @bci=4, line=670 (Compiled frame) > - > org.apache.spark.util.JsonProtocol$$anonfun$taskMetricsFromJson$2.apply(java.lang.Object) > @bci=5, line=670 (Compiled frame) > - scala.Option.map(scala.Function1) @bci=22, line=145 (Compiled frame) > - > org.apache.spark.util.JsonProtocol$.taskMetricsFromJson(org.json4s.JsonAST$JValue) > @bci=414, line=670 (Compiled frame) > - > org.apache.spark.util.JsonProtocol$.taskEndFromJson(org.json4s.JsonAST$JValue) > @bci=174, line=508 (Compiled frame) > - > org.apache.spark.util.JsonProtocol$.sparkEventFromJson(org.json4s.JsonAST$JValue) > @bci=389, line=464 (Compiled frame) > - > org.apache.spark.scheduler.ReplayListenerBus$$anonfun$replay$1.apply(java.lang.String) > @bci=34, line=51 (Compiled frame) > - > org.apache.spark.scheduler.ReplayListenerBus$$anonfun$replay$1.apply(java.lang.Object) > @bci=5, line=49 (Compiled frame) > - scala.collection.Iterator$class.foreach(scala.collection.Iterator, > scala.Function1) @bci=16, line=743 (Compiled frame) > - scala.collection.AbstractIterator.foreach(scala.Function1) @bci=2, > line=1177 (Interpreted frame) > - > org.apache.spark.scheduler.ReplayListenerBus.replay(java.io.InputStream, > java.lang.String) @bci=42, line=49 (Interpreted frame) > - > org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(org.apache.hadoop.fs.FileStatus, > org.apache.spark.scheduler.ReplayListenerBus) @bci=69, line=260 > (Interpreted frame) > - > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$6.apply(org.apache.hadoop.fs.FileStatus) > @bci=19, line=190 (Interpreted frame) > - > org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$6.apply(java.lang.Object) > @bci=5, line=188 (Interpreted frame) > - > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(java.lang.Object) > @bci=9, line=252 (Interpreted frame) > - > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(java.lang.Object) > @bci=2, line=252 (Interpreted frame) > - > scala.collection.IndexedSeqOptimized$class.foreach(scala.collection.IndexedSeqOptimized, > scala.Function1) @bci=22, line=33 (Interpreted frame) > - scala.collection.mutable.WrappedArray.foreach(scala.Function1) @bci=2, > line=35 (Interpreted frame) > - > scala.collection.TraversableLike$class.flatMap(scala.collection.TraversableLike, > scala.Function1, scala.collection.generic.CanBuildFrom) @bci=17, line=252 > (Interpreted frame) > - scala.collection.AbstractTraversable.flatMap(scala.Function1, > scala.collection.generic.CanBuildFrom) @bci=3, line=104 (Interpreted frame) > - org.apache.spark.deploy.history.FsHistoryProvider.checkForLogs() > @bci=110, line=188 (Interpreted frame) > - org.apache.spark.deploy.history.FsHistoryProvider.initialize() @bci=38, > line=116 (Interpreted frame) > - > org.apache.spark.deploy.history.FsHistoryProvider.<init>(org.apache.spark.SparkConf) > @bci=214, line=99 (Interpreted frame) > - > sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, > java.lang.Object[]) @bci=0 (Interpreted frame) > - > sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) > @bci=72, line=57 (Interpreted frame) > - > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) > @bci=5, line=45 (Interpreted frame) > - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, > line=526 (Interpreted frame) > - org.apache.spark.deploy.history.HistoryServer$.main(java.lang.String[]) > @bci=89, line=185 (Interpreted frame) > - org.apache.spark.deploy.history.HistoryServer.main(java.lang.String[]) > @bci=4 (Interpreted frame) > > > > On Thu, May 7, 2015 at 2:17 PM, Koert Kuipers <ko...@tresata.com> wrote: > >> good idea i will take a look. it does seem to be spinning one cpu at >> 100%... >> >> On Thu, May 7, 2015 at 2:03 PM, Marcelo Vanzin <van...@cloudera.com> >> wrote: >> >>> Can you get a jstack for the process? Maybe it's stuck somewhere. >>> >>> On Thu, May 7, 2015 at 11:00 AM, Koert Kuipers <ko...@tresata.com> >>> wrote: >>> >>>> i am trying to launch the spark 1.3.1 history server on a secure >>>> cluster. >>>> >>>> i can see in the logs that it successfully logs into kerberos, and it >>>> is replaying all the logs, but i never see the log message that indicate >>>> the web server is started (i should see something like "Successfully >>>> started service on port 18080." or "Started HistoryServer at >>>> http://somehost:18080";). yet the daemon stays alive... >>>> >>>> any idea why the history server would never start the web service? >>>> >>>> thanks! >>>> >>>> >>> >>> >>> -- >>> Marcelo >>> >> >> > -- Marcelo
