Sebastian YEPES FERNANDEZ created SPARK-11170: -------------------------------------------------
Summary: EOFException on History server reading in progress lz4 Key: SPARK-11170 URL: https://issues.apache.org/jira/browse/SPARK-11170 Project: Spark Issue Type: Bug Components: Web UI, YARN Affects Versions: 1.5.1 Environment: HDP: 2.3.2.0-2950 (Hadoop 2.7.1.2.3.2.0-2950) Spark: 1.5.x (c27e1904) Reporter: Sebastian YEPES FERNANDEZ The Spark History server is not able to read/save the jobs history if Spark is configured to use "spark.io.compression.codec=org.apache.spark.io.LZ4CompressionCodec", it continuously generated the following error: {code} ERROR 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: Exception encountered when attempting to load application log hdfs://DATA/user/spark/his tory/application_1444297190346_0073_1.lz4.inprogress java.io.EOFException: Stream ended prematurely at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218) at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150) at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) at java.io.BufferedReader.readLine(BufferedReader.java:389) at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67) at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:55) at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:457) at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:292) at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:289) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:289) at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$anon$2.run(FsHistoryProvider.scala:210) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) INFO 2015-10-16 16:21:39 org.apache.spark.deploy.history.FsHistoryProvider: Replaying log path: hdfs://DATA/user/spark/history/application_1444297190346_0072_1.lz4.i nprogress {code} As a workaround setting "spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec" makes the History server work correctly -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org