Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/23241#discussion_r239476570 --- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala --- @@ -197,4 +201,8 @@ class ZStdCompressionCodec(conf: SparkConf) extends CompressionCodec { // avoid overhead excessive of JNI call while trying to uncompress small amount of data. new BufferedInputStream(new ZstdInputStream(s), bufferSize) } + + override def zstdEventLogCompressedInputStream(s: InputStream): InputStream = { + new BufferedInputStream(new ZstdInputStream(s).setContinuous(true), bufferSize) --- End diff -- BTW it seems like 'continuous' changes behavior very little: https://github.com/luben/zstd-jni/blob/master/src/main/java/com/github/luben/zstd/ZstdInputStream.java#L147 I agree with your concern to keep the change minimal. I'm trying to think if this would break anything if everything were read as 'continuous'. It wouldn't fail fast in some case?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org