Github user shahidki31 commented on a diff in the pull request:
https://github.com/apache/spark/pull/23241#discussion_r239494478
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -197,4 +201,8 @@ class ZStdCompressionCodec(conf: SparkConf) extends
CompressionCodec {
// avoid overhead excessive of JNI call while trying to uncompress
small amount of data.
new BufferedInputStream(new ZstdInputStream(s), bufferSize)
}
+
+ override def zstdEventLogCompressedInputStream(s: InputStream):
InputStream = {
+ new BufferedInputStream(new ZstdInputStream(s).setContinuous(true),
bufferSize)
--- End diff --
Hi @srowen Thanks for the comment.
Yes. This parameter is, by default false, intended for continuous stream
reading. So, if some classes doesn't need continuously read the data, do we
need to set `isContinuous` as true.
This method is called by other classes, like 'UnsafeShuffleWriter' etc.
which are performance sensitive. If it try to read from the open frames, this
issue (read error exception) will happen in other classes as well. But, other
than from 'EventLoggingListener', this issue hasn't reported. That is why, I
tried to limit it to the EventLoggingListener call.
Yes. If we do 'continuous' true for all, then this code will be much
simplified.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
