zuston commented on issue #198: URL: https://github.com/apache/incubator-uniffle/issues/198#issuecomment-1238046717
> Revisit: do we need to fail fast? As I know, the exception of BufferUnderflowException is caused by the incomplete `FileBasedShuffleSegment`. In localfile reader, it will always get the n pieces of `FileBasedShuffleSegment` and then wont throw the exception. However in HDFS index read, it may return the incomplete data ((this problem will happen when spill data from memory to HDFS, so we also should do similar operation like local index file read). After that, we should do fail fast in https://github.com/apache/incubator-uniffle/blob/07f70ed872b87107fc7028577bd9e66d8349fd6c/common/src/main/java/org/apache/uniffle/common/util/RssUtils.java#L224 > Is inconsistent blocks caused by this `BufferUnderflowException`? The operation of spilling is a common action for uniffle and it has done the operation of filtering blocks. So this inconsistent blocks means it exist bug when spilling -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
