Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20184 >I think that a lazy buffer allocation can not thoroughly solve this problem because UnsafeSorterSpillReader has BufferedFileInputStream witch will allocate off heap memory. Can you please explain more. From my understanding the off heap memory in `BufferedFileInputStream` is the key issue for your scenario here. I don't think the logics you changed in `ChainedIterator` matters a lot. So a lazy allocation of off-heap memory should be enough IIUC.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org