Ngone51 commented on issue #23497: [SPARK-26578][CORE] Synchronize putBytes's memory allocation and putting block on memoryManager URL: https://github.com/apache/spark/pull/23497#issuecomment-457466509 > If we put some blocks into entries before putting block1, task 2 will evict them all first. So we can say we evicted all entries but still got smaller free space than whole pool size? Do you mean other blocks are all in the status as block1, which has already acquired memory but havn't put into entries ? (Ok, let's ignore the possibility of the case temporarily.) And at this point, the question may be *does the total size of blocks in entries is always equal to storage pool size*, right ? Follow the test above, for example, there may be another task 3 get a chance to do putIterator before task 2 to do putBytes. And task 3 may reserved a lot of storage unroll memory during the process. So, when task 2 got a changce to accquire storage memory and trigger evicting blocks, it will find the pool size still do not increase obviously. So, there may be do not exists strongly consistency between entries and stroage pool.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
