attilapiros commented on pull request #32790: URL: https://github.com/apache/spark/pull/32790#issuecomment-855688142
> However, looking at removeBlock, it shows it only cleans rdd blocks indeed. So it makes me think whether we missed other block types. Fortunately, we only have broadcast block and rdd block here so it won't leak anything. @Ngone51 I am not so sure about that. 1.) > However, looking at removeBlock, it shows it only cleans rdd blocks indeed. Why do you sure about that? Just looking only these lines? https://github.com/apache/spark/blob/6de20f10ca60c778db25ba3d5975f85e5c23aedb/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L906-L914 2. > Fortunately, we only have broadcast block and rdd block here so it won't leak anything. Do we? https://github.com/apache/spark/blob/6de20f10ca60c778db25ba3d5975f85e5c23aedb/core/src/main/scala/org/apache/spark/storage/BlockId.scala#L96 Related to both points: https://github.com/apache/spark/blob/6de20f10ca60c778db25ba3d5975f85e5c23aedb/core/src/main/scala/org/apache/spark/executor/Executor.scala#L617 https://github.com/apache/spark/blob/6de20f10ca60c778db25ba3d5975f85e5c23aedb/core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala#L80 I am sorry but now I have a limited time for this (starting tomorrow I will be without my laptop for the next 10 days) and within the scope "Fix memory leak in BlockManagerMasterEndpoint removeRdd" I intend to fix the existing / proved memory leaks and avoid introducing any new errors or performance degradation by being not careful enough. Moreover I do not want to leave any potential error for the others (unattended by me). With my phone I cannot fix it (although a vim probably would run just fine on it ;) but having no keyboard no sbt..) so it would disturb me a lot and it could ruin my vacation what I do not want risk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
