Please take a look at the design doc attached to SPARK-10000
The answer is on page 2 of that doc.
On Mon, Oct 12, 2015 at 8:55 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> Please note the block where cleanUpAllAllocatedMemory() is called:
> } finally {
> val freedMemory = taskMemoryManager.cleanUpAllAllocatedMemory()
> if (freedMemory > 0) {
> I think the intention is that allocated memory should have been freed by
> the time we reach the finally block in normal program execution.
> Hence the check for the return value for memory leak.
>
> Experts may have better explanation.
>
> On Mon, Oct 12, 2015 at 12:28 AM, Lei Wu <wulei.bj...@gmail.com> wrote:
>
>> Dear all,
>>
>> I'm reading source code of TaskMemoryManager.java and I got stuck in the
>> last function, that is cleanUpAllAllocatedMemory. What confuses me is the
>> comments for this function : "A non-zero return value can be used to detect
>> memory leaks".
>>
>> And from Executor.scala where this function is called, the code is like
>> this :
>> val freedMemory = taskMemoryManager.cleanUpAllAllocatedMemory()
>> if(freedMemory > 0) {
>> val errMsg = "Managed memory leak detected; ..."
>> }
>>
>> My question is : this function cleanUpAllAllocatedMemory is used to clean
>> up all the allocated memory and pages, and computes the bytes that are
>> freed. Of course this value will be greater than 0 since we consumed that
>> many amount of memory. What's the logic of detecting memory leak by
>> checking this returned value is greater than 0?
>>
>
>
