Hello,
I am sorry about my first explanation, was not concrete. Well I
will explain further about TaskMemoryManager. TaskMemoryManager manages
the execution memory of each task application as follow:
1. MemoryConsumer is the entry for the Spark task to run.
MemoryConsumer requests execution memory from TaskMemoryManager.
2. TaskMemoryManager requests memory from ExeuctionMemoryPool. If
the execution memory is insufficient, it will borrow storage memory. If
it is not enough, it will force the cached data in storage to be flushed
to disk to free memory.
3. If the Memory returned by the ExecutionMemoryPool is
insufficient, the MemoryConsumer.spill method is called to flush the
memory data occupied by the MemoryConsumer to the disk to free the memory.
4. Then use HeapMemoryAllocator to allocate memory for
MemoryConsumer OnHeap or use UnsafeMemoryAllocator to allocate memory
for MemoryConsumer OffHeap, based on the MemoryMode.
5. Wrap the allocation memory to a MemoryBlock and each MemoryBlock
is corresponding to a page.
6. The taskpageManager maintains the page Table of the task, and
the task can query the corresponding MemoryBlock by page number.
So using the name "TungstenConsumer" I mean the MemoryConsumer that use
offHeap memory execution. Running some tests to see when
HeapMemoryAllocator is called, I see that for some applications
HeapMemoryAllocator is called and for some others not. Could you please
explain me why this happens ? HeapMemoryAllocator would not always
called by MemoryConsumer ?
--Iacovos
On 11/02/2019 11:06 πμ, Wenchen Fan wrote:
what do you mean by ''Tungsten Consumer"?
On Fri, Feb 8, 2019 at 6:11 PM Jack Kolokasis <[email protected]
<mailto:[email protected]>> wrote:
Hello all,
I am studying about Tungsten Project and I am wondering when
Spark
creates a Tungsten consumer. While I am running some applications,
I see
that Spark creates Tungsten Consumer while in other applications not
(using the same configuration). When does this happens ?
I am looking forward for your reply.
--Jack Kolokasis
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
<mailto:[email protected]>
--
Iacovos Kolokasis
Email: [email protected]
Postgraduate Student CSD, University of Crete
Researcher in CARV Lab ICS FORTH
