[ 
https://issues.apache.org/jira/browse/IMPALA-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6425.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 3.1.0

> Change Mempool memory allocation size to be <1MB to avoid allocating from 
> CentralFreeList
> -----------------------------------------------------------------------------------------
>
>                 Key: IMPALA-6425
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6425
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 2.11.0
>            Reporter: Mostafa Mokhtar
>            Assignee: Tim Armstrong
>            Priority: Major
>              Labels: perfomance
>             Fix For: Impala 3.1.0
>
>
> While [~tlipcon] was investigating KRPC contention he noticed that 
> MemPool::Allocate is doing 1MB allocations, which is somewhat of an 
> anti-pattern with tcmalloc. 
> During the tests MemPool was doing several thousand 1MB allocs per second and 
> those have to do a full scan of the tcmalloc span linked list, which is very 
> slow and only gets slower
>  1040384 bytes on the other hand is constant time. 
>  
> It is not clear if a Power of 2 allocation size would help, worth 
> experimenting with 512KB and 1040384 bytes. 
> {code}
>  /// The maximum size of chunk that should be allocated. Allocations larger 
> than this
>  /// size will get their own individual chunk.
>  static const int MAX_CHUNK_SIZE = 8192*127
> {code}
>  
> {code}
> #0  0x0000000002097407 in base::internal::SpinLockDelay(int volatile*, int, 
> int) ()
> #1  0x00000000020e2049 in SpinLock::SlowLock() ()
> #2  0x0000000002124348 in tcmalloc::CentralFreeList::Populate() ()
> #3  0x0000000002124458 in 
> tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) ()
> #4  0x00000000021244e8 in tcmalloc::CentralFreeList::RemoveRange(void**, 
> void**, int) ()
> #5  0x0000000002131ee5 in 
> tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* 
> (*)(unsigned long)) ()
> #6  0x0000000000b2879a in impala::MemPool::FindChunk(long, bool) ()
> #7  0x0000000000b364f6 in impala::MemPool::Allocate(long) ()
> #8  0x0000000000b36674 in impala::FreePool::Allocate(long) ()
> #9  0x0000000000b353db in impala::RowBatch::Deserialize(kudu::Slice const&, 
> kudu::Slice const&, long, bool, impala::FreePool*) ()
> #10 0x0000000000b35795 in impala::RowBatch::RowBatch(impala::RowDescriptor 
> const*, impala::RowBatchHeaderPB const&, kudu::Slice const&, kudu::Slice 
> const&, impala::FreePool*) ()
> #11 0x0000000000b1644f in 
> impala::KrpcDataStreamRecvr::SenderQueue::AddBatchWork(long, 
> impala::RowBatchHeaderPB const&, kudu::Slice const&, kudu::Slice const&, 
> boost::unique_lock<impala::SpinLock>*) ()
> #12 0x0000000000b19135 in 
> impala::KrpcDataStreamRecvr::SenderQueue::AddBatch(impala::TransmitDataRequestPB
>  const*, kudu::rpc::RpcContext*) ()
> #13 0x0000000000b0ee30 in 
> impala::KrpcDataStreamMgr::AddData(impala::TransmitDataRequestPB const*, 
> kudu::rpc::RpcContext*) ()
> #14 0x0000000001187035 in 
> kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) ()
> #15 0x00000000011bc1cd in impala::ImpalaServicePool::RunThread(long) ()
> {code}
>  
> Also it appears that the thread above was a victim of thread below, yet 
> allocations <1MB will make MemPool::Allocate content less over the 
> CentralFreeList lock. 
> {code}
> #0 0x0000003173ae5407 in madvise () from /lib64/libc.so.6
> #1 0x0000000002131cca in TCMalloc_SystemRelease(void*, unsigned long) ()
> #2 0x000000000212f26a in tcmalloc::PageHeap::DecommitSpan(tcmalloc::Span*) ()
> #3 0x000000000212f505 in 
> tcmalloc::PageHeap::MergeIntoFreeList(tcmalloc::Span*) ()
> #4 0x000000000212f864 in tcmalloc::PageHeap::Delete(tcmalloc::Span*) ()
> #5 0x0000000002123cf7 in tcmalloc::CentralFreeList::ReleaseToSpans(void*) ()
> #6 0x0000000002123d9b in tcmalloc::CentralFreeList::ReleaseListToSpans(void*) 
> ()
> #7 0x0000000002124067 in tcmalloc::CentralFreeList::InsertRange(void*, void*, 
> int) ()
> #8 0x00000000021320a4 in 
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
>  unsigned int, int) ()
> #9 0x0000000002132575 in 
> tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned 
> int) ()
> #10 0x0000000000b276e0 in impala::MemPool::FreeAll() ()
> #11 0x0000000000b34655 in impala::RowBatch::Reset() ()
> #12 0x0000000000fe882f in 
> impala::PartitionedAggregationNode::GetRowsStreaming(impala::RuntimeState*, 
> impala::RowBatch*) ()
> #13 0x0000000000fe9771 in 
> impala::PartitionedAggregationNode::GetNext(impala::RuntimeState*, 
> impala::RowBatch*, bool*) ()
> #14 0x0000000000b78352 in impala::FragmentInstanceState::ExecInternal() ()
> #15 0x0000000000b7adc2 in impala::FragmentInstanceState::Exec() ()
> #16 0x0000000000b6a0da in 
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) ()
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to