Hi Sumudu, Thanks for sharing your findings! Unfortunately, the attachments (fig_1 and fig_2) are missing. They were probably ignored by the mailing list. Could you upload them to Google Drive and share the links?
BTW, is it the same query you shared at this thread? https://lists.apache.org/thread/dstobv61xjhm18vddv264s66sx2oh22w I think we need some time to dig into the details. Not sure if the issue still occurs in 4.1.0. Thanks, Quanlong On Tue, Jun 7, 2022 at 11:42 PM Sumudu Madushanka < [email protected]> wrote: > Hi all, > > We are using kudu 1.15.0, and impala 3.4.0-RELEASE > > In our impala select query, we are scanning *10 kudu partitions/tablets* > per impala daemon host in *6 node cluster* with *16 cores* per machine ( > *c5.4xlarge*). > > The kudu threads (around 9 threads) are working in parallel to do the > scanning as in given in fig_2. But, in Impala *only one reactor thread > (rpc reactor-227 thread)* is working and it’s at* 99.99%* which creates a > bottleneck(fig_1). There are other reactor threads at idle. Please note > that the query results are not cached to disk. > > Here are our questions. > > 1. Why only one rpc reactor thread is working and creating a bottleneck > for multi-tablets scan query in an impala daemon host and why is it always > *rpc > reactor-227*? (We also suspect that this could be the rpc reactor thread > that runs inside kudu client inside the impala daemon) > > 2. How can we solve this issue for other rpc reactor threads to work as > well? > > This is creating a huge bottleneck in the query scanning. Could there be > other reasons why this behavior occurs? > > Appreciate your help! > > gstack 22788 > Thread 1 (process 22788): > #0 0x00007f15a5ebeaeb in recv () from /usr/lib64/libpthread.so.0 > #1 0x00007f15a42763b4 in kudu::Socket::Recv (this=0xd5588f0, > buf=0x17324004 "\f\b\325\300\b\020", amt=1185345, nread=0x7f14c81d9940) at > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/net/socket.cc:517 > #2 0x00007f15a419172e in kudu::rpc::InboundTransfer::ReceiveBuffer > (this=0xd507840, socket=...) at > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/transfer.cc:144 > #3 0x00007f15a4182fd0 in ReadHandler (revents=<optimized out>, > watcher=..., this=0xdb56000) at > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/connection.cc:653 > #4 ev::base<ev_io, ev::io>::method_thunk<kudu::rpc::Connection, > &kudu::rpc::Connection::ReadHandler> (loop=<optimized out>, w=<optimized > out>, revents=<optimized out>) at > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:479 > #5 0x00007f15a43c1f1b in ev_invoke_pending (loop=0xd552480) at > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3155 > #6 0x00007f15a41611dd in kudu::rpc::ReactorThread::InvokePendingCb > (loop=0xd552480) at > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:196 > #7 0x00007f15a43c55f4 in ev_run (loop=0xd552480, flags=<optimized out>) at > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3555 > #8 0x00007f15a41617ab in run (flags=0, this=0x9c03088) at > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:211 > #9 kudu::rpc::ReactorThread::RunThread (this=0x9c03080) at > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:497 > #10 0x00007f15a429df06 in operator() (this=0x9ffdb28) at > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771 > #11 kudu::Thread::SuperviseThread (arg=0x9ffdb00) at > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/thread.cc:675 > #12 0x00007f15a5eb7ea5 in start_thread () from /usr/lib64/libpthread.so.0 > #13 0x00007f15a252b9fd in clone () from /usr/lib64/libc.so.6 > > > > Thank you > > > > Best Regards, > *Sumudu Madushanka* >
