[jira] [Updated] (KUDU-3107) TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service queue is full
[
https://issues.apache.org/jira/browse/KUDU-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liusheng updated KUDU-3107:
---
Issue Type: Task (was: Bug)
> TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service
> queue is full
> ---
>
> Key: KUDU-3107
> URL: https://issues.apache.org/jira/browse/KUDU-3107
> Project: Kudu
> Issue Type: Task
>Reporter: liusheng
>Priority: Major
> Attachments: rpc-test.txt
>
>
> The test TestRpc.TestCancellationMultiThreads fail sometimes on ARM mechine
> due the the "service queue full" error. related error message:
> {code:java}
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 318)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 319)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 320)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 321)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 324)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 332)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 334)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 335)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 336)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 337)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 338)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 339)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 340)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 341)
> F0416 13:01:38.616358 31937 rpc-test.cc:1471] Check failed:
> controller.status().IsAborted() || controller.status().IsServiceUnavailable()
> || controller.status().ok() Remote error: Service unavailable: PushStrings
> request on kudu.rpc.GenericCalculatorService from 127.0.0.1:41516 dropped due
> to backpressure. The service queue is full; it has 100 items.
> *** Check failure stack trace: ***
> PC: @0x0 (unknown)
> *** SIGABRT (@0x3e86bbf) received by PID 27583 (TID 0x84b1f050) from
> PID 27583; stack trace: ***
> @ 0x93cf0464 raise at ??:0
> @ 0x93cf18b4 abort at ??:0
> @ 0x942c5fdc google::logging_fail() at ??:0
> @ 0x942c7d40 google::LogMessage::Fail() at ??:0
> @ 0x942c9c78 google::LogMessage::SendToLog() at ??:0
> @ 0x942c7874 google::LogMessage::Flush() at ??:0
> @ 0x942ca4fc google::LogMessageFatal::~LogMessageFatal() at ??:0
> @ 0xdcee4940 kudu::rpc::SendAndCancelRpcs() at ??:0
> @ 0xdcee4b98
> _ZZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvENKUlvE_clEv
> at ??:0
> @ 0xdcee76bc
> _ZSt13__invoke_implIvZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_
> at ??:0
> @ 0xdcee7484
> _ZSt8__invokeIZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_
> at ??:0
> @ 0xdcee8208
> _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEE9_M_invokeIJLm0DTcl8__invokespcl10_S_declvalIXT_ESt12_Index_tupleIJXspT_EEE
> at ??:0
> @ 0xdcee8168
> _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEEclEv
> at ??:0
> @ 0xdcee8110
> _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_E6_M_runEv
> at ??:0
> @ 0x93f22e94 (unknown) at ??:0
> @ 0x93e1e088 start_thread at ??:0
> @ 0x93d8e4ec (unknown) at ??:0
> {code}
> The attatchment is the full test log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3107) TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service queue is full
[
https://issues.apache.org/jira/browse/KUDU-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liusheng updated KUDU-3107:
---
Parent: KUDU-3007
Issue Type: Sub-task (was: Task)
> TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service
> queue is full
> ---
>
> Key: KUDU-3107
> URL: https://issues.apache.org/jira/browse/KUDU-3107
> Project: Kudu
> Issue Type: Sub-task
>Reporter: liusheng
>Priority: Major
> Attachments: rpc-test.txt
>
>
> The test TestRpc.TestCancellationMultiThreads fail sometimes on ARM mechine
> due the the "service queue full" error. related error message:
> {code:java}
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 318)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 319)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 320)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 321)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 324)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 332)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 334)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 335)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 336)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 337)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 338)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 339)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 340)
> Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516
> (request call id 341)
> F0416 13:01:38.616358 31937 rpc-test.cc:1471] Check failed:
> controller.status().IsAborted() || controller.status().IsServiceUnavailable()
> || controller.status().ok() Remote error: Service unavailable: PushStrings
> request on kudu.rpc.GenericCalculatorService from 127.0.0.1:41516 dropped due
> to backpressure. The service queue is full; it has 100 items.
> *** Check failure stack trace: ***
> PC: @0x0 (unknown)
> *** SIGABRT (@0x3e86bbf) received by PID 27583 (TID 0x84b1f050) from
> PID 27583; stack trace: ***
> @ 0x93cf0464 raise at ??:0
> @ 0x93cf18b4 abort at ??:0
> @ 0x942c5fdc google::logging_fail() at ??:0
> @ 0x942c7d40 google::LogMessage::Fail() at ??:0
> @ 0x942c9c78 google::LogMessage::SendToLog() at ??:0
> @ 0x942c7874 google::LogMessage::Flush() at ??:0
> @ 0x942ca4fc google::LogMessageFatal::~LogMessageFatal() at ??:0
> @ 0xdcee4940 kudu::rpc::SendAndCancelRpcs() at ??:0
> @ 0xdcee4b98
> _ZZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvENKUlvE_clEv
> at ??:0
> @ 0xdcee76bc
> _ZSt13__invoke_implIvZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_
> at ??:0
> @ 0xdcee7484
> _ZSt8__invokeIZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_
> at ??:0
> @ 0xdcee8208
> _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEE9_M_invokeIJLm0DTcl8__invokespcl10_S_declvalIXT_ESt12_Index_tupleIJXspT_EEE
> at ??:0
> @ 0xdcee8168
> _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEEclEv
> at ??:0
> @ 0xdcee8110
> _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_E6_M_runEv
> at ??:0
> @ 0x93f22e94 (unknown) at ??:0
> @ 0x93e1e088 start_thread at ??:0
> @ 0x93d8e4ec (unknown) at ??:0
> {code}
> The attatchment is the full test log
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KUDU-2986) Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables
[ https://issues.apache.org/jira/browse/KUDU-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Serbin resolved KUDU-2986. - Fix Version/s: 1.12.0 Resolution: Fixed > Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables > -- > > Key: KUDU-2986 > URL: https://issues.apache.org/jira/browse/KUDU-2986 > Project: Kudu > Issue Type: Bug > Components: CLI, client, master, metrics >Affects Versions: 1.11.0 >Reporter: YifanZhang >Assignee: LiFu He >Priority: Major > Fix For: 1.12.0 > > > When we upgraded the cluster with pre-1.11.0 tables, we got inconsistent > values for the 'live_row_count' metric of these tables: > When visiting masterURL:port/metrics, we got 0 for old tables, and got a > positive integer for a old table with a newly added partition, which is the > count of rows in the newly added partition. > When getting table statistics via `kudu table statistics` CLI tool, we got 0 > for old tables and the old table with a new parition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3108) Tablet server crashes when handle scan request
[
https://issues.apache.org/jira/browse/KUDU-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YifanZhang updated KUDU-3108:
-
Description:
When we use KuduBackup Spark job to backup tables in a cluster with 20
tservers, 3 tservers crashed, coredump stacks are the same:
{code:java}
Program terminated with signal 11, Segmentation fault.Program terminated with
signal 11, Segmentation fault.
#0 kudu::Schema::Compare
(this=0x25b883680, lhs=..., rhs=...) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267
267 /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h: No such file or
directory.
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-gssapi-2.1.26-20.el7_2.x86_64
cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 cyrus-sasl-md5-2.1.26-20.el7_2.x86_64
cyrus-sasl-plain-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64
elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7_3.1.x86_64
keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64
libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64
libcom_err-1.42.9-9.el7.x86_64 libdb-5.3.21-19.el7.x86_64
libgcc-4.8.5-28.el7_5.1.x86_64 libselinux-2.5-6.el7.x86_64
ncurses-libs-5.9-13.20130511.el7.x86_64
nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64
openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64
systemd-libs-219-30.el7_3.8.x86_64 xz-libs-5.2.2-1.el7.x86_64
zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 kudu::Schema::Compare
(this=0x25b883680, lhs=..., rhs=...) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267
#1 0x01da51fb in kudu::MergeIterator::RefillHotHeap
(this=this@entry=0x78f6ec500) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:720
#2 0x01da622b in kudu::MergeIterator::AdvanceAndReheap
(this=this@entry=0x78f6ec500, state=0xd1661a000,
num_rows_to_advance=num_rows_to_advance@entry=1) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:690
#3 0x01da7927 in kudu::MergeIterator::MaterializeOneRow
(this=this@entry=0x78f6ec500, dst=dst@entry=0x7f0d5cc9ffc0,
dst_row_idx=dst_row_idx@entry=0x7f0d5cc9fbb0) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:894
#4 0x01da7de3 in kudu::MergeIterator::NextBlock (this=0x78f6ec500,
dst=0x7f0d5cc9ffc0) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:796
#5 0x00a9ff19 in kudu::tablet::Tablet::Iterator::NextBlock
(this=, dst=) at
/home/zhangyifan8/work/kudu-xm/src/kudu/tablet/tablet.cc:2499
#6 0x0095475c in
kudu::tserver::TabletServiceImpl::HandleContinueScanRequest
(this=this@entry=0x53b5a90, req=req@entry=0x7f0d5cca0720,
rpc_context=rpc_context@entry=0x5e512a460,
result_collector=result_collector@entry=0x7f0d5cca0a00,
has_more_results=has_more_results@entry=0x7f0d5cca0886,
error_code=error_code@entry=0x7f0d5cca0888) at
/home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2565
#7 0x00966564 in
kudu::tserver::TabletServiceImpl::HandleNewScanRequest
(this=this@entry=0x53b5a90, replica=0xf5c0189c0, req=req@entry=0x2a15c240,
rpc_context=rpc_context@entry=0x5e512a460,
result_collector=result_collector@entry=0x7f0d5cca0a00,
scanner_id=scanner_id@entry=0x7f0d5cca0940,
snap_timestamp=snap_timestamp@entry=0x7f0d5cca0950,
has_more_results=has_more_results@entry=0x7f0d5cca0886,
error_code=error_code@entry=0x7f0d5cca0888) at
/home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2476
#8 0x00967f4b in kudu::tserver::TabletServiceImpl::Scan
(this=0x53b5a90, req=0x2a15c240, resp=0x56f9be6c0, context=0x5e512a460) at
/home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:1674
#9 0x01d2e449 in operator() (__args#2=0x5e512a460,
__args#1=0x56f9be6c0, __args#0=, this=0x497ecdd8) at
/usr/include/c++/4.8.2/functional:2471
#10 kudu::rpc::GeneratedServiceIf::Handle (this=0x53b5a90, call=) at /home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_if.cc:139
#11 0x01d2eb49 in kudu::rpc::ServicePool::RunThread (this=0x2ab69560)
at /home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_pool.cc:225
#12 0x01e9e924 in operator() (this=0x90fb52e8) at
/home/zhangyifan8/work/kudu-xm/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771
#13 kudu::Thread::SuperviseThread (arg=0x90fb52c0) at
/home/zhangyifan8/work/kudu-xm/src/kudu/util/thread.cc:657
#14 0x7f103b20cdc5 in start_thread () from /lib64/libpthread.so.0
#15 0x7f103956673d in clone () from /lib64/libc.so.6
{code}
was:
When we use KuduBackup{{}} Spark job to backup tables in a cluster with 20
tservers, 3 tservers crashed, coredump stacks are the same:
{code:java}
[Thread debugging using libthread_db enabled][Thread debugging using
libthread_db enabled]Using host libthread_db library
"/lib64/libthread_db.so.1".Missing
[jira] [Created] (KUDU-3108) Tablet server crashes when handle scan request
YifanZhang created KUDU-3108:
Summary: Tablet server crashes when handle scan request
Key: KUDU-3108
URL: https://issues.apache.org/jira/browse/KUDU-3108
Project: Kudu
Issue Type: Bug
Affects Versions: 1.10.1
Reporter: YifanZhang
When we use KuduBackup{{}} Spark job to backup tables in a cluster with 20
tservers, 3 tservers crashed, coredump stacks are the same:
{code:java}
[Thread debugging using libthread_db enabled][Thread debugging using
libthread_db enabled]Using host libthread_db library
"/lib64/libthread_db.so.1".Missing separate debuginfo for
/home/work/app/kudu/zjyprc-hadoop/tablet_server/package/libstdc++.so.6Try: yum
--enablerepo='*debug*' install
/usr/lib/debug/.build-id/b3/d9128bcf6786292a339a477953167d0ddab5ba.debugCore
was generated by
`/home/work/app/kudu/zjyprc-hadoop/tablet_server/package/kudu_tablet_server
-tse'.Program terminated with signal 11, Segmentation fault.#0
kudu::Schema::Compare (this=0x25b883680,
lhs=..., rhs=...) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267267
/home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h: No such file or
directory.Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-gssapi-2.1.26-20.el7_2.x86_64
cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 cyrus-sasl-md5-2.1.26-20.el7_2.x86_64
cyrus-sasl-plain-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64
elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7_3.1.x86_64
keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64
libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64
libcom_err-1.42.9-9.el7.x86_64 libdb-5.3.21-19.el7.x86_64
libgcc-4.8.5-28.el7_5.1.x86_64 libselinux-2.5-6.el7.x86_64
ncurses-libs-5.9-13.20130511.el7.x86_64
nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64
openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64
systemd-libs-219-30.el7_3.8.x86_64 xz-libs-5.2.2-1.el7.x86_64
zlib-1.2.7-17.el7.x86_64(gdb) bt#0 kudu::Schema::Compare (this=0x25b883680, lhs=..., rhs=...) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267#1
0x01da51fb in kudu::MergeIterator::RefillHotHeap
(this=this@entry=0x78f6ec500) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:720#2
0x01da622b in kudu::MergeIterator::AdvanceAndReheap
(this=this@entry=0x78f6ec500, state=0xd1661a000,
num_rows_to_advance=num_rows_to_advance@entry=1) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:690#3
0x01da7927 in kudu::MergeIterator::MaterializeOneRow
(this=this@entry=0x78f6ec500, dst=dst@entry=0x7f0d5cc9ffc0,
dst_row_idx=dst_row_idx@entry=0x7f0d5cc9fbb0) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:894#4
0x01da7de3 in kudu::MergeIterator::NextBlock (this=0x78f6ec500,
dst=0x7f0d5cc9ffc0) at
/home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:796#5
0x00a9ff19 in kudu::tablet::Tablet::Iterator::NextBlock
(this=, dst=) at
/home/zhangyifan8/work/kudu-xm/src/kudu/tablet/tablet.cc:2499#6
0x0095475c in
kudu::tserver::TabletServiceImpl::HandleContinueScanRequest
(this=this@entry=0x53b5a90, req=req@entry=0x7f0d5cca0720,
rpc_context=rpc_context@entry=0x5e512a460,
result_collector=result_collector@entry=0x7f0d5cca0a00,
has_more_results=has_more_results@entry=0x7f0d5cca0886,
error_code=error_code@entry=0x7f0d5cca0888) at
/home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2565#7
0x00966564 in kudu::tserver::TabletServiceImpl::HandleNewScanRequest
(this=this@entry=0x53b5a90, replica=0xf5c0189c0, req=req@entry=0x2a15c240,
rpc_context=rpc_context@entry=0x5e512a460,
result_collector=result_collector@entry=0x7f0d5cca0a00,
scanner_id=scanner_id@entry=0x7f0d5cca0940,
snap_timestamp=snap_timestamp@entry=0x7f0d5cca0950,
has_more_results=has_more_results@entry=0x7f0d5cca0886,
error_code=error_code@entry=0x7f0d5cca0888) at
/home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2476#8
0x00967f4b in kudu::tserver::TabletServiceImpl::Scan (this=0x53b5a90,
req=0x2a15c240, resp=0x56f9be6c0, context=0x5e512a460) at
/home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:1674#9
0x01d2e449 in operator() (__args#2=0x5e512a460, __args#1=0x56f9be6c0,
__args#0=, this=0x497ecdd8) at
/usr/include/c++/4.8.2/functional:2471#10 kudu::rpc::GeneratedServiceIf::Handle
(this=0x53b5a90, call=) at
/home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_if.cc:139#11
0x01d2eb49 in kudu::rpc::ServicePool::RunThread (this=0x2ab69560) at
/home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_pool.cc:225#12
0x01e9e924 in operator() (this=0x90fb52e8) at
