[
https://issues.apache.org/jira/browse/KUDU-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328464#comment-15328464
]
Adar Dembo commented on KUDU-1472:
----------------------------------
[~bruceSz] to build with ASAN, see the AddressSanitizer section of README.adoc:
{noformat}
=== Running tests with the clang AddressSanitizer enabled
AddressSanitizer is a nice clang feature which can detect many types of memory
errors. The Jenkins setup for kudu runs these tests automatically on a regular
basis, but if you make large changes it can be a good idea to run it locally
before pushing. To do so, you'll need to build using `clang`:
[source,bash]
----
$ mkdir -p build/asan
$ cd build/asan
$ CC=../../thirdparty/clang-toolchain/bin/clang \
CXX=../../thirdparty/clang-toolchain/bin/clang++ \
cmake -DKUDU_USE_ASAN=1 ../..
$ make -j8
{noformat}
Once you do this, you'll see a kudu-tserver binary in the bin/ subdirectory.
Run that one instead of the usual one. It'll be a lot slower, but if you're
able to reproduce the crash, the ASAN instrumentation will hopefully yield
useful information in the tserver log.
> kudu-tserver crash unexpected
> -----------------------------
>
> Key: KUDU-1472
> URL: https://issues.apache.org/jira/browse/KUDU-1472
> Project: Kudu
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.9.0
> Reporter: zhangsong
> Priority: Critical
>
> kudu-tserver will crash under some case, in jd.com 200-node env, it occurring
> frequently.
> some crash info from core file:
> (gdb) bt
> #0 0x0000000000a2489f in kudu::tablet::RowSetDataPB::SharedDtor
> (this=0x58fb5b180)
> at /export/ldb/kudu-master/build/release/src/kudu/tablet/metadata.pb.cc:815
> #1 kudu::tablet::RowSetDataPB::~RowSetDataPB (this=0x58fb5b180,
> __in_chrg=<optimized out>)
> at /export/ldb/kudu-master/build/release/src/kudu/tablet/metadata.pb.cc:809
> #2 kudu::tablet::RowSetDataPB::~RowSetDataPB (this=0x58fb5b180,
> __in_chrg=<optimized out>)
> at /export/ldb/kudu-master/build/release/src/kudu/tablet/metadata.pb.cc:810
> #3
> google::protobuf::internal::GenericTypeHandler<kudu::tablet::RowSetDataPB>::Delete
> (value=0x58fb5b180)
> at
> /export/ldb/kudu-master/thirdparty/installed-deps/include/google/protobuf/repeated_field.h:363
> #4
> google::protobuf::internal::RepeatedPtrFieldBase::Destroy<google::protobuf::RepeatedPtrField<kudu::tablet::RowSetDataPB>::TypeHandler>
> (
> this=<optimized out>, this=<optimized out>) at
> /export/ldb/kudu-master/thirdparty/installed-deps/include/google/protobuf/repeated_field.h:869
> Backtrace stopped: Cannot access memory at address 0x7fc1f230fd08
> after crash , kudu-tserver will not be restarted successfully, due to some pb
> validation check failed, for example:
> check failed: _s.ok() Bad status: IO error: Could not init Tablet Manager:
> Failed to open tablet metadata for tablet: 260359a41a134c1f91631e9094847bcf:
> Failed to load tablet metadata for tablet id
> 260359a41a134c1f91631e9094847bcf: Could not load tablet metadata from
> /export/servers/kudu/tserver_data_7052/tablet-meta/260359a41a134c1f91631e9094847bcf:
> Unable to parse PB from path:
> /export/servers/kudu/tserver_data_7052/tablet-meta/260359a41a134c1f91631e9094847bcf
> kudu version is 0.9.0-snapshot, last commit id :
> be10f8514c48950b64c7d59bbce848f3792ec52d
> workload is: several write tasks keeps inserting into kudu table, some task
> using java api, while others using impala.
> kudu-table will be scanned while whose tasks are running.
> almost everyday there will be a crash case. same phenomenon as described
> above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
