Hello Marton Greber, Attila Bukor, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23222
to look at the new patch set (#6).
Change subject: WIP: KUDU-3635 call OPENSSL_cleanup() explicitly
......................................................................
WIP: KUDU-3635 call OPENSSL_cleanup() explicitly
WIP:
* the issue is fixed and don't manifest itself in RELEASE builds,
but there is still something that makes kudu CLI crash on exit;
the stack trace is doesn't involve OpenSSL symbols, though
With this changelist, the global state of the OpenSSL library is now
explicitly cleaned up when shutting down a process. The rationale
for this is outlined in [1].
This is applicable to the OpenSSL library of versions 1.1.1 and newer,
and it covers all the contemporary Linux distributions at the time
of writing. It works even at Ubuntu 18.04.1 LTS.
As for testing, I verified that the issue is gone after seeing its
manifestation with the frequency of about 1 in 10 runs of the kudu CLI
tool with RELEASE bits on RHEL8.8 x86_64 and RHEL9.2 x86_64 without
the patch. I ran the following for the Kudu code built with and without
this changelist:
./bin/kudu-tool-test --gtest_repeat=100
Without the patch, I'd see many core files left by the kudu CLI binary
during every run, where many of of the core files had stack traces
similar to the one described in the JIRA item. With this patch,
not a single crash has been observed and no core files have been
generated after more than twenty of the hundred repetition test runs
with Kudu bits built in RELEASE mode.
While running the bits without the patch, I'd also see core files with
stack traces attributable to KUDU-2439. With this patch, the issue
hasn't manifested itself in many thousand of the kudu CLI runs,
so I think we can declare that it's been addressed up to some extent.
It's still a stop-gap, but it's better than nothing.
[1]
https://developers.redhat.com/articles/2022/10/31/best-practices-application-shutdown-openssl
Change-Id: Ib08310d66a7eabb1996bde901f39f36f54bff483
---
M src/kudu/master/master_main.cc
M src/kudu/rpc/messenger.cc
M src/kudu/rpc/messenger.h
M src/kudu/security/ca/cert_management.cc
M src/kudu/security/crypto.cc
M src/kudu/tools/tool_main.cc
M src/kudu/tserver/tablet_server_main.cc
A src/kudu/util/before_and_after_main.h
A src/kudu/util/entry_exit_wrapper.h
M src/kudu/util/openssl_util.cc
M src/kudu/util/openssl_util.h
M src/kudu/util/process_memory.cc
M src/kudu/util/test_main.cc
M src/kudu/util/thread.cc
M src/kudu/util/thread.h
M thirdparty/build-definitions.sh
16 files changed, 426 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/22/23222/6
--
To view, visit http://gerrit.cloudera.org:8080/23222
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib08310d66a7eabb1996bde901f39f36f54bff483
Gerrit-Change-Number: 23222
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Attila Bukor <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
