[
https://issues.apache.org/jira/browse/KUDU-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012922#comment-18012922
]
ASF subversion and git services commented on KUDU-3635:
-------------------------------------------------------
Commit 72fdc7f8c89ae03e195d9f5d11186ce9604f39af in kudu's branch
refs/heads/branch-1.18.x from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=72fdc7f8c ]
[security] make sure OpenSSL initialized as expected
Per OpenSSL's documentation [1][2], numerous internal OpenSSL functions
call OPENSSL_init_{crypto,ssl}(). Therefore, in order to perform
nondefault initialisation which Kudu does since addressing KUDU-3635,
OPENSSL_init_{crypto,ssl}() MUST be called by application code prior
to any other OpenSSL function calls. And this is also applicable to
ERR_peek_error() which calls OPENSSL_init_crypt with the
OPENSSL_INIT_BASE_ONLY option, at least in OpenSSL 1.1.1. Meanwhile,
ERR_peek_error() is called in the constructor and the destructor
of the underlying object in the SCOPED_OPENSSL_NO_PENDING_ERRORS macro.
Even if Kudu adds a non-default OPENSSL_INIT_NO_ATEXIT flag only when
initializing the OpenSSL library early in the process start-up phase,
it makes sense to address this to avoid unexpected surprises if any
extra flag is added. Also, placing SCOPED_OPENSSL_NO_PENDING_ERRORS
before InitializeOpenSSL() was an apparent typo, so it needs to be fixed
anyway.
This is a follow-up 5f1ca4f3948a61b22946255e4ada895c77bc6adf.
[1] https://docs.openssl.org/1.1.1/man3/OPENSSL_init_crypto/
[2] https://docs.openssl.org/1.1.1/man3/OPENSSL_init_ssl/
Change-Id: If56b5d23e1f974aa2ab5677458b125cb011a36dc
Reviewed-on: http://gerrit.cloudera.org:8080/23268
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Abhishek Chennaka <[email protected]>
(cherry picked from commit b92f16d1c86a753c597b46c7575bfa6a1479726a)
Reviewed-on: http://gerrit.cloudera.org:8080/23272
Reviewed-by: Alexey Serbin <[email protected]>
> kudu CLI tool sometimes crashes on exit with SIGSEGV in OPENSSL_cleanup
> -----------------------------------------------------------------------
>
> Key: KUDU-3635
> URL: https://issues.apache.org/jira/browse/KUDU-3635
> Project: Kudu
> Issue Type: Bug
> Components: CLI
> Affects Versions: 1.17.0, 1.18.0, 1.17.1
> Reporter: Alexey Serbin
> Assignee: Alexey Serbin
> Priority: Major
>
> The kudu CLI tools sometimes crash on exit with SIGSEGV.
> I haven't had a chance looking at this closely, but it seems the problem is
> related to the order of cleanup of different libraries and overall unexpected
> state of the runtime when the implicitly installed cleanup handler for the
> OpenSSL library is being called.
> Below is a snippet from the output of the
> {{RebalanceIgnoredTserversTest.Basic}} test scenario. That was generated by
> Kudu bits built in RELEASE configuration on Ubuntu 18.04.6 LTS machine and
> run via dist-test on Ubuntu 18.04.6 LTS as well.
> BTW, we have been suppressing TSAN warnings in the OpenSSL cleanup paths for
> a long time due to well-known issue in the OpenSSL library (see [this TSAN
> suppression|https://github.com/apache/kudu/blob/2b9a2012f6d7b59931119dfad03e8d40e3031a0e/src/kudu/util/sanitizer_options.cc#L177-L184]),
> so there might be some other issues around that we haven't paid attention
> for a long time.
> Probably, it's time to follow [best practices for at-exit cleanup of
> applications using
> OpenSSL|https://developers.redhat.com/articles/2022/10/31/best-practices-application-shutdown-openssl#].
> In essence, that works at least with v1.1.1 and newer versions of the
> OpenSSL library: use the {{OPENSSL_INIT_NO_ATEXIT}} option for
> {{OPENSSL_init_ssl()}} at initialization and then explicitly call
> {{OPENSSL_cleanup()}} upon exit/shutdown.
> {noformat}
> *** SIGSEGV (@0x10000562bd5) received by PID 1447 (TID 0x7fb1cda47480) from
> PID 5647317; stack trace: ***
> @ 0x7fb1d6307980 (unknown) at ??:0
>
> @ 0x7fb1d5a37873 tcmalloc::ThreadCache::ReleaseToCentralCache() at
> ??:0
> @ 0x7fb1d5a37be7 tcmalloc::ThreadCache::Scavenge() at ??:0
>
> @ 0x7fb1d3bce271 OPENSSL_LH_free at ??:0
>
> @ 0x7fb1d3bacbfd (unknown) at ??:0
>
> @ 0x7fb1d3bcbe10 OPENSSL_cleanup at ??:0
>
> @ 0x7fb1d434e161 (unknown) at ??:0
>
> @ 0x7fb1d434e25a exit at ??:0
>
> @ 0x7fb1d432cbfe __libc_start_main at ??:0
>
> @ 0x562bc9f8300a _start at ??:0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
