Wenhai Wang created ZOOKEEPER-5073:
--------------------------------------
Summary: C client hangs indefinitely when pthread_create fails
during initialization
Key: ZOOKEEPER-5073
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5073
Project: ZooKeeper
Issue Type: Bug
Affects Versions: 3.7.0, 3.5.9, 3.10.0
Environment: OS: Debian10
Architecture: x86_64
Compiler: GCC 8.3.0
ZooKeeper version: 3.10.0
Client mode: multi-threaded
Reporter: Wenhai Wang
Attachments: README.md, build.sh, error.log, pthread_create_fail.c,
run_fault_test.sh, zk_init_probe.c
The ZooKeeper multi-threaded C client may hang indefinitely when either of the
worker threads fails to start during client initialization.
h3. Affected code
The issue is located in:
{{zookeeper-client/zookeeper-client-c/src/mt_adaptor.c}}
The {{start_threads()}} function initializes the startup barrier with:
{{adaptor->threadsToWait = 2;}}
It then creates the IO thread and completion thread:
{{rc = pthread_create(&adaptor->io, 0, do_io, zh);
assert("pthread_create() failed for the IO thread" && !rc);
rc = pthread_create(&adaptor->completion, 0, do_completion, zh);
assert("pthread_create() failed for the completion thread" && !rc);
wait_for_others(zh);}}
h3. Problem
When assertions are disabled, for example in a release build compiled with
{{{}NDEBUG{}}}, the return values of {{pthread_create()}} are effectively
ignored.
If either {{pthread_create()}} call fails, the corresponding worker thread is
not created. However, {{threadsToWait}} remains initialized to {{{}2{}}}.
Each successfully created worker thread calls {{notify_thread_ready()}} and
decrements {{threadsToWait}} once. The initialization thread calls
{{wait_for_others()}} and waits until {{threadsToWait}} becomes zero.
For example, if the IO thread is created successfully but the completion thread
fails to start:
# {{threadsToWait}} is initialized to {{{}2{}}}.
# The IO thread decrements it to {{{}1{}}}.
# The completion thread does not exist and therefore cannot perform the second
decrement.
# The IO thread waits in {{{}notify_thread_ready(){}}}.
# The initialization thread waits in {{{}wait_for_others(){}}}.
# {{threadsToWait}} can never become zero.
As a result, {{zookeeper_init()}} does not return and the client initialization
hangs indefinitely.
When assertions are enabled, the process aborts instead of returning an
initialization error, which also means the runtime resource failure is not
handled gracefully.
h3. Expected behavior
The C client should detect a non-zero return value from
{{{}pthread_create(){}}}, stop and join any worker thread that was already
created, clean up the initialized resources, and return an initialization error
to the caller.
h3. Actual behavior
When assertions are disabled, the client waits indefinitely on the startup
condition variable.
When assertions are enabled, the process terminates through {{{}assert(){}}}.
h3. Reproduction
The issue can be reproduced deterministically by:
# Building the ZooKeeper C client with {{NDEBUG}} enabled.
# Injecting an {{EAGAIN}} failure into either the first or second
{{pthread_create()}} call.
# Calling {{{}zookeeper_init(){}}}.
# Observing that {{zookeeper_init()}} does not return.
A typical blocked thread state is:
{{Initialization thread:
pthread_cond_wait
wait_for_others
start_threads
adaptor_init
zookeeper_init
Successfully created worker thread:
pthread_cond_wait
notify_thread_ready
do_io or do_completion}}
h3. Root cause
{{start_threads()}} assumes that both {{pthread_create()}} calls always succeed
and uses assertions instead of normal runtime error handling.
The startup barrier count is not adjusted when thread creation fails, and the
failure cannot be propagated because {{start_threads()}} currently returns
{{{}void{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
