[
https://issues.apache.org/jira/browse/ZOOKEEPER-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenhai Wang updated ZOOKEEPER-5073:
-----------------------------------
Description:
The ZooKeeper multi-threaded C client may hang indefinitely when either worker
thread fails to start during client initialization.
h3. Affected code
The issue is located in:
{{zookeeper-client/zookeeper-client-c/src/mt_adaptor.c}}
The {{start_threads()}} function initializes the startup barrier with:
{code:c}
adaptor->threadsToWait = 2;
{code}
It then creates the IO thread and completion thread:
{code:c}
rc = pthread_create(&adaptor->io, 0, do_io, zh);
assert("pthread_create() failed for the IO thread" && !rc);
rc = pthread_create(&adaptor->completion, 0, do_completion, zh);
assert("pthread_create() failed for the completion thread" && !rc);
wait_for_others(zh);
{code}
h3. Problem
When assertions are disabled, for example in a release build compiled with
{{NDEBUG}}, the return values of {{pthread_create()}} are effectively ignored.
If either {{pthread_create()}} call fails, the corresponding worker thread is
not created. However, {{threadsToWait}} remains initialized to {{2}}.
Each successfully created worker thread calls {{notify_thread_ready()}} and
decrements {{threadsToWait}} once. The initialization thread calls
{{wait_for_others()}} and waits until {{threadsToWait}} becomes zero.
For example, if the IO thread is created successfully but the completion thread
fails to start:
# {{threadsToWait}} is initialized to {{2}}.
# The IO thread decrements it to {{1}}.
# The completion thread does not exist and therefore cannot perform the second
decrement.
# The IO thread waits in {{notify_thread_ready()}}.
# The initialization thread waits in {{wait_for_others()}}.
# {{threadsToWait}} can never become zero.
As a result, {{zookeeper_init()}} does not return, and the client
initialization hangs indefinitely.
When assertions are enabled, the process aborts instead of returning an
initialization error. Therefore, the runtime resource failure is not handled
gracefully in either build mode.
h3. Expected behavior
The C client should detect a non-zero return value from {{pthread_create()}},
stop and join any worker thread that was already created, clean up the
initialized resources, and return an initialization error to the caller.
h3. Actual behavior
When assertions are disabled, the client waits indefinitely on the startup
condition variable.
When assertions are enabled, the process terminates through {{assert()}}.
h3. Reproduction
The issue can be reproduced deterministically by:
# Building the ZooKeeper C client with {{NDEBUG}} enabled.
# Injecting an {{EAGAIN}} failure into either the first or second
{{pthread_create()}} call.
# Calling {{zookeeper_init()}}.
# Observing that {{zookeeper_init()}} does not return.
A typical blocked thread state is:
{noformat}
Initialization thread:
pthread_cond_wait
wait_for_others
start_threads
adaptor_init
zookeeper_init
Successfully created worker thread:
pthread_cond_wait
notify_thread_ready
do_io or do_completion
{noformat}
h3. Root cause
{{start_threads()}} assumes that both {{pthread_create()}} calls always succeed
and uses assertions instead of normal runtime error handling.
The startup barrier count is not adjusted when thread creation fails. In
addition, the failure cannot be propagated because {{start_threads()}}
currently returns {{void}}.
was:
The ZooKeeper multi-threaded C client may hang indefinitely when either of the
worker threads fails to start during client initialization.
h3. Affected code
The issue is located in:
{{zookeeper-client/zookeeper-client-c/src/mt_adaptor.c}}
The {{start_threads()}} function initializes the startup barrier with:
{{adaptor->threadsToWait = 2;}}
It then creates the IO thread and completion thread:
{{rc = pthread_create(&adaptor->io, 0, do_io, zh);
assert("pthread_create() failed for the IO thread" && !rc);
rc = pthread_create(&adaptor->completion, 0, do_completion, zh);
assert("pthread_create() failed for the completion thread" && !rc);
wait_for_others(zh);}}
h3. Problem
When assertions are disabled, for example in a release build compiled with
{{{}NDEBUG{}}}, the return values of {{pthread_create()}} are effectively
ignored.
If either {{pthread_create()}} call fails, the corresponding worker thread is
not created. However, {{threadsToWait}} remains initialized to {{{}2{}}}.
Each successfully created worker thread calls {{notify_thread_ready()}} and
decrements {{threadsToWait}} once. The initialization thread calls
{{wait_for_others()}} and waits until {{threadsToWait}} becomes zero.
For example, if the IO thread is created successfully but the completion thread
fails to start:
# {{threadsToWait}} is initialized to {{{}2{}}}.
# The IO thread decrements it to {{{}1{}}}.
# The completion thread does not exist and therefore cannot perform the second
decrement.
# The IO thread waits in {{{}notify_thread_ready(){}}}.
# The initialization thread waits in {{{}wait_for_others(){}}}.
# {{threadsToWait}} can never become zero.
As a result, {{zookeeper_init()}} does not return and the client initialization
hangs indefinitely.
When assertions are enabled, the process aborts instead of returning an
initialization error, which also means the runtime resource failure is not
handled gracefully.
h3. Expected behavior
The C client should detect a non-zero return value from
{{{}pthread_create(){}}}, stop and join any worker thread that was already
created, clean up the initialized resources, and return an initialization error
to the caller.
h3. Actual behavior
When assertions are disabled, the client waits indefinitely on the startup
condition variable.
When assertions are enabled, the process terminates through {{{}assert(){}}}.
h3. Reproduction
The issue can be reproduced deterministically by:
# Building the ZooKeeper C client with {{NDEBUG}} enabled.
# Injecting an {{EAGAIN}} failure into either the first or second
{{pthread_create()}} call.
# Calling {{{}zookeeper_init(){}}}.
# Observing that {{zookeeper_init()}} does not return.
A typical blocked thread state is:
{{Initialization thread:
pthread_cond_wait
wait_for_others
start_threads
adaptor_init
zookeeper_init
Successfully created worker thread:
pthread_cond_wait
notify_thread_ready
do_io or do_completion}}
h3. Root cause
{{start_threads()}} assumes that both {{pthread_create()}} calls always succeed
and uses assertions instead of normal runtime error handling.
The startup barrier count is not adjusted when thread creation fails, and the
failure cannot be propagated because {{start_threads()}} currently returns
{{{}void{}}}.
> C client hangs indefinitely when pthread_create fails during initialization
> ---------------------------------------------------------------------------
>
> Key: ZOOKEEPER-5073
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5073
> Project: ZooKeeper
> Issue Type: Bug
> Affects Versions: 3.5.9, 3.7.0, 3.10.0
> Environment: OS: Debian10
> Architecture: x86_64
> Compiler: GCC 8.3.0
> ZooKeeper version: 3.10.0
> Client mode: multi-threaded
> Reporter: Wenhai Wang
> Priority: Major
> Labels: hang
> Attachments: README.md, build.sh, error.log, pthread_create_fail.c,
> run_fault_test.sh, zk_init_probe.c
>
>
> The ZooKeeper multi-threaded C client may hang indefinitely when either
> worker thread fails to start during client initialization.
> h3. Affected code
> The issue is located in:
> {{zookeeper-client/zookeeper-client-c/src/mt_adaptor.c}}
> The {{start_threads()}} function initializes the startup barrier with:
> {code:c}
> adaptor->threadsToWait = 2;
> {code}
> It then creates the IO thread and completion thread:
> {code:c}
> rc = pthread_create(&adaptor->io, 0, do_io, zh);
> assert("pthread_create() failed for the IO thread" && !rc);
> rc = pthread_create(&adaptor->completion, 0, do_completion, zh);
> assert("pthread_create() failed for the completion thread" && !rc);
> wait_for_others(zh);
> {code}
> h3. Problem
> When assertions are disabled, for example in a release build compiled with
> {{NDEBUG}}, the return values of {{pthread_create()}} are effectively ignored.
> If either {{pthread_create()}} call fails, the corresponding worker thread is
> not created. However, {{threadsToWait}} remains initialized to {{2}}.
> Each successfully created worker thread calls {{notify_thread_ready()}} and
> decrements {{threadsToWait}} once. The initialization thread calls
> {{wait_for_others()}} and waits until {{threadsToWait}} becomes zero.
> For example, if the IO thread is created successfully but the completion
> thread fails to start:
> # {{threadsToWait}} is initialized to {{2}}.
> # The IO thread decrements it to {{1}}.
> # The completion thread does not exist and therefore cannot perform the
> second decrement.
> # The IO thread waits in {{notify_thread_ready()}}.
> # The initialization thread waits in {{wait_for_others()}}.
> # {{threadsToWait}} can never become zero.
> As a result, {{zookeeper_init()}} does not return, and the client
> initialization hangs indefinitely.
> When assertions are enabled, the process aborts instead of returning an
> initialization error. Therefore, the runtime resource failure is not handled
> gracefully in either build mode.
> h3. Expected behavior
> The C client should detect a non-zero return value from {{pthread_create()}},
> stop and join any worker thread that was already created, clean up the
> initialized resources, and return an initialization error to the caller.
> h3. Actual behavior
> When assertions are disabled, the client waits indefinitely on the startup
> condition variable.
> When assertions are enabled, the process terminates through {{assert()}}.
> h3. Reproduction
> The issue can be reproduced deterministically by:
> # Building the ZooKeeper C client with {{NDEBUG}} enabled.
> # Injecting an {{EAGAIN}} failure into either the first or second
> {{pthread_create()}} call.
> # Calling {{zookeeper_init()}}.
> # Observing that {{zookeeper_init()}} does not return.
> A typical blocked thread state is:
> {noformat}
> Initialization thread:
> pthread_cond_wait
> wait_for_others
> start_threads
> adaptor_init
> zookeeper_init
> Successfully created worker thread:
> pthread_cond_wait
> notify_thread_ready
> do_io or do_completion
> {noformat}
> h3. Root cause
> {{start_threads()}} assumes that both {{pthread_create()}} calls always
> succeed and uses assertions instead of normal runtime error handling.
> The startup barrier count is not adjusted when thread creation fails. In
> addition, the failure cannot be propagated because {{start_threads()}}
> currently returns {{void}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
