[
https://issues.apache.org/jira/browse/ZOOKEEPER-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ZOOKEEPER-5073:
--------------------------------------
Labels: hang pull-request-available (was: hang)
> C client hangs indefinitely when pthread_create fails during initialization
> ---------------------------------------------------------------------------
>
> Key: ZOOKEEPER-5073
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5073
> Project: ZooKeeper
> Issue Type: Bug
> Affects Versions: 3.5.9, 3.7.0, 3.10.0
> Environment: OS: Debian10
> Architecture: x86_64
> Compiler: GCC 8.3.0
> ZooKeeper version: 3.10.0
> Client mode: multi-threaded
> Reporter: Wenhai Wang
> Priority: Major
> Labels: hang, pull-request-available
> Attachments: README.md, build.sh, error.log, pthread_create_fail.c,
> run_fault_test.sh, zk_init_probe.c
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The ZooKeeper multi-threaded C client may hang indefinitely when either
> worker thread fails to start during client initialization.
> h3. Affected code
> The issue is located in:
> {{zookeeper-client/zookeeper-client-c/src/mt_adaptor.c}}
> The {{start_threads()}} function initializes the startup barrier with:
> {code:c}
> adaptor->threadsToWait = 2;
> {code}
> It then creates the IO thread and completion thread:
> {code:c}
> rc = pthread_create(&adaptor->io, 0, do_io, zh);
> assert("pthread_create() failed for the IO thread" && !rc);
> rc = pthread_create(&adaptor->completion, 0, do_completion, zh);
> assert("pthread_create() failed for the completion thread" && !rc);
> wait_for_others(zh);
> {code}
> h3. Problem
> When assertions are disabled, for example in a release build compiled with
> {{NDEBUG}}, the return values of {{pthread_create()}} are effectively ignored.
> If either {{pthread_create()}} call fails, the corresponding worker thread is
> not created. However, {{threadsToWait}} remains initialized to {{2}}.
> Each successfully created worker thread calls {{notify_thread_ready()}} and
> decrements {{threadsToWait}} once. The initialization thread calls
> {{wait_for_others()}} and waits until {{threadsToWait}} becomes zero.
> For example, if the IO thread is created successfully but the completion
> thread fails to start:
> * {{threadsToWait}} is initialized to {{2}}.
> * The IO thread decrements {{threadsToWait}} to {{1}}.
> * The completion thread does not exist and therefore cannot perform the
> second decrement.
> * The IO thread waits in {{notify_thread_ready()}}.
> * The initialization thread waits in {{wait_for_others()}}.
> * {{threadsToWait}} can never become zero.
> As a result, {{zookeeper_init()}} does not return, and client initialization
> hangs indefinitely.
> When assertions are enabled, the process aborts instead of returning an
> initialization error. Therefore, the runtime resource failure is not handled
> gracefully in either build mode.
> h3. Expected behavior
> The C client should:
> * Detect a non-zero return value from {{pthread_create()}}.
> * Stop and join any worker thread that was already created.
> * Clean up initialized synchronization and adaptor resources.
> * Propagate an initialization error to the caller.
> h3. Actual behavior
> * When assertions are disabled, the client waits indefinitely on the startup
> condition variable.
> * When assertions are enabled, the process terminates through {{assert()}}.
> h3. Reproduction
> The issue can be reproduced deterministically as follows:
> * Build the ZooKeeper C client with {{NDEBUG}} enabled.
> * Inject an {{EAGAIN}} failure into either the first or second
> {{pthread_create()}} call.
> * Call {{zookeeper_init()}}.
> * Observe that {{zookeeper_init()}} does not return.
> A typical blocked thread state is:
> {noformat}
> Initialization thread:
> pthread_cond_wait
> wait_for_others
> start_threads
> adaptor_init
> zookeeper_init
> Successfully created worker thread:
> pthread_cond_wait
> notify_thread_ready
> do_io or do_completion
> {noformat}
> h3. Root cause
> {{start_threads()}} assumes that both {{pthread_create()}} calls always
> succeed and uses assertions instead of normal runtime error handling.
> When thread creation fails, the startup barrier count is not adjusted.
> Therefore, {{threadsToWait}} remains greater than zero and the condition
> required to release the waiting threads can never be satisfied.
> In addition, the failure cannot be propagated to the caller because
> {{start_threads()}} currently returns {{void}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
