[
https://issues.apache.org/jira/browse/ZOOKEEPER-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenhai Wang updated ZOOKEEPER-5073:
-----------------------------------
Tags: (was: hang, c client)
> C client hangs indefinitely when pthread_create fails during initialization
> ---------------------------------------------------------------------------
>
> Key: ZOOKEEPER-5073
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5073
> Project: ZooKeeper
> Issue Type: Bug
> Affects Versions: 3.5.9, 3.7.0, 3.10.0
> Environment: OS: Debian10
> Architecture: x86_64
> Compiler: GCC 8.3.0
> ZooKeeper version: 3.10.0
> Client mode: multi-threaded
> Reporter: Wenhai Wang
> Priority: Major
> Attachments: README.md, build.sh, error.log, pthread_create_fail.c,
> run_fault_test.sh, zk_init_probe.c
>
>
> The ZooKeeper multi-threaded C client may hang indefinitely when either of
> the worker threads fails to start during client initialization.
> h3. Affected code
> The issue is located in:
> {{zookeeper-client/zookeeper-client-c/src/mt_adaptor.c}}
> The {{start_threads()}} function initializes the startup barrier with:
> {{adaptor->threadsToWait = 2;}}
> It then creates the IO thread and completion thread:
>
> {{rc = pthread_create(&adaptor->io, 0, do_io, zh);
> assert("pthread_create() failed for the IO thread" && !rc);
> rc = pthread_create(&adaptor->completion, 0, do_completion, zh);
> assert("pthread_create() failed for the completion thread" && !rc);
> wait_for_others(zh);}}
> h3. Problem
> When assertions are disabled, for example in a release build compiled with
> {{{}NDEBUG{}}}, the return values of {{pthread_create()}} are effectively
> ignored.
> If either {{pthread_create()}} call fails, the corresponding worker thread is
> not created. However, {{threadsToWait}} remains initialized to {{{}2{}}}.
> Each successfully created worker thread calls {{notify_thread_ready()}} and
> decrements {{threadsToWait}} once. The initialization thread calls
> {{wait_for_others()}} and waits until {{threadsToWait}} becomes zero.
> For example, if the IO thread is created successfully but the completion
> thread fails to start:
> # {{threadsToWait}} is initialized to {{{}2{}}}.
> # The IO thread decrements it to {{{}1{}}}.
> # The completion thread does not exist and therefore cannot perform the
> second decrement.
> # The IO thread waits in {{{}notify_thread_ready(){}}}.
> # The initialization thread waits in {{{}wait_for_others(){}}}.
> # {{threadsToWait}} can never become zero.
> As a result, {{zookeeper_init()}} does not return and the client
> initialization hangs indefinitely.
> When assertions are enabled, the process aborts instead of returning an
> initialization error, which also means the runtime resource failure is not
> handled gracefully.
> h3. Expected behavior
> The C client should detect a non-zero return value from
> {{{}pthread_create(){}}}, stop and join any worker thread that was already
> created, clean up the initialized resources, and return an initialization
> error to the caller.
> h3. Actual behavior
> When assertions are disabled, the client waits indefinitely on the startup
> condition variable.
> When assertions are enabled, the process terminates through {{{}assert(){}}}.
> h3. Reproduction
> The issue can be reproduced deterministically by:
> # Building the ZooKeeper C client with {{NDEBUG}} enabled.
> # Injecting an {{EAGAIN}} failure into either the first or second
> {{pthread_create()}} call.
> # Calling {{{}zookeeper_init(){}}}.
> # Observing that {{zookeeper_init()}} does not return.
> A typical blocked thread state is:
>
> {{Initialization thread:
> pthread_cond_wait
> wait_for_others
> start_threads
> adaptor_init
> zookeeper_init
> Successfully created worker thread:
> pthread_cond_wait
> notify_thread_ready
> do_io or do_completion}}
> h3. Root cause
> {{start_threads()}} assumes that both {{pthread_create()}} calls always
> succeed and uses assertions instead of normal runtime error handling.
> The startup barrier count is not adjusted when thread creation fails, and the
> failure cannot be propagated because {{start_threads()}} currently returns
> {{{}void{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
