saravanakumar1987 opened a new pull request, #30066: URL: https://github.com/apache/camel/pull/30066
# Description Fixes [CAMEL-26971](https://issues.apache.org/jira/browse/CAMEL-26971). ## What was wrong Two defects in how `AlibabaOtsEndpoint` creates its `SyncClient`. They were listed in the description of apache/camel#29991 (CAMEL-23832) and found by reading the code. 1. `AlibabaOtsEndpoint.initClient`, which `AlibabaOtsProducer.process` calls for each exchange, read and wrote the `otsClient` field without a lock, and the field was not volatile. Messages processed at the same time before a client existed could each create a `SyncClient`. The endpoint kept only the last one, and `doStop` shut down only that one, so the others were never shut down. 2. `AlibabaOtsUtils.createClient` called `new SyncClient(endpoint, accessKey, secretKey, instanceName)`. In Tablestore SDK 5.17.14, this constructor ends in `this(endpoint, CredentialsProviderFactory.newDefaultCredentialProvider(...), instanceName, config, new ResourceManager(config, callbackExecutor))` (`InternalClient.java:124`), so the `ResourceManager` exists before the constructor that it calls checks anything. Creating it creates `ClientResources`, whose `AsyncServiceClient` starts the reactor thread of the HTTP client, which starts the I/O dispatcher threads (one per CPU by default), and an `IdleConnectionEvictor` thread. Nothing shuts it down when that constructor then rejects the endpoint or the instance name: - in its first checks (`InternalClient.java:130-140`): an endpoint that does not start with `http://` or `https://`, or an instance name with a multibyte character or a character other than letters, digits, `_` and `-`; - later, when `new LauncherFactory(...)` creates the `OTSUri` objects (`InternalClient.java:167`): an endpoint that does not match `Constants.ENDPOINT_REGEX`, for example one with a path. The field stays null, so the creation was tried again for each message, and each message left these threads behind: 18 threads per message on the 16-CPU machine of the tests below. The reactor and I/O dispatcher threads are not daemon threads, so they can also keep a JVM from exiting. ## The change ### One client per endpoint - `otsClient` is volatile. `initClient` reads it once; when it is null, it takes the `lock` that the endpoint inherits from `BaseService`, reads the field again, and creates the client only if it is still null. This is the lazy creation of `MilvusEndpoint.getClient`, `QdrantEndpoint.getClient` and `WeaviateVectorDbEndpoint.getClient`, which create their client when their producer starts and close it in `doStop`. Some other endpoints use a private lock for the same pattern; the inherited one also makes the creation and `doStop` exclude each other. Unlike those three, `initClient` returns the value it read or created, not the field read again after the lock is released. Once the client exists, a message does not take the lock. The client is still created when the producer processes its first message, as the description of the `otsClient` option says. - `setOtsClient` and `doStop` take the same lock, so the client and the `autowiredOtsClient` flag change together, and `doStop` cannot run in the middle of a creation. `BaseService.stop()` already holds this lock when it calls `doStop`, and the lock is reentrant. - The creation is in a new package-private method `createClient()`, which calls `AlibabaOtsUtils.createClient(this)`. The concurrency test overrides it to hold the creation. ### No SDK threads left behind by a failed creation - `AlibabaOtsUtils.createClient` creates the `CredentialsProvider` first (`CredentialsProviderFactory.newDefaultCredentialProvider`, which rejects an empty AccessKey ID or secret and an AccessKey ID of an invalid format; as before, these are rejected before any thread is started), then one default `ClientConfiguration`, then `new ResourceManager(configuration)`, and passes the three to the public constructor `SyncClient(endpoint, CredentialsProvider, instanceName, ClientConfiguration, ResourceManager)`. When this constructor throws, it shuts the `ResourceManager` down and rethrows the exception (a failure of the shutdown is added to it as suppressed). - A `ResourceManager` created with its `ClientConfiguration` constructor owns its resources (`owned = true`), so `SyncClient.shutdown()` still releases them through `InternalClient.shutdown()` and `ResourceManager.shutdown()` when the Camel endpoint stops. - For valid options, the client is the same as before: a default `ClientConfiguration`, no STS token, and the callback executor that the SDK creates. Before, the `ResourceManager` and the client each had their own default `ClientConfiguration`; now they share one. - Passing `null` as the `ResourceManager` is not enough: `InternalClient` then runs its first checks before it creates a `ResourceManager`, but it creates one before `OTSUri` rejects an endpoint, so an endpoint with a path still leaks (verified, see below). The ticket also suggests checking the endpoint and the instance name before the `SyncClient` constructor is called; that would copy the rules of the SDK (prefix, multibyte characters, the pattern of the instance name, `ENDPOINT_REGEX`) into Camel, where they could drift from the SDK, while shutting the `ResourceManager` down covers every failure of the constructor. No option, header or default changes, so there is no upgrade-guide entry, and no generated file changes. ## Tests Two new tests in `AlibabaOtsEndpointTest`, and `org.awaitility:awaitility` as a test dependency of the module: - `concurrentInitClientCreatesOneClient`: eight threads call `initClient` on an endpoint whose `createClient` records the calling thread, waits on a latch, and returns a new mock `SyncClient`. The latch is released only when each caller has either entered `createClient` or waits for a lock that a thread in `createClient` holds (`ThreadMXBean.getThreadInfo(...).getLockOwnerId()`), so the outcome does not depend on how the threads are scheduled: without the lock, all eight enter `createClient`; with it, one does and seven wait. The test then checks that one client was created, that each caller got it, that the endpoint holds it, and that `doStop` shuts it down. - `failedClientCreationLeavesNoSdkThread`, for `my-instance.cn-hangzhou.ots.aliyuncs.com` (rejected by the first checks of `InternalClient`) and `https://my-instance.cn-hangzhou.ots.aliyuncs.com/path` (rejected by `OTSUri`): it calls `initClient` three times, checks that each call throws an `IllegalArgumentException` and that the endpoint keeps no client, then waits, with Awaitility for at most 10 seconds, until no SDK thread is alive besides those that were alive before the test. An SDK thread is a live thread whose class, or a frame of whose stack, is in `com.alicloud.openservices.tablestore.` or in `com.aliyun.ots.`, the package of the HTTP client that the SDK shades (artifact `tablestore-shaded-httpasyncclient`). The thread names cannot be used: the evictor and the reactor have the JDK default names `Thread-N` and `pool-N-thread-1`, and the `tablestore-` pools of the SDK start no thread before a request is sent. ## How it was verified - Without the fixes, with only `createClient()` added to the endpoint so that the first test can run: - `concurrentInitClientCreatesOneClient` fails with `Expected size: 1 but was: 8`. Run 200 times (with `@RepeatedTest(200)`, not committed), it failed 200 times, each time with 8 clients. - `failedClientCreationLeavesNoSdkThread` fails for both values with a `ConditionTimeoutException` after 10 seconds, which lists 54 extra threads per value: for each of the three calls, the `IdleConnectionEvictor` (`Thread-N`), the reactor thread (`pool-N-thread-1`) and 16 `I/O dispatcher N` threads. - With only the fix of `initClient`, the first test passes and the second still fails for both values. - With a `createClient` that passes `null` as the `ResourceManager` (not committed), the second test passes for the endpoint without `https://` and fails for the endpoint with a path, with the same 54 extra threads. - With both fixes, both tests pass. Run 200 times, the first test passed 200 times. - With both fixes, all 13 tests of camel-alibaba-ots pass, with no failure, error or rerun. ## Limits, in the SDK - `AsyncServiceClient.shutdown()` of the SDK ignores an `InterruptedException` from joining its evictor thread and then does not close its HTTP client. So when the thread that shuts the client down is interrupted, the reactor and I/O dispatcher threads stay alive, and the interrupt flag is lost. This applies to the shutdown after a failed creation as it already did to `SyncClient.shutdown()` in `doStop`. - In the shaded httpcore-nio 4.3.2, a reactor that is shut down before its thread has started to run does not close its `Selector`. A failed creation shuts the `ResourceManager` down right after it is created, so this can happen: in 1,600 failed creations of a scratch program that repeats the new failure path (not committed), no thread was left, but 69 `Selector`s stayed open (42 of 800 for the endpoint without a scheme, 27 of 800 for the endpoint with a path). Before the change, every failed creation left the threads and this `Selector` open. - A message to an endpoint whose client cannot be created still creates and shuts down a `ResourceManager`, which starts and stops these threads. In 800 of these failed creations, the shutdown took about 20 ms on average, but twice more than one second (1.5 s and 2.5 s), and the messages that wait for the lock in `initClient` wait for it. ## Other Alibaba endpoints The other Alibaba endpoints create their client with the same unsynchronized check-then-create in `initClient`. They are not changed here: | Module | Method | Their `doStop` | |---|---|---| | camel-alibaba-eventbridge | `AlibabaEventBridgeEndpoint.initClient` | drops the reference | | camel-alibaba-fc | `FCEndpoint.initClient` | drops the reference | | camel-alibaba-kms | `KMSEndpoint.initClient` | drops the reference | | camel-alibaba-mns | `MNSEndpoint.initClient` | closes the client | | camel-alibaba-oss | `OSSEndpoint.initClient` | closes the client | | camel-alibaba-sls | `AlibabaSlsEndpoint.initClient` | drops the reference | | camel-alibaba-sms | `SMSEndpoint.initClient` | drops the reference | With mns and oss, a client created by a second message at the same time is never closed, as it was with ots. Whether their SDKs also leave threads behind when a creation fails was not checked. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [x] I have run `mvn clean install -Dquickly -DskipTests` from the root folder on upstream `main` (31e144597200, the parent of this commit), and then, with the change, `mvn clean install -Psourcecheck` in camel-alibaba-ots, which also runs its code generators, the formatter and the import sorter. They changed no file, so there are no auto-generated changes to commit. The commands that were run: ``` # in components/camel-alibaba/camel-alibaba-ots, without the fixes and with only the fix of initClient mvn -o -B test -Dtest='AlibabaOtsEndpointTest#concurrentInitClientCreatesOneClient+failedClientCreationLeavesNoSdkThread' -Dsurefire.failIfNoSpecifiedTests=false # with the null ResourceManager variant mvn -o -B test -Dtest='AlibabaOtsEndpointTest#failedClientCreationLeavesNoSdkThread' -Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0 # with both fixes mvn -o -B test -Dtest='AlibabaOtsEndpointTest#concurrentInitClientCreatesOneClient+failedClientCreationLeavesNoSdkThread' -Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0 # with @RepeatedTest(200) on the first test, with both fixes and without the lock mvn -o -B test -Dtest='AlibabaOtsEndpointTest#concurrentInitClientCreatesOneClient' -Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0 # with both fixes mvn -o -B formatter:format impsort:sort mvn -o -B clean install -Psourcecheck ``` # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. The change, the tests and this text were drafted with Claude Code (Anthropic), from the ticket and the sources of Tablestore SDK 5.17.14. The tests were run as listed above. The claims about the SDK and the change were checked by independent AI reviews. _Claude Code on behalf of saravanakumar1987_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
