saravanakumar1987 opened a new pull request, #30120: URL: https://github.com/apache/camel/pull/30120
# Description Fixes [CAMEL-27003](https://issues.apache.org/jira/browse/CAMEL-27003). ## What was wrong `MNSEndpoint.initClient` and `OSSEndpoint.initClient` created the client of the endpoint with the check-then-create that `AlibabaOtsEndpoint.initClient` had before apache/camel#30066 (CAMEL-26971): they read and wrote the `mnsClient` and `ossClient` fields without a lock, and the fields were not volatile. This was suggested in the review of apache/camel#30066 and found by reading the code. 1. camel-alibaba-mns: `MNSProducer.process` calls `initClient` for each exchange, and `MNSConsumer.poll` for each poll. Calls made at the same time before a client existed could each create an `MNSClient`. The endpoint kept only the last one, and `doStop` closed only that one. In aliyun-sdk-mns 2.0.0, the clients that Camel builds share one service client: `MNSClientBuilder.build()` without a `ClientConfiguration` makes `CloudAccount` create a default one (`CloudAccount.java:126`), `ServiceClientFactory.createServiceClient` caches the `DefaultServiceClient` by its `ClientConfiguration` (`ServiceClientFactory.java:35-40`), which overrides `equals` and `hashCode` (`ClientConfiguration.java:60`, `:80`), and counts the clients that use it; `MNSClient.close()` decrements that count, and the service client closes its HTTP client when the count reaches 0. So a client that the endpoint lost and never closed keeps the shared HTTP client open, with its reactor thread and its `I/O dispatcher` threads (one per CPU by default), none of them daemon threads, also after every MNS endpoint has stopped. A scratch program (not committed) showed it on the 16-CPU machine of the tests below: after two creations through `MNSUtils.createClient`, one service client; after cl osing one client, still one service client, the reactor thread and 16 `I/O dispatcher` threads; after closing both, no service client and none of these threads. 2. camel-alibaba-oss: `OSSProducer.process` calls `initClient` while the producer has no client, and `OSSConsumer.doStart` calls it when the consumer starts. Calls made at the same time before a client existed could each create an `OSSClient`. The endpoint kept only the last one, and `doStop` closed only that one; the other client, and its connection pool, were never closed. ## The change The first part of the fix of CAMEL-26971 (apache/camel#30066), "one client per endpoint", in both endpoints: - `mnsClient` and `ossClient` are volatile. `initClient` reads the field; when it is null, it takes the `lock` that the endpoint inherits from `BaseService`, reads the field again, and creates the client only if it is still null. Once the client exists, a call does not take the lock. - `setMnsClient`, `setOssClient` and `doStop` take the same lock, so the client and the `autowiredMnsClient` / `autowiredOssClient` flag change together, and `doStop` cannot run in the middle of a creation. `BaseService.stop()` already holds this lock when it calls `doStop`, and the lock is reentrant. - The creation is in a new package-private method `createClient()` of each endpoint, which calls `MNSUtils.createClient(this)` or `OSSUtils.createClient(this)`. The new tests override it to hold the creation. - The signatures and the moment of the creation do not change. `MNSEndpoint.initClient` is still `void`: the producer and the consumer still get the client with `getMnsClient()` after it, and the client is still created by the first message of the producer or the first poll of the consumer. `OSSEndpoint.initClient` still returns the client, now the value it read or created rather than the field read again, and the client is still created by the first message of the producer or when the consumer starts. No option, header or default changes, so there is no upgrade-guide entry, and no generated file changes. ## Tests One new test in each module, as `concurrentInitClientCreatesOneClient` of apache/camel#30066, and `org.awaitility:awaitility` as a test dependency of both modules: - `MNSEndpointTest#concurrentInitClientCreatesOneClient` and `OSSEndpointTest#concurrentInitClientCreatesOneClient`: eight threads call `initClient` on an endpoint whose `createClient` records the calling thread, waits on a latch, and returns a new mock client. The latch is released only when each caller has either entered `createClient` or waits for a lock that a thread in `createClient` holds (`ThreadMXBean.getThreadInfo(...).getLockOwnerId()`), so the outcome does not depend on how the threads are scheduled: without the lock, all eight enter `createClient`; with it, one does and seven wait. The test then checks that one client was created, that each caller got it (in mns, with `getMnsClient()` after `initClient`, as the producer and the consumer do), that the endpoint holds it, and that `doStop` closes it and clears the field. ## How it was verified - Without the fix, with only `createClient()` added to each endpoint so that the tests can run: - `MNSEndpointTest#concurrentInitClientCreatesOneClient` fails with `Expected size: 1 but was: 8`. Run 200 times (with `@RepeatedTest(200)`, not committed), it failed 200 times, each time with 8 clients. - `OSSEndpointTest#concurrentInitClientCreatesOneClient` fails with `Expected size: 1 but was: 8`. Run 200 times, it failed 200 times, each time with 8 clients. - With the fix, both tests pass. Run 200 times, each passed 200 times. - With the fix, all 28 tests of camel-alibaba-mns and all 28 tests of camel-alibaba-oss pass, with no failure, error or rerun. ## A failed client creation, in the SDKs The ticket left open whether a failed client creation leaves resources of the SDK behind, as it did in camel-alibaba-ots, where each message then tried again and left more. It was checked in the sources and the bytecode of aliyun-sdk-mns 2.0.0 and alibabacloud-oss-v2 0.6.1 and of the HTTP clients that the two modules resolve (httpasyncclient 4.1.5 with httpcore-nio 4.4.15 for mns, httpclient5 5.5 with httpcore5 5.3.4 for oss: these modules do not use the versions of `parent/pom.xml`), and with scratch programs (not committed) on the test classpath of each module, on Windows with JDK 21.0.2. In camel-alibaba-oss, nothing is left behind. In camel-alibaba-mns, nothing is left behind by the failures that the options of the endpoint can cause, but a failure while the SDK builds its HTTP client leaves resources behind, and as the endpoint keeps no client, each later message or poll tries again and leaves more. This change does not change that. camel-alibaba-mns: - `MNSUtils.createClient` checks the AccessKey pair, `accountEndpoint` and `region` first. The SDK then checks the account endpoint in the `CloudAccount` constructor (`CloudAccount.java:84`, `Utils.getHttpURI`): it rejects an endpoint without `http://` or `https://`, one with a URI syntax error, and one without a host. Nothing has been created at that point. In a scratch program, 400 failed creations (100 each for an endpoint without a scheme, one without a host, one with a URI syntax error, and a blank AccessKey ID) left no service client and no thread, and 200 more for an endpoint without a scheme left no OS handle. - The SDK creates its HTTP client in `CloudAccount.getMNSClient`, in `ServiceClientFactory.createServiceClient` (`CloudAccount.java:99`), only when no service client with an equal `ClientConfiguration` exists: for Camel, at the first creation, or at the first one after every client was closed. The `DefaultMNSClient` constructor that follows checks the already normalized endpoint again, which cannot fail, and `getMNSClient` would release the service client if it did (`CloudAccount.java:102-107`). - When the constructor of `DefaultServiceClient` fails, nothing closes what it has created, as the service client is neither cached nor returned yet: - The JVM cannot create a TLS `SSLContext`, for example because `jdk.tls.client.protocols` or `jdk.tls.server.protocols` names a protocol that does not exist, such as `TLSv1.2 TLSv1.3`, separated by a space instead of a comma. `HttpFactory.createConnectionManager` has opened the `Selector` of the I/O reactor (`HttpFactory.java:120`) when `new PoolingNHttpClientConnectionManager(ioReactor)` (`HttpFactory.java:138`) fails in `SSLContexts.createDefault()`, which it calls for its `https` strategy, so also for an `http://` account endpoint. In a scratch program run with `-Djdk.tls.client.protocols="TLSv1.2 TLSv1.3"`, each creation failed with an `SSLInitializationException`, and each failed creation left about 8 OS handles open, also after a full GC, and no thread; with the same setting and an account endpoint without a scheme, the handle count did not grow. So a consumer or a producer of such an endpoint leaks these handles (file descriptors on Linux) with each poll or message, unti l the process runs out of them. - `IdleConnectionMonitor.addConnMgr` throws a `ClientException` (`Too Many ServiceClient created.`) when more than 50 connection managers are registered (`HttpFactory.java:310-317`), after the HTTP client has been started (`DefaultServiceClient.java:111-114`). Each attempt then leaves a running HTTP client: its reactor thread and one `I/O dispatcher` thread per CPU, none of them daemon threads. Camel alone cannot reach it, as its clients share one service client: other code in the same class loader must keep more than 50 MNS service clients with different, non-default configurations open, or lower the limit with `IdleConnectionMonitor.set_CONNECTION_MANAGER_LIMIT`, before Camel creates its first client. - After an `Error` only: if the static initializer of `HttpFactory$IdleConnectionMonitor` fails once, for example with an `OutOfMemoryError` when its `IdleConnectionMonitorThread` cannot be started, each later creation starts an HTTP client and then fails with a `NoClassDefFoundError`, which leaves that client running. The last two were found in the sources and the bytecode, and not run. camel-alibaba-oss: - `OSSUtils.createClient` checks the AccessKey pair and that a region or an endpoint is set, and creates the `StaticCredentialsProvider`, before the SDK creates anything. - `OSSClientBuilder.build()` creates the HTTP transport first (`DefaultOSSClientBuilder.java:23`). `Apache5HttpClientBuilder.build()` sets up TLS, which throws before anything is created if it fails, then creates a connection pool and registers it with the SDK's `IdleConnectionReaper` (`Apache5HttpClientBuilder.java:259`). Then it creates the client (`DefaultOSSClientBuilder.java:25`), whose `ClientImpl` constructor does not reject an invalid region or endpoint: `resolveEndpoint` keeps a null endpoint (`ClientImpl.java:514`), and the first operation fails with `endpoint or region is invalid` (`ClientImpl.java:282`). Its only exception, `credentialsProvider is null` (`ClientImpl.java:459`), cannot happen with the call of `OSSUtils`. So once the transport exists, the creation does not fail for the options that Camel passes, and a client with an invalid region or endpoint is kept and closed in `doStop`, as before. - In a scratch program, the creation succeeded for an invalid region (`cn hangzhou!`), an invalid endpoint (`http://exa mple.com`), an endpoint with a path, and a valid region; the first operation failed with `endpoint or region is invalid` for the first two. The only thread these clients started was the `idle-connection-reaper` daemon thread that all clients of the SDK share, and it stopped when the last of them was closed. - After the registration, the SDK fails only for an explicit proxy host with a port above 65535 (`applyProxy`; Camel sets no proxy) or with an `Error`. Such a failure would leave a connection pool without connections, which the reaper holds weakly (`IdleConnectionReaper.java:32`), so that the garbage collector reclaims it, and, if this registration started it, the reaper's daemon thread, until a later close finds no pool registered. ## Not changed - `MNSEndpoint.initClient` stays `void`, so `MNSProducer` and `MNSConsumer.poll` read the field again with `getMnsClient()` after it, and `MNSConsumer.processCommit` reads it when an exchange completes. If the endpoint stops between these reads, they get `null` and fail with a `NullPointerException` (in `processCommit`, given to the exception handler, so the message is not deleted). And a call of `initClient` after the endpoint has stopped creates a client that nothing closes, which keeps the shared HTTP client of the MNS SDK open. Both need an exchange that is still in flight when the endpoint stops: the routes are shut down before the endpoints, and their in-flight exchanges are waited for up to the shutdown timeout. This was so before the change, and is so in camel-alibaba-ots. - `OSSProducer` keeps the client that `initClient` returned in its own field, and calls `initClient` only while that field is null, so a producer that is used after its endpoint has stopped uses the closed client. A route that starts again creates a new producer, and `OSSConsumer` gets the client again each time it starts. This was so before the change. - The other Alibaba endpoints (eventbridge, fc, kms, sls, sms) create their client in the same way, but their `doStop` only drops the reference, as the ticket says; they are not part of this ticket. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [x] I have run `mvn clean install -Dquickly -DskipTests` from the root folder on upstream `main` (31e144597200, the parent of this commit), and then, with the change, `mvn clean install -Psourcecheck` in camel-alibaba-mns and then in camel-alibaba-oss, which also runs their code generators, the formatter and the import sorter. They changed no file, so there are no auto-generated changes to commit. The commands that were run: ``` # in components/camel-alibaba/camel-alibaba-mns, without the fix (only createClient() added) and with the fix mvn -o -B test -Dtest='MNSEndpointTest#concurrentInitClientCreatesOneClient' -Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0 # in components/camel-alibaba/camel-alibaba-oss, without the fix (only createClient() added) and with the fix mvn -o -B test -Dtest='OSSEndpointTest#concurrentInitClientCreatesOneClient' -Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0 # the same two commands with @RepeatedTest(200) on the new test, without and with the fix # with the fix, in camel-alibaba-mns and then in camel-alibaba-oss mvn -o -B formatter:format impsort:sort mvn -o -B clean install -Psourcecheck ``` # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. The change, the tests and this text were drafted with Claude Code (Anthropic), from the ticket, the fix of CAMEL-26971 and the sources of aliyun-sdk-mns 2.0.0 and alibabacloud-oss-v2 0.6.1. The tests and the scratch programs were run as described above. The claims about the SDKs and the change were checked by independent AI reviews. _Claude Code on behalf of saravanakumar1987_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
