saravanakumar1987 opened a new pull request, #30120:
URL: https://github.com/apache/camel/pull/30120

   # Description
   
   Fixes [CAMEL-27003](https://issues.apache.org/jira/browse/CAMEL-27003).
   
   ## What was wrong
   
   `MNSEndpoint.initClient` and `OSSEndpoint.initClient` created the client of 
the endpoint with the check-then-create that `AlibabaOtsEndpoint.initClient` 
had before apache/camel#30066 (CAMEL-26971): they read and wrote the 
`mnsClient` and `ossClient` fields without a lock, and the fields were not 
volatile. This was suggested in the review of apache/camel#30066 and found by 
reading the code.
   
   1. camel-alibaba-mns: `MNSProducer.process` calls `initClient` for each 
exchange, and `MNSConsumer.poll` for each poll. Calls made at the same time 
before a client existed could each create an `MNSClient`. The endpoint kept 
only the last one, and `doStop` closed only that one.
   
      In aliyun-sdk-mns 2.0.0, the clients that Camel builds share one service 
client: `MNSClientBuilder.build()` without a `ClientConfiguration` makes 
`CloudAccount` create a default one (`CloudAccount.java:126`), 
`ServiceClientFactory.createServiceClient` caches the `DefaultServiceClient` by 
its `ClientConfiguration` (`ServiceClientFactory.java:35-40`), which overrides 
`equals` and `hashCode` (`ClientConfiguration.java:60`, `:80`), and counts the 
clients that use it; `MNSClient.close()` decrements that count, and the service 
client closes its HTTP client when the count reaches 0. So a client that the 
endpoint lost and never closed keeps the shared HTTP client open, with its 
reactor thread and its `I/O dispatcher` threads (one per CPU by default), none 
of them daemon threads, also after every MNS endpoint has stopped. A scratch 
program (not committed) showed it on the 16-CPU machine of the tests below: 
after two creations through `MNSUtils.createClient`, one service client; after 
cl
 osing one client, still one service client, the reactor thread and 16 `I/O 
dispatcher` threads; after closing both, no service client and none of these 
threads.
   
   2. camel-alibaba-oss: `OSSProducer.process` calls `initClient` while the 
producer has no client, and `OSSConsumer.doStart` calls it when the consumer 
starts. Calls made at the same time before a client existed could each create 
an `OSSClient`. The endpoint kept only the last one, and `doStop` closed only 
that one; the other client, and its connection pool, were never closed.
   
   ## The change
   
   The first part of the fix of CAMEL-26971 (apache/camel#30066), "one client 
per endpoint", in both endpoints:
   
   - `mnsClient` and `ossClient` are volatile. `initClient` reads the field; 
when it is null, it takes the `lock` that the endpoint inherits from 
`BaseService`, reads the field again, and creates the client only if it is 
still null. Once the client exists, a call does not take the lock.
   - `setMnsClient`, `setOssClient` and `doStop` take the same lock, so the 
client and the `autowiredMnsClient` / `autowiredOssClient` flag change 
together, and `doStop` cannot run in the middle of a creation. 
`BaseService.stop()` already holds this lock when it calls `doStop`, and the 
lock is reentrant.
   - The creation is in a new package-private method `createClient()` of each 
endpoint, which calls `MNSUtils.createClient(this)` or 
`OSSUtils.createClient(this)`. The new tests override it to hold the creation.
   - The signatures and the moment of the creation do not change. 
`MNSEndpoint.initClient` is still `void`: the producer and the consumer still 
get the client with `getMnsClient()` after it, and the client is still created 
by the first message of the producer or the first poll of the consumer. 
`OSSEndpoint.initClient` still returns the client, now the value it read or 
created rather than the field read again, and the client is still created by 
the first message of the producer or when the consumer starts.
   
   No option, header or default changes, so there is no upgrade-guide entry, 
and no generated file changes.
   
   ## Tests
   
   One new test in each module, as `concurrentInitClientCreatesOneClient` of 
apache/camel#30066, and `org.awaitility:awaitility` as a test dependency of 
both modules:
   
   - `MNSEndpointTest#concurrentInitClientCreatesOneClient` and 
`OSSEndpointTest#concurrentInitClientCreatesOneClient`: eight threads call 
`initClient` on an endpoint whose `createClient` records the calling thread, 
waits on a latch, and returns a new mock client. The latch is released only 
when each caller has either entered `createClient` or waits for a lock that a 
thread in `createClient` holds 
(`ThreadMXBean.getThreadInfo(...).getLockOwnerId()`), so the outcome does not 
depend on how the threads are scheduled: without the lock, all eight enter 
`createClient`; with it, one does and seven wait. The test then checks that one 
client was created, that each caller got it (in mns, with `getMnsClient()` 
after `initClient`, as the producer and the consumer do), that the endpoint 
holds it, and that `doStop` closes it and clears the field.
   
   ## How it was verified
   
   - Without the fix, with only `createClient()` added to each endpoint so that 
the tests can run:
     - `MNSEndpointTest#concurrentInitClientCreatesOneClient` fails with 
`Expected size: 1 but was: 8`. Run 200 times (with `@RepeatedTest(200)`, not 
committed), it failed 200 times, each time with 8 clients.
     - `OSSEndpointTest#concurrentInitClientCreatesOneClient` fails with 
`Expected size: 1 but was: 8`. Run 200 times, it failed 200 times, each time 
with 8 clients.
   - With the fix, both tests pass. Run 200 times, each passed 200 times.
   - With the fix, all 28 tests of camel-alibaba-mns and all 28 tests of 
camel-alibaba-oss pass, with no failure, error or rerun.
   
   ## A failed client creation, in the SDKs
   
   The ticket left open whether a failed client creation leaves resources of 
the SDK behind, as it did in camel-alibaba-ots, where each message then tried 
again and left more. It was checked in the sources and the bytecode of 
aliyun-sdk-mns 2.0.0 and alibabacloud-oss-v2 0.6.1 and of the HTTP clients that 
the two modules resolve (httpasyncclient 4.1.5 with httpcore-nio 4.4.15 for 
mns, httpclient5 5.5 with httpcore5 5.3.4 for oss: these modules do not use the 
versions of `parent/pom.xml`), and with scratch programs (not committed) on the 
test classpath of each module, on Windows with JDK 21.0.2.
   
   In camel-alibaba-oss, nothing is left behind. In camel-alibaba-mns, nothing 
is left behind by the failures that the options of the endpoint can cause, but 
a failure while the SDK builds its HTTP client leaves resources behind, and as 
the endpoint keeps no client, each later message or poll tries again and leaves 
more. This change does not change that.
   
   camel-alibaba-mns:
   
   - `MNSUtils.createClient` checks the AccessKey pair, `accountEndpoint` and 
`region` first. The SDK then checks the account endpoint in the `CloudAccount` 
constructor (`CloudAccount.java:84`, `Utils.getHttpURI`): it rejects an 
endpoint without `http://` or `https://`, one with a URI syntax error, and one 
without a host. Nothing has been created at that point. In a scratch program, 
400 failed creations (100 each for an endpoint without a scheme, one without a 
host, one with a URI syntax error, and a blank AccessKey ID) left no service 
client and no thread, and 200 more for an endpoint without a scheme left no OS 
handle.
   - The SDK creates its HTTP client in `CloudAccount.getMNSClient`, in 
`ServiceClientFactory.createServiceClient` (`CloudAccount.java:99`), only when 
no service client with an equal `ClientConfiguration` exists: for Camel, at the 
first creation, or at the first one after every client was closed. The 
`DefaultMNSClient` constructor that follows checks the already normalized 
endpoint again, which cannot fail, and `getMNSClient` would release the service 
client if it did (`CloudAccount.java:102-107`).
   - When the constructor of `DefaultServiceClient` fails, nothing closes what 
it has created, as the service client is neither cached nor returned yet:
     - The JVM cannot create a TLS `SSLContext`, for example because 
`jdk.tls.client.protocols` or `jdk.tls.server.protocols` names a protocol that 
does not exist, such as `TLSv1.2 TLSv1.3`, separated by a space instead of a 
comma. `HttpFactory.createConnectionManager` has opened the `Selector` of the 
I/O reactor (`HttpFactory.java:120`) when `new 
PoolingNHttpClientConnectionManager(ioReactor)` (`HttpFactory.java:138`) fails 
in `SSLContexts.createDefault()`, which it calls for its `https` strategy, so 
also for an `http://` account endpoint. In a scratch program run with 
`-Djdk.tls.client.protocols="TLSv1.2 TLSv1.3"`, each creation failed with an 
`SSLInitializationException`, and each failed creation left about 8 OS handles 
open, also after a full GC, and no thread; with the same setting and an account 
endpoint without a scheme, the handle count did not grow. So a consumer or a 
producer of such an endpoint leaks these handles (file descriptors on Linux) 
with each poll or message, unti
 l the process runs out of them.
     - `IdleConnectionMonitor.addConnMgr` throws a `ClientException` (`Too Many 
ServiceClient created.`) when more than 50 connection managers are registered 
(`HttpFactory.java:310-317`), after the HTTP client has been started 
(`DefaultServiceClient.java:111-114`). Each attempt then leaves a running HTTP 
client: its reactor thread and one `I/O dispatcher` thread per CPU, none of 
them daemon threads. Camel alone cannot reach it, as its clients share one 
service client: other code in the same class loader must keep more than 50 MNS 
service clients with different, non-default configurations open, or lower the 
limit with `IdleConnectionMonitor.set_CONNECTION_MANAGER_LIMIT`, before Camel 
creates its first client.
     - After an `Error` only: if the static initializer of 
`HttpFactory$IdleConnectionMonitor` fails once, for example with an 
`OutOfMemoryError` when its `IdleConnectionMonitorThread` cannot be started, 
each later creation starts an HTTP client and then fails with a 
`NoClassDefFoundError`, which leaves that client running.
   
     The last two were found in the sources and the bytecode, and not run.
   
   camel-alibaba-oss:
   
   - `OSSUtils.createClient` checks the AccessKey pair and that a region or an 
endpoint is set, and creates the `StaticCredentialsProvider`, before the SDK 
creates anything.
   - `OSSClientBuilder.build()` creates the HTTP transport first 
(`DefaultOSSClientBuilder.java:23`). `Apache5HttpClientBuilder.build()` sets up 
TLS, which throws before anything is created if it fails, then creates a 
connection pool and registers it with the SDK's `IdleConnectionReaper` 
(`Apache5HttpClientBuilder.java:259`). Then it creates the client 
(`DefaultOSSClientBuilder.java:25`), whose `ClientImpl` constructor does not 
reject an invalid region or endpoint: `resolveEndpoint` keeps a null endpoint 
(`ClientImpl.java:514`), and the first operation fails with `endpoint or region 
is invalid` (`ClientImpl.java:282`). Its only exception, `credentialsProvider 
is null` (`ClientImpl.java:459`), cannot happen with the call of `OSSUtils`. So 
once the transport exists, the creation does not fail for the options that 
Camel passes, and a client with an invalid region or endpoint is kept and 
closed in `doStop`, as before.
   - In a scratch program, the creation succeeded for an invalid region (`cn 
hangzhou!`), an invalid endpoint (`http://exa mple.com`), an endpoint with a 
path, and a valid region; the first operation failed with `endpoint or region 
is invalid` for the first two. The only thread these clients started was the 
`idle-connection-reaper` daemon thread that all clients of the SDK share, and 
it stopped when the last of them was closed.
   - After the registration, the SDK fails only for an explicit proxy host with 
a port above 65535 (`applyProxy`; Camel sets no proxy) or with an `Error`. Such 
a failure would leave a connection pool without connections, which the reaper 
holds weakly (`IdleConnectionReaper.java:32`), so that the garbage collector 
reclaims it, and, if this registration started it, the reaper's daemon thread, 
until a later close finds no pool registered.
   
   ## Not changed
   
   - `MNSEndpoint.initClient` stays `void`, so `MNSProducer` and 
`MNSConsumer.poll` read the field again with `getMnsClient()` after it, and 
`MNSConsumer.processCommit` reads it when an exchange completes. If the 
endpoint stops between these reads, they get `null` and fail with a 
`NullPointerException` (in `processCommit`, given to the exception handler, so 
the message is not deleted). And a call of `initClient` after the endpoint has 
stopped creates a client that nothing closes, which keeps the shared HTTP 
client of the MNS SDK open. Both need an exchange that is still in flight when 
the endpoint stops: the routes are shut down before the endpoints, and their 
in-flight exchanges are waited for up to the shutdown timeout. This was so 
before the change, and is so in camel-alibaba-ots.
   - `OSSProducer` keeps the client that `initClient` returned in its own 
field, and calls `initClient` only while that field is null, so a producer that 
is used after its endpoint has stopped uses the closed client. A route that 
starts again creates a new producer, and `OSSConsumer` gets the client again 
each time it starts. This was so before the change.
   - The other Alibaba endpoints (eventbridge, fc, kms, sls, sms) create their 
client in the same way, but their `doStop` only drops the reference, as the 
ticket says; they are not part of this ticket.
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [x] I have run `mvn clean install -Dquickly -DskipTests` from the root 
folder on upstream `main` (31e144597200, the parent of this commit), and then, 
with the change, `mvn clean install -Psourcecheck` in camel-alibaba-mns and 
then in camel-alibaba-oss, which also runs their code generators, the formatter 
and the import sorter. They changed no file, so there are no auto-generated 
changes to commit.
   
   The commands that were run:
   
   ```
   # in components/camel-alibaba/camel-alibaba-mns, without the fix (only 
createClient() added) and with the fix
   mvn -o -B test -Dtest='MNSEndpointTest#concurrentInitClientCreatesOneClient' 
-Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0
   # in components/camel-alibaba/camel-alibaba-oss, without the fix (only 
createClient() added) and with the fix
   mvn -o -B test -Dtest='OSSEndpointTest#concurrentInitClientCreatesOneClient' 
-Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0
   # the same two commands with @RepeatedTest(200) on the new test, without and 
with the fix
   # with the fix, in camel-alibaba-mns and then in camel-alibaba-oss
   mvn -o -B formatter:format impsort:sort
   mvn -o -B clean install -Psourcecheck
   ```
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
   
   The change, the tests and this text were drafted with Claude Code 
(Anthropic), from the ticket, the fix of CAMEL-26971 and the sources of 
aliyun-sdk-mns 2.0.0 and alibabacloud-oss-v2 0.6.1. The tests and the scratch 
programs were run as described above. The claims about the SDKs and the change 
were checked by independent AI reviews.
   
   _Claude Code on behalf of saravanakumar1987_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to