Croway opened a new pull request, #30088:
URL: https://github.com/apache/camel/pull/30088

   [CAMEL-26998](https://issues.apache.org/jira/browse/CAMEL-26998)
   
   ## What was wrong
   
   `ScheduledPollConsumerHealthCheck` (the consumer health check of every 
`ScheduledPollConsumer`: file, ftp/sftp, sql, jpa, mail, aws2-s3, ...) reports 
from the state a poll leaves behind once it is done: the error counter and last 
error, which `ScheduledPollConsumer.doRun` updates after `poll()` returned or 
threw, and `isConsumerReady()`. A poll that does not return is invisible, so 
readiness stays UP for as long as it hangs. It is also UP when the first poll 
hangs with `camel.health.initialState=UP`, and when a consumer marked itself 
ready before its poll is done (`forceConsumerAsReady()`: sql after the query, 
ftp after the listing).
   
   Found with fault-injection tests (Toxiproxy blackhole / silent drop in front 
of a real PostgreSQL and SFTP server): camel-sql with pgjdbc's default 
`socketTimeout=0` waits in a socket read until TCP gives up or for ever; sftp 
with the default `soTimeout` (the JSch session timeout) hangs ~600 s. No error, 
no WARN, readiness UP.
   
   ## The fix
   
   - `ScheduledPollConsumer` times the most recent poll (a new `StopWatch` in a 
volatile field, set just before `pollingCounter` is incremented around 
begin/poll/commit), and has `getPollingDuration()`: the time since the poll in 
progress started, 0 when not polling. Each greedy run and each retry is a poll 
of its own.
   - New consumer option `healthCheckMaxPollDuration` on 
`ScheduledPollEndpoint` (so on every polling component), passed to the 
consumer. Default `0` = off. When set, the consumer health check is DOWN while 
a poll has been running longer than that, before any other rule (also before 
the first poll is done and for a consumer forced ready), with a `poll.duration` 
detail and the message:
     `Consumer has been polling for 10m5s, longer than 
healthCheckMaxPollDuration (10m0s) route: route1 (sftp://myserver/inbox)`.
     When the poll is done, the check is UP or DOWN by its result, as before. 
The poll is not interrupted.
   - Excluded from the AS2 and Box endpoints, which exclude the other scheduler 
options.
   - Docs: new section "Detecting a poll that hangs" in `health-check.adoc`, 
and a pointer from `batch-consumer.adoc`. The rest of the diff is the 
regenerated component JSON, configurers, URI factories, catalog and endpoint 
DSL for the new option.
   
   ## Design choices for reviewers
   
   - **Opt-in, no default.** A poll includes routing what it polled (a batch 
consumer routes the whole batch within `poll()`), so a big file, a large batch 
or a slow query is a long poll with nothing wrong. Any default (such as N × 
`delay`) would flip readiness for existing users; CAMEL-20189 shows how 
sensitive readiness during a long first poll is. No upgrade-guide entry, since 
nothing changes unless the option is set.
   - **Not a result threshold (CAMEL-17587).** CAMEL-17587 removed 
failure/success thresholds because a monitoring system can apply them to the 
results. Here there is no result to apply them to: while a poll hangs the check 
returns UP, and only Camel knows a poll is in progress.
   - **Per endpoint, not `camel.health.*`.** How long a poll may take depends 
on the endpoint (an SFTP route with big files vs. a SQL route), and it sits 
with the other scheduler options. A global default could come later if wanted.
   - **Unit:** milliseconds, typed as a duration (`10m` works), not affected by 
`timeUnit` (which the docs say is for `initialDelay`/`delay`).
   - **Concurrent polls** (`scheduler.concurrentConsumers` > 1): one watch, of 
the most recent poll to start, so the check is DOWN when polls are in progress 
and none has started for longer than the option; it never reports DOWN for a 
poll that has run less. Tracking the oldest of several polls would need a 
collection on every poll for a rare setup.
   - **Clock:** the duration is measured with `StopWatch`, which follows 
`TimeSource` (durations do; the timestamps of CAMEL-26715 stay on the system 
clock, so `ScheduledPollConsumer` does not call `TimeSource` itself and stays 
out of `TimeSourceDirectClockCallsTest`). The unit tests move a manual clock 
instead of waiting, and a deterministic simulation test gets it in virtual time.
   - Not done here: showing the polling duration in JMX / the consumer dev 
console.
   
   ## Tests
   
   - `ScheduledPollConsumerHangHealthCheckTest` (camel-core, manual clock, 
`@Isolated`): UP up to the max, DOWN at max + 1 ms with the message and detail, 
UP when the poll is done; a poll that fails after hanging is reported as a 
failed poll; off by default (1 h hang stays UP); a hanging first poll with each 
initial state; a consumer forced ready before its poll is done; each greedy run 
is a poll of its own; concurrent polls.
   - `FileConsumerHangHealthCheckTest` (camel-core): 
`file:...?healthCheckMaxPollDuration=200` through `ConsumerHealthCheck`, a 
route that blocks on a latch: UP, DOWN while blocked, UP after.
   - Fail before: on unmodified `main`, a consumer whose poll has been blocked 
for 2 s reports readiness UP with no message (`polling=true state=UP`). With 
the health-check part of this change removed (option and timing kept), 8 of the 
9 unit tests fail (`expected: <DOWN> but was: <UP>`; the "off by default" one 
passes, as it should) and the file test times out waiting for DOWN.
   - `mvnd test -pl core/camel-core,core/camel-management -am 
-Dtest='*HealthCheck*Test,*ScheduledPoll*Test,TimeSource*Test,*PollingConsumer*Test,*Greedy*Test,*Backoff*Test,*PollEnrich*Test'`:
 camel-core 124, camel-management 5, camel-util 13 tests, all pass (including 
`TimeSourceDirectClockCallsTest`).
   - Generated files: the 74 changed component JSON files only gain the new 
option (checked by comparing them with the index fields removed); 69 endpoint 
DSL factories gain the two setters; AS2 and Box JSON are unchanged.
   
   _Claude Code on behalf of Croway_
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to