ruchirvaninasdaq opened a new issue #1341:
URL: https://github.com/apache/camel-kafka-connector/issues/1341


   Hello team, 
   
   I am using Camel-kafka-s3 connector version 3.3.0 and apache camel 3.7.0. I 
have been using for more than a year and had never seen this issue.  We have 
kafka cluster strimi 0.25.0 for kafka 2.8.0.
   
   I noticed that for some issue (most probably custom broker certificate 
expired) on kafka-cluster, our kafka connector went on dead loop of re-trying 
to connect to broker. Our cluster came back in 10-15 mins but the 
kafka-connector process never came back to a healthy status.   Here is what I 
see in logs:
   
   ```
   2022-02-28 02:11:33,287 WARN [Worker clientId=connect-1, 
groupId=s3-connector] Didn't reach end of config log quickly enough 
(org.apache.kafka.connect.runtime.distributed.DistributedHerder) 
[DistributedHerder-connect-1-1]
   java.util.concurrent.TimeoutException: Timed out waiting for future
   at 
org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:106)
   at 
org.apache.kafka.connect.storage.KafkaConfigBackingStore.refresh(KafkaConfigBackingStore.java:441)
   at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.readConfigToEnd(DistributedHerder.java:1188)
   at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:342)
   at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:316)
   at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   at java.base/java.lang.Thread.run(Thread.java:829)
   2022-02-28 02:16:45,296 INFO [Worker clientId=connect-1, 
groupId=s3-connector] Reading to end of config log; current config state 
offset: 2233 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) 
[DistributedHerder-connect-1-1]
   ```
   
   Follwing the log from where exception started:
   ```
   2022-02-27 19:18:30,203 INFO [AdminClient clientId=adminclient-8] Failed 
authentication with 
kafka-cluster-kafka-15.kafka-cluster-kafka-brokers.mdic-strimzi-pro.svc/100.80.52.142
 (Token validation failed: Unknown signing key (kid:{keyid}) (ErrId: 26617d8d)) 
(org.apache.kafka.common.network.Selector) [kafka-admin-client-thread | 
adminclient-8]
   2022-02-27 19:18:30,203 ERROR [AdminClient clientId=adminclient-8] 
Connection to node 15 
(kafka-cluster-kafka-15.kafka-cluster-kafka-brokers.mdic-strimzi-pro.svc/100.80.52.142:9093)
 failed authentication due to: Token validation failed: Unknown signing key 
(kid:{keyid}) (ErrId: 26617d8d) (org.apache.kafka.clients.NetworkClient) 
[kafka-admin-client-thread | adminclient-8]
   2022-02-27 19:18:30,203 WARN [AdminClient clientId=adminclient-8] Metadata 
update failed due to authentication error 
(org.apache.kafka.clients.admin.internals.AdminMetadataManager) 
[kafka-admin-client-thread | adminclient-8]
   org.apache.kafka.common.errors.SaslAuthenticationException: Token validation 
failed: Unknown signing key (kid:m45ykTYExWg7_fe2lwAqTCHevnt-1c064bM9vRSw5V4) 
(ErrId: 26617d8d)
   2022-02-27 19:18:30,218 INFO [AdminClient clientId=adminclient-8] Metadata 
update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) 
[kafka-admin-client-thread | adminclient-8]
   org.apache.kafka.common.errors.TimeoutException: 
Call(callName=fetchMetadata, deadlineMs=1645989510217, tries=1, 
nextAllowedTryMs=1645989510318) timed out at 1645989510218 after 1 attempt(s)
   Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out 
waiting to send the call. Call: fetchMetadata
   2022-02-27 19:18:30,318 INFO [AdminClient clientId=adminclient-8] Metadata 
update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) 
[kafka-admin-client-thread | adminclient-8]
   org.apache.kafka.common.errors.TimeoutException: 
Call(callName=fetchMetadata, deadlineMs=1645989510317, tries=1, 
nextAllowedTryMs=1645989510418) timed out at 1645989510318 after 1 attempt(s)
   Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out 
waiting to send the call. Call: fetchMetadata
   2022-02-27 19:18:30,418 INFO [AdminClient clientId=adminclient-8] Metadata 
update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) 
[kafka-admin-client-thread | adminclient-8]
   org.apache.kafka.common.errors.TimeoutException: 
Call(callName=fetchMetadata, deadlineMs=1645989510417, tries=1, 
nextAllowedTryMs=1645989510518) timed out at 1645989510418 after 1 attempt(s)
   Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out 
waiting to send the call. Call: fetchMetadata
   2022-02-27 19:18:30,518 INFO [AdminClient clientId=adminclient-8] Metadata 
update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) 
[kafka-admin-client-thread | adminclient-8]
   org.apache.kafka.common.errors.TimeoutException: 
Call(callName=fetchMetadata, deadlineMs=1645989510517, tries=1, 
nextAllowedTryMs=1645989510618) timed out at 1645989510518 after 1 attempt(s)
   Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out 
waiting to send the call. Call: fetchMetadata
   ```
   
   I have few following question:
   
   1. My question is that why didn't the connector die and restarted the 
process? 
   2. Is there a way we can propagate the Admin client exception in application?
   3. If not, How can I monitor my application health?
   
   Thank you.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to