Hello,
We work in a multinational company that produces diesel engines and is working 
on an IoT platform to analyze engine performance based on sensor data. We are 
using Flink for deploying analytics stream processing jobs. We recently 
integrated these jobs with Couchbase (serving as a Cache) and are monitoring 
the performance of these jobs in our test environment.
Flink Cluster
Two Job Managers (2 cpu, 8 GB, Centos 7, 50 GB Disk, Apache Flink 1.9.0)
Six Task Managers (4 cpu, 16 GB, Centos 7, 50 GB Disk, Apache Flink 1.9.0)
Couchbase Cluster
Two nodes (4 cpu, 16 GB, Amazon Linux 2, 25 GB Disk, Community Edition 6.5.1 
build 6299)
Couchbase SDK
java-client (3.0.10)
We noticed that after re-deploying our analytics Flink jobs, high CPU usage 
alert is seen on Flink Job Manager.
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
1831 centos    20   0   11.3g   5.8g   6076 S 162.5 77.3  17181:30 java
It was observed that the Job Manager process (1831) comprised of 60.9% (873 out 
of 1433) threads related to couchbase timers of some sort (e.g. cb-timer-1-1, 
cb-events, cb-tracing-1, cb-orphan-1 etc.). This may be the reason for high CPU 
usage.
$ ps -eT | grep 1831 | grep -i cb | wc -l
873

$ ps -eT | grep 1831 | wc -l
1433

$ ps -eT | grep 1831 | grep -i cb
1831 11539 ?        01:45:52 cb-timer-1-1
1831 11541 ?        00:11:53 cb-events
1831 11542 ?        00:10:13 cb-tracing-1
1831 11543 ?        00:10:08 cb-orphan-1
1831 11545 ?        00:04:47 cb-comp-1
1831 11546 ?        00:06:41 cb-comp-2
1831 11547 ?        00:34:57 cb-io-kv-5-1
1831 11549 ?        00:17:48 cb-io-kv-5-2
1831 24911 ?        00:43:07 cb-timer-1-1
1831 24912 ?        00:05:34 cb-events
1831 24913 ?        00:04:02 cb-tracing-1
1831 24914 ?        00:04:02 cb-orphan-1
1831 24915 ?        00:02:35 cb-comp-1
1831 24916 ?        00:04:17 cb-comp-2
1831 24917 ?        00:23:41 cb-io-kv-5-1
1831 24919 ?        00:13:08 cb-io-kv-5-2
1831 24966 ?        00:44:37 cb-timer-1-1
1831 24967 ?        00:05:43 cb-events
...
If there is any issue in connecting to Couchbase from the Flink jobs running on 
Flink task managers then those may result in errors being logged in task 
manager or cause some other issue (e.g. CPU) in task managers. However, I am 
wondering why high CPU is seen on Flink Job Manager instead and not on Flink 
task managers. Please help in understanding possible reasons for creation of 
numerous cb-* threads on Flink Job Manager. If you have any other suggestions 
or commands to better troubleshoot this issue, those are welcome too.
Thank you.
Kind regards,
Vinaya

Reply via email to