Hey All, We have a Kafka Streams application which ingests from a topic to which more than 15K messages are generated per second. The app filters a few of them, counts the number of unique filtered messages (based on one particular field) within a 1 min time window, and dumps it back to Kafka.
The issue that we are facing is that for certain minutes, there is no data in the sink topic. I have attached the data from 03:30AM to 10:00 AM today morning with this mail. And if you notice closely, the data for quite a few minutes is missing. One thing that we have noticed is that the skipped-records-rate metrics emitted by Kafka is around 200 for each thread. By the way, what does metric indicate? Does this represent the filtered out messages? We have checked the raw data in the source topic and didn't find any discrepancy. We even checked the logs on the stream app boxes and the only errors we found were GC errors. Other relevant info: Kafka version: 0.10.2.0 Number of partitions for source topic: 50 Stream App cluster: 5 machines with 10 threads each How do we debug this? What could be the cause?
2017-04-27T03:30:00 1 2017-04-27T03:36:00 2 2017-04-27T03:50:00 16 2017-04-27T04:03:00 9 2017-04-27T04:12:00 3 2017-04-27T04:33:00 2 2017-04-27T04:38:00 3 2017-04-27T04:44:00 2 2017-04-27T04:50:00 3 2017-04-27T04:53:00 0 2017-04-27T05:02:00 3 2017-04-27T05:10:00 0 2017-04-27T05:14:00 4 2017-04-27T05:18:00 1 2017-04-27T05:20:00 3 2017-04-27T05:22:00 13 2017-04-27T05:24:00 7 2017-04-27T05:25:00 6 2017-04-27T05:26:00 2 2017-04-27T05:30:00 10 2017-04-27T05:32:00 5 2017-04-27T05:34:00 8 2017-04-27T05:36:00 9 2017-04-27T05:37:00 1 2017-04-27T05:38:00 17 2017-04-27T05:39:00 0 2017-04-27T05:40:00 6 2017-04-27T05:44:00 1 2017-04-27T05:45:00 8 2017-04-27T05:48:00 9 2017-04-27T05:49:00 24 2017-04-27T05:52:00 1 2017-04-27T05:53:00 22 2017-04-27T05:54:00 24 2017-04-27T05:55:00 20 2017-04-27T05:56:00 11 2017-04-27T05:57:00 1 2017-04-27T05:58:00 2 2017-04-27T05:59:00 10 2017-04-27T06:01:00 6 2017-04-27T06:07:00 4 2017-04-27T06:09:00 6 2017-04-27T06:10:00 23 2017-04-27T06:12:00 38 2017-04-27T06:13:00 9 2017-04-27T06:14:00 4 2017-04-27T06:15:00 34 2017-04-27T06:16:00 25 2017-04-27T06:17:00 11 2017-04-27T06:18:00 19 2017-04-27T06:19:00 32 2017-04-27T06:21:00 68 2017-04-27T06:22:00 55 2017-04-27T06:25:00 57 2017-04-27T06:26:00 38 2017-04-27T06:27:00 65 2017-04-27T06:28:00 18 2017-04-27T06:29:00 38 2017-04-27T06:30:00 66 2017-04-27T06:31:00 64 2017-04-27T06:32:00 30 2017-04-27T06:35:00 46 2017-04-27T06:36:00 6 2017-04-27T06:38:00 43 2017-04-27T06:39:00 19 2017-04-27T06:40:00 30 2017-04-27T06:41:00 23 2017-04-27T06:42:00 82 2017-04-27T06:43:00 27 2017-04-27T06:44:00 20 2017-04-27T06:45:00 19 2017-04-27T06:46:00 45 2017-04-27T06:48:00 27 2017-04-27T06:49:00 4 2017-04-27T06:50:00 67 2017-04-27T06:51:00 111 2017-04-27T06:52:00 90 2017-04-27T06:53:00 95 2017-04-27T06:55:00 83 2017-04-27T06:57:00 51 2017-04-27T06:58:00 110 2017-04-27T06:59:00 45 2017-04-27T07:01:00 10 2017-04-27T07:02:00 119 2017-04-27T07:03:00 4 2017-04-27T07:04:00 63 2017-04-27T07:06:00 75 2017-04-27T07:07:00 63 2017-04-27T07:08:00 30 2017-04-27T07:09:00 36 2017-04-27T07:10:00 94 2017-04-27T07:11:00 95 2017-04-27T07:12:00 62 2017-04-27T07:13:00 57 2017-04-27T07:14:00 59 2017-04-27T07:15:00 20 2017-04-27T07:16:00 56 2017-04-27T07:17:00 85 2017-04-27T07:18:00 106 2017-04-27T07:19:00 71 2017-04-27T07:20:00 111 2017-04-27T07:21:00 23 2017-04-27T07:22:00 2 2017-04-27T07:23:00 103 2017-04-27T07:24:00 26 2017-04-27T07:25:00 74 2017-04-27T07:26:00 126 2017-04-27T07:27:00 57 2017-04-27T07:28:00 61 2017-04-27T07:29:00 69 2017-04-27T07:30:00 59 2017-04-27T07:31:00 95 2017-04-27T07:32:00 26 2017-04-27T07:33:00 43 2017-04-27T07:34:00 61 2017-04-27T07:35:00 62 2017-04-27T07:36:00 106 2017-04-27T07:37:00 80 2017-04-27T07:38:00 90 2017-04-27T07:39:00 118 2017-04-27T07:40:00 123 2017-04-27T07:41:00 59 2017-04-27T07:42:00 73 2017-04-27T07:43:00 66 2017-04-27T07:44:00 120 2017-04-27T07:45:00 50 2017-04-27T07:46:00 23 2017-04-27T07:47:00 64 2017-04-27T07:48:00 40 2017-04-27T07:49:00 28 2017-04-27T07:50:00 36 2017-04-27T07:51:00 90 2017-04-27T07:52:00 21 2017-04-27T07:53:00 48 2017-04-27T07:54:00 64 2017-04-27T07:56:00 186 2017-04-27T07:57:00 154 2017-04-27T07:58:00 152 2017-04-27T07:59:00 79 2017-04-27T08:00:00 186 2017-04-27T08:01:00 192 2017-04-27T08:02:00 11 2017-04-27T08:03:00 74 2017-04-27T08:04:00 200 2017-04-27T08:05:00 100 2017-04-27T08:06:00 48 2017-04-27T08:07:00 179 2017-04-27T08:08:00 28 2017-04-27T08:09:00 25 2017-04-27T08:10:00 203 2017-04-27T08:11:00 216 2017-04-27T08:12:00 122 2017-04-27T08:13:00 122 2017-04-27T08:14:00 59 2017-04-27T08:15:00 89 2017-04-27T08:16:00 123 2017-04-27T08:17:00 19 2017-04-27T08:18:00 17 2017-04-27T08:19:00 10 2017-04-27T08:20:00 117 2017-04-27T08:21:00 193 2017-04-27T08:22:00 76 2017-04-27T08:23:00 93 2017-04-27T08:24:00 52 2017-04-27T08:25:00 51 2017-04-27T08:26:00 188 2017-04-27T08:27:00 143 2017-04-27T08:28:00 0 2017-04-27T08:29:00 79 2017-04-27T08:31:00 140 2017-04-27T08:32:00 142 2017-04-27T08:34:00 41 2017-04-27T08:35:00 118 2017-04-27T08:36:00 93 2017-04-27T08:37:00 170 2017-04-27T08:38:00 183 2017-04-27T08:39:00 95 2017-04-27T08:40:00 61 2017-04-27T08:41:00 32 2017-04-27T08:42:00 83 2017-04-27T08:44:00 52 2017-04-27T08:45:00 136 2017-04-27T08:46:00 190 2017-04-27T08:47:00 100 2017-04-27T08:48:00 19 2017-04-27T08:49:00 29 2017-04-27T08:50:00 54 2017-04-27T08:51:00 51 2017-04-27T08:52:00 81 2017-04-27T08:54:00 86 2017-04-27T08:55:00 9 2017-04-27T08:56:00 163 2017-04-27T08:57:00 58 2017-04-27T08:58:00 135 2017-04-27T08:59:00 164 2017-04-27T09:00:00 169 2017-04-27T09:01:00 11 2017-04-27T09:02:00 150 2017-04-27T09:03:00 251 2017-04-27T09:05:00 11 2017-04-27T09:06:00 174 2017-04-27T09:07:00 206 2017-04-27T09:08:00 83 2017-04-27T09:09:00 172 2017-04-27T09:10:00 247 2017-04-27T09:11:00 4 2017-04-27T09:12:00 140 2017-04-27T09:13:00 148 2017-04-27T09:16:00 160 2017-04-27T09:17:00 42 2017-04-27T09:18:00 11 2017-04-27T09:19:00 120 2017-04-27T09:20:00 294 2017-04-27T09:21:00 69 2017-04-27T09:22:00 277 2017-04-27T09:24:00 7 2017-04-27T09:27:00 45 2017-04-27T09:31:00 111 2017-04-27T09:34:00 194 2017-04-27T09:35:00 156 2017-04-27T09:37:00 160 2017-04-27T09:42:00 58 2017-04-27T09:45:00 152 2017-04-27T09:47:00 198 2017-04-27T09:49:00 30 2017-04-27T09:56:00 200 2017-04-27T09:59:00 158
