[
https://issues.apache.org/jira/browse/KAFKA-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105539#comment-15105539
]
Michal Turek edited comment on KAFKA-2985 at 1/18/16 5:19 PM:
--------------------------------------------------------------
I'm probably experiencing the same issue, I found the following in the logs.
{noformat}
2016-01-18 15:40:37.380 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
00355837-e81b-49ab-a62c-9c5d1dd041e2, 71, kafka-test/7/38888
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.380 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
0292119d-ef0f-409e-8ee8-577ead5ddfd3, 34, kafka-test/29/43293
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.381 ERROR o.a.k.c.c.i.ConsumerCoordinator
[AutoCommitConsumer-JDARXqhv-worker-0]: Error UNKNOWN_MEMBER_ID occurred while
committing offsets for group KafkaTestsAutoCommit (ConsumerCoordinator.java:550)
2016-01-18 15:40:37.381 WARN o.a.k.c.c.i.ConsumerCoordinator
[AutoCommitConsumer-JDARXqhv-worker-0]: Auto offset commit failed: Commit
cannot be completed due to group rebalance (ConsumerCoordinator.java:424)
2016-01-18 15:40:37.383 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
84062194-86f0-47a9-8c44-b7dcbe32bac0, 9, kafka-test/16/42101
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.383 ERROR o.a.k.c.c.i.ConsumerCoordinator
[AutoCommitConsumer-JDARXqhv-worker-0]: Error UNKNOWN_MEMBER_ID occurred while
committing offsets for group KafkaTestsAutoCommit (ConsumerCoordinator.java:550)
2016-01-18 15:40:37.384 WARN o.a.k.c.c.i.ConsumerCoordinator
[AutoCommitConsumer-JDARXqhv-worker-0]: Auto offset commit failed:
(ConsumerCoordinator.java:445)
2016-01-18 15:40:37.384 INFO c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Rebalance callback, revoked:
[kafka-test-23] (AutoCommitConsumer.java:87)
2016-01-18 15:40:37.386 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
5c664f3c-9d03-4c3a-9285-7e847e94b2c7, 58, kafka-test/4/39899
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.388 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:00355837-e81b-49ab-a62c-9c5d1dd041e2, 71
(RedisStateDao.java:81)
2016-01-18 15:40:37.388 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
e748f05b-024f-4509-a0c2-7b3c2b762baa, 38, kafka-test/27/34377
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.395 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
ec9d9502-54de-4f38-930a-38c38b9ce583, 59, kafka-test/29/43294
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.398 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
84062194-86f0-47a9-8c44-b7dcbe32bac0, 10, kafka-test/16/42102
(AutoCommitConsumer.java:74)
{noformat}
I'm running our artificial integration test
https://github.com/avast/kafka-tests (critical KAFKA-2978 has been found using
it) with Kafka brokers and clients from 0.9.0 branch (commit
3e1290c6928a6001263acfd7181bd768b231afc0, Wed Dec 30). We have 3 machines for
the tests with 3 brokers in a cluster, single topic with 30 partitions and
replication factor 2. Producers and consumers are randomly created and removed,
min. 5 per machine and max. 15 per machine, change is made every 3 minutes.
Each produced message is verified that it was consumed. I have just started
analysis of the logs and found that rebalance callback "revoked" is not
sometimes logged and "assigned" is instead logged multiple times. Same message
was consumed twice nearly at the same time and it was consumed for the third
time after very long 15 minutes!. About 700 000 of messages were produced
total, but consumer read about 5 000 000 of duplications before I killed the
tests. I have just started the analysis and I'm trying to reproduce the issue
with a smaller setup...
I would attach full logs but they have about 3 * 1.5 GB ;-)
{noformat}
...
2016-01-18 15:39:25.058 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 61, kafka-test/6/34961
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.060 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
fd6f2596-8198-4fd0-8169-e07c58bd4603, 44, kafka-test/5/38644
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.060 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
6c1a6a02-9593-4757-94c3-eb7bfabca4ed, 13, kafka-test/9/38012
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.068 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 62, kafka-test/6/34962
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.070 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
fd6f2596-8198-4fd0-8169-e07c58bd4603, 45, kafka-test/5/38645
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.071 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
6c1a6a02-9593-4757-94c3-eb7bfabca4ed, 14, kafka-test/9/38013
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.080 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 63, kafka-test/6/34963
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 63 (machine 01)
2016-01-18 15:39:25.083 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
fd6f2596-8198-4fd0-8169-e07c58bd4603, 46, kafka-test/5/38646
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.083 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
6c1a6a02-9593-4757-94c3-eb7bfabca4ed, 15, kafka-test/9/38014
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.092 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
3e1700ce-6308-4266-917b-be11a08232af, 96, kafka-test/7/36592
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.095 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
fd6f2596-8198-4fd0-8169-e07c58bd4603, 47, kafka-test/5/38647
(AutoCommitConsumer.java:74)
...
2016-01-18 15:39:29.381 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 13, kafka-test/8/37332
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.385 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 94, kafka-test/7/37090
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.390 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 82, kafka-test/5/38997
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.391 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 14, kafka-test/8/37333
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.394 INFO c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Rebalance callback, assigned:
[kafka-test-23] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Missing "Rebalance callback, revoked"?
2016-01-18 15:39:29.395 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 95, kafka-test/7/37091
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.400 INFO c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Rebalance callback, assigned:
[kafka-test-28] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.403 INFO c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Rebalance callback, assigned:
[kafka-test-13, kafka-test-12] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.403 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 0, kafka-test/28/39200
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.403 INFO c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Rebalance callback, assigned:
[kafka-test-26] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.405 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 15, kafka-test/5/38998
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.406 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
fad65154-dbf7-4320-ba40-728323a2f779, 17, kafka-test/26/30428
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.406 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
933258de-fe01-4a67-b153-510268979262, 47, kafka-test/12/36290
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.408 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 15, kafka-test/8/37334
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.414 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 96, kafka-test/7/37092
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.414 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 48, kafka-test/23/29822
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.419 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 83, kafka-test/5/38999
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.420 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 16, kafka-test/8/37335
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.421 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 1, kafka-test/28/39201
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.421 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
71dff061-ec06-4f6d-98cb-3dc992e2bc88, 11, kafka-test/26/30429
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.422 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
9cd4c002-90c8-4ac6-82aa-e406ab9a34b5, 32, kafka-test/12/36291
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.426 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 97, kafka-test/7/37093
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.428 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 49, kafka-test/23/29823
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.434 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 16, kafka-test/5/39000
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.434 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 17, kafka-test/8/37336
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.435 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 2, kafka-test/28/39202
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.435 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
fad65154-dbf7-4320-ba40-728323a2f779, 18, kafka-test/26/30430
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.436 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
4f7118a5-a199-4e3d-8b7e-1a9b1aa52040, 11, kafka-test/12/36292
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.443 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 98, kafka-test/7/37094
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.444 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 50, kafka-test/23/29824
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.450 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 17, kafka-test/5/39001
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.450 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 18, kafka-test/8/37337
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.451 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 3, kafka-test/28/39203
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.452 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
71dff061-ec06-4f6d-98cb-3dc992e2bc88, 12, kafka-test/26/30431
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.452 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
9cd4c002-90c8-4ac6-82aa-e406ab9a34b5, 33, kafka-test/12/36293
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.459 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 99, kafka-test/7/37095
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.461 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 51, kafka-test/23/29825
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.467 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 84, kafka-test/5/39002
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.468 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 19, kafka-test/8/37338
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.469 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 4, kafka-test/28/39204
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.469 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
fad65154-dbf7-4320-ba40-728323a2f779, 19, kafka-test/26/30432
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.469 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
933258de-fe01-4a67-b153-510268979262, 48, kafka-test/12/36294
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.477 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64, kafka-test/6/34964
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 64 (machine 01)
2016-01-18 15:39:29.478 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 52, kafka-test/23/29826
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.488 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64
(RedisStateDao.java:81) <<<<<<<<<<<<<< message 64 is duplication (machine 01)
2016-01-18 15:39:29.488 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 18, kafka-test/5/39003
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.489 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 20, kafka-test/8/37339
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.490 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 5, kafka-test/28/39205
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.490 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
71dff061-ec06-4f6d-98cb-3dc992e2bc88, 13, kafka-test/26/30433
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.490 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
4f7118a5-a199-4e3d-8b7e-1a9b1aa52040, 12, kafka-test/12/36295
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.497 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 53, kafka-test/23/29827
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.503 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65, kafka-test/6/34965
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 65 (machine 01)
2016-01-18 15:39:29.503 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 85, kafka-test/5/39004
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.503 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 21, kafka-test/8/37340
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.504 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 6, kafka-test/28/39206
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.504 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
fad65154-dbf7-4320-ba40-728323a2f779, 20, kafka-test/26/30434
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.505 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
9cd4c002-90c8-4ac6-82aa-e406ab9a34b5, 34, kafka-test/12/36296
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.510 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65
(RedisStateDao.java:81) <<<<<<<<<<<<<< message 65 is duplication (machine 01)
2016-01-18 15:39:29.512 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 54, kafka-test/23/29828
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.517 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 19, kafka-test/5/39005
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.518 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 22, kafka-test/8/37341
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.519 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 7, kafka-test/28/39207
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.519 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
71dff061-ec06-4f6d-98cb-3dc992e2bc88, 14, kafka-test/26/30435
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.519 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
933258de-fe01-4a67-b153-510268979262, 49, kafka-test/12/36297
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.524 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66, kafka-test/6/34966
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.526 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 55, kafka-test/23/29829
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.531 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66
(RedisStateDao.java:81)
2016-01-18 15:39:29.531 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 86, kafka-test/5/39006
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.532 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 23, kafka-test/8/37342
(AutoCommitConsumer.java:74)
...
------------------------------------------------------------------
...
2016-01-18 15:39:23.980 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
e34dc2db-943a-4d3e-b999-24445737948c, 96, kafka-test/1/35073
(AutoCommitConsumer.java:74)
2016-01-18 15:39:23.998 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
e34dc2db-943a-4d3e-b999-24445737948c, 97, kafka-test/1/35074
(AutoCommitConsumer.java:74)
2016-01-18 15:39:24.013 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
e34dc2db-943a-4d3e-b999-24445737948c, 98, kafka-test/1/35075
(AutoCommitConsumer.java:74)
2016-01-18 15:39:24.023 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
e34dc2db-943a-4d3e-b999-24445737948c, 99, kafka-test/1/35076
(AutoCommitConsumer.java:74)
2016-01-18 15:39:24.055 INFO c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Rebalance callback, revoked:
[kafka-test-1, kafka-test-0] (AutoCommitConsumer.java:87)
2016-01-18 15:39:29.395 INFO c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Rebalance callback, assigned:
[kafka-test-1, kafka-test-0] (AutoCommitConsumer.java:92)
2016-01-18 15:39:29.396 INFO c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Rebalance callback, assigned:
[kafka-test-20] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.397 INFO c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Rebalance callback, assigned:
[kafka-test-24] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.399 INFO c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Rebalance callback, assigned:
[kafka-test-18, kafka-test-19] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.399 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 83, kafka-test/0/40983
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.400 TRACE c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Message consumed:
a0f35c3a-fc6a-4491-9a38-12b00ce1eda0, 0, kafka-test/20/35300
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.403 INFO c.a.k.AutoCommitConsumer.zHh6AHaD
[AutoCommitConsumer-zHh6AHaD-worker-0]: Rebalance callback, assigned:
[kafka-test-9, kafka-test-8] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.403 INFO c.a.k.AutoCommitConsumer.9GWf2Hp9
[AutoCommitConsumer-9GWf2Hp9-worker-0]: Rebalance callback, assigned:
[kafka-test-7, kafka-test-6] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.406 TRACE c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Message consumed:
22964313-8770-42bc-a790-73233c997441, 0, kafka-test/24/35100
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.408 TRACE c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Message consumed:
e59b54a5-2584-4212-b795-89d40da2647a, 0, kafka-test/18/36206
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.409 TRACE c.a.k.AutoCommitConsumer.zHh6AHaD
[AutoCommitConsumer-zHh6AHaD-worker-0]: Message consumed:
423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 37, kafka-test/8/37156
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.415 TRACE c.a.k.AutoCommitConsumer.9GWf2Hp9
[AutoCommitConsumer-9GWf2Hp9-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64, kafka-test/6/34964
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 64 (machine 03)
2016-01-18 15:39:29.417 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-zHh6AHaD-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 37
(RedisStateDao.java:81)
2016-01-18 15:39:29.419 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 84, kafka-test/0/40984
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.419 TRACE c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Message consumed:
a0f35c3a-fc6a-4491-9a38-12b00ce1eda0, 1, kafka-test/20/35301
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.423 TRACE c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Message consumed:
22964313-8770-42bc-a790-73233c997441, 1, kafka-test/24/35101
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.427 TRACE c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Message consumed:
e59b54a5-2584-4212-b795-89d40da2647a, 1, kafka-test/18/36207
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.432 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 85, kafka-test/0/40985
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.432 TRACE c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Message consumed:
a0f35c3a-fc6a-4491-9a38-12b00ce1eda0, 2, kafka-test/20/35302
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.433 TRACE c.a.k.AutoCommitConsumer.9GWf2Hp9
[AutoCommitConsumer-9GWf2Hp9-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65, kafka-test/6/34965
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 65 (machine 03)
2016-01-18 15:39:29.439 TRACE c.a.k.AutoCommitConsumer.zHh6AHaD
[AutoCommitConsumer-zHh6AHaD-worker-0]: Message consumed:
423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 38, kafka-test/8/37157
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.442 TRACE c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Message consumed:
22964313-8770-42bc-a790-73233c997441, 2, kafka-test/24/35102
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.446 TRACE c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Message consumed:
f349d95d-e456-4869-9d5c-b5d25919ab78, 0, kafka-test/18/36208
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.446 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-zHh6AHaD-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 38
(RedisStateDao.java:81)
2016-01-18 15:39:29.448 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 86, kafka-test/0/40986
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.448 TRACE c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Message consumed:
a0f35c3a-fc6a-4491-9a38-12b00ce1eda0, 3, kafka-test/20/35303
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.457 TRACE c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Message consumed:
22964313-8770-42bc-a790-73233c997441, 3, kafka-test/24/35103
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.458 TRACE c.a.k.AutoCommitConsumer.9GWf2Hp9
[AutoCommitConsumer-9GWf2Hp9-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66, kafka-test/6/34966
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.462 TRACE c.a.k.AutoCommitConsumer.zHh6AHaD
[AutoCommitConsumer-zHh6AHaD-worker-0]: Message consumed:
423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 39, kafka-test/8/37158
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.463 TRACE c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Message consumed:
e59b54a5-2584-4212-b795-89d40da2647a, 2, kafka-test/18/36209
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.465 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 87, kafka-test/0/40987
(AutoCommitConsumer.java:74)
...
------------------------------------------------------------------
...
2016-01-18 15:53:05.529 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
36c8b5e8-547b-42ab-9baa-a41ef1bd17c4, 93, kafka-test/23/37783
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.532 TRACE c.a.k.AutoCommitConsumer.F26Cvza0
[AutoCommitConsumer-F26Cvza0-worker-0]: Message consumed:
ade9750c-abad-4d75-a5c5-14f60fe1cdaa, 38, kafka-test/7/43434
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.532 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
4ed79598-c2de-4cfa-8b5b-48d5ffd6f601, 20, kafka-test/1/39797
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.535 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
08e5655c-f07d-4315-910c-ef0205a3cc60, 85, kafka-test/8/42922
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.535 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:36c8b5e8-547b-42ab-9baa-a41ef1bd17c4, 93
(RedisStateDao.java:81)
2016-01-18 15:53:05.542 INFO c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Rebalance callback, assigned:
[kafka-test-1, kafka-test-2, kafka-test-0, kafka-test-9, kafka-test-7,
kafka-test-8, kafka-test-5, kafka-test-6, kafka-test-3, kafka-test-4]
(AutoCommitConsumer.java:92) <<<<<<<<<<<<<< Missing "Rebalance callback,
revoked"?
2016-01-18 15:53:05.546 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:4ed79598-c2de-4cfa-8b5b-48d5ffd6f601, 20
(RedisStateDao.java:81)
2016-01-18 15:53:05.546 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
6884a9e7-bc13-4b3e-b28c-f850e58aaf9b, 80, kafka-test/11/41977
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.546 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-F26Cvza0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:ade9750c-abad-4d75-a5c5-14f60fe1cdaa, 38
(RedisStateDao.java:81)
2016-01-18 15:53:05.549 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
8d62899b-251a-4ff7-9693-903ddac5a941, 16, kafka-test/23/37784
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.550 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
36949bdd-5a38-466e-ac07-f143330050de, 18, kafka-test/8/42923
(AutoCommitConsumer.java:74)
...
2016-01-18 15:53:54.417 TRACE c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Message consumed:
cd09a2d8-b7e4-4388-a7ae-0909fce70653, 12, kafka-test/1/40689
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.420 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 56, kafka-test/7/37152
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.422 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
882a3c46-d84e-4999-9e47-5d05037b473a, 60, kafka-test/23/40434
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.422 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-F26Cvza0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:c8988a07-dd14-4275-9218-57a31970c43d, 18
(RedisStateDao.java:81)
2016-01-18 15:53:54.425 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 70, kafka-test/11/44471
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.426 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 56
(RedisStateDao.java:81)
2016-01-18 15:53:54.426 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-khLIQrfi-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:cd09a2d8-b7e4-4388-a7ae-0909fce70653, 12
(RedisStateDao.java:81)
2016-01-18 15:53:54.430 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 70
(RedisStateDao.java:81)
2016-01-18 15:53:54.431 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:882a3c46-d84e-4999-9e47-5d05037b473a, 60
(RedisStateDao.java:81)
2016-01-18 15:53:54.431 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64, kafka-test/6/34964
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 64 (machine 02)
2016-01-18 15:53:54.435 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 57, kafka-test/7/37153
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.437 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-EGuftZNP-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64
(RedisStateDao.java:81) <<<<<<<<<<<<<< message 64 is duplication, after 15
minutes! (machine 02)
2016-01-18 15:53:54.437 TRACE c.a.k.AutoCommitConsumer.F26Cvza0
[AutoCommitConsumer-F26Cvza0-worker-0]: Message consumed:
c8988a07-dd14-4275-9218-57a31970c43d, 19, kafka-test/4/41953
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.440 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
04ccc3fd-b82e-4ef9-9fe3-c7d28962e0dc, 72, kafka-test/11/44472
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.441 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 57
(RedisStateDao.java:81)
2016-01-18 15:53:54.442 TRACE c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Message consumed:
cd09a2d8-b7e4-4388-a7ae-0909fce70653, 13, kafka-test/1/40690
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.446 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:04ccc3fd-b82e-4ef9-9fe3-c7d28962e0dc, 72
(RedisStateDao.java:81)
2016-01-18 15:53:54.446 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
882a3c46-d84e-4999-9e47-5d05037b473a, 61, kafka-test/23/40435
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.446 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-F26Cvza0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:c8988a07-dd14-4275-9218-57a31970c43d, 19
(RedisStateDao.java:81)
2016-01-18 15:53:54.447 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-khLIQrfi-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:cd09a2d8-b7e4-4388-a7ae-0909fce70653, 13
(RedisStateDao.java:81)
2016-01-18 15:53:54.451 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 58, kafka-test/7/37154
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.454 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:882a3c46-d84e-4999-9e47-5d05037b473a, 61
(RedisStateDao.java:81)
2016-01-18 15:53:54.454 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65, kafka-test/6/34965
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 65 (machine 02)
2016-01-18 15:53:54.456 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 71, kafka-test/11/44473
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.457 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 58
(RedisStateDao.java:81)
2016-01-18 15:53:54.463 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 71
(RedisStateDao.java:81)
2016-01-18 15:53:54.463 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-EGuftZNP-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65
(RedisStateDao.java:81) <<<<<<<<<<<<<< message 65 is duplication, after 15
minutes! (machine 02)
2016-01-18 15:53:54.464 TRACE c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Message consumed:
cd09a2d8-b7e4-4388-a7ae-0909fce70653, 14, kafka-test/1/40691
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.464 TRACE c.a.k.AutoCommitConsumer.F26Cvza0
[AutoCommitConsumer-F26Cvza0-worker-0]: Message consumed:
c8988a07-dd14-4275-9218-57a31970c43d, 20, kafka-test/4/41954
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.467 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 59, kafka-test/7/37155
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.470 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
882a3c46-d84e-4999-9e47-5d05037b473a, 62, kafka-test/23/40436
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.470 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-khLIQrfi-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:cd09a2d8-b7e4-4388-a7ae-0909fce70653, 14
(RedisStateDao.java:81)
2016-01-18 15:53:54.470 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-F26Cvza0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:c8988a07-dd14-4275-9218-57a31970c43d, 20
(RedisStateDao.java:81)
2016-01-18 15:53:54.473 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
04ccc3fd-b82e-4ef9-9fe3-c7d28962e0dc, 73, kafka-test/11/44474
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.473 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 59
(RedisStateDao.java:81)
2016-01-18 15:53:54.478 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:04ccc3fd-b82e-4ef9-9fe3-c7d28962e0dc, 73
(RedisStateDao.java:81)
2016-01-18 15:53:54.478 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:882a3c46-d84e-4999-9e47-5d05037b473a, 62
(RedisStateDao.java:81)
2016-01-18 15:53:54.479 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66, kafka-test/6/34966
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.483 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 60, kafka-test/7/37156
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.485 TRACE c.a.k.AutoCommitConsumer.F26Cvza0
[AutoCommitConsumer-F26Cvza0-worker-0]: Message consumed:
c8988a07-dd14-4275-9218-57a31970c43d, 21, kafka-test/4/41955
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.485 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-EGuftZNP-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66
(RedisStateDao.java:81)
2016-01-18 15:53:54.485 TRACE c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Message consumed:
cd09a2d8-b7e4-4388-a7ae-0909fce70653, 15, kafka-test/1/40692
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.488 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 72, kafka-test/11/44475
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.488 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 60
(RedisStateDao.java:81)
2016-01-18 15:53:54.493 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 72
(RedisStateDao.java:81)
2016-01-18 15:53:54.494 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
882a3c46-d84e-4999-9e47-5d05037b473a, 63, kafka-test/23/40437
(AutoCommitConsumer.java:74)
...
{noformat}
was (Author: tu...@avast.com):
I'm probably experiencing the same issue, I found the following in the logs.
{noformat}
2016-01-18 15:40:37.380 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
00355837-e81b-49ab-a62c-9c5d1dd041e2, 71, kafka-test/7/38888
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.380 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
0292119d-ef0f-409e-8ee8-577ead5ddfd3, 34, kafka-test/29/43293
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.381 ERROR o.a.k.c.c.i.ConsumerCoordinator
[AutoCommitConsumer-JDARXqhv-worker-0]: Error UNKNOWN_MEMBER_ID occurred while
committing offsets for group KafkaTestsAutoCommit (ConsumerCoordinator.java:550)
2016-01-18 15:40:37.381 WARN o.a.k.c.c.i.ConsumerCoordinator
[AutoCommitConsumer-JDARXqhv-worker-0]: Auto offset commit failed: Commit
cannot be completed due to group rebalance (ConsumerCoordinator.java:424)
2016-01-18 15:40:37.383 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
84062194-86f0-47a9-8c44-b7dcbe32bac0, 9, kafka-test/16/42101
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.383 ERROR o.a.k.c.c.i.ConsumerCoordinator
[AutoCommitConsumer-JDARXqhv-worker-0]: Error UNKNOWN_MEMBER_ID occurred while
committing offsets for group KafkaTestsAutoCommit (ConsumerCoordinator.java:550)
2016-01-18 15:40:37.384 WARN o.a.k.c.c.i.ConsumerCoordinator
[AutoCommitConsumer-JDARXqhv-worker-0]: Auto offset commit failed:
(ConsumerCoordinator.java:445)
2016-01-18 15:40:37.384 INFO c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Rebalance callback, revoked:
[kafka-test-23] (AutoCommitConsumer.java:87)
2016-01-18 15:40:37.386 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
5c664f3c-9d03-4c3a-9285-7e847e94b2c7, 58, kafka-test/4/39899
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.388 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:00355837-e81b-49ab-a62c-9c5d1dd041e2, 71
(RedisStateDao.java:81)
2016-01-18 15:40:37.388 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
e748f05b-024f-4509-a0c2-7b3c2b762baa, 38, kafka-test/27/34377
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.395 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
ec9d9502-54de-4f38-930a-38c38b9ce583, 59, kafka-test/29/43294
(AutoCommitConsumer.java:74)
2016-01-18 15:40:37.398 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
84062194-86f0-47a9-8c44-b7dcbe32bac0, 10, kafka-test/16/42102
(AutoCommitConsumer.java:74)
{noformat}
I'm running our artificial integration test
https://github.com/avast/kafka-tests (critical KAFKA-2978 has been found using
it) with Kafka brokers and clients from 0.9.0 branch (commit
3e1290c6928a6001263acfd7181bd768b231afc0, Wed Dec 30). We have 3 machines for
the tests with 3 brokers in a cluster, single topic with 30 partitions and
replication factor 2. Producers and consumers are randomly created and removed,
min. 5 per machine and max. 15 per machine, change is made every 3 minutes.
Each produced message is verified that it was consumed. I have just started
analysis of the logs and found that rebalance callback "revoked" is not
sometimes logged and "assigned" is instead logged multiple times. Same message
was consumed twice nearly at the same time and it was consumed for the third
time after very long 15 minutes (!). About 700 000 of messages were produced
total, but consumer read about 5 000 000 of duplications before I killed the
tests. I have just started the analysis and I'm trying to reproduce the issue
with a smaller setup...
I would attach full logs but they have about 3 * 1.5 GB ;-)
{noformat}
...
2016-01-18 15:39:25.058 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 61, kafka-test/6/34961
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.060 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
fd6f2596-8198-4fd0-8169-e07c58bd4603, 44, kafka-test/5/38644
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.060 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
6c1a6a02-9593-4757-94c3-eb7bfabca4ed, 13, kafka-test/9/38012
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.068 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 62, kafka-test/6/34962
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.070 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
fd6f2596-8198-4fd0-8169-e07c58bd4603, 45, kafka-test/5/38645
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.071 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
6c1a6a02-9593-4757-94c3-eb7bfabca4ed, 14, kafka-test/9/38013
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.080 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 63, kafka-test/6/34963
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< offset 63 (machine 01)
2016-01-18 15:39:25.083 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
fd6f2596-8198-4fd0-8169-e07c58bd4603, 46, kafka-test/5/38646
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.083 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
6c1a6a02-9593-4757-94c3-eb7bfabca4ed, 15, kafka-test/9/38014
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.092 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
3e1700ce-6308-4266-917b-be11a08232af, 96, kafka-test/7/36592
(AutoCommitConsumer.java:74)
2016-01-18 15:39:25.095 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
fd6f2596-8198-4fd0-8169-e07c58bd4603, 47, kafka-test/5/38647
(AutoCommitConsumer.java:74)
...
2016-01-18 15:39:29.381 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 13, kafka-test/8/37332
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.385 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 94, kafka-test/7/37090
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.390 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 82, kafka-test/5/38997
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.391 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 14, kafka-test/8/37333
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.394 INFO c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Rebalance callback, assigned:
[kafka-test-23] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Missing "Rebalance callback, revoked"?
2016-01-18 15:39:29.395 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 95, kafka-test/7/37091
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.400 INFO c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Rebalance callback, assigned:
[kafka-test-28] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.403 INFO c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Rebalance callback, assigned:
[kafka-test-13, kafka-test-12] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.403 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 0, kafka-test/28/39200
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.403 INFO c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Rebalance callback, assigned:
[kafka-test-26] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.405 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 15, kafka-test/5/38998
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.406 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
fad65154-dbf7-4320-ba40-728323a2f779, 17, kafka-test/26/30428
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.406 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
933258de-fe01-4a67-b153-510268979262, 47, kafka-test/12/36290
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.408 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 15, kafka-test/8/37334
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.414 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 96, kafka-test/7/37092
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.414 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 48, kafka-test/23/29822
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.419 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 83, kafka-test/5/38999
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.420 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 16, kafka-test/8/37335
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.421 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 1, kafka-test/28/39201
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.421 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
71dff061-ec06-4f6d-98cb-3dc992e2bc88, 11, kafka-test/26/30429
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.422 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
9cd4c002-90c8-4ac6-82aa-e406ab9a34b5, 32, kafka-test/12/36291
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.426 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 97, kafka-test/7/37093
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.428 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 49, kafka-test/23/29823
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.434 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 16, kafka-test/5/39000
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.434 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 17, kafka-test/8/37336
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.435 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 2, kafka-test/28/39202
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.435 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
fad65154-dbf7-4320-ba40-728323a2f779, 18, kafka-test/26/30430
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.436 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
4f7118a5-a199-4e3d-8b7e-1a9b1aa52040, 11, kafka-test/12/36292
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.443 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 98, kafka-test/7/37094
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.444 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 50, kafka-test/23/29824
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.450 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 17, kafka-test/5/39001
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.450 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 18, kafka-test/8/37337
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.451 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 3, kafka-test/28/39203
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.452 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
71dff061-ec06-4f6d-98cb-3dc992e2bc88, 12, kafka-test/26/30431
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.452 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
9cd4c002-90c8-4ac6-82aa-e406ab9a34b5, 33, kafka-test/12/36293
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.459 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
7041f37c-499d-473e-abd2-a42639b33f60, 99, kafka-test/7/37095
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.461 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 51, kafka-test/23/29825
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.467 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 84, kafka-test/5/39002
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.468 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 19, kafka-test/8/37338
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.469 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 4, kafka-test/28/39204
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.469 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
fad65154-dbf7-4320-ba40-728323a2f779, 19, kafka-test/26/30432
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.469 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
933258de-fe01-4a67-b153-510268979262, 48, kafka-test/12/36294
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.477 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64, kafka-test/6/34964
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 64 (machine 01)
2016-01-18 15:39:29.478 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 52, kafka-test/23/29826
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.488 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64
(RedisStateDao.java:81) <<<<<<<<<<<<<< message 64 is duplication (machine 01)
2016-01-18 15:39:29.488 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 18, kafka-test/5/39003
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.489 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 20, kafka-test/8/37339
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.490 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 5, kafka-test/28/39205
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.490 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
71dff061-ec06-4f6d-98cb-3dc992e2bc88, 13, kafka-test/26/30433
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.490 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
4f7118a5-a199-4e3d-8b7e-1a9b1aa52040, 12, kafka-test/12/36295
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.497 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 53, kafka-test/23/29827
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.503 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65, kafka-test/6/34965
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 65 (machine 01)
2016-01-18 15:39:29.503 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 85, kafka-test/5/39004
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.503 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 21, kafka-test/8/37340
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.504 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 6, kafka-test/28/39206
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.504 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
fad65154-dbf7-4320-ba40-728323a2f779, 20, kafka-test/26/30434
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.505 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
9cd4c002-90c8-4ac6-82aa-e406ab9a34b5, 34, kafka-test/12/36296
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.510 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65
(RedisStateDao.java:81) <<<<<<<<<<<<<< message 65 is duplication (machine 01)
2016-01-18 15:39:29.512 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 54, kafka-test/23/29828
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.517 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
1bf39a85-be30-4a73-93ae-5acc5f9ffc4d, 19, kafka-test/5/39005
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.518 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 22, kafka-test/8/37341
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.519 TRACE c.a.k.AutoCommitConsumer.MpWr175j
[AutoCommitConsumer-MpWr175j-worker-0]: Message consumed:
8a3056e5-aff1-4ae0-a101-e02a16134ff6, 7, kafka-test/28/39207
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.519 TRACE c.a.k.AutoCommitConsumer.3V4DKI4g
[AutoCommitConsumer-3V4DKI4g-worker-0]: Message consumed:
71dff061-ec06-4f6d-98cb-3dc992e2bc88, 14, kafka-test/26/30435
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.519 TRACE c.a.k.AutoCommitConsumer.jrJlnDvc
[AutoCommitConsumer-jrJlnDvc-worker-0]: Message consumed:
933258de-fe01-4a67-b153-510268979262, 49, kafka-test/12/36297
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.524 TRACE c.a.k.AutoCommitConsumer.0iz7YzoQ
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66, kafka-test/6/34966
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.526 TRACE c.a.k.AutoCommitConsumer.JDARXqhv
[AutoCommitConsumer-JDARXqhv-worker-0]: Message consumed:
a994a5d0-d3ac-4e71-bf74-6da4656b280f, 55, kafka-test/23/29829
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.531 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-0iz7YzoQ-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66
(RedisStateDao.java:81)
2016-01-18 15:39:29.531 TRACE c.a.k.AutoCommitConsumer.G6UvNiyb
[AutoCommitConsumer-G6UvNiyb-worker-0]: Message consumed:
ea0bbd86-e303-4989-9f1c-37e290bb08ea, 86, kafka-test/5/39006
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.532 TRACE c.a.k.AutoCommitConsumer.PbB13rFA
[AutoCommitConsumer-PbB13rFA-worker-0]: Message consumed:
2250aadb-6695-4ef2-9e38-beaaa3d04ba2, 23, kafka-test/8/37342
(AutoCommitConsumer.java:74)
...
------------------------------------------------------------------
...
2016-01-18 15:39:23.980 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
e34dc2db-943a-4d3e-b999-24445737948c, 96, kafka-test/1/35073
(AutoCommitConsumer.java:74)
2016-01-18 15:39:23.998 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
e34dc2db-943a-4d3e-b999-24445737948c, 97, kafka-test/1/35074
(AutoCommitConsumer.java:74)
2016-01-18 15:39:24.013 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
e34dc2db-943a-4d3e-b999-24445737948c, 98, kafka-test/1/35075
(AutoCommitConsumer.java:74)
2016-01-18 15:39:24.023 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
e34dc2db-943a-4d3e-b999-24445737948c, 99, kafka-test/1/35076
(AutoCommitConsumer.java:74)
2016-01-18 15:39:24.055 INFO c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Rebalance callback, revoked:
[kafka-test-1, kafka-test-0] (AutoCommitConsumer.java:87)
2016-01-18 15:39:29.395 INFO c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Rebalance callback, assigned:
[kafka-test-1, kafka-test-0] (AutoCommitConsumer.java:92)
2016-01-18 15:39:29.396 INFO c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Rebalance callback, assigned:
[kafka-test-20] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.397 INFO c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Rebalance callback, assigned:
[kafka-test-24] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.399 INFO c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Rebalance callback, assigned:
[kafka-test-18, kafka-test-19] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.399 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 83, kafka-test/0/40983
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.400 TRACE c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Message consumed:
a0f35c3a-fc6a-4491-9a38-12b00ce1eda0, 0, kafka-test/20/35300
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.403 INFO c.a.k.AutoCommitConsumer.zHh6AHaD
[AutoCommitConsumer-zHh6AHaD-worker-0]: Rebalance callback, assigned:
[kafka-test-9, kafka-test-8] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.403 INFO c.a.k.AutoCommitConsumer.9GWf2Hp9
[AutoCommitConsumer-9GWf2Hp9-worker-0]: Rebalance callback, assigned:
[kafka-test-7, kafka-test-6] (AutoCommitConsumer.java:92)
<<<<<<<<<<<<<< Multiple "Rebalance callback, assigned"
2016-01-18 15:39:29.406 TRACE c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Message consumed:
22964313-8770-42bc-a790-73233c997441, 0, kafka-test/24/35100
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.408 TRACE c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Message consumed:
e59b54a5-2584-4212-b795-89d40da2647a, 0, kafka-test/18/36206
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.409 TRACE c.a.k.AutoCommitConsumer.zHh6AHaD
[AutoCommitConsumer-zHh6AHaD-worker-0]: Message consumed:
423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 37, kafka-test/8/37156
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.415 TRACE c.a.k.AutoCommitConsumer.9GWf2Hp9
[AutoCommitConsumer-9GWf2Hp9-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64, kafka-test/6/34964
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 64 (machine 03)
2016-01-18 15:39:29.417 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-zHh6AHaD-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 37
(RedisStateDao.java:81)
2016-01-18 15:39:29.419 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 84, kafka-test/0/40984
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.419 TRACE c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Message consumed:
a0f35c3a-fc6a-4491-9a38-12b00ce1eda0, 1, kafka-test/20/35301
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.423 TRACE c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Message consumed:
22964313-8770-42bc-a790-73233c997441, 1, kafka-test/24/35101
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.427 TRACE c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Message consumed:
e59b54a5-2584-4212-b795-89d40da2647a, 1, kafka-test/18/36207
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.432 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 85, kafka-test/0/40985
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.432 TRACE c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Message consumed:
a0f35c3a-fc6a-4491-9a38-12b00ce1eda0, 2, kafka-test/20/35302
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.433 TRACE c.a.k.AutoCommitConsumer.9GWf2Hp9
[AutoCommitConsumer-9GWf2Hp9-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65, kafka-test/6/34965
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 65 (machine 03)
2016-01-18 15:39:29.439 TRACE c.a.k.AutoCommitConsumer.zHh6AHaD
[AutoCommitConsumer-zHh6AHaD-worker-0]: Message consumed:
423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 38, kafka-test/8/37157
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.442 TRACE c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Message consumed:
22964313-8770-42bc-a790-73233c997441, 2, kafka-test/24/35102
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.446 TRACE c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Message consumed:
f349d95d-e456-4869-9d5c-b5d25919ab78, 0, kafka-test/18/36208
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.446 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-zHh6AHaD-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 38
(RedisStateDao.java:81)
2016-01-18 15:39:29.448 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 86, kafka-test/0/40986
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.448 TRACE c.a.k.AutoCommitConsumer.KqtDMWde
[AutoCommitConsumer-KqtDMWde-worker-0]: Message consumed:
a0f35c3a-fc6a-4491-9a38-12b00ce1eda0, 3, kafka-test/20/35303
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.457 TRACE c.a.k.AutoCommitConsumer.v5nsAHAG
[AutoCommitConsumer-v5nsAHAG-worker-0]: Message consumed:
22964313-8770-42bc-a790-73233c997441, 3, kafka-test/24/35103
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.458 TRACE c.a.k.AutoCommitConsumer.9GWf2Hp9
[AutoCommitConsumer-9GWf2Hp9-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66, kafka-test/6/34966
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.462 TRACE c.a.k.AutoCommitConsumer.zHh6AHaD
[AutoCommitConsumer-zHh6AHaD-worker-0]: Message consumed:
423eb17c-4a4e-4559-aa91-ca46dcc1f5c3, 39, kafka-test/8/37158
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.463 TRACE c.a.k.AutoCommitConsumer.ZicrSnNN
[AutoCommitConsumer-ZicrSnNN-worker-0]: Message consumed:
e59b54a5-2584-4212-b795-89d40da2647a, 2, kafka-test/18/36209
(AutoCommitConsumer.java:74)
2016-01-18 15:39:29.465 TRACE c.a.k.AutoCommitConsumer.1U0dWzgq
[AutoCommitConsumer-1U0dWzgq-worker-0]: Message consumed:
be8da206-01e2-415c-84f9-ebc3afb2df86, 87, kafka-test/0/40987
(AutoCommitConsumer.java:74)
...
------------------------------------------------------------------
...
2016-01-18 15:53:05.529 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
36c8b5e8-547b-42ab-9baa-a41ef1bd17c4, 93, kafka-test/23/37783
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.532 TRACE c.a.k.AutoCommitConsumer.F26Cvza0
[AutoCommitConsumer-F26Cvza0-worker-0]: Message consumed:
ade9750c-abad-4d75-a5c5-14f60fe1cdaa, 38, kafka-test/7/43434
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.532 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
4ed79598-c2de-4cfa-8b5b-48d5ffd6f601, 20, kafka-test/1/39797
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.535 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
08e5655c-f07d-4315-910c-ef0205a3cc60, 85, kafka-test/8/42922
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.535 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:36c8b5e8-547b-42ab-9baa-a41ef1bd17c4, 93
(RedisStateDao.java:81)
2016-01-18 15:53:05.542 INFO c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Rebalance callback, assigned:
[kafka-test-1, kafka-test-2, kafka-test-0, kafka-test-9, kafka-test-7,
kafka-test-8, kafka-test-5, kafka-test-6, kafka-test-3, kafka-test-4]
(AutoCommitConsumer.java:92) <<<<<<<<<<<<<< Missing "Rebalance callback,
revoked"?
2016-01-18 15:53:05.546 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:4ed79598-c2de-4cfa-8b5b-48d5ffd6f601, 20
(RedisStateDao.java:81)
2016-01-18 15:53:05.546 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
6884a9e7-bc13-4b3e-b28c-f850e58aaf9b, 80, kafka-test/11/41977
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.546 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-F26Cvza0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:ade9750c-abad-4d75-a5c5-14f60fe1cdaa, 38
(RedisStateDao.java:81)
2016-01-18 15:53:05.549 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
8d62899b-251a-4ff7-9693-903ddac5a941, 16, kafka-test/23/37784
(AutoCommitConsumer.java:74)
2016-01-18 15:53:05.550 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
36949bdd-5a38-466e-ac07-f143330050de, 18, kafka-test/8/42923
(AutoCommitConsumer.java:74)
...
2016-01-18 15:53:54.417 TRACE c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Message consumed:
cd09a2d8-b7e4-4388-a7ae-0909fce70653, 12, kafka-test/1/40689
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.420 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 56, kafka-test/7/37152
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.422 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
882a3c46-d84e-4999-9e47-5d05037b473a, 60, kafka-test/23/40434
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.422 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-F26Cvza0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:c8988a07-dd14-4275-9218-57a31970c43d, 18
(RedisStateDao.java:81)
2016-01-18 15:53:54.425 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 70, kafka-test/11/44471
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.426 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 56
(RedisStateDao.java:81)
2016-01-18 15:53:54.426 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-khLIQrfi-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:cd09a2d8-b7e4-4388-a7ae-0909fce70653, 12
(RedisStateDao.java:81)
2016-01-18 15:53:54.430 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 70
(RedisStateDao.java:81)
2016-01-18 15:53:54.431 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:882a3c46-d84e-4999-9e47-5d05037b473a, 60
(RedisStateDao.java:81)
2016-01-18 15:53:54.431 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64, kafka-test/6/34964
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 64 (machine 02)
2016-01-18 15:53:54.435 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 57, kafka-test/7/37153
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.437 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-EGuftZNP-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 64
(RedisStateDao.java:81) <<<<<<<<<<<<<< message 64 is duplication, after 15
minutes! (machine 02)
2016-01-18 15:53:54.437 TRACE c.a.k.AutoCommitConsumer.F26Cvza0
[AutoCommitConsumer-F26Cvza0-worker-0]: Message consumed:
c8988a07-dd14-4275-9218-57a31970c43d, 19, kafka-test/4/41953
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.440 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
04ccc3fd-b82e-4ef9-9fe3-c7d28962e0dc, 72, kafka-test/11/44472
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.441 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 57
(RedisStateDao.java:81)
2016-01-18 15:53:54.442 TRACE c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Message consumed:
cd09a2d8-b7e4-4388-a7ae-0909fce70653, 13, kafka-test/1/40690
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.446 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:04ccc3fd-b82e-4ef9-9fe3-c7d28962e0dc, 72
(RedisStateDao.java:81)
2016-01-18 15:53:54.446 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
882a3c46-d84e-4999-9e47-5d05037b473a, 61, kafka-test/23/40435
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.446 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-F26Cvza0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:c8988a07-dd14-4275-9218-57a31970c43d, 19
(RedisStateDao.java:81)
2016-01-18 15:53:54.447 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-khLIQrfi-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:cd09a2d8-b7e4-4388-a7ae-0909fce70653, 13
(RedisStateDao.java:81)
2016-01-18 15:53:54.451 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 58, kafka-test/7/37154
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.454 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:882a3c46-d84e-4999-9e47-5d05037b473a, 61
(RedisStateDao.java:81)
2016-01-18 15:53:54.454 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65, kafka-test/6/34965
(AutoCommitConsumer.java:74) <<<<<<<<<<<<<< message 65 (machine 03)
2016-01-18 15:53:54.456 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 71, kafka-test/11/44473
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.457 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 58
(RedisStateDao.java:81)
2016-01-18 15:53:54.463 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 71
(RedisStateDao.java:81)
2016-01-18 15:53:54.463 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-EGuftZNP-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 65
(RedisStateDao.java:81) <<<<<<<<<<<<<< message 65 is duplication, after 15
minutes! (machine 02)
2016-01-18 15:53:54.464 TRACE c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Message consumed:
cd09a2d8-b7e4-4388-a7ae-0909fce70653, 14, kafka-test/1/40691
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.464 TRACE c.a.k.AutoCommitConsumer.F26Cvza0
[AutoCommitConsumer-F26Cvza0-worker-0]: Message consumed:
c8988a07-dd14-4275-9218-57a31970c43d, 20, kafka-test/4/41954
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.467 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 59, kafka-test/7/37155
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.470 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
882a3c46-d84e-4999-9e47-5d05037b473a, 62, kafka-test/23/40436
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.470 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-khLIQrfi-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:cd09a2d8-b7e4-4388-a7ae-0909fce70653, 14
(RedisStateDao.java:81)
2016-01-18 15:53:54.470 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-F26Cvza0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:c8988a07-dd14-4275-9218-57a31970c43d, 20
(RedisStateDao.java:81)
2016-01-18 15:53:54.473 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
04ccc3fd-b82e-4ef9-9fe3-c7d28962e0dc, 73, kafka-test/11/44474
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.473 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 59
(RedisStateDao.java:81)
2016-01-18 15:53:54.478 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:04ccc3fd-b82e-4ef9-9fe3-c7d28962e0dc, 73
(RedisStateDao.java:81)
2016-01-18 15:53:54.478 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:882a3c46-d84e-4999-9e47-5d05037b473a, 62
(RedisStateDao.java:81)
2016-01-18 15:53:54.479 TRACE c.a.k.AutoCommitConsumer.EGuftZNP
[AutoCommitConsumer-EGuftZNP-worker-0]: Message consumed:
e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66, kafka-test/6/34966
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.483 TRACE c.a.k.AutoCommitConsumer.g6pSRtp0
[AutoCommitConsumer-g6pSRtp0-worker-0]: Message consumed:
3565c8f1-73f7-4be1-be21-1fb42333d11f, 60, kafka-test/7/37156
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.485 TRACE c.a.k.AutoCommitConsumer.F26Cvza0
[AutoCommitConsumer-F26Cvza0-worker-0]: Message consumed:
c8988a07-dd14-4275-9218-57a31970c43d, 21, kafka-test/4/41955
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.485 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-EGuftZNP-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:e6e55abe-60d3-45a7-9b6f-2d9726e23088, 66
(RedisStateDao.java:81)
2016-01-18 15:53:54.485 TRACE c.a.k.AutoCommitConsumer.khLIQrfi
[AutoCommitConsumer-khLIQrfi-worker-0]: Message consumed:
cd09a2d8-b7e4-4388-a7ae-0909fce70653, 15, kafka-test/1/40692
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.488 TRACE c.a.k.AutoCommitConsumer.GoN5xCst
[AutoCommitConsumer-GoN5xCst-worker-0]: Message consumed:
8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 72, kafka-test/11/44475
(AutoCommitConsumer.java:74)
2016-01-18 15:53:54.488 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-g6pSRtp0-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:3565c8f1-73f7-4be1-be21-1fb42333d11f, 60
(RedisStateDao.java:81)
2016-01-18 15:53:54.493 WARN com.avast.kafkatests.RedisStateDao
[AutoCommitConsumer-GoN5xCst-worker-0]: Duplication:
duplications_consume:autocommit,
consume:autocommit:8be2991f-a4f0-4ae1-bdde-cc34be2b9661, 72
(RedisStateDao.java:81)
2016-01-18 15:53:54.494 TRACE c.a.k.AutoCommitConsumer.nqWHI5Rc
[AutoCommitConsumer-nqWHI5Rc-worker-0]: Message consumed:
882a3c46-d84e-4999-9e47-5d05037b473a, 63, kafka-test/23/40437
(AutoCommitConsumer.java:74)
...
{noformat}
> Consumer group stuck in rebalancing state
> -----------------------------------------
>
> Key: KAFKA-2985
> URL: https://issues.apache.org/jira/browse/KAFKA-2985
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.9.0.0
> Environment: Kafka 0.9.0.0.
> Kafka Java consumer 0.9.0.0
> 2 Java producers.
> 3 Java consumers using the new consumer API.
> 2 kafka brokers.
> Reporter: Jens Rantil
> Assignee: Jason Gustafson
>
> We've doing some load testing on Kafka. _After_ the load test when our
> consumers and have two times now seen Kafka become stuck in consumer group
> rebalancing. This is after all our consumers are done consuming and
> essentially polling periodically without getting any records.
> The brokers list the consumer group (named "default"), but I can't query the
> offsets:
> {noformat}
> jrantil@queue-0:/srv/kafka/kafka$ ./bin/kafka-consumer-groups.sh
> --new-consumer --bootstrap-server localhost:9092 --list
> default
> jrantil@queue-0:/srv/kafka/kafka$ ./bin/kafka-consumer-groups.sh
> --new-consumer --bootstrap-server localhost:9092 --describe --group
> default|sort
> Consumer group `default` does not exist or is rebalancing.
> {noformat}
> Retrying to query the offsets for 15 minutes or so still said it was
> rebalancing. After restarting our first broker, the group immediately started
> rebalancing. That broker was logging this before restart:
> {noformat}
> [2015-12-12 13:09:48,517] INFO [Group Metadata Manager on Broker 0]: Removed
> 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
> [2015-12-12 13:10:16,139] INFO [GroupCoordinator 0]: Stabilized group default
> generation 16 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:10:16,141] INFO [GroupCoordinator 0]: Assignment received from
> leader for group default for generation 16
> (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:10:16,575] INFO [GroupCoordinator 0]: Preparing to restabilize
> group default with old generation 16 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:11:15,141] INFO [GroupCoordinator 0]: Stabilized group default
> generation 17 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:11:15,143] INFO [GroupCoordinator 0]: Assignment received from
> leader for group default for generation 17
> (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:11:15,314] INFO [GroupCoordinator 0]: Preparing to restabilize
> group default with old generation 17 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:12:14,144] INFO [GroupCoordinator 0]: Stabilized group default
> generation 18 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:12:14,145] INFO [GroupCoordinator 0]: Assignment received from
> leader for group default for generation 18
> (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:12:14,340] INFO [GroupCoordinator 0]: Preparing to restabilize
> group default with old generation 18 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:13:13,146] INFO [GroupCoordinator 0]: Stabilized group default
> generation 19 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:13:13,148] INFO [GroupCoordinator 0]: Assignment received from
> leader for group default for generation 19
> (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:13:13,238] INFO [GroupCoordinator 0]: Preparing to restabilize
> group default with old generation 19 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:14:12,148] INFO [GroupCoordinator 0]: Stabilized group default
> generation 20 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:14:12,149] INFO [GroupCoordinator 0]: Assignment received from
> leader for group default for generation 20
> (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:14:12,360] INFO [GroupCoordinator 0]: Preparing to restabilize
> group default with old generation 20 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:15:11,150] INFO [GroupCoordinator 0]: Stabilized group default
> generation 21 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:15:11,152] INFO [GroupCoordinator 0]: Assignment received from
> leader for group default for generation 21
> (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:15:11,217] INFO [GroupCoordinator 0]: Preparing to restabilize
> group default with old generation 21 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:16:10,152] INFO [GroupCoordinator 0]: Stabilized group default
> generation 22 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:16:10,154] INFO [GroupCoordinator 0]: Assignment received from
> leader for group default for generation 22
> (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:16:10,339] INFO [GroupCoordinator 0]: Preparing to restabilize
> group default with old generation 22 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:17:09,155] INFO [GroupCoordinator 0]: Stabilized group default
> generation 23 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:17:09,157] INFO [GroupCoordinator 0]: Assignment received from
> leader for group default for generation 23
> (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:17:09,262] INFO [GroupCoordinator 0]: Preparing to restabilize
> group default with old generation 23 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:18:08,157] INFO [GroupCoordinator 0]: Stabilized group default
> generation 24 (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:18:08,159] INFO [GroupCoordinator 0]: Assignment received from
> leader for group default for generation 24
> (kafka.coordinator.GroupCoordinator)
> [2015-12-12 13:18:08,333] INFO [GroupCoordinator 0]: Preparing to restabilize
> group default with old generation 24 (kafka.coordinator.GroupCoordinator)
> {noformat}
> Our consumers were logging:
> {noformat}
> Dec 12 13:09:17 X.X.X.110 system[27782]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator Marking the
> coordinator 2147483647 dead.
> Dec 12 13:09:17 X.X.X.110 system[27782]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Error
> UNKNOWN_MEMBER_ID occurred while committing offsets for group default
> Dec 12 13:09:17 X.X.X.110 system[27782]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Auto offset
> commit failed: Commit cannot be completed due to group rebalance
> Dec 12 13:09:17 X.X.X.144 system[9915]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator Marking the
> coordinator 2147483647 dead.
> Dec 12 13:09:17 X.X.X.144 system[9915]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Error
> UNKNOWN_MEMBER_ID occurred while committing offsets for group default
> Dec 12 13:09:17 X.X.X.144 system[9915]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Auto offset
> commit failed: Commit cannot be completed due to group rebalance
> Dec 12 13:09:17 X.X.X.110 system[27782]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Error
> UNKNOWN_MEMBER_ID occurred while committing offsets for group default
> Dec 12 13:09:17 X.X.X.110 system[27782]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Auto offset
> commit failed:
> Dec 12 13:09:17 X.X.X.110 system[27782]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator Attempt to
> join group default failed due to unknown member id, resetting and retrying.
> Dec 12 13:09:17 X.X.X.144 system[9915]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Error
> UNKNOWN_MEMBER_ID occurred while committing offsets for group default
> Dec 12 13:09:17 X.X.X.144 system[9915]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Auto offset
> commit failed:
> Dec 12 13:09:17 X.X.X.144 system[9915]: [KafkaTaskExecutorConsumer]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator Attempt to
> join group default failed due to unknown member id, resetting and retrying.
> {noformat}
> I understand that the broker might start rebalancing if my consumers hasn't
> reported heartbeat in session timeout. This might well have happened during
> my load test. However, the issue here is that the rebalancing doesn't
> stabilize/finish after the load test is done.
> Let me know if I can be of any assistance to track this down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
