Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0
Hi All, We are currently trying to migrate Confluent replicator to Apache Open Source Mirror Maker v2.0. We are facing an issue where the messages which are already replicated by replicator is getting replicated again when the mirror maker is started on the same topic. This should not happen as messages are getting duplicated at the target cluster. Here are more details: 1. RCA: replicator assign a consumer group for replicating messages. This consumer group maintains the offset of the source topic. But we are not able to assign same consumer group to the Consumer config in mirror maker 2. 2. Mirror Maker 1.0 : working as same consumer group can be assigned in consumer.properties file and the messages are picked right after where replicator was stopped. 3. Tried running and configuring source.cluster.consumer.group.id in mirror maker 2.0 in all available options (in cluster mode, in connect-standalone and connect-distributed mode) but mirror maker 2.0 is assigning consumer group id as null while replicating messages. Any pointers if anyone has done same and tried to maintain the same offset with mirror maker 2.0. Thanks and regards, Amit This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
[VOTE] 2.6.1 RC2
Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 2.6.1. Since RC1, the following JIRAs have been fixed: KAFKA-10758 Release notes for the 2.6.1 release: https://home.apache.org/~mimaison/kafka-2.6.1-rc2/RELEASE_NOTES.html *** Please download, test and vote by Wednesday, December 2, 5PM PT Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~mimaison/kafka-2.6.1-rc2/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~mimaison/kafka-2.6.1-rc2/javadoc/ * Tag to be voted upon (off 2.6 branch) is the 2.6.1 tag: https://github.com/apache/kafka/releases/tag/2.6.1-rc2 * Documentation: https://kafka.apache.org/26/documentation.html * Protocol: https://kafka.apache.org/26/protocol.html * Successful Jenkins builds for the 2.6 branch: Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/55/ /** Thanks, Mickael
Re: [VOTE] 2.7.0 RC3
Hi Tom, Thanks for reviewing the docs and providing the fix. I agree that this doesn't necessarily meet the bar for a new RC, we'll fix the 2.7 HTML -Bill On Wed, Nov 25, 2020 at 11:17 AM Tom Bentley wrote: > Hi Bill, > > One very minor problem I just spotted is > https://kafka.apache.org/27/documentation.html#brokerconfigs_listeners, > because the tag is not properly closed the HTML doesn't render > property (in Chromium and Firefox at least) and all the rest of the configs > are shown in a monospaced font. I opened a PR > https://github.com/apache/kafka/pull/9655, but unless there's some other > reason for an RC4 it might be better to just fix the generated HTML for 2.7 > > Kind regards, > > Tom > > On Wed, Nov 25, 2020 at 3:37 PM Bill Bejeck wrote: > > > This is the fourth candidate for the release of Apache Kafka 2.7.0. > > > > This is a major release that includes many new features, including: > > > > * Configurable TCP connection timeout and improve the initial metadata > > fetch > > * Enforce broker-wide and per-listener connection creation rate (KIP-612, > > part 1) > > * Throttle Create Topic, Create Partition and Delete Topic Operations > > * Add TRACE-level end-to-end latency metrics to Streams > > * Add Broker-side SCRAM Config API > > * Support PEM format for SSL certificates and private key > > * Add RocksDB Memory Consumption to RocksDB Metrics > > * Add Sliding-Window support for Aggregations > > > > This release also includes a few other features, 53 improvements, and 84 > > bug fixes. > > > > Release notes for the 2.7.0 release: > > https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/RELEASE_NOTES.html > > > > *** Please download, test and vote by Wednesday, December 2, 12PM ET > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > https://kafka.apache.org/KEYS > > > > * Release artifacts to be voted upon (source and binary): > > https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/ > > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > * Javadoc: > > https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/javadoc/ > > > > * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag: > > https://github.com/apache/kafka/releases/tag/2.7.0-rc3 > > > > * Documentation: > > https://kafka.apache.org/27/documentation.html > > > > * Protocol: > > https://kafka.apache.org/27/protocol.html > > > > * Successful Jenkins builds for the 2.7 branch: > > Unit/integration tests: (link to follow) > > System tests: (link to follow) > > > > Thanks, > > Bill > > >
Re: [VOTE] 2.7.0 RC3
Hi Bill, One very minor problem I just spotted is https://kafka.apache.org/27/documentation.html#brokerconfigs_listeners, because the tag is not properly closed the HTML doesn't render property (in Chromium and Firefox at least) and all the rest of the configs are shown in a monospaced font. I opened a PR https://github.com/apache/kafka/pull/9655, but unless there's some other reason for an RC4 it might be better to just fix the generated HTML for 2.7 Kind regards, Tom On Wed, Nov 25, 2020 at 3:37 PM Bill Bejeck wrote: > This is the fourth candidate for the release of Apache Kafka 2.7.0. > > This is a major release that includes many new features, including: > > * Configurable TCP connection timeout and improve the initial metadata > fetch > * Enforce broker-wide and per-listener connection creation rate (KIP-612, > part 1) > * Throttle Create Topic, Create Partition and Delete Topic Operations > * Add TRACE-level end-to-end latency metrics to Streams > * Add Broker-side SCRAM Config API > * Support PEM format for SSL certificates and private key > * Add RocksDB Memory Consumption to RocksDB Metrics > * Add Sliding-Window support for Aggregations > > This release also includes a few other features, 53 improvements, and 84 > bug fixes. > > Release notes for the 2.7.0 release: > https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/RELEASE_NOTES.html > > *** Please download, test and vote by Wednesday, December 2, 12PM ET > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Release artifacts to be voted upon (source and binary): > https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/ > > * Maven artifacts to be voted upon: > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > * Javadoc: > https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/javadoc/ > > * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag: > https://github.com/apache/kafka/releases/tag/2.7.0-rc3 > > * Documentation: > https://kafka.apache.org/27/documentation.html > > * Protocol: > https://kafka.apache.org/27/protocol.html > > * Successful Jenkins builds for the 2.7 branch: > Unit/integration tests: (link to follow) > System tests: (link to follow) > > Thanks, > Bill >
[VOTE] 2.7.0 RC3
This is the fourth candidate for the release of Apache Kafka 2.7.0. This is a major release that includes many new features, including: * Configurable TCP connection timeout and improve the initial metadata fetch * Enforce broker-wide and per-listener connection creation rate (KIP-612, part 1) * Throttle Create Topic, Create Partition and Delete Topic Operations * Add TRACE-level end-to-end latency metrics to Streams * Add Broker-side SCRAM Config API * Support PEM format for SSL certificates and private key * Add RocksDB Memory Consumption to RocksDB Metrics * Add Sliding-Window support for Aggregations This release also includes a few other features, 53 improvements, and 84 bug fixes. Release notes for the 2.7.0 release: https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/RELEASE_NOTES.html *** Please download, test and vote by Wednesday, December 2, 12PM ET Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~bbejeck/kafka-2.7.0-rc3/javadoc/ * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag: https://github.com/apache/kafka/releases/tag/2.7.0-rc3 * Documentation: https://kafka.apache.org/27/documentation.html * Protocol: https://kafka.apache.org/27/protocol.html * Successful Jenkins builds for the 2.7 branch: Unit/integration tests: (link to follow) System tests: (link to follow) Thanks, Bill
Re: Many dups received by consumer (kafka_2.13)
Liam, many thanks! We already jumped into v.2.6.0 :) I appreciate your help
Regards,
Den
ср, 25 нояб. 2020 г. в 15:10, Liam Clarke-Hutchinson <
liam.cla...@adscale.co.nz>:
> Can you upgrade Kafka to 2.5.1? This problem was fixed in that release.
> https://issues.apache.org/jira/browse/KAFKA-9839
>
> On Wed, Nov 25, 2020 at 9:44 PM Dev Op wrote:
>
> > Hello community! Hasn't anyone faced a similar problem? I see nobody can
> > give me advice on what's happening with our Kafka cluster. :(
> >
> > пн, 9 нояб. 2020 г. в 10:57, Dev Op :
> >
> > > Hello all!
> > >
> > > Please, help me to understand why my consumer start receives the
> > > duplicates. I think it is because of problems on my kafka1 node.
> > >
> > > Cluster consists of three nodes: kafka1 (192.168.137.19, id=1),
> > > kafka2 (192.168.137.20, id=2), kafka3 ( 192.168.137.21, id=3)
> > > Version of Kafka: kafka_2.13-2.4.1
> > > Configs:
> > > - Broker config (server.properties from kafka1):
> > > https://pastebin.com/MR20rZdQ
> > > - Zookeeper config (zookeeper.properties from kafka1):
> > > https://pastebin.com/vCpFU0gp
> > >
> > > /opt/kafka_2.13-2.4.1/bin/kafka-topics.sh --describe --topic in_raw
> > > --zookeeper localhost:2181
> > > Topic: in_raw PartitionCount: 1 ReplicationFactor: 3
> Configs:
> > > Topic: in_raw Partition: 0Leader: 1 Replicas: 1,3,2
> > > Isr: 1,2,3
> > >
> > > Producer put one msg in `in_raw' topic msg, after it our consumer
> starts
> > > receive many dups from that topic every 10 minutes:
> > >
> > > The first duplicate occurrence was at 20:01:
> > > $ xzcat parsers.log-20201105.xz | perl -MData::Dumper -lne 'if
> > > (/(unitId=\d+, unitDate=\d+, msgNumber=\d+)/) { ++$a->{$1}; die "$_\n"
> if
> > > $a->{$1} > 1; }'
> > > 2020-11-04 20:01:47.173
> > > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> > ParserService
> > > - PARSE: Event{id='aeb33e8d-f7e5-4734-bc41-3c93da3cf94f', unitId=1073,
> > > unitDate=1604519428552, msgNumber=6948}
> > > ...
> > >
> > > A couple of record from log file:
> > > 2020-11-04 19:54:52.740
> > > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> > ParserService
> > > - PARSE: Event{id='86cc792b-fb5e-4ebb-be49-7a51f3a1c954', unitId=1073,
> > > unitDate=1604519428552, msgNumber=6948}
> > > 2020-11-04 20:01:47.173
> > > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> > ParserService
> > > - PARSE: Event{id='aeb33e8d-f7e5-4734-bc41-3c93da3cf94f', unitId=1073,
> > > unitDate=1604519428552, msgNumber=6948}
> > > 2020-11-04 20:11:47.217
> > > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> > ParserService
> > > - PARSE: Event{id='05a059e0-8002-48d0-b2da-269e42b879a0', unitId=1073,
> > > unitDate=1604519428552, msgNumber=6948}
> > > 2020-11-04 20:21:47.185
> > > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> > ParserService
> > > - PARSE: Event{id='5b590bde-9e86-4660-8916-db4a590ba12e', unitId=1073,
> > > unitDate=1604519428552, msgNumber=6948}
> > > ..and so on.
> > >
> > > Something went wrong earlier at 19:50.
> > >
> > > Log from kafka1 broker:
> > >
> > > [2020-11-04 19:04:02,195] INFO [GroupMetadataManager brokerId=1]
> Removed
> > 0
> > > expired offsets in 0 milliseconds.
> > > (kafka.coordinator.group.GroupMetadataManager)
> > > [2020-11-04 19:14:02,195] INFO [GroupMetadataManager brokerId=1]
> Removed
> > 0
> > > expired offsets in 0 milliseconds.
> > > (kafka.coordinator.group.GroupMetadataManager)
> > > [2020-11-04 19:24:02,195] INFO [GroupMetadataManager brokerId=1]
> Removed
> > 0
> > > expired offsets in 0 milliseconds.
> > > (kafka.coordinator.group.GroupMetadataManager)
> > > [2020-11-04 19:34:02,195] INFO [GroupMetadataManager brokerId=1]
> Removed
> > 0
> > > expired offsets in 0 milliseconds.
> > > (kafka.coordinator.group.GroupMetadataManager)
> > > [2020-11-04 19:44:02,195] INFO [GroupMetadataManager brokerId=1]
> Removed
> > 0
> > > expired offsets in 0 milliseconds.
> > > (kafka.coordinator.group.GroupMetadataManager)
> > > [2020-11-04 19:50:19,506] WARN Client session timed out, have not heard
> > > from server in 7997ms for sessionid 0x160d4310001
> > > (org.apache.zookeeper.ClientCnxn)
> > > [2020-11-04 19:50:19,526] INFO Client session timed out, have not heard
> > > from server in 7997ms for sessionid 0x160d4310001, closing socket
> > > connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> > > [2020-11-04 19:50:20,774] INFO Opening socket connection to server
> > > kafka2.8m.local/192.168.137.20:2181. Will not attempt to authenticate
> > > using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
> > > [2020-11-04 19:50:20,775] INFO Socket connection established,
> initiating
> > > session, client: /192.168.137.19:57606, server: kafka2.8m.local/
> > > 192.168.137.20:2181 (org.apache.zookeeper.ClientCnxn)
> > > [2020-11-04 19:50:22,776] WARN Client session timed out, have not heard
> > > from server in 2002ms for sessionid 0x160d4310001
>
Re: Reg: Max number of partitions in a topic
Yes, you can have 1000 partitions. But, there are implications of having a large number. Each partition has a leader. Clients downloading metadata to find those 1000 leaders will be a bit slower than finding 100 leaders. Producers that are buffering messages in order to use batching create a buffer per partition, so you may see increased memory usage. Likewise, if one of your brokers failed, the more partition leaders on it, the more leader elections that have to occur. I'd suggest benchmarking your use case with different partition counts and see where your sweet spot is. This old blog post has some good ideas: https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ Cheers, Liam Clarke On Tue, Nov 24, 2020 at 10:51 PM Gowtham S wrote: > Hi, > Can we have 1000 partitions in a Single topic? If not how many partitions > will a single topic have at the max? > Anyone, please tell. > > Thanks and regards, > Gowtham S >
Re: Many dups received by consumer (kafka_2.13)
Can you upgrade Kafka to 2.5.1? This problem was fixed in that release.
https://issues.apache.org/jira/browse/KAFKA-9839
On Wed, Nov 25, 2020 at 9:44 PM Dev Op wrote:
> Hello community! Hasn't anyone faced a similar problem? I see nobody can
> give me advice on what's happening with our Kafka cluster. :(
>
> пн, 9 нояб. 2020 г. в 10:57, Dev Op :
>
> > Hello all!
> >
> > Please, help me to understand why my consumer start receives the
> > duplicates. I think it is because of problems on my kafka1 node.
> >
> > Cluster consists of three nodes: kafka1 (192.168.137.19, id=1),
> > kafka2 (192.168.137.20, id=2), kafka3 ( 192.168.137.21, id=3)
> > Version of Kafka: kafka_2.13-2.4.1
> > Configs:
> > - Broker config (server.properties from kafka1):
> > https://pastebin.com/MR20rZdQ
> > - Zookeeper config (zookeeper.properties from kafka1):
> > https://pastebin.com/vCpFU0gp
> >
> > /opt/kafka_2.13-2.4.1/bin/kafka-topics.sh --describe --topic in_raw
> > --zookeeper localhost:2181
> > Topic: in_raw PartitionCount: 1 ReplicationFactor: 3Configs:
> > Topic: in_raw Partition: 0Leader: 1 Replicas: 1,3,2
> > Isr: 1,2,3
> >
> > Producer put one msg in `in_raw' topic msg, after it our consumer starts
> > receive many dups from that topic every 10 minutes:
> >
> > The first duplicate occurrence was at 20:01:
> > $ xzcat parsers.log-20201105.xz | perl -MData::Dumper -lne 'if
> > (/(unitId=\d+, unitDate=\d+, msgNumber=\d+)/) { ++$a->{$1}; die "$_\n" if
> > $a->{$1} > 1; }'
> > 2020-11-04 20:01:47.173
> > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> ParserService
> > - PARSE: Event{id='aeb33e8d-f7e5-4734-bc41-3c93da3cf94f', unitId=1073,
> > unitDate=1604519428552, msgNumber=6948}
> > ...
> >
> > A couple of record from log file:
> > 2020-11-04 19:54:52.740
> > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> ParserService
> > - PARSE: Event{id='86cc792b-fb5e-4ebb-be49-7a51f3a1c954', unitId=1073,
> > unitDate=1604519428552, msgNumber=6948}
> > 2020-11-04 20:01:47.173
> > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> ParserService
> > - PARSE: Event{id='aeb33e8d-f7e5-4734-bc41-3c93da3cf94f', unitId=1073,
> > unitDate=1604519428552, msgNumber=6948}
> > 2020-11-04 20:11:47.217
> > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> ParserService
> > - PARSE: Event{id='05a059e0-8002-48d0-b2da-269e42b879a0', unitId=1073,
> > unitDate=1604519428552, msgNumber=6948}
> > 2020-11-04 20:21:47.185
> > [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1]
> ParserService
> > - PARSE: Event{id='5b590bde-9e86-4660-8916-db4a590ba12e', unitId=1073,
> > unitDate=1604519428552, msgNumber=6948}
> > ..and so on.
> >
> > Something went wrong earlier at 19:50.
> >
> > Log from kafka1 broker:
> >
> > [2020-11-04 19:04:02,195] INFO [GroupMetadataManager brokerId=1] Removed
> 0
> > expired offsets in 0 milliseconds.
> > (kafka.coordinator.group.GroupMetadataManager)
> > [2020-11-04 19:14:02,195] INFO [GroupMetadataManager brokerId=1] Removed
> 0
> > expired offsets in 0 milliseconds.
> > (kafka.coordinator.group.GroupMetadataManager)
> > [2020-11-04 19:24:02,195] INFO [GroupMetadataManager brokerId=1] Removed
> 0
> > expired offsets in 0 milliseconds.
> > (kafka.coordinator.group.GroupMetadataManager)
> > [2020-11-04 19:34:02,195] INFO [GroupMetadataManager brokerId=1] Removed
> 0
> > expired offsets in 0 milliseconds.
> > (kafka.coordinator.group.GroupMetadataManager)
> > [2020-11-04 19:44:02,195] INFO [GroupMetadataManager brokerId=1] Removed
> 0
> > expired offsets in 0 milliseconds.
> > (kafka.coordinator.group.GroupMetadataManager)
> > [2020-11-04 19:50:19,506] WARN Client session timed out, have not heard
> > from server in 7997ms for sessionid 0x160d4310001
> > (org.apache.zookeeper.ClientCnxn)
> > [2020-11-04 19:50:19,526] INFO Client session timed out, have not heard
> > from server in 7997ms for sessionid 0x160d4310001, closing socket
> > connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> > [2020-11-04 19:50:20,774] INFO Opening socket connection to server
> > kafka2.8m.local/192.168.137.20:2181. Will not attempt to authenticate
> > using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
> > [2020-11-04 19:50:20,775] INFO Socket connection established, initiating
> > session, client: /192.168.137.19:57606, server: kafka2.8m.local/
> > 192.168.137.20:2181 (org.apache.zookeeper.ClientCnxn)
> > [2020-11-04 19:50:22,776] WARN Client session timed out, have not heard
> > from server in 2002ms for sessionid 0x160d4310001
> > (org.apache.zookeeper.ClientCnxn)
> > [2020-11-04 19:50:22,776] INFO Client session timed out, have not heard
> > from server in 2002ms for sessionid 0x160d4310001, closing socket
> > connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> > [2020-11-04 19:50:23,360] INFO Opening socket connection to server
> > kafka3.8m.local/192.168.137.21:2181. Will not attempt to authe
Re: Many dups received by consumer (kafka_2.13)
Hello community! Hasn't anyone faced a similar problem? I see nobody can
give me advice on what's happening with our Kafka cluster. :(
пн, 9 нояб. 2020 г. в 10:57, Dev Op :
> Hello all!
>
> Please, help me to understand why my consumer start receives the
> duplicates. I think it is because of problems on my kafka1 node.
>
> Cluster consists of three nodes: kafka1 (192.168.137.19, id=1),
> kafka2 (192.168.137.20, id=2), kafka3 ( 192.168.137.21, id=3)
> Version of Kafka: kafka_2.13-2.4.1
> Configs:
> - Broker config (server.properties from kafka1):
> https://pastebin.com/MR20rZdQ
> - Zookeeper config (zookeeper.properties from kafka1):
> https://pastebin.com/vCpFU0gp
>
> /opt/kafka_2.13-2.4.1/bin/kafka-topics.sh --describe --topic in_raw
> --zookeeper localhost:2181
> Topic: in_raw PartitionCount: 1 ReplicationFactor: 3Configs:
> Topic: in_raw Partition: 0Leader: 1 Replicas: 1,3,2
> Isr: 1,2,3
>
> Producer put one msg in `in_raw' topic msg, after it our consumer starts
> receive many dups from that topic every 10 minutes:
>
> The first duplicate occurrence was at 20:01:
> $ xzcat parsers.log-20201105.xz | perl -MData::Dumper -lne 'if
> (/(unitId=\d+, unitDate=\d+, msgNumber=\d+)/) { ++$a->{$1}; die "$_\n" if
> $a->{$1} > 1; }'
> 2020-11-04 20:01:47.173
> [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1] ParserService
> - PARSE: Event{id='aeb33e8d-f7e5-4734-bc41-3c93da3cf94f', unitId=1073,
> unitDate=1604519428552, msgNumber=6948}
> ...
>
> A couple of record from log file:
> 2020-11-04 19:54:52.740
> [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1] ParserService
> - PARSE: Event{id='86cc792b-fb5e-4ebb-be49-7a51f3a1c954', unitId=1073,
> unitDate=1604519428552, msgNumber=6948}
> 2020-11-04 20:01:47.173
> [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1] ParserService
> - PARSE: Event{id='aeb33e8d-f7e5-4734-bc41-3c93da3cf94f', unitId=1073,
> unitDate=1604519428552, msgNumber=6948}
> 2020-11-04 20:11:47.217
> [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1] ParserService
> - PARSE: Event{id='05a059e0-8002-48d0-b2da-269e42b879a0', unitId=1073,
> unitDate=1604519428552, msgNumber=6948}
> 2020-11-04 20:21:47.185
> [parser-59bb945d-1e57-454e-a152-b6c92b6bf981-StreamThread-1] ParserService
> - PARSE: Event{id='5b590bde-9e86-4660-8916-db4a590ba12e', unitId=1073,
> unitDate=1604519428552, msgNumber=6948}
> ..and so on.
>
> Something went wrong earlier at 19:50.
>
> Log from kafka1 broker:
>
> [2020-11-04 19:04:02,195] INFO [GroupMetadataManager brokerId=1] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)
> [2020-11-04 19:14:02,195] INFO [GroupMetadataManager brokerId=1] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)
> [2020-11-04 19:24:02,195] INFO [GroupMetadataManager brokerId=1] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)
> [2020-11-04 19:34:02,195] INFO [GroupMetadataManager brokerId=1] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)
> [2020-11-04 19:44:02,195] INFO [GroupMetadataManager brokerId=1] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)
> [2020-11-04 19:50:19,506] WARN Client session timed out, have not heard
> from server in 7997ms for sessionid 0x160d4310001
> (org.apache.zookeeper.ClientCnxn)
> [2020-11-04 19:50:19,526] INFO Client session timed out, have not heard
> from server in 7997ms for sessionid 0x160d4310001, closing socket
> connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2020-11-04 19:50:20,774] INFO Opening socket connection to server
> kafka2.8m.local/192.168.137.20:2181. Will not attempt to authenticate
> using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2020-11-04 19:50:20,775] INFO Socket connection established, initiating
> session, client: /192.168.137.19:57606, server: kafka2.8m.local/
> 192.168.137.20:2181 (org.apache.zookeeper.ClientCnxn)
> [2020-11-04 19:50:22,776] WARN Client session timed out, have not heard
> from server in 2002ms for sessionid 0x160d4310001
> (org.apache.zookeeper.ClientCnxn)
> [2020-11-04 19:50:22,776] INFO Client session timed out, have not heard
> from server in 2002ms for sessionid 0x160d4310001, closing socket
> connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2020-11-04 19:50:23,360] INFO Opening socket connection to server
> kafka3.8m.local/192.168.137.21:2181. Will not attempt to authenticate
> using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2020-11-04 19:50:23,361] INFO Socket connection established, initiating
> session, client: /192.168.137.19:54702, server: kafka3.8m.local/
> 192.168.137.21:2181 (org.apache.zookeeper.ClientCnxn)
> [2020-11-04 19:50:23,373] WARN Unable to reconnect to ZooKeeper service,
> session 0x160d4310001 has expi
