We did an upgrade from Kafka 2.2 to 2.6, followed by a migration (through reassign-partitions) from old to new brokers.
As described in https://stackoverflow.com/questions/64514851/apache-kafka-kafka-common-offsetsoutoforderexception-when-reassigning-consume, all but 1 partition (__consumer_offsets-18) were successfully migrated. But that one stubbornly refuses to migrate, because new replicas complain about OutOfOrderOffset. When we looked today at the exception message more closely, we discovered that really in huge list of offsets, there are 2 non-monotic offsets [2020-10-24 15:04:54,528] ERROR [ReplicaFetcher replicaId=10, leaderId=3, fetcherId=0] Unexpected error occurred while processing data for partition __consumer_offsets-18 at offset 1545264631 (kafka.server.ReplicaFetcherThread) kafka.common.OffsetsOutOfOrderException: Out of order offsets found in append to __consumer_offsets-18: ArrayBuffer(1545264631, 1545264632, ..., 1545271418, 1545271419, 1, 1, 1545271422, 1545271423, ... 1545272005, 1545272006, 1545272007) Apparently, the leader - when asked for the list of offsets to replicate - returns that list with the two weird '1' offsets. One would expect 1545271420, 1545271421 in place of the two '1' entries. Does this mean that the data is really somehow corrupted? That offset range (probably no by accident) is about when we started the upgrade of 2.2 to 2.6. The last snapshot file written on the leader with the old 2.2 version had offset 1545271411, only 8 messages before the one that is now apparently causing issue. Any idea what could have happened here? And more importantly, any idea how to get out of this mess :-) We would be able to accept the loss of all data on __consumer_offsets-18. It happens to contain only "non-essential" consumer groups. So that would also be an option. CU, Joe
