[jira] [Created] (KAFKA-1749) Brokers continually throw exceptions when there are hundreds of topic being fetched by mirrormaker

[Min Zhou (JIRA)]

Min Zhou created KAFKA-1749:
-------------------------------

             Summary: Brokers continually throw exceptions when there are 
hundreds of topic being fetched by mirrormaker
                 Key: KAFKA-1749
                 URL: https://issues.apache.org/jira/browse/KAFKA-1749
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.8.1.1, 0.8.2
            Reporter: Min Zhou


Here is one piece of  millions of the exceptions. 

{noformat}
kafka.common.KafkaException: This operation cannot be completed on a complete 
request.
at kafka.network.Transmission$class.expectIncomplete(Transmission.scala:34)
at kafka.api.FetchResponseSend.expectIncomplete(FetchResponse.scala:191)
at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:214)
at kafka.network.Processor.write(SocketServer.scala:375)
at kafka.network.Processor.run(SocketServer.scala:247)
at java.lang.Thread.run(Thread.java:662)
{noformat}

We use tools to hook function kafka.api.FetchResponseSend.writeTo, found 
 fetchResponse.sizeInBytes was overflow.  Which means below code will get a 
result over the limit of integer type

{noformat}
val sizeInBytes =
    FetchResponse.headerSize +
    dataGroupedByTopic.foldLeft(0) ((folded, curr) => {
      val topicData = TopicData(curr._1, curr._2.map {
        case (topicAndPartition, partitionData) => 
(topicAndPartition.partition, partitionData)
      })
      folded + topicData.sizeInBytes
    })

{noformat} 

 If we just fetch few topics by mirrormaker, the brokers ran well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)