Hi Dave, Do you get any errors logged in the broker when you get ACK error 2 (InvalidMessage) while producing requests to a mixed version cluster? It would be helpful to see them.
With regards to the kafka-console-producer.sh error, did you use the 0.9.0.0 console producer with a mixed version cluster (ie some brokers were on 0.8.2.1 while others were on 0.9.0.0)? If so, it is expected that it won't work correctly. All the brokers should be upgraded before the clients are upgraded (otherwise the 0.8.2.1 broker will send a response that the newer clients cannot handle). Ismael On Fri, Jan 15, 2016 at 7:52 PM, Dave Peterson <d...@academia.edu> wrote: > Hi Ismael, > > I'm using bruce (https://github.com/ifwe/bruce) to send the produce > requests, with a RequiredAcks value of 1. Everything works fine when > all brokers are running 0.8.2.1. Also if I set up a new 0.9.0.0 > cluster from scratch rather than trying to upgrade, everything works > fine. The problem only occurs after upgrading one broker in the > 3-broker cluster. > > The topic I am sending to has 8 partitions numbered 0-7. Doing > further experimentation I see that the ACK error 2 occurs only when > I send to partition 7. No problems occur when sending to partitions > 0-6. If it helps I can send output from "kafka-topics.sh --describe" > as well as tcpdump output showing the produce requests and responses. > > For comparison I tried using the 0.9.0.0 version of > kafka-console-producer.sh to send messages. With the default > RequiredAcks value of 0, it worked although I don't know which > partition it sent to. With a RequiredAcks value of 1 I get the > output shown below. > > Thanks, > Dave > > > > [2016-01-15 10:33:28,843] ERROR Uncaught error in kafka producer I/O > thread: (org.apache.kafka.clients.producer.internals.Sender) > org.apache.kafka.common.protocol.types.SchemaException: Error reading field > 'throttle_time_ms': java.nio.BufferUnderflowException > at > org.apache.kafka.common.protocol.types.Schema.read(Schema.java:71) > at > > org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:464) > at > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:279) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > at java.lang.Thread.run(Thread.java:745) > > > On Fri, Jan 15, 2016 at 1:06 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi Dave, > > > > On Fri, Jan 15, 2016 at 2:04 AM, Dave Peterson <d...@academia.edu> > wrote: > > > > > I was trying to upgrade an 0.8.2.1 broker cluster to 0.9.0.0 by > following > > > the instructions here: > > > > > > http://kafka.apache.org/documentation.html#upgrade > > > > > > After upgrading one broker, with inter.broker.protocol.version=0.8.2.X > > > set, I get ACK error 2 (InvalidMessage) when I try to send produce > > > requests. > > > > > > I haven't seen other reports of this issue yet. Also, we have a system > test > > that covers this scenario: > > > > > > > https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/upgrade_test.py > > > > Just to double-check, what is the version of the producer that you are > > using to send produce requests to the 0.9.0.0 broker when you get the > > error? > > > > Ismael > > >
