Do you mind filing a bug and attaching the reproducible test case there ? Thanks, Neha
On Wednesday, March 20, 2013, 王国栋 wrote: > Hi Jun, > > We use one thread with one sync produce to send data to broker > (QPS:10k-15k, each log is about 1k bytes). The problem is reproduced. > > We have used Producer and SyncProducer in our test. The same Exception > appears. > > Thanks. > > > > On Thu, Mar 21, 2013 at 12:19 PM, Jun Rao <jun...@gmail.com> wrote: > > > How many threads are you using? > > > > Thanks, > > > > Jun > > > > On Wed, Mar 20, 2013 at 7:33 PM, Yang Zhou <zhou.yang.h...@gmail.com> > > wrote: > > > > > Sorry, I made a mistake, we use many threads producing at same time. > > > > > > > > > 2013/3/20 Jun Rao <jun...@gmail.com> > > > > > > > How many producer instances do you have? Can you reproduce the > problem > > > with > > > > a single producer? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Wed, Mar 20, 2013 at 12:29 AM, 王国栋 <wangg...@gmail.com> wrote: > > > > > > > > > Hi Jun, > > > > > > > > > > we do not use any compression in our test. > > > > > > > > > > We deploy producer and broker in the same machine. The problem > still > > > > > exists. We use sync producer, and send one message at a time(no > batch > > > > now). > > > > > We find that when the qps reaches more than 40k, the exception > > appears. > > > > So > > > > > I don't think it's the underlying system error. > > > > > > > > > > Any suggestions if we want to do some debug on kafka > > > > > serialization/deserialization? > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 20, 2013 at 12:10 AM, Jun Rao <jun...@gmail.com> > wrote: > > > > > > > > > > > It basically means that the broker is expecting to read certain > > > number > > > > of > > > > > > bytes in a buffer received from socket, but there are fewer bytes > > > than > > > > > > expected in the buffer. Possible causes are (1) a bug in Kafka > > > request > > > > > > serialization/deserialization logic; (2) corruption in the > > underlying > > > > > > system such as network. > > > > > > > > > > > > BTW, did you enable compression in your producer? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Mon, Mar 18, 2013 at 10:12 PM, Helin Xiang <xkee...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > thanks Jun. > > > > > > > > > > > > > > we are using java producer. > > > > > > > does the last exception > > > > > > > "java.lang.IllegalArgumentException > > > > > > > at java.nio.Buffer.limit(Buffer.java:266) > > > > > > > " > > > > > > > also means the broker received corrupted messages ? sorry i am > > not > > > > > > > familiar with java nio. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2013 at 12:58 PM, Jun Rao <jun...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > Hmm, both log4j messages suggest that the broker received > some > > > > > > corrupted > > > > > > > > produce requests. Are you using the java producer? Also, we > > have > > > > seen > > > > > > > that > > > > > > > > network router problems caused corrupted requests before. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > On Mon, Mar 18, 2013 at 8:22 PM, Helin Xiang < > -- > Guodong Wang > 王国栋 >
