Could you reduce the max message size? Do you really expect to have a single message of 600MB? After that, you can reduce the fetch size.
Thanks, Jun On Thu, Jan 2, 2014 at 8:06 AM, Gerrit Jansen van Vuuren < gerrit...@gmail.com> wrote: > There is a particular topic that has allot of data in each message, there > is nothing I can do about it. > Because I have so much data I try to split the data over 8-12 partitions, > if I reduce the partitions I won't have enough consumers to consume the > data in time. > > > On Thu, Jan 2, 2014 at 4:50 PM, Jun Rao <jun...@gmail.com> wrote: > > > 600mb for fetch size is considerably larger than the default size. Is > there > > a particular reason for this? Also, how many partitions do you have? You > > may have to reduce the fetch size further if there are multiple > partitions. > > > > Thanks, > > > > Jun > > > > > > On Thu, Jan 2, 2014 at 2:42 AM, Gerrit Jansen van Vuuren < > > gerrit...@gmail.com> wrote: > > > > > Hi, > > > > > > I just double checked my configuration and the broker has > > message.max.bytes > > > set to 1 gig, the consumers have the same setting for max fetch size. > > I've > > > lowered this to 600 mb and still see the same error :(, > > > > > > at the moment kafka is un-usable for me, the the only other alternative > > is > > > writing my own client (as i'm doing with the producer), what a pain! > > > > > > > > > > > > > > > > > > On Wed, Jan 1, 2014 at 6:45 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > In our wire protocol, we expect the first 4 bytes for a response to > be > > > its > > > > size. If the actual size is larger than 2GB, what's stored in the > > those 4 > > > > bytes is the overflowed value. This could cause some of the buffer > size > > > to > > > > be smaller than it should be later on. If #partitions * fetch_size > is > > > > larger than 2GB in a single fetch request, you could hit this > problem. > > > You > > > > can try reducing the fetch size. Ideally, the sender should catch > this > > > and > > > > throw an exception, which we don't do currently. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Wed, Jan 1, 2014 at 9:27 AM, Gerrit Jansen van Vuuren < > > > > gerrit...@gmail.com> wrote: > > > > > > > > > Mm... Could be Im not sure if in a single request though. I am > moving > > > > allot > > > > > of data. Any pointer at were in the code the overflow might start? > > > > > On 1 Jan 2014 18:13, "Jun Rao" <jun...@gmail.com> wrote: > > > > > > > > > > > Are you fetching more than 2GB of data in a single fetch response > > > > (across > > > > > > all partitions)? Currently, we don't handle integer overflow > > > properly. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > On Wed, Jan 1, 2014 at 4:24 AM, Gerrit Jansen van Vuuren < > > > > > > gerrit...@gmail.com> wrote: > > > > > > > > > > > > > While consuming from the topics I get an > IlegalArgumentException > > > and > > > > > all > > > > > > > consumption stops, the error keeps on throwing. > > > > > > > > > > > > > > I've tracked it down to FectchResponse.scala line 33 > > > > > > > > > > > > > > The error happens when the FetchResponsePartitionData object's > > > > readFrom > > > > > > > method calls: > > > > > > > messageSetBuffer.limit(messageSetSize) > > > > > > > > > > > > > > I put in some debug code the the messageSetSize is 671758648, > > while > > > > the > > > > > > > buffer.capacity() gives 155733313, for some reason the buffer > is > > > > > smaller > > > > > > > than the required message size. > > > > > > > > > > > > > > I don't know the consumer code enough to debug this. It doesn't > > > > matter > > > > > if > > > > > > > compression is used or not. > > > > > > > > > > > > > > I've created a jira ticket for this: > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1196 > > > > > > > > > > > > > > this is a real pain for me because I'm unable to consume from > > kafka > > > > at > > > > > > all > > > > > > > :( > > > > > > > > > > > > > > > > > > > > > Any ideas on possible config? or code changes I could try to > fix? > > > > > > > > > > > > > > Regards, > > > > > > > Gerrit > > > > > > > > > > > > > > > > > > > > > > > > > > > >
