Raymond, Your understanding is correct. There are 2 common reasons for getting InvalidMessageSizeException: (1) fetch size smaller than the largest message (compressed messages in the same batch is treated as 1 message), (2) the offset in the fetch request is wrong. Both can be verified using the DumpLogSegment tool.
Thanks, Jun On Tue, Sep 25, 2012 at 7:25 AM, Raymond Ng <[email protected]> wrote: > Hi > > I'd like to know how SimpleConsumer.fetch and > ByteBufferMessageSet.iterartor work in terms of dealing with GZIP'd > messages > > is it right to say that SimpleConsumer.fetch will fetch upto the amount > specified in the fetch size regardless whether a complete GZIP batch has > been retrieved, and ByteBufferMessageSet.iterartor is responsible for > converting the fetched MessageSet into a set of complete MessageAndOffset > objects and disgarding any incomplete batch due to fetch size limit? > > I'm asking because I'm regularly getting InvalidMessageSizeException after > a consumer fetch, and I can see this topic being discussed in other mail > threads already, and I've also used the DumpLogSegments to check the kafka > file reported in the exception, nothing negative reported by the tool in > terms of the offset > > I'm using GZIP compression to send msgs to kafka, and a limit of 3MB of raw > data batch has been imposed and the batch size will be smaller once GZIP'ed > to kafka > > On the consumer the fetch size is set to 10 MB to allow for plenty of > leeway to fetch at least 1 full batch of GZIP'd content > > what else can be wrong in this setup ? > > -- > Rgds > Ray >
