Thanks for heads up Joel! Kind regards, Stevo Slavic.
On Thu, Jul 2, 2015 at 10:17 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > A somewhat related request came up in another thread and I think it is > reasonable to provide this. However, there are already some indicators > that you can use: > - The consumer iterator throws a MessageSizeTooLargeException if it > cannot extract any messages out of the next chunk. > - If you are using the simple consumer, the fetch response includes > the high watermark - so if the HW > your fetch offset but there are > no iterable messages, then you will know that your fetch size needs > to increase. > > Thanks, > > Joel > > On Thu, Jul 02, 2015 at 05:32:20PM +0200, Stevo Slavić wrote: > > Hello Apache Kafka community, > > > > Couldn't broker return a special error code in FetchResponse for a given > > partition(s) where it detects that there was something to return/read > from > > partition but actual FetchResponse contains no messages for that > partition > > since fetch size in FetchRequest for that partition is too small? > > > > I don't see it already in existing Kafka wire protocol docs > > < > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol > >. > > It doesn't seem to be supported, since docs for broker and consumer > > configuration mention that the maximums should be in sync, otherwise too > > large message could block consumer. > > > > If it was available this feature would be useful for simple consumer API > > users, in the cases when range of possible message sizes on same topic is > > relatively large (e.g. most very small messages, but some small fraction > of > > relatively large messages). Would help with being able to soft and hard > > maximums, so most of the time one could read with soft max, and only in > > case of this error code is on, attempt a read using hard max limit. > > > > Assuming this feature is not there, I guess what would need to be done at > > minimum is: > > - minimal change in protocol and docs, just one new error code > > - accompanying detection of this condition on broker side, so extend > > handling of FetchRequest > > - maybe in high level consumer make use of this error code in a special > way > > (e.g. just log a warning/error that there is potential misconfiguration). > > > > Kind regards, > > Stevo Slavic. > >
