What would you recommend in this case? Bump up the max.message.bytes, use sync producer or lower batch.num.messages? Also does it matter on which side mirrormaker is located, source or target one?
And another related question: as I understand, potentially offsets might be different for source and target, and if the whole datacenter is lost, you probably loose offsets in zookeeper as well. What is your strategy to switch over producers and consumers to the new cluster, given consumer should pickup from the same point where it was dropped in another datacenter? Thanks. On Sat, Jun 7, 2014 at 3:49 PM, Guozhang Wang <wangg...@gmail.com> wrote: > In the old producer yes, in the new producer (available in 0.8.1.1) the > batch size is by bytes instead of #. messages, which gives you a better > control. > > Guozhang > > > On Sat, Jun 7, 2014 at 2:48 PM, Kane Kane <kane.ist...@gmail.com> wrote: > >> Ah, that makes sense then. As I understand right now there is no >> something like queue.buffering.max.size to control batch size? >> >> On Sat, Jun 7, 2014 at 2:32 PM, Guozhang Wang <wangg...@gmail.com> wrote: >> > With compression, the batch of messages are compressed in to a "single" >> > wrapper message and sent to the broker, and the broker will reject the >> > request if this single message's size is larger than 1MB. So you need to >> > either change your max request size on broker or reduce your producer >> batch >> > size. >> > >> > Guozhang >> > >> > >> > On Sat, Jun 7, 2014 at 2:09 PM, Kane Kane <kane.ist...@gmail.com> wrote: >> > >> >> Yes, messages were compressed with gzip and I've enabled the same >> >> compression in mirrormaker producer. >> >> >> >> On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wangg...@gmail.com> >> wrote: >> >> > Kane, >> >> > >> >> > Did you use any compression method? >> >> > >> >> > Guozhang >> >> > >> >> > >> >> > On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <kane.ist...@gmail.com> >> wrote: >> >> > >> >> >> I've tried to run mirrormaker tools in async mode and I get >> >> >> WARN Produce request with correlation id 263 failed due to >> >> >> [benchmark1,30]: kafka.common.MessageSizeTooLargeException >> >> >> (kafka.producer.async.DefaultEventHandler) >> >> >> >> >> >> I don't get error in sync mode. My message.max.bytes is default >> >> >> (1000000). As I understand the difference between sync and async mode >> >> >> is that async collects batches before sending. Does it mean that >> batch >> >> >> should be less than message.max.bytes, or any single message within >> >> >> batch should be less than that. >> >> >> >> >> >> Thanks. >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > -- Guozhang >> >> >> > >> > >> > >> > -- >> > -- Guozhang >> > > > > -- > -- Guozhang
