You can try to reduce batch.num.messages first and see if the throughput
gets affected.
For your second question, we do not have a good solution to that since the
offsets are not consistent across data centers like you said. One way we
did is to have the consumer consuming both data centers, but k
What would you recommend in this case? Bump up the max.message.bytes,
use sync producer or lower batch.num.messages? Also does it matter on
which side mirrormaker is located, source or target one?
And another related question: as I understand, potentially offsets
might be different for source and
In the old producer yes, in the new producer (available in 0.8.1.1) the
batch size is by bytes instead of #. messages, which gives you a better
control.
Guozhang
On Sat, Jun 7, 2014 at 2:48 PM, Kane Kane wrote:
> Ah, that makes sense then. As I understand right now there is no
> something like
Ah, that makes sense then. As I understand right now there is no
something like queue.buffering.max.size to control batch size?
On Sat, Jun 7, 2014 at 2:32 PM, Guozhang Wang wrote:
> With compression, the batch of messages are compressed in to a "single"
> wrapper message and sent to the broker,
With compression, the batch of messages are compressed in to a "single"
wrapper message and sent to the broker, and the broker will reject the
request if this single message's size is larger than 1MB. So you need to
either change your max request size on broker or reduce your producer batch
size.
Yes, messages were compressed with gzip and I've enabled the same
compression in mirrormaker producer.
On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang wrote:
> Kane,
>
> Did you use any compression method?
>
> Guozhang
>
>
> On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane wrote:
>
>> I've tried to run
Kane,
Did you use any compression method?
Guozhang
On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane wrote:
> I've tried to run mirrormaker tools in async mode and I get
> WARN Produce request with correlation id 263 failed due to
> [benchmark1,30]: kafka.common.MessageSizeTooLargeException
> (kafka.
I've tried to run mirrormaker tools in async mode and I get
WARN Produce request with correlation id 263 failed due to
[benchmark1,30]: kafka.common.MessageSizeTooLargeException
(kafka.producer.async.DefaultEventHandler)
I don't get error in sync mode. My message.max.bytes is default
(100). As
