The config name on the brokers is "message.max.bytes". It defaults to
roughly 1 MB.
Thanks,
Neha
On Sat, May 17, 2014 at 8:46 AM, Jun Rao wrote:
> You just need to make sure that fetch size is larger than max.message.size
> set in the brokers.
>
> Thanks,
>
> Jun
>
>
> On Thu, May 15, 2014 at
You just need to make sure that fetch size is larger than max.message.size
set in the brokers.
Thanks,
Jun
On Thu, May 15, 2014 at 3:55 PM, MB JA wrote:
> Hi.
>
> I´m using Kafka, Can you help me please? :)
>
> I´m have the problem to read the next message available when is larger
> than the
Hi.
I´m using Kafka, Can you help me please? :)
I´m have the problem to read the next message available when is larger
than the maximum fetch size you have specified, i do not manage several
size´s for the message becase the size is very diferente of one message to
other. Do you now the more eff
And to be clear, if uncompressed messages come in, they remain uncompressed
in the broker, correct?
Correct
Currently, only the broker has knowledge of the offsets for a partition and
hence is the right place to assign the offsets. Even if the producer sends
metadata, the broker still needs to de
Ah,
I think I remember a previous discussion on a way to avoid the double
compression
So would it be possible for the producer to send metadata with a compressed
batch that includes the logical offset info for the batch? Can this info
just be a count of how many messages are in the batch?
A
The broker only recompresses the messages if the producer sent them
compressed. And it has to recompress to assign the logical offsets to the
individual messages inside the compressed message.
Thanks,
Neha
On Oct 7, 2013 11:36 PM, "Jason Rosenberg" wrote:
> Neha,
>
> Does the broker store messag
Neha,
Does the broker store messages compressed, even if the producer doesn't
compress them when sending them to the broker?
Why does the broker re-compress message batches? Does it not have enough
info from the producer request to know the number of messages in the batch?
Jason
On Mon, Oct 7
the total message size of the batch should be less than
message.max.bytes or is that for each individual message?
The former is correct.
When you batch, I am assuming that the producer sends some sort of flag
that this is a batch, and then the broker will split up those messages to
individual mes
I don't think the batch referred to initially is a Kafka API batch, hence
the confusion. I'm sure someone from LinkedIn can clarify.
On Oct 7, 2013 9:27 AM, "S Ahmed" wrote:
> When you batch things on the producer, say you batch 1000 messages or by
> time whatever, the total message size of the b
When you batch things on the producer, say you batch 1000 messages or by
time whatever, the total message size of the batch should be less than
message.max.bytes or is that for each individual message?
When you batch, I am assuming that the producer sends some sort of flag
that this is a batch, an
The message size limit is imposed on the compressed message. To answer your
question about the effect of large messages - they cause memory pressure on
the Kafka brokers as well as on the consumer since we re-compress messages
on the broker and decompress messages on the consumer.
I'm not so sure
I see, so that is one thing to consider is if I have 20 KB messages, I
shouldn't batch too many together as that will increase latency and the
memory usage footprint on the producer side of things.
On Mon, Oct 7, 2013 at 11:55 AM, Jun Rao wrote:
> At LinkedIn, our message size can be 10s of KB.
At LinkedIn, our message size can be 10s of KB. This is mostly because we
batch a set of messages and send them as a single compressed message.
Thanks,
Jun
On Mon, Oct 7, 2013 at 7:44 AM, S Ahmed wrote:
> When people using message queues, the message size is usually pretty small.
>
> I want t
When people using message queues, the message size is usually pretty small.
I want to know who out there is using kafka with larger payload sizes?
In the configuration, the maximum message size by default is set to 1
megabyte (
message.max.bytes100)
My message sizes will be probably be aroun
14 matches
Mail list logo
