Re: Oversized Message 40k

[Felipe Santos]

Thanks guys, for your information, I will do some performance tests

Em qua, 23 de nov de 2016 às 05:14, Ignacio Solis <iso...@igso.net>
escreveu:

> At LinkedIn we have a number of use cases for large messages.  We stick to
> the 1MB message limit at the high end though.
>
> Nacho
>
> On Tue, Nov 22, 2016 at 6:11 PM, Gwen Shapira <g...@confluent.io> wrote:
>
> > This has been our experience as well. I think the largest we've seen
> > in production is 50MB.
> >
> > If you have performance numbers you can share for the large messages,
> > I think we'll all appreciate :)
> >
> > On Tue, Nov 22, 2016 at 1:04 PM, Tauzell, Dave
> > <dave.tauz...@surescripts.com> wrote:
> > > I ran tests with a mix of messages, some as large as 20MB.   These
> large
> > messages do slow down processing, but it still works.
> > >
> > > -Dave
> > >
> > > -Original Message-
> > > From: h...@confluent.io [mailto:h...@confluent.io]
> > > Sent: Tuesday, November 22, 2016 1:41 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Oversized Message 40k
> > >
> > > The default config handles messages up to 1MB so you should be fine.
> > >
> > > -hans
> > >
> > >> On Nov 22, 2016, at 4:00 AM, Felipe Santos <felip...@gmail.com>
> wrote:
> > >>
> > >> I read on documentation that kafka is not optimized for big messages,
> > >> what is considered a big message?
> > >>
> > >> For us the messages will be on average from 20k ~ 40k? Is this a real
> > >> problem?
> > >>
> > >> Thanks
> > >> --
> > >> Felipe Santos
> > > This e-mail and any files transmitted with it are confidential, may
> > contain sensitive information, and are intended solely for the use of the
> > individual or entity to whom they are addressed. If you have received
> this
> > e-mail in error, please notify the sender by reply e-mail immediately and
> > destroy all copies of the e-mail and any attachments.
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>
>
>
> --
> Nacho - Ignacio Solis - iso...@igso.net
>

Re: Oversized Message 40k

[Ignacio Solis]

At LinkedIn we have a number of use cases for large messages.  We stick to
the 1MB message limit at the high end though.

Nacho

On Tue, Nov 22, 2016 at 6:11 PM, Gwen Shapira <g...@confluent.io> wrote:

> This has been our experience as well. I think the largest we've seen
> in production is 50MB.
>
> If you have performance numbers you can share for the large messages,
> I think we'll all appreciate :)
>
> On Tue, Nov 22, 2016 at 1:04 PM, Tauzell, Dave
> <dave.tauz...@surescripts.com> wrote:
> > I ran tests with a mix of messages, some as large as 20MB.   These large
> messages do slow down processing, but it still works.
> >
> > -Dave
> >
> > -Original Message-
> > From: h...@confluent.io [mailto:h...@confluent.io]
> > Sent: Tuesday, November 22, 2016 1:41 PM
> > To: users@kafka.apache.org
> > Subject: Re: Oversized Message 40k
> >
> > The default config handles messages up to 1MB so you should be fine.
> >
> > -hans
> >
> >> On Nov 22, 2016, at 4:00 AM, Felipe Santos <felip...@gmail.com> wrote:
> >>
> >> I read on documentation that kafka is not optimized for big messages,
> >> what is considered a big message?
> >>
> >> For us the messages will be on average from 20k ~ 40k? Is this a real
> >> problem?
> >>
> >> Thanks
> >> --
> >> Felipe Santos
> > This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>



-- 
Nacho - Ignacio Solis - iso...@igso.net

Re: Oversized Message 40k

[Gwen Shapira]

This has been our experience as well. I think the largest we've seen
in production is 50MB.

If you have performance numbers you can share for the large messages,
I think we'll all appreciate :)

On Tue, Nov 22, 2016 at 1:04 PM, Tauzell, Dave
<dave.tauz...@surescripts.com> wrote:
> I ran tests with a mix of messages, some as large as 20MB.   These large 
> messages do slow down processing, but it still works.
>
> -Dave
>
> -Original Message-
> From: h...@confluent.io [mailto:h...@confluent.io]
> Sent: Tuesday, November 22, 2016 1:41 PM
> To: users@kafka.apache.org
> Subject: Re: Oversized Message 40k
>
> The default config handles messages up to 1MB so you should be fine.
>
> -hans
>
>> On Nov 22, 2016, at 4:00 AM, Felipe Santos <felip...@gmail.com> wrote:
>>
>> I read on documentation that kafka is not optimized for big messages,
>> what is considered a big message?
>>
>> For us the messages will be on average from 20k ~ 40k? Is this a real
>> problem?
>>
>> Thanks
>> --
>> Felipe Santos
> This e-mail and any files transmitted with it are confidential, may contain 
> sensitive information, and are intended solely for the use of the individual 
> or entity to whom they are addressed. If you have received this e-mail in 
> error, please notify the sender by reply e-mail immediately and destroy all 
> copies of the e-mail and any attachments.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

RE: Oversized Message 40k

[Tauzell, Dave]

I ran tests with a mix of messages, some as large as 20MB.   These large 
messages do slow down processing, but it still works.

-Dave

-Original Message-
From: h...@confluent.io [mailto:h...@confluent.io]
Sent: Tuesday, November 22, 2016 1:41 PM
To: users@kafka.apache.org
Subject: Re: Oversized Message 40k

The default config handles messages up to 1MB so you should be fine.

-hans

> On Nov 22, 2016, at 4:00 AM, Felipe Santos <felip...@gmail.com> wrote:
>
> I read on documentation that kafka is not optimized for big messages,
> what is considered a big message?
>
> For us the messages will be on average from 20k ~ 40k? Is this a real
> problem?
>
> Thanks
> --
> Felipe Santos
This e-mail and any files transmitted with it are confidential, may contain 
sensitive information, and are intended solely for the use of the individual or 
entity to whom they are addressed. If you have received this e-mail in error, 
please notify the sender by reply e-mail immediately and destroy all copies of 
the e-mail and any attachments.

Re: Oversized Message 40k

[hans]

The default config handles messages up to 1MB so you should be fine.

-hans

> On Nov 22, 2016, at 4:00 AM, Felipe Santos  wrote:
> 
> I read on documentation that kafka is not optimized for big messages, what
> is considered a big message?
> 
> For us the messages will be on average from 20k ~ 40k? Is this a real
> problem?
> 
> Thanks
> -- 
> Felipe Santos

Re: Oversized Message 40k

[Dominik Safaric]

Big is a relative term.

And the question you ask is quite difficult to answer because not other 
information is available - including the configuration of the Kafka cluster, 
hardware specification etcetera. 

I suggest the following: (1) read a couple of benchmarks such as [1], (2) 
investigate onto this question of matter yourself by designing and running a 
microbenchmark - since you’re the only one who knows what throughput or for 
example latency you’d like to achieve.

But in general, I do believe that those messages are larger then the expected. 
Hence, you may also want to take a look at compression in Kafka - although 
you’ll lose performance.  

[1] 
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

> On 22 Nov 2016, at 13:00, Felipe Santos  wrote:
> 
> I read on documentation that kafka is not optimized for big messages, what
> is considered a big message?
> 
> For us the messages will be on average from 20k ~ 40k? Is this a real
> problem?
> 
> Thanks
> -- 
> Felipe Santos