The best practice I think is to just put large objects in a blob store and have messages embed references to those blobs. Interestingly we ended up having to implement large-message-support at LinkedIn but for various reasons were forced to put messages inline (i.e., against the above recommendation). So we ended up having to break up large messages into smaller chunks. This obviously adds considerable complexity to the consumer since the checkpointing can become pretty complicated. There are other nuances as well - we can probably do a short talk on this at an upcoming meetup.
Joel On Mon, Oct 5, 2015 at 9:31 PM, Rahul Jain <rahul...@gmail.com> wrote: > In addition to the config changes mentioned in that post, you may also have > to change producer config if you are using the new producer. > > Specifically, *max.request.size* and *request.timeout.ms > <http://request.timeout.ms>* have to be increased to allow the producer to > send large messages. > > > On 6 Oct 2015 02:02, "James Cheng" <jch...@tivo.com> wrote: > >> Here’s an article that Gwen wrote earlier this year on handling large >> messages in Kafka. >> >> http://ingest.tips/2015/01/21/handling-large-messages-kafka/ >> >> -James >> >> > On Oct 5, 2015, at 11:20 AM, Pradeep Gollakota <pradeep...@gmail.com> >> wrote: >> > >> > Fellow Kafkaers, >> > >> > We have a pretty heavyweight legacy event logging system for batch >> > processing. We're now sending the events into Kafka now for realtime >> > analytics. But we have some pretty large messages (> 40 MB). >> > >> > I'm wondering if any of you have use cases where you have to send large >> > messages to Kafka and how you're dealing with them. >> > >> > Thanks, >> > Pradeep >> >> >> ________________________________ >> >> This email and any attachments may contain confidential and privileged >> material for the sole use of the intended recipient. Any review, copying, >> or distribution of this email (or any attachments) by others is prohibited. >> If you are not the intended recipient, please contact the sender >> immediately and permanently delete this email and any attachments. No >> employee or agent of TiVo Inc. is authorized to conclude any binding >> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo >> Inc. may only be made by a signed written agreement. >>
