Hi Jeff & Santiago, Thanks for your help!! I realized that just after posting that question. Sorry for not updating it earlier.
Thanks Mohit On Thu, Oct 30, 2014 at 5:36 PM, Santiago Mola <[email protected]> wrote: > Hi Mohit, > > 2014-10-27 19:30 GMT+01:00 Mohit Durgapal <[email protected]>: > >> >> I am using rsyslog to send messages to flume nodes via AWS ELB. On flume >> nodes I am using the source type *syslogtcp * where the ELB forwards >> the messages. Now I see the messages that are over 2k in size are being >> broken into chunks of size 2k when I receive them in flume. As my messages >> are in JSON this breaks the structure and my application reading the logs >> from hdfs see them as ill-formed JSON and skip those records. >> > > > You need to increase the message size both in rsyslog (as you did) *and* > in your Flume agent. > > The Syslog source has an eventSize option that defaults to 2500 bytes. You > can check the details in the docs: > > https://flume.apache.org/FlumeUserGuide.html#syslog-tcp-source > > Best, > -- > > Santiago M. Mola > > > <http://www.stratio.com/> > Avenida de Europa, 26. Ática 5. 3ª Planta > 28224 Pozuelo de Alarcón, Madrid > Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>* >
