What about your flume config? Did you try increasing the eventSize? On Mon, Oct 27, 2014 at 11:30 AM, Mohit Durgapal <[email protected]> wrote:
> Hi, > > I am using rsyslog to send messages to flume nodes via AWS ELB. On flume > nodes I am using the source type *syslogtcp * where the ELB forwards the > messages. Now I see the messages that are over 2k in size are being broken > into chunks of size 2k when I receive them in flume. As my messages are in > JSON this breaks the structure and my application reading the logs from > hdfs see them as ill-formed JSON and skip those records. > > I know that the default message size of rsyslog is 2k. I even increased > that with : > > $MaxMessageSize 50k > > > My rsyslog.conf looks like this: > > > > > > > > > *# rsyslog v5 configuration file# For more information see > /usr/share/doc/rsyslog-*/rsyslog_conf.html# If you experience problems, see > http://www.rsyslog.com/doc/troubleshoot.html > <http://www.rsyslog.com/doc/troubleshoot.html>#### MODULES ####* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *$MaxMessageSize 50k$ModLoad imuxsock # provides support for local system > logging (e.g. via logger command)$ModLoad imklog # provides kernel > logging support (previously done by rklogd)#$ModLoad immark # provides > --MARK-- message capability# Provides UDP syslog reception#$ModLoad > imudp#$UDPServerRun 514# Provides TCP syslog reception#$ModLoad > imtcp#$InputTCPServerRun 5140#### GLOBAL DIRECTIVES ##### Use default > timestamp format$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat# > File syncing capability is disabled by default. This feature is usually not > required,# not useful and an extreme performance hit#$ActionFileEnableSync > on# Include all config files in /etc/rsyslog.d/$IncludeConfig > /etc/rsyslog.d/*.conf#### RULES ##### Log all kernel messages to the > console.# Logging much else clutters up the > screen.#kern.* > /dev/console# Log anything (except mail) of level info or higher.# Don't > log private authentication > messages!#*.info;mail.none;authpriv.none;cron.none > /var/log/messages*.info;mail.none;authpriv.none;cron.none;local2.none;local3.none > /var/log/messages# The authpriv file has restricted > access.authpriv.* > /var/log/secure# Log all the mail messages in one > place.mail.* > -/var/log/maillog# Log cron > stuffcron.* /var/log/cron# > Everybody gets emergency > messages*.emerg > :omusrmsg:*# Save news errors of level crit and higher in a special > file.uucp,news.crit > /var/log/spooler# Save boot messages also to > boot.loglocal7.* > /var/log/boot.log$template RTFormat,"%msg%\n"# An on-disk queue is created > for this action. If the remote host is# down, messages are spooled to disk > and sent when it is up again.$WorkDirectory /var/lib/rsyslog # where to > place spool files$ActionQueueFileName fwdRuleRTLogs1 # unique name prefix > for spool files$ActionQueueMaxDiskSpace 5g # 4gb space limit (use as much > as possible)$ActionQueueSaveOnShutdown on # save messages to disk on > shutdown$ActionQueueType LinkedList # run > asynchronously$ActionResumeRetryCount -1 # infinite retries if host is > down# remote host is: name/ip:port, e.g. 192.168.0.1:514 > <http://192.168.0.1:514>, port optionallocal2.* > @@internal-load-balancer:5149;RTFormat# ### end of the forwarding rule ###* > > > > Any help would be great. > > > Regards > Mohit >
