Hi, I am using rsyslog to send messages to flume nodes via AWS ELB. On flume nodes I am using the source type *syslogtcp * where the ELB forwards the messages. Now I see the messages that are over 2k in size are being broken into chunks of size 2k when I receive them in flume. As my messages are in JSON this breaks the structure and my application reading the logs from hdfs see them as ill-formed JSON and skip those records.
I know that the default message size of rsyslog is 2k. I even increased that with : $MaxMessageSize 50k My rsyslog.conf looks like this: *# rsyslog v5 configuration file# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html <http://www.rsyslog.com/doc/troubleshoot.html>#### MODULES ####* *$MaxMessageSize 50k$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)$ModLoad imklog # provides kernel logging support (previously done by rklogd)#$ModLoad immark # provides --MARK-- message capability# Provides UDP syslog reception#$ModLoad imudp#$UDPServerRun 514# Provides TCP syslog reception#$ModLoad imtcp#$InputTCPServerRun 5140#### GLOBAL DIRECTIVES ##### Use default timestamp format$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat# File syncing capability is disabled by default. This feature is usually not required,# not useful and an extreme performance hit#$ActionFileEnableSync on# Include all config files in /etc/rsyslog.d/$IncludeConfig /etc/rsyslog.d/*.conf#### RULES ##### Log all kernel messages to the console.# Logging much else clutters up the screen.#kern.* /dev/console# Log anything (except mail) of level info or higher.# Don't log private authentication messages!#*.info;mail.none;authpriv.none;cron.none /var/log/messages*.info;mail.none;authpriv.none;cron.none;local2.none;local3.none /var/log/messages# The authpriv file has restricted access.authpriv.* /var/log/secure# Log all the mail messages in one place.mail.* -/var/log/maillog# Log cron stuffcron.* /var/log/cron# Everybody gets emergency messages*.emerg :omusrmsg:*# Save news errors of level crit and higher in a special file.uucp,news.crit /var/log/spooler# Save boot messages also to boot.loglocal7.* /var/log/boot.log$template RTFormat,"%msg%\n"# An on-disk queue is created for this action. If the remote host is# down, messages are spooled to disk and sent when it is up again.$WorkDirectory /var/lib/rsyslog # where to place spool files$ActionQueueFileName fwdRuleRTLogs1 # unique name prefix for spool files$ActionQueueMaxDiskSpace 5g # 4gb space limit (use as much as possible)$ActionQueueSaveOnShutdown on # save messages to disk on shutdown$ActionQueueType LinkedList # run asynchronously$ActionResumeRetryCount -1 # infinite retries if host is down# remote host is: name/ip:port, e.g. 192.168.0.1:514 <http://192.168.0.1:514>, port optionallocal2.* @@internal-load-balancer:5149;RTFormat# ### end of the forwarding rule ###* Any help would be great. Regards Mohit
