[rsyslog] imuxsock needs UseSpecialParser='off" to parse /var/run/log correctly on FreeBSD
Hi,
Since r332100, FreeBSD uses RFC5424 format internally in libc[1]
This is in both 11.2R and 12.0R the 2 current supported releases.
imuxsock is unable to parse this format without setting "UseSpecialParser=off"
or it generates incorrect data for downstream consumption, like GELF and similar
formats, as well as making it messy in omfile output.
TLDR:
I'm not clear if this is a FreeBSD-induced bug, or rsyslog, or some combination
of the two. Should I file a bug, provide a docs patch for UseSpecialParser
(what does it actually do?), or something else?
Details - for example:
rsyslog 8.40.0 (from ports):
Using standard logger tool as test:
$ logger -H hostname -p local0.notice -s -i -t tag msg_woop
standard FreeBSD log output using rfc5424 output:
# syslogd -dFHO rfc5424
<133>1 2019-06-05T21:21:33.438075+08:00 j01 tag 19190 - - msg_woop
rsyslog using "UseSpecialParser=off":
# rsyslogd -d -f /usr/local/etc/rsyslog.conf -n -dq
2019-06-05T21:30:16.211530+08:00 j01 tag[19351] msg_woop
which produces the expected gelf_JSON output:
{
"version": "1.1",
"host": "j01",
"short_message": "msg_woop",
"timestamp": 1559741402,
"level": "5",
"facility": "local0",
"syslog-tag": "tag[19343]",
"source": "tag",
"program": "tag",
"procid": "19343"
}
Without UseSpecialParser, the datetime.c parser returns an error, presumably
because the wrong parsing format has been chosen?
the input line looks reasonable:
inputname: imuxsock rawmsg: '<133>1 2019-06-05T21:36:53.619029+08:00 j01 tag
19524 - - msg_woop'
but ends up with an odd TIMESTAMP:
FROMHOST: 'jmp', fromhost-ip: '127.0.0.1', HOSTNAME: 'jmp', PRI: 133,
syslogtag '1', programname: '1', APP-NAME: '1', PROCID: '-', MSGID: '-',
TIMESTAMP: 'Jun 5 21:36:53', STRUCTURED-DATA: '-',
msg: ' 2019-06-05T21:36:53.619029+08:00 j01 tag 19524 - - msg_woop'
escaped msg: ' 2019-06-05T21:36:53.619029+08:00 j01 tag 19524 - - msg_woop'
inputname: imuxsock rawmsg: '<133>1 2019-06-05T21:36:53.619029+08:00 j01 tag
19524 - - msg_woop'
$!:
$.:
$/:
And thus the output has the timestamp doubled up (one from the message, one
from rsyslog):
2019-06-05T21:33:47.925435+08:00 jmp 1 2019-06-05T21:33:47.925275+08:00 j01 tag
19455 - - msg_woop
debug output to console from rsyslog:
1813.748464568:main Q:Reg/w0 : wti.c: main Q:Reg/w0: worker IDLE, waiting for
work.
1813.923548068:imuxsock.c : imuxsock.c: Message from UNIX socket: #8, size
138
1813.923601868:imuxsock.c : datetime.c: ParseTIMESTAMP3339: invalid year:
1, pszTS: '2'
1813.923614768:imuxsock.c : main Q: queue.c: qqueueAdd: entry added, size
now log 1, phys 1 entries
1813.923627768:imuxsock.c : main Q: queue.c: EnqueueMsg advised worker start
1813.923634468:imuxsock.c : imuxsock.c: imuxsock calling poll() on
3 fds
1813.923648268:main Q:Reg/w0 : wti 0x8009c2b00: wti.c: worker awoke from idle
processing
1813.923657368:main Q:Reg/w0 : queue.c: DeleteProcessedBatch: we deleted 0
objects and enqueued 0 objects
1813.923663668:main Q:Reg/w0 : queue.c: doDeleteBatch: delete batch from
store, new sizes: log 1, phys 1
1813.923669668:main Q:Reg/w0 : main Q: queue.c: dequeued 1 consumable
elements, szlog 0 sz phys 1
1813.923677668:main Q:Reg/w0 : ruleset.c: processBATCH: batch of 1 elements
must be processed
The documentation [2] refers briefly to `useSpecialParser` but the
source code doesn't offer much further clarification either.
thanks
Dave
[1]:
https://github.com/freebsd/freebsd/commits/591ef7c8076109cff3c41f9bb50da996a34121e9
[2]: https://www.rsyslog.com/tag/8-9-0/
commit 591ef7c8076109cff3c41f9bb50da996a34121e9
Author: ed
Date: Fri Apr 6 13:00:45 2018 +
Let syslog(3) use RFC 5424.
With r332099 changing syslogd(8) to parse RFC 5424 formatted syslog
messages, go ahead and also change the syslog(3) libc function to
generate them. Compared to RFC 3164, RFC 5424 has various advantages,
such as sub-second precision for log entry timestamps.
As this change could have adverse effects when not updating syslogd(8)
or using a different system logging daemon, add a notice to UPDATING and
increase __FreeBSD_version.
Differential Revision: https://reviews.freebsd.org/D14926
Notes:
svn path=/head/; revision=332100
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] output numbers in JSON template
On Wed, 9 Jan 2019, at 04:21, Dave Cottlehuber wrote: > Is it possible to output unquoted numbers in a JSON template? using a > format string? > > For example: > > property(outname="timestamp" name="timereported" > dateFormat="unixtimestamp" format="jsonf") > > produces "timestamp": "123456789". > > obviously the much more error-prone does work, but then the > option.jsonf can't be used via template(name="JSON" type="list" > option.jsonf="on") any more: > > constant(value="\",\"timestamp\":") > property(name="timegenerated" dateformat="unixtimestamp") > constant(value=",\... In case others look for this in future, I added this to https://github.com/rsyslog/rsyslog/issues/2827 which seems to cover a broader range of JSON-related formatting concerns. A+ Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] output numbers in JSON template
Is it possible to output unquoted numbers in a JSON template? using a format string? For example: property(outname="timestamp" name="timereported" dateFormat="unixtimestamp" format="jsonf") produces "timestamp": "123456789". obviously the much more error-prone does work, but then the option.jsonf can't be used via template(name="JSON" type="list" option.jsonf="on") any more: constant(value="\",\"timestamp\":") property(name="timegenerated" dateformat="unixtimestamp") constant(value=",\... thanks Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] on-disk queue has many small files instead of a few larger ones
> -Original Message- > From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of > Dave Cottlehuber > Sent: Monday, January 29, 2018 8:39 > To: rsyslog@lists.adiscon.com > Subject: [rsyslog] on-disk queue has many small files instead of a few > larger ones > > Hi, > > I've got a queue that spools to disk when the log db isn't available. > It's working fine except the file sizes are 1Mb in size instead of > "something much larger" -- I'd like something over 1Gb ideally. The > spool dir is on FreeBSD zfs dataset and so a larger file size reaps huge > wins due to inbuilt compression. > > I don't particularly care what it is, but I'm not clear what settings > are causing this - what am I missing? > On Mon, 29 Jan 2018, at 22:51, Chivian, John ZKTJB5F via rsyslog wrote: > queue.maxfilesize="1g" Thanks John - perfect! A+ Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] on-disk queue has many small files instead of a few larger ones
Hi, I've got a queue that spools to disk when the log db isn't available. It's working fine except the file sizes are 1Mb in size instead of "something much larger" -- I'd like something over 1Gb ideally. The spool dir is on FreeBSD zfs dataset and so a larger file size reaps huge wins due to inbuilt compression. I don't particularly care what it is, but I'm not clear what settings are causing this - what am I missing? # forward to graylog without altering the template # buffer on disk action(type="omfwd" target="10.241.0.6" port="1514" protocol="tcp" # queue queue.fileName="graylog" queue.type="linkedlist" # ram queue.size="1" # max # of messages to hold in memory # durability queue.saveonshutdown="on" # dont lose messages on clean restart queue.maxdiskspace="50g"# approximate also compressed via zfs queue.highwatermark="9000" # start spooling to disk queue.lowwatermark="200"# resume in ram # flow control queue.TimeoutEnqueue="0"# dont throttle senders action.resumeRetryCount="-1"# infinite retries action.resumeInterval="17" # resume after connectivity loss action.reportSuspension="on"# notify when we stop, and again on start action.reportSuspensionContinuation="on" queue.dequeueBatchSize="100"# batches improve throughput ) .. -rw--- 1 root wheel 1.0M Jan 29 14:29 graylog.00011502 -rw--- 1 root wheel 1.0M Jan 29 14:29 graylog.00011503 -rw--- 1 root wheel 1.0M Jan 29 14:29 graylog.00011504 -rw--- 1 root wheel 1.0M Jan 29 14:29 graylog.00011505 -rw--- 1 root wheel 1.0M Jan 29 14:29 graylog.00011506 -rw--- 1 root wheel 1.0M Jan 29 14:29 graylog.00011507 -rw--- 1 root wheel 1.0M Jan 29 14:31 graylog.00011508 -rw--- 1 root wheel 1.0M Jan 29 14:31 graylog.00011509 -rw--- 1 root wheel 1.0M Jan 29 14:31 graylog.00011510 -rw--- 1 root wheel 1.0M Jan 29 14:31 graylog.00011511 -rw--- 1 root wheel 1.0M Jan 29 14:31 graylog.00011512 -rw--- 1 root wheel 1.0M Jan 29 14:31 graylog.00011513 -rw--- 1 root wheel 1.0M Jan 29 14:31 graylog.00011514 -rw--- 1 root wheel 1.0M Jan 29 14:34 graylog.00011515 -rw--- 1 root wheel 1.0M Jan 29 14:34 graylog.00011516 -rw--- 1 root wheel 1.0M Jan 29 14:34 graylog.00011517 -rw--- 1 root wheel 1.0M Jan 29 14:34 graylog.00011518 -rw--- 1 root wheel 1.0M Jan 29 14:34 graylog.00011519 -rw--- 1 root wheel 1.0M Jan 29 14:34 graylog.00011520 -rw--- 1 root wheel 1.0M Jan 29 14:34 graylog.00011521 -rw--- 1 root wheel 280K Jan 29 14:34 graylog.00011522 -rw--- 1 root wheel 583B Jan 29 14:34 graylog.qi # Each file has around 1200 messages in it: # strings graylog.00011528 |egrep \>End | wc -l 1296 The full config file is https://git.io/vNSet if needed. A+ Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Add the file name to syslog data
On Sun, 1 Oct 2017, at 11:50, David Lang wrote: > any time you have a question like this, first log locally with the format > RSYSLOG_DebugFormat so that you can see exactly what data you have where. > > In this case, you will need to enable metadata in your imfile config, > this will > add the filename (and other information), but it doesn't do this by > changing the > message itself, it does this by creating file (under $!) > > so you will need to look at the debug output and then decide how you are > going > to format the output message so that it contains the data. > > do you want to add a field at the beginning of the message? (will the > things you > are sending it to know what to do? or will it confuse their parsers?) > > What I like to do is to send everything as a RFC3164 message, but with > the body > of the message being a JSON structure. I set $!msg to the contents of > $msg (if a > parser hasn't already done this for me), and I create a $!trusted branch > that I > can add various metadata to (not just filename, but what input, what > machine > sent it, what machine received it, what time each machine touched the > message, > etc) > > at the final destination, I have all that data available and can either > use it, > or create a template that just writes out a RFC3164 style message with > the > original message content. Is there any reason why you prefer RFC3164 vs the later RFC5424 http://datatracker.ietf.org/doc/rfc5424 ? Thanks Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] 100% cpu lockup rsyslog 8.29 on FreeBSD 11.0Rp12 amd64 - bad config?
On Mon, 25 Sep 2017, at 08:18, Andreas Wehrmann via rsyslog wrote: > Hey Dave, > > can you check whether _this_ might be the issue: > http://lists.adiscon.net/pipermail/rsyslog/2017-August/044683.html > > I ran into the same problem (rsyslog locking up the CPU). > > Regards. > Andreas > > ___ Thanks Andreas & Andre and David for your replies. Following on from David's comments about queue configs, I re-did my configs from scratch and re-tested the last few days - no more lockups, and the desired buffering if the central logging server is unavailable/unreachable (spool to memory & then to disk if needed, and respool when remote is available) works fine! The config follows, if anybody has further suggestions I'll gladly incorporate them. This is for the client side, the central logging server is largely the same, with different forwarder requirements. I still need to check out a custom app that sends occasional invalid multiline messages, however, and see if I can tidy up the forwarding templates - graylog ultimately ends up with some messages having an incorrect hostname. I do have dtrace/dtruss messages but was unable to attach via gdb for the original high CPU issues on request, due to data in them I don't want to post publically. # Load Modules module(load="imtcp") module(load="imudp") module(load="imrelp") module(load="omrelp") module(load="imuxsock") module(load="immark") module(load="impstats") # FreeBSD kernel / console logging module(load="imklog" permitnonkernelfacility="on") # debugging config # http://www.rsyslog.com/doc/master/troubleshooting/debug.html # http://www.rsyslog.com/how-to-use-debug-on-demand/ # $DebugLevel 2 # $DebugFile /var/log/rsyslog/debug.log # buffer 10k events in RAM main_queue( queue.fileName="main" queue.type="linkedlist" # ram queue.size="1" # max # of messages to hold in memory # durability queue.saveonshutdown="on" # dont lose messages on clean restart queue.maxdiskspace="1g" # approximate also compressed via zfs queue.highwatermark="9000" # start spooling to disk queue.lowwatermark="1000"# resume in ram # flow control queue.TimeoutEnqueue="0"# dont throttle senders action.resumeRetryCount="-1"# infinite retries action.resumeInterval="17" # resume after connectivity loss action.reportSuspension="on"# notify when we stop, and again on start action.reportSuspensionContinuation="on" queue.dequeueBatchSize="100"# batches improve throughput # async flushing ioBufferSize="64k" flushOnTXEnd="off" asyncWriting="on" ) # Disk Queues $WorkDirectory /var/spool/rsyslog # dont do reverse DNS lookups it gets confusing with RFC1918 and IPv6 lookups global(net.enableDNS="off") # rsyslog templates ## handle multiline messages (e.g. from various APIs safely) # $EscapeControlCharactersOnReceive="off" ## ensure hostnames make sense when coming via UDP / jails template(name="normalised" type="string" string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n") # rsyslog input modules input(type="imtcp" address="127.0.0.1" port="514") input(type="imudp" rcvbufSize="1m" address="127.0.0.1" port="514") # allow logging via jail hosts input(type="imtcp" address="10.241.0.0" port="514") input(type="imudp" rcvbufSize="1m" address="10.241.0.0" port="514") # filters :msg, contains, "NSSWITCH(_nsdispatch)" ~ # rulesets # rsyslog output modules # base servers forward logs via relp using the hostname-fixing template action(type="omrelp" target="127.0.0.1" port="14514" template="normalised" # queue queue.fileName="relp" queue.type="linkedlist" # ram queue.size="1" # max # of messages to hold in memory # durability queue.saveonshutdown="on" # dont lose messages on clean restart queue.maxdiskspace="5g" # approximate also compressed via zfs queue.highwatermark="9000" # start spooling to disk queue.lowwatermark="1000"# resume in ram early # flow control queue.TimeoutEnqueue="0"# dont throttle senders action.resumeRetryCount="-1"# infinite retries action.resumeInterval="17" # resume after connectivity loss action.reportSuspension="on"# notify when we stop, and again on start action.reportSuspensionContinuation="on" queue.dequeueBatchSize="100"# batches improve throughput ) # log locally just like normal syslog with jail hostnames adjusted action(type="omfile" template="normalised" file="/var/log/messages" # async flushing ioBufferSize="64k" flushOnTXEnd="off" asyncWriting="on" ) ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsys
[rsyslog] 100% cpu lockup rsyslog 8.29 on FreeBSD 11.0Rp12 amd64 - bad config?
hi folks, for the last few weeks I have been seeing rsyslog locking up a single cpu at 100% on multiple servers. All are running FreeBSD 11.0R amd64p12, and rsyslog 8.29, although the lockup has been seen with prior patch levels and rsyslog versions. The general architecture is as follows: - rsyslog replaces normal freebsd syslog - listens on UDP & TCP both on jails and on main host - does basic template tidyups to correct hostnames - forwards these over a spiped[1] secure connection rather than inbuilt TLS to a centralised rsyslog logging host - central rsyslog sticks these in a file and also throws them into graylog[2] - the local rsyslog is supposed to be resilient to reboots of the central system, and to buffer locally via queue any messages in case of e.g. reboots or transient network issues, but I don't believe this is working as expected Is it possible my config[3] is somehow incorrect, and somehow causing this lock up? [1]: http://www.tarsnap.com/spiped.html [2]: https://graylog.org/ [3]: https://gist.github.com/dch/c5dcb7b6159dc2b070aefe89207c3a92#file-rsyslog-conf # /usr/local/etc/rsyslog.conf # central logging server is largely the same except has an additional stanza for graylog # Load Modules module(load="imtcp") module(load="imklog") module(load="imudp") module(load="imrelp") module(load="omrelp") module(load="imuxsock") module(load="immark") module(load="impstats") module(load="imfile" PollingInterval="5") # Disk Queues # zfs create -o canmount=off zroot/var/spool # zfs create zroot/var/spool/rsyslog # mkdir -m0700 /var/log/rsyslog $WorkDirectory /var/spool/rsyslog $MaxMessageSize 64k $ActionQueueFileName mainq $ActionQueueType LinkedList $ActionQueueSaveOnShutDown on $ActionQueueMaxDiskSpace 10g $ActionSendResendLastMsgOnReconnect on $ActionResumeRetryCount -1 # FreeBSD kernel / console logging $KLogPermitNonKernelFacility on $KlogConsoleLogLevel 10 # rsyslog templates ## handle multiline messages (e.g. from various APIs safely) $EscapeControlCharactersOnReceive="off" ## ensure hostnames make sense when coming via UDP / jails template(name="normalised" type="string" string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% example.org %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n") # rsyslog input modules input(type="imtcp" address="127.0.0.1" port="514") input(type="imudp" rcvbufSize="1m" address="127.0.0.1" port="514") # allow logging via jail hosts input(type="imtcp" address="10.241.0.0" port="514") input(type="imudp" rcvbufSize="1m" address="10.241.0.0" port="514") # filters :msg, contains, "NSSWITCH(_nsdispatch)" ~ # rulesets # rsyslog output modules # base servers forward logs using the hostname-fixing template action(type="omrelp" target="127.0.0.1" template="normalised" port="14514" ) # log locally just like normal syslog with hostnames adjusted action(type="omfile" queue.type="linkedlist" template="normalised" file="/var/log/messages" ) ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] collect and forward w/o change
On Fri, 23 Dec 2016, at 19:16, Woodruff, Dan wrote: > Then later on if you want to get real fancy, you can set up a queue on > the > ForwardToQRadar ruleset so when QRadar is down for patching logs will be > buffered and forwarded once QRadar is back up. I just got this worked a > few > weeks ago and can share the full ruleset config if you're interested. > > Hope that helps, > Dan An example of buffer/forward would be super interesting - please share Dan. A+ Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] omriemann Re: Are we building an ERK stack?
On Wed, 23 Nov 2016, at 19:18, David Lang wrote: > On Wed, 23 Nov 2016, Bob Gregory wrote: > > > For that, I'd like to see better support for GeoIP tagging, a Riemann > > output plugin, some better guidance on "failed message queues", etc. etc. > > etc. > > With a bit of digging, I can't find where Riemann defines what the > over-the-wire > format is that you would need to deliver logs to it. > > I see hints that it uses protobuf to serialize things, and has an > application-level ack mechanism similar to what we have in relp, but the > levels > of indirection are stacked high, and the API documenation only points you > at the > function defintions. > > David Lang Hi David, Bob, https://github.com/algernon/riemann-c-client may be of interest to use it directly -- its been dropped into collectd as a library now as well, and is ported to Debian & FreeBSD already, that I know of. The protobuf wire format is https://github.com/algernon/riemann-c-client/blob/master/lib/riemann/proto/riemann.proto if that's helpful. License is LGPL3 and I can't work out what rsyslog mainly comes under. What I've found useful with collectd and riemann was to be able to set specific custom tags per instance (rsyslog server in our case) which makes the sorting in riemann very easy prior to parsing any specific message output. Mainly source & instance type: A+ Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] imudp missing packets on FreeBSD 11R amd64 & 8.22.0
> > hmm, I thought recvmmsg was a linux-only thing. Is it supported on > > freebsd? and > > if so, does it work the same as on Linux? > > > > try disabling it and try again. > > Thanks David. Hi David, This syscall has been added in 11.0-RELEASE. You were spot on - thanks for the tip! This patch fixes the issue in FreeBSD itself: https://reviews.freebsd.org/rS307311 https://svnweb.freebsd.org/base/head/lib/libc/gen/recvmmsg.c?view=log&pathrev=307036 It's in 11-STABLE & 12-CURRENT. rsyslog works as expected using recvmmsg in these builds. Matthew Seaman provided a patch for FreeBSD ports tree to exclude the recvmmsg usage at configure time only if the FreeBSD version is not recent enough to contain that patch, so future builds of rsyslog against a suitable FreeBSD release will use recvmmsg correctly. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214428 The patch has been applied to both the trunk & the quarterly release branch: https://svnweb.freebsd.org/ports/head/sysutils/rsyslog8/Makefile?revision=425998&view=markup https://svnweb.freebsd.org/ports/branches/2016Q4/sysutils/rsyslog8/Makefile?view=patch&r1=426016&r2=426015&pathrev=426016 which means other FreeBSD users should get an update in the near future too. Thanks Matthew & David for your timely expertise! Cheers Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] imudp missing packets on FreeBSD 11R amd64 & 8.22.0
> > - rsyslog doesn't have any log entry, just 2 cryptic lines: > > > > 3092.216287535:imudp.c: imudp: recvmmsg returned 1 > > 3092.216304260:imudp.c: imudp: recvmmsg returned -1 > > hmm, I thought recvmmsg was a linux-only thing. Is it supported on > freebsd? and > if so, does it work the same as on Linux? > > try disabling it and try again. Thanks David. recvmmsg is definitely a FreeBSD thing but how compatible is it I can't tell. https://www.freebsd.org/cgi/man.cgi?query=recvmmsg&manpath=FreeBSD+11.0-RELEASE I can't see any mechanism in ./configure to disable this however. I did run it under dtrace (see below for details) and got -1 and EAGAIN which looks reasonable: The recv(), recvfrom() and recvmsg() return the length of the message on successful completion, whereas recvmmsg() returns the number of received messages. If a message is too long to fit in the supplied buffer, excess bytes may be discarded depending on the type of socket the message is received from (see socket(2)). If no messages are available at the socket, the receive call waits for a message to arrive, unless the socket is non-blocking (see fcntl(2)) in which case the value -1 is returned and the global variable errno is set to EAGAIN. The receive calls except recvmmsg() normally return any data available, up to the requested amount, rather than waiting for receipt of the full amount requested; this behavior is affected by the socket-level options SO_RCVLOWAT and SO_RCVTIMEO described in getsockopt(2). The recvmmsg() function implements this behaviour for each message in the vector. & from here on I'm a bit lost to be honest! I'm happy to grab anything else useful. same content at https://gist.github.com/dch/7e7d2be70e419eb2cad55e00a0430d5b # /usr/local/share/dtrace-toolkit/dtruss -p 46181 -a -s PID/LWPRELATIVE ELAPSDCPU SYSCALL(args) = return 46181/100880:26 74 8 write(0x1, "8713.574640499:main thread: janitorRun() called\n\0", 0x34) = 52 0 libc.so.7`_write+0xa rsyslogd`dbgprint+0x211 rsyslogd`dbgprintf+0xe9 rsyslogd`janitorRun+0x18 rsyslogd`main+0x613 rsyslogd`_start+0x17f ld-elf.so.1`0x8006b3000 46181/101196:10 55 3 recvmsg(0x5, 0x801E3BE00, 0x0) = 17 0 libc.so.7`__sys_recvmsg+0xa imudp.so`wrkr+0x432 imudp.so`runInput+0xba rsyslogd`thrdStarter+0x9d libthr.so.3`0x8014eab55 46181/101196:11 1 0 recvmsg(0x5, 0x801E3BE38, 0x0) = -1 Err#35 libc.so.7`__sys_recvmsg+0xa imudp.so`wrkr+0x432 imudp.so`runInput+0xba rsyslogd`thrdStarter+0x9d libthr.so.3`0x8014eab55 46181/101196:24 4 2 write(0x1, "8804.086743385:imudp.c: imudp: recvmmsg returned 1\n\0", 0x3B) = 59 0 libc.so.7`_write+0xa rsyslogd`dbgprint+0x211 rsyslogd`dbgprintf+0xe9 imudp.so`wrkr+0x468 imudp.so`runInput+0xba rsyslogd`thrdStarter+0x9d libthr.so.3`0x8014eab55 46181/101196:30 1 0 recvmsg(0x5, 0x801E3BE00, 0x0) = -1 Err#35 libc.so.7`__sys_recvmsg+0xa imudp.so`wrkr+0x432 imudp.so`runInput+0xba rsyslogd`thrdStarter+0x9d libthr.so.3`0x8014eab55 46181/101196:33 2 0 write(0x1, "8804.086771347:imudp.c: imudp: recvmmsg returned -1\n\0", 0x3C) = 60 0 libc.so.7`_write+0xa rsyslogd`dbgprint+0x211 rsyslogd`dbgprintf+0xe9 imudp.so`wrkr+0x468 imudp.so`runInput+0xba rsyslogd`thrdStarter+0x9d libthr.so.3`0x8014eab55 46181/101196:36 2 0 getsockname(0x5, 0x7FFFDF7F98A0, 0x7FFFDF7F989C) = 0 0 libc.so.7`_getsockname+0xa imudp.so`wrkr+0x22a imudp.so`runInput+0xba rsyslogd`thrdStarter+0x9d libthr.so.3`0x8014eab55 46181/101196:41 4 2 write(0x1, "8804.086789418:imudp.c: Listening on UDP syslogd socket 5 (IPv4/port 514).\n\0", 0x53)= 83 0 libc.so.7`_write+0xa rsyslogd`dbgprint+0x211 rsyslogd`dbgprintf+0xe9 lmnet.so`debugListenInfo+0xa7 imudp.so`wrkr+0x22a imudp.so`runInput+0xba rsyslogd`thrdStarter+0x9d libthr.so.3`0x8014eab55 46181/101196:45 2 1 write(0x1, "8804.086801418:imudp.c: imUDP calling select,
[rsyslog] imudp missing packets on FreeBSD 11R amd64 & 8.22.0
Hi folks, I'm trying to understand why I don't have any logfile entries from rsyslog over UDP. Have I missed something very obvious? - FreeBSD 11.0-p3 amd64 - rsyslog 8.22.0 from ports - /dev/log works - I can see the outgoing UDP packet on lo0 - rsyslog doesn't have any log entry, just 2 cryptic lines: 3092.216287535:imudp.c: imudp: recvmmsg returned 1 3092.216304260:imudp.c: imudp: recvmmsg returned -1 # /usr/local/etc/rsyslog.conf # # /usr/local/etc/rsyslog.conf module(load="imklog") module(load="imudp") input(type="imudp") action(type="omfile" queue.type="linkedlist" name="rsyslog" file="/var/log/messages" ) # check /dev/log is working # root@wintermute /u/h/dch# jobs Job Group State Command 1 44200 running tail -F /var/log/messages & root@wintermute /u/h/dch# sockstat -46l |grep 514 root spiped 1064 3 tcp4 *:4514*:* root@wintermute /u/h/dch# env RSYSLOG_DEBUG=LogFuncFlow /usr/local/sbin/rsyslogd -i /var/run/syslog.pid -f /usr/local/etc/rsyslog.conf -dn & 5550.429656821:main thread: rsyslogd 8.22.0 startup, module path '', cwd:/usr/home/dch ... root@wintermute /u/h/dch# sockstat -46l |grep 514 root rsyslogd 44540 5 tcp4 127.0.0.1:514 *:* root rsyslogd 44540 7 udp6 *:514 *:* root rsyslogd 44540 8 udp4 *:514 *:* root rsyslogd 44540 9 tcp6 *:43514 *:* root rsyslogd 44540 10 tcp4 *:43514 *:* root@wintermute /u/h/dch# logger test 5560.724784811:imuxsock.c : Message from UNIX socket: #11 ... 2016-11-11T23:06:00.724791+00:00 wintermute dch: test # try same thing over network # root@wintermute /u/h/dch# ifconfig lo0 -rxcsum -txcsum root@wintermute /u/h/dch# ifconfig lo0 lo0: flags=8149 metric 0 mtu 16384 options=60 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff00 nd6 options=21 groups: lo root@wintermute /u/h/dch# jobs Job Group State Command 2 46225 running tcpdump -i lo0 -n -v -v -s 1500 -X port 514 & 1 46198 running tail -F /var/log/messages & root@wintermute /u/h/dch# env RSYSLOG_DEBUG=LogFuncFlow /usr/local/sbin/rsyslogd -i /var/run/syslog.pid -f /usr/local/etc/rsyslog.conf -dn & root@wintermute /u/h/dch# logger -4 -h 127.0.0.1 test 01:11:32.216222 IP (tos 0x0, ttl 64, id 24624, offset 0, flags [none], proto UDP (17), length 41) 127.0.0.1.18606 > 127.0.0.1.514: [udp sum ok] SYSLOG, length: 13 Facility user (1), Severity notice (5) Msg: dch: test 0x: 3c31 333e 6463 683a 2074 6573 74 0x: 4500 0029 6030 4011 1c92 7f00 0001 E..)`0..@... 0x0010: 7f00 0001 48ae 0202 0015 811c 3c31 333e H...<13> 0x0020: 6463 683a 2074 6573 74 dch:.test 3092.216287535:imudp.c: imudp: recvmmsg returned 1 3092.216304260:imudp.c: imudp: recvmmsg returned -1 3092.216312801:imudp.c: Listening on UDP syslogd socket 4 (IPv4/port 514). 3092.216316025:imudp.c: imUDP calling select, active file descriptors (max 4): 4 more details at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214428 A+ Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
