Re: [rsyslog] Metrics: rsyslog_queue_full_counter vs rsyslog_queue_discarded_full_counter?

2024-06-04 Thread David Lang via rsyslog
my software can handle the load? Best regards Johan Ryberg On Fri, 31 May 2024 at 20:40, Rainer Gerhards via rsyslog < rsyslog@lists.adiscon.com> wrote: Plus we wait a bit (pushback to sender) when the next messages come in. But all depends on queue config. Rainer Sent from phone, thus b

Re: [rsyslog] Metrics: rsyslog_queue_full_counter vs rsyslog_queue_discarded_full_counter?

2024-06-04 Thread David Lang via rsyslog
here any other tuning that might be suggested if we play with the thought that my software can handle the load? Best regards Johan Ryberg On Fri, 31 May 2024 at 20:40, Rainer Gerhards via rsyslog < rsyslog@lists.adiscon.com> wrote: Plus we wait a bit (pushback to sender) when the next messages

Re: [rsyslog] Question on legacy syntax vs action RainerScript

2024-05-31 Thread David Lang via rsyslog
Rory Toma wrote: However, we are missing some messages on the backed that get logged locally. I'm trying to eliminate causes, and want to make sure of something. Before, with legacy syntax, we'd use *.* Syntax so everything was logged. Is this the default or do I need to add something to

Re: [rsyslog] Metrics: rsyslog_queue_full_counter vs rsyslog_queue_discarded_full_counter?

2024-05-31 Thread David Lang via rsyslog
when the queue hits full, you you aren't yet losing a message, it's the next message that arrives while the queue is full that is lost. David Lang ___ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] unencrypted rsyslog through a wireguard terminal

2024-05-30 Thread David Lang via rsyslog
resending the configs On Tue, 28 May 2024, David Lang wrote: let's simplify this to the minimum needed *Server** # I've tried both with and without the line below $ModLoad imtcp $InputTCPServerRun 12345 $template LDSTag, "<%PRI>%TIMESTAMP [nameofsystem] %syslog% %msg%" # 10.50.x.x is

Re: [rsyslog] unencrypted rsyslog through a wireguard terminal

2024-05-30 Thread David Lang via rsyslog
. It's primarily there for technical reasons, and it is the default. I strongly advise against defining encryption settings and then disabling them this way. Rainer El mié, 29 may 2024 a las 11:26, David Lang via rsyslog () escribió: On Wed, 29 May 2024, Kathy Lyons wrote: which part sets encry

Re: [rsyslog] unencrypted rsyslog through a wireguard terminal

2024-05-30 Thread David Lang via rsyslog
ult. I strongly advise against defining encryption settings and then disabling them this way. Rainer El mié, 29 may 2024 a las 11:26, David Lang via rsyslog () escribió: On Wed, 29 May 2024, Kathy Lyons wrote: which part sets encryption? I thought these options set encryption to 0, or disab

Re: [rsyslog] why is my config ignored

2024-05-29 Thread David Lang via rsyslog
it can be matched, it's just not what you thought it was. log with the RSYSLOG_DebugFormat template and you will see what $syslogtag contains. David Lang On Wed, 29 May 2024, sacawulu via rsyslog wrote: ok... but then... what's the use of being able to assign a tag with "logger -t TAG"

Re: [rsyslog] why is my config ignored

2024-05-29 Thread David Lang via rsyslog
log the message with the template RSYSLOG_DebugFormat so you can see all the details about how rsyslog is seeing the message. I'm not sure if _ is valid as a syslog tag, but if the debug output shows that it's not being parsed into the tag field, try without that. we really would need to see

Re: [rsyslog] unencrypted rsyslog through a wireguard terminal

2024-05-29 Thread David Lang via rsyslog
Rainer Gerhards wrote: Mode 0 indeed does turn any encryption off. It's primarily there for technical reasons, and it is the default. I strongly advise against defining encryption settings and then disabling them this way. Rainer El mié, 29 may 2024 a las 11:26, David Lang via rsyslog

Re: [rsyslog] unencrypted rsyslog through a wireguard terminal

2024-05-29 Thread David Lang via rsyslog
On Wed, 29 May 2024, Kathy Lyons wrote: which part sets encryption? I thought these options set encryption to 0, or disabled. leave out all the encryption settings to have them be disabled, setting the mode to anon turns on encryption, but accepting any cert. David Lang On Tue, May 28,

Re: [rsyslog] unencrypted rsyslog through a wireguard terminal

2024-05-28 Thread David Lang via rsyslog
On Tue, 28 May 2024, Kathy Lyons wrote: let's simplify this to the minimum needed *Server** # I've tried both with and without the line below $ModLoad imtcp $InputTCPServerRun 12345 $template LDSTag, "<%PRI>%TIMESTAMP [nameofsystem] %syslog% %msg%" # 10.50.x.x is where the server sends its

Re: [rsyslog] unencrypted rsyslog through a wireguard terminal

2024-05-28 Thread David Lang via rsyslog
On Tue, 28 May 2024, Kathy Lyons wrote: I hope this is better. much better We have a server and two clients with wireguard on them. The server is 10.10.10.1, the first client is 10.10.10.2 and the second client is 10.10.10.3. The tunnel works and I can ping from server -> clients and vice

Re: [rsyslog] unencrypted rsyslog through a wireguard terminal

2024-05-28 Thread David Lang via rsyslog
your message is badly linewrapped, can you please try again? also note that while you can ping between the systems, that doesn't mean that port 514 (TCP or UDP) can get through, either due to firewalls at the network layer or iptables on the systems David Lang On Tue, 28 May 2024, Kathy

Re: [rsyslog] need help with rsyslog

2024-05-27 Thread David Lang via rsyslog
8.24 is ancient (with some unknown additional backports by redhat), so it's very possible that you are using options that it doesn't know about if you do rsyslogd -N1 does it report any errors in the config file? I know that imfile has been re-written at least once since 8.24 It would be far

Re: [rsyslog] Problem with omprog

2024-05-24 Thread David Lang via rsyslog
If you specify omprog in your config and then try to start rsyslog, do you get any error messages? if the omprog module is not installed, you should get an error trying to load it. David Lang On Fri, 24 May 2024, Mårten Persson via rsyslog wrote: Date: Fri, 24 May 2024 21:03:56 +0200 From:

Re: [rsyslog] Stop actions

2024-05-24 Thread David Lang via rsyslog
if you start rsyslog with the -o /path/to/file option, it will write a copy of the config file as it sees it with all includes, that is what you should look at to figure the order of things. Many distros put the includes late in the config, so putting things in an included file may be too late

Re: [rsyslog] Stop actions

2024-05-24 Thread David Lang via rsyslog
or you have other actions in the config that happen before your stop takes place. David Lang On Fri, 24 May 2024, Rainer Gerhards via rsyslog wrote: Date: Fri, 24 May 2024 13:57:07 +0200 From: Rainer Gerhards via rsyslog To: Thomas Raef Cc: Rainer Gerhards , rsyslog-users Subject: Re:

Re: [rsyslog] Unable to re-use variable generated by mmnormalize

2024-05-22 Thread David Lang via rsyslog
On Wed, 22 May 2024, Adam Cecile via rsyslog wrote: Yes I'll consider that if needed, those are old servers, most of other are a lot newer and are running recent rsyslogd, so maybe I'll leave it like this. Can you explain me a bit more what kind of input should I use to re-inject my imfile

Re: [rsyslog] Unable to re-use variable generated by mmnormalize

2024-05-22 Thread David Lang via rsyslog
8.24 was released back in 2017. RedHat has backported some fixes and features from newer versions of rsyslog (which were released every 6 weeks for years, now every 8 weeks), but only they track what is and isn't in there. If you are going to really start using the power of rsyslog, I would

Re: [rsyslog] Unable to re-use variable generated by mmnormalize

2024-05-21 Thread David Lang via rsyslog
Adam Cecile wrote: You got it ! It does not like rules setting, switching to external file with rulebase works... Probably a too old version. probably It also does not seems to be able to set path, to nest new properties, but this is not really important. that is not something that is

Re: [rsyslog] Unable to re-use variable generated by mmnormalize

2024-05-21 Thread David Lang via rsyslog
if you do rsyslogd -N1 does it complain about anything? David Lang On Wed, 22 May 2024, Adam Cecile via rsyslog wrote: Date: Wed, 22 May 2024 00:32:25 +0200 From: Adam Cecile via rsyslog To: Adam Cecile via rsyslog Cc: Adam Cecile Subject: Re: [rsyslog] Unable to re-use variable generated

Re: [rsyslog] Unable to re-use variable generated by mmnormalize

2024-05-21 Thread David Lang via rsyslog
if you post that exact text into your liblognorm test, what do you get? David Lang On Wed, 22 May 2024, Adam Cecile via rsyslog wrote: Date: Wed, 22 May 2024 00:24:08 +0200 From: Adam Cecile via rsyslog To: Adam Cecile via rsyslog Cc: Adam Cecile Subject: Re: [rsyslog] Unable to re-use

Re: [rsyslog] Unable to re-use variable generated by mmnormalize

2024-05-21 Thread David Lang via rsyslog
if you look at the msg field in the RSYSLOG_DebugFormat output, you will see that it does have a leading space. your pattern doesn't David Lang On Tue, 21 May 2024, Adam Cecile via rsyslog wrote: Date: Tue, 21 May 2024 23:58:23 +0200 From: Adam Cecile via rsyslog To: Adam Cecile via rsyslog

Re: [rsyslog] Unable to re-use variable generated by mmnormalize

2024-05-21 Thread David Lang via rsyslog
log the message with RSYSLOG_DebugFormat so that you can see the variables that exist. my guess is that your rule needs a leading space, because the msg field you are parsing starts with a space (a very common problem when you are starting to use mmnormalize) David Lang On Tue, 21 May

Re: [rsyslog] [Maybe OFF-TOPIC] Add TAG in syslogd/klogd

2024-05-05 Thread David Lang via rsyslog
On Sun, 5 May 2024, Alberto via rsyslog wrote: El 5/5/24 a las 22:02, David Lang escribió: ... I only need filter by source, but all fields (FROMHOST, HOSTNAME, FROMHOST-IP...) that can give me any information are useless because appears Docker host IP, not real source host IP, and I cannot

Re: [rsyslog] [Maybe OFF-TOPIC] Add TAG in syslogd/klogd

2024-05-05 Thread David Lang via rsyslog
On Sun, 5 May 2024, Alberto via rsyslog wrote: Hi David, This system don't have any MAN. I only need filter by source, but all fields (FROMHOST, HOSTNAME, FROMHOST-IP...) that can give me any information are useless because appears Docker host IP, not real source host IP, and I cannot

Re: [rsyslog] [Maybe OFF-TOPIC] Add TAG in syslogd/klogd

2024-05-05 Thread David Lang via rsyslog
On Sun, 5 May 2024, Alberto via rsyslog wrote: I have a host with very old firmware that I cannot update, with syslogd/klogd 1.5.0. I'm sending their logs to remote Rsyslog server (Docker container actually), but when I filter for get files by hostname/source IP..., I don't get real

Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20

2024-04-23 Thread David Lang via rsyslog
file = "/var/log/node/Tlog.log" ) cheers ian -----Original Message- From: rsyslog On Behalf Of David Lang via rsyslog Sent: Friday, April 19, 2024 12:44 PM To: David Lang via rsyslog Cc: David Lang Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade

Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20

2024-04-22 Thread David Lang via rsyslog
= "imfile" pollingInterval = "1" statefile.directory = "/var/log/node" ) input( type = "imfile" tag = "tserv-stdout" facility = "local4" severity = "info" file = "/var/log/node/Tlog.log" ) cheer

Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20

2024-04-22 Thread David Lang via rsyslog
On Mon, 22 Apr 2024, Ian Diddams wrote: Is there any chance that they are getting logged under a different hostname? I've done an extensive search in the rsyslog server this morning, and the answer is sadly no. try logging the local4 facility to a different, fixed file (rather than a

Re: [rsyslog] Too many queue files

2024-04-19 Thread David Lang via rsyslog
This is showing that your omfwd is running into grief delivering messages. I have also seen the queue sizes exceed what's configured by a few percentage, I don't know the reason for that. I suspect that it has something to do with the batch size and batches being counted rather than individual

Re: [rsyslog] Too many queue files

2024-04-19 Thread David Lang via rsyslog
stats every 60s is not a problem I think Rainer or someone else from Adiscon will need to weigh in. They are based in Germany so we may not see anything from them until after the weekend. your explination of the watermarks is not quite how I've understood them, but I haven't used them much

Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20

2024-04-19 Thread David Lang via rsyslog
that puts them in different directories based on the hostname. David Lang On Fri, 19 Apr 2024, David Lang via rsyslog wrote: Date: Fri, 19 Apr 2024 03:59:53 -0700 (PDT) From: David Lang via rsyslog To: Ian Diddams via rsyslog Cc: David Lang Subject: Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog

Re: [rsyslog] [EXTERNAL] Re: imfile rsyslog config sporadic since upgrade to ubuntu20

2024-04-19 Thread David Lang via rsyslog
Is there any chance that they are getting logged under a different hostname? David Lang On Fri, 19 Apr 2024, Ian Diddams via rsyslog wrote: Date: Fri, 19 Apr 2024 09:24:03 + From: Ian Diddams via rsyslog To: "rsyslog@lists.adiscon.com" Cc: Ian Diddams Subject: Re: [rsyslog] [EXTERNAL]

Re: [rsyslog] imfile rsyslog config sporadic since upgrade to ubuntu20

2024-04-19 Thread David Lang via rsyslog
Caveat: I've recently inherited a bunch of ubuntu systems with very little historical knowledge available to me. The issue described below has occurred only since the inline ubuntu upgrade was done. However, Ive no vision of whether this is solely an issue with ubuntu implementation of rsyslog,

Re: [rsyslog] Possibility to introduce a new log crypto provider

2024-04-17 Thread David Lang via rsyslog
On Wed, 17 Apr 2024, Attila Lakatos via rsyslog wrote: On Tue, Apr 16, 2024 at 1:17 PM Derek Atkins via rsyslog < rsyslog@lists.adiscon.com> wrote: Hi David, On Tue, April 16, 2024 6:32 am, David Lang via rsyslog wrote: > Is there any way to duplicate the existing functionality wit

Re: [rsyslog] Possibility to introduce a new log crypto provider

2024-04-16 Thread David Lang via rsyslog
While this approach makes it easier to add new algorithms, it isn't going to do anything to reduce the load on the maintainers. Is there any way to duplicate the existing functionality with openssl or gnutls libraries? given that some people prefer openssl and some prefer gnutls, I think

Re: [rsyslog] rate limiting logs written to a file

2024-04-09 Thread David Lang via rsyslog
On Mon, 8 Apr 2024, Prasad Koya wrote: Yes, we get a lot of flexibility with omprog. However, changing a syslog's facility/priority isn't possible right? Please see below. if ($syslogfacility-text == 'daemon' and $msg contains "Out of memory") then { action(type="omfile"

Re: [rsyslog] rate limiting logs written to a file

2024-04-08 Thread David Lang via rsyslog
not easily within rsyslog, with an event correlation engine, you have a lot more capability (which is why I keep pushing you that way :-) ) you can create a template to use when outputting a log, and that log can say anything (it can be arbitrary text unrelated to the log you received), but

Re: [rsyslog] rate limiting logs written to a file

2024-04-08 Thread David Lang via rsyslog
you would need to do the filtering with omprog on the sending machines. I would suggest that rather than throwing them away, you generate a log every rate-limiting period along the lines of "X number of messages happened in the last period" This is really a job for an event correlation

Re: [rsyslog] RHEL-7.9 rsyslog with FileOwner

2024-04-06 Thread David Lang via rsyslog
: with CAP_CHOWN) may    change the group arbitrarily. So if your rsyslogd is an unprivileged process (it does not have CAP_CHOWN granted explicitly and it does not run as root user), you won' be able to create files as a different user. MK On 6.04.2024 07:20, David Lang via rsyslog wrote

Re: [rsyslog] RHEL-7.9 rsyslog with FileOwner

2024-04-05 Thread David Lang via rsyslog
if you are using the action() syntax, you set the ownership as part of the action. if you post your full config (including included files) we can better guess what's wrong with it. David Lang On Sat, 6 Apr 2024, warron.french via rsyslog wrote: I am running multiple servers on RHEL-7.9 at

Re: [rsyslog] rate limiting logs written to a file

2024-04-02 Thread David Lang via rsyslog
rate limiting output will just cause things to back up, it doesn't throw away the messages. It would be better for you to detect these messages and feed them into an external event correlation engine (Simple Event Correltator for example), and have that engine then send you logs that you keep

Re: [rsyslog] Bash history --> rsyslog relay --> Elastic

2024-03-25 Thread David Lang via rsyslog
Ok, the fact that you are getting other logs remotely does eliminate the permission/network problems. That just means that the filters you are applying to find the bash logs are not matching the log contents. To figure this out, you need to figure out what is actually being sent (since it's

Re: [rsyslog] Bash history --> rsyslog relay --> Elastic

2024-03-25 Thread David Lang via rsyslog
on the receiving system, log the messages with the template RSYSLOG_DebugFormat and give us a sample message. note that there are other reasons why you may see a log message with tcpdump but rsyslog will not process it, including if there is not a route back to the sender, or if there are

Re: [rsyslog] question: decoding cisco syslog messages using pmciscoios

2024-03-22 Thread David Lang via rsyslog
good to hear, If you can identify what in the logging configuration was different that caused this problem, please post it to the list so that others can learn from it. David Lang On Fri, 22 Mar 2024, Pedro Caetano via rsyslog wrote: Starting from a minimal configuration on the switch

Re: [rsyslog] question: decoding cisco syslog messages using pmciscoios

2024-03-21 Thread David Lang via rsyslog
ok, the rawmsg field is telling us that the body of the message sent by the device is a bunch of nonsense (control characters with a lot of nulls) so the problem is on the sending side, not on the rsyslog side, look at your options there.. David Lang On Thu, 21 Mar 2024, Pedro Caetano via

Re: [rsyslog] Capturing shutdown logs

2024-03-20 Thread David Lang via rsyslog
when you use imjournal with rsyslog, journald is storing the logs in it's database, then rsyslog is periodically querying the database for new logs. that database can be all in ram, or partially on disk. David Lang On Wed, 20 Mar 2024, David Lang via rsyslog wrote: Date: Wed, 20 Mar 2024 12

Re: [rsyslog] Capturing shutdown logs

2024-03-20 Thread David Lang via rsyslog
 PM Peter Portante via rsyslog < rsyslog@lists.adiscon.com> wrote: Attila, any reason you can't just use persistent journald? That is what we did to solve the lost shutdown and crash logs. -Peter On Fri, Mar 15, 2024 at 12:31 PM David Lang via rsyslog wrote: > > imjournal uses the

Re: [rsyslog] question: decoding cisco syslog messages using pmciscoios

2024-03-20 Thread David Lang via rsyslog
please log some messages with the template RSYSLOG_DebugFormat so we can see exactly what is being sent, along with all the variables that it's being parsed into. (and go ahead and send those in the email rather than posting to pastebin) David Lang On Wed, 20 Mar 2024, Pedro Caetano via

Re: [rsyslog] Logs received via tls are binary

2024-03-19 Thread David Lang via rsyslog
we would need to see your full configs on both the sender and receiving sides to make a guess as to what is being done wrong. David Lang On Tue, 19 Mar 2024, Brian via rsyslog wrote: I will have to take a closer look at logs but yes, the logs in the receiving syslog server are showing up as

Re: [rsyslog] PMCISCOIOS Module Issue

2024-03-19 Thread David Lang via rsyslog
if Redhat does not provide you with the pmciscoios module, then you need to upgrade to a version that the community supports install that package from the community repo. see https://www.rsyslog.com/rhelcentos-rpms/ for instructions. David Lang P.S. this list is very much still alive and the

Re: [rsyslog] Bash history --> rsyslog relay --> Elastic

2024-03-19 Thread David Lang via rsyslog
please post your full configs, it's likely that there are other things in the config that are causing issues. note that when you are within an if statement, you don't need to do the &, just a bare stop will apply to everything that the if matched. David Lang On Tue, 19 Mar 2024, Kees de

Re: [rsyslog] custom properties and arm performance

2024-03-15 Thread David Lang via rsyslog
On Fri, 15 Mar 2024, John Chivian via rsyslog wrote: Is there a way to add custom/user properties? That’s what the STRUCTURED_DATA header element is for. in theory yes, in practice RFC5424 does not have broad support for things like structured data. What is becoming more common in

Re: [rsyslog] Capturing shutdown logs

2024-03-15 Thread David Lang via rsyslog
imjournal uses the journal api to fetch the logs (fetching them in near-real-time), journald keeps files internally to support it. David Lang On Fri, 15 Mar 2024, Attila Lakatos via rsyslog wrote: The solution is clean to me, however I think this could be a bottleneck for busy systems. Also,

Re: [rsyslog] Capturing shutdown logs

2024-03-13 Thread David Lang via rsyslog
you could put the remote sender things in a seprate ruleset with a queue on that ruleset, that would let the rest of the config run without the network (accumulating early logs and gathering shutdown logs up to the point that rsyslog gets shut down) you can configure rsyslog to save the queue

Re: [rsyslog] Journald sending log to remote Rsyslog

2024-03-12 Thread David Lang via rsyslog
journald does not have the ability to send over the network to a syslog server. when you configure journald to send to a syslog daemon, journald throws away a lot of details that it knows. The other option is to use imjournal in rsyslog to fetch the logs from journald David Lang On Tue, 12

Re: [rsyslog] Tuning rsyslog to forward logs

2024-03-05 Thread David Lang via rsyslog
The queue fills up because rsyslog is not able to deliver the logs fast enough. You are sending the logs via TCP (encrypted) so the sending is throttled to the speed that the receiving system can accept them. Are you using Splunk as the syslog listener to accept the messages? Splunk is a very

Re: [rsyslog] Short source hostname

2024-02-24 Thread David Lang via rsyslog
On Sat, 24 Feb 2024, Mariusz Kruk via rsyslog wrote: On 23.02.2024 20:29, Frank Morawietz via rsyslog wrote: You could also just do your own field based on the $fromhost value, just split at first dot. set $.loghost=field($hostname,46,1); Then you can use the $.loghost variable in your

Re: [rsyslog] Short source hostname

2024-02-23 Thread David Lang via rsyslog
$fromhost is the result of a name lookup of the IP of the connection, it is always the full result of that lookup (which will normally be a FQDN from DNS David Lang On Fri, 23 Feb 2024, Frank Morawietz via rsyslog wrote: Thanks for your reply, Mariusz. Also remember that fromhost and

Re: [rsyslog] bind one queue to a ruleset for all syslogs going to one server.

2024-02-14 Thread David Lang via rsyslog
On Tue, 13 Feb 2024, Prasad Koya via rsyslog wrote: While these messages are deep copied and enqueued to respective action queues, it's possible that the second message may reach the remote syslog server before the first message. I'd like to avoid that situation. How do I define one queue for

Re: [rsyslog] Cant parse incoming json via relp

2024-02-14 Thread David Lang via rsyslog
what is the config of the receiver? I'll note that what you are sending is valid json, but not a valid syslog message, that could be why you are running into grief. log using the template RSYSLOG_DebugFormat on the receiver so we can see exactly what it's getting. Also, you have a comment

Re: [rsyslog] Rsyslog stop working when output destination (rabbitmq) is unavialable

2024-02-07 Thread David Lang via rsyslog
on. Perhaps I'm missing some other "action" setting? Appreciate if you can give me a pointer to a sample configuration or point me to relevant documentation. Thank you. On Mon, Feb 5, 2024 at 11:11 AM David Lang via rsyslog < rsyslog@lists.adiscon.com> wrote: > you hav

Re: [rsyslog] Rsyslog stop working when output destination (rabbitmq) is unavialable

2024-02-06 Thread David Lang via rsyslog
is not helping in my situation. Perhaps I'm missing some other "action" setting? Appreciate if you can give me a pointer to a sample configuration or point me to relevant documentation. Thank you. On Mon, Feb 5, 2024 at 11:11 AM David Lang via rsyslog < rsyslog@lists.adiscon.com>

Re: [rsyslog] Action queue with omkafka

2024-02-05 Thread David Lang via rsyslog
you should be able to configure kafka to not throw away logs in it's queue. In your example below, you have configured rsyslog to throw away messages when the queue fills up to 6 messages. can you show the pstats data that shows that rsyslog is dropping messages? David Lang On Tue, 6

Re: [rsyslog] imfile readTimeout behavior question

2024-02-05 Thread David Lang via rsyslog
I remember hearing about this sort of problem before, the version you are using is 2 years old, please update to a current version and check again. note that the version you are running is different from the 8.2202 that the rsyslog project shipped, and is entirely supported by redhat as a

[rsyslog] doc request

2024-02-05 Thread David Lang via rsyslog
As the docs are getting cleaned up, these three pages should either be combined or put close enough to each other that when you find one the other two are just a click away https://www.rsyslog.com/doc/concepts/queues.html https://www.rsyslog.com/doc/whitepapers/queues_analogy.html

Re: [rsyslog] Rsyslog stop working when output destination (rabbitmq) is unavialable

2024-02-05 Thread David Lang via rsyslog
you have a queue of 1024 for rabbitmq, if there are more messages than that pending, other processing will stop until the queue can accept more messages. Setup a larger queue (potentially a disk assisted queue) to handle longer outages. you may also want to consider configuring the queue to

Re: [rsyslog] Ubuntu 22.04 rsyslog client connection over RELP over TLS

2024-02-04 Thread David Lang via rsyslog
The error is very clear that rsyslog is not able to read the file, so either you have the wrong path, or there is a permission problem (classic/apparmor/selinux permissions) If you enable debug logging and capture that, you can go through it looking for the error message and see exactly what

Re: [rsyslog] Ubuntu 22.04 rsyslog client connection over RELP over TLS

2024-02-04 Thread David Lang via rsyslog
on many systems, the permissions of a program started at boot are no longer simple root (systemd is being configured to to retrict the programs significantly So I would suggest that you try starting rsyslog as root manually and see if that avoids this error message. If so, then it's a

Re: [rsyslog] Installing rsyslog/omhttp on Rocky linux 9

2024-01-23 Thread David Lang via rsyslog
the build instructions for how they are compiled is in the rsyslog git repos, different rpm based distros have different versions of packages on them that rsyslog depends on. If the wrong ones change in incompatible ways, it won't work. If the Rocky devs are shipping rsyslog, see what their

Re: [rsyslog] New Doc Effort

2024-01-06 Thread David Lang via rsyslog
On Sat, 6 Jan 2024, Rainer Gerhards via rsyslog wrote: Hi all, thanks for the great feedback! Any help is appreciated. I am actually looking for four kind of things right now in regard to the core doc: * how would a beginner's guide structure best be? What do we need to describe for someone

Re: [rsyslog] Syslog configuration issues

2024-01-03 Thread David Lang via rsyslog
the error messages you are posting say you are running an amazon-modified version of 8.24 rsyslogd: version 8.24.0-57.amzn2.2.0.2, config validation run (level 1), master config /etc/rsyslog.conf RedHat (which amazon linux is a fork of) used 8.24 on redhat 7 (released in June 2014), RedHat

Re: [rsyslog] Syslog configuration issues

2024-01-03 Thread David Lang via rsyslog
This looks to me like it's a problem with the library, not with the ca cert also, rsyslog 8.24 is very old, and there have been a lot of improvements since, especially related to TLS connections. based on the package name, I would guess this is an amazon AWS image, and you should look to

Re: [rsyslog] Does HUP signal cause a TLS certificate reload?

2023-12-30 Thread David Lang via rsyslog
a HUP will reconnect, but I don't think that a HUP will reload the certificates from disk. David Lang On Sat, 30 Dec 2023, John Chivian via rsyslog wrote: I believe restarting is the only way possible to achieve this. Certificates are connection based and therefore you must force the client

Re: [rsyslog] Confirming the behavior of omhttp in the face of 4xx errors

2023-12-06 Thread David Lang via rsyslog
we would be open to a patch for a flag that dropped the offending message and kept going (see my other message about batch handling) but not to just ignore the message and retry. David Lang On Wed, 6 Dec 2023, Peter Portante via rsyslog wrote: Hello Rsyslog People! I have been working with

Re: [rsyslog] logrotate wont compress-rotate syslog templated files

2023-11-30 Thread David Lang via rsyslog
On my central rsyslog servers, I have rsyslog write the logs out to a directory tree, then have a cron job that does a mv to a parallel directory structure on the same filesystem (this is very fast and atomic, even when there are a lot of files), then kick rsyslog to have it start writing to

Re: [rsyslog] logrotate wont compress-rotate syslog templated files

2023-11-14 Thread David Lang via rsyslog
it very much looks like a logrotate issue. David Lang On Tue, 14 Nov 2023, Dimi Onobodies via rsyslog wrote: So I added "-v" option on the cronjob and redirected output to a file. I observed the following: considering log /data/servers/rsyslog/ldap-access.log log needs rotating rotating

[rsyslog] variable related feature request

2023-11-03 Thread David Lang via rsyslog
I was explaining the different variable/property types and the history around the inconsistancy yesterday. Today I thought of a couple possible config options that can't be made default without breaking things, but could be default on new configs and simplify the variable mess. 1st,

Re: [rsyslog] [rsyslog/rsyslog-doc] documentation is hard to use and badly structured (#394)

2023-11-02 Thread David Lang via rsyslog
On Thu, 2 Nov 2023, computerquip-work wrote: This is a bit unorganized of a take so I'm going to apologize ahead of time. These are the things I could think of off the top of my head. 1. Documentation is unclear and doesn't take itself seriously. What I mean by this is that it states things

Re: [rsyslog] Hostname field changed in rsyslog proxy

2023-11-02 Thread David Lang via rsyslog
hostname should not change, fromhost and fromhost-ip will change. David Lang On Thu, 2 Nov 2023, Martin Passard via rsyslog wrote: Date: Thu, 2 Nov 2023 14:46:34 + From: Martin Passard via rsyslog To: "rsyslog@lists.adiscon.com" Cc: Martin Passard Subject: [rsyslog] Hostname field

Re: [rsyslog] [rsyslog/rsyslog-doc] documentation is hard to use and badly structured (#394)

2023-11-02 Thread David Lang via rsyslog
We have received complaints about rsyslog documentation repeatedly, We have a lot of detail, but it's all written for someone already fairly familiar with things. Here is a 3am first pass from me at writing an overview of how rsyslog works, with the idea that this could be made pretty with

Re: [rsyslog] HOSTNAME property truncating strings with certain special characters('.')

2023-11-02 Thread David Lang via rsyslog
There is an option to allow FQDNs in the hostname (it's a violation of the RFC, but commonly needed) see https://www.rsyslog.com/doc/master/rainerscript/global.html (the new way of setting global parameters) or https://www.rsyslog.com/doc/master/configuration/global/index.html (the old way,

Re: [rsyslog] RFC: removing rsyslog.service samples from git

2023-10-20 Thread David Lang via rsyslog
Makes sense to me, they are far more dependent on the distro decisions than anything we provide. David Lang On Fri, 20 Oct 2023, John Chivian via rsyslog wrote: Hi Rainer: Our source of truth for rsyslog.service has always been the copy packaged with the OS, and all modifications to it

Re: [rsyslog] Rsyslog error: imfile: wd 1 already in wdmap

2023-10-11 Thread David Lang via rsyslog
please post your full config (you can have rsyslog combine all include files into one file to see them as rsyslog does by starting rsyslog with -o /path/to/file) It sounds as if you have additional imfile inputs that already specify these files, but without the full config, it's hard to guess

Re: [rsyslog] Need help with Rsyslog Parser

2023-10-11 Thread David Lang via rsyslog
P.S. this confusion of templates being useful for parsing messages seems to be a common one, any suggestions on what we should put in the documentation to make clear that they are for output only, not for parsing messages? David Lang On Wed, 11 Oct 2023, Gundlapally, Navanitha via rsyslog

Re: [rsyslog] Need help with Rsyslog Parser

2023-10-11 Thread David Lang via rsyslog
Templates are how you format messages that you are outputting, they have nothing to do with parsing messages. I would first suggest that you log the message with the template RSYSLOG_DebugFormat so that you can see all the variables that get parsed out of the message already, and what is

Re: [rsyslog] Help in getting omudpspoof.so of rsyslog

2023-10-11 Thread David Lang via rsyslog
most distros have additional rsyslog-* packages that include modules that have other dependencies, and it's common for omudpspoof to be in those additional packages. David Lang On Wed, 11 Oct 2023, Raghunatha Reddy wrote: Dear David & Rainer, How are you doing? As part of my work, I need

Re: [rsyslog] [RFC] locking down rsyslog.service on Debian

2023-10-10 Thread David Lang via rsyslog
On Tue, 10 Oct 2023, Michael Biebl wrote: Am Di., 10. Okt. 2023 um 21:49 Uhr schrieb David Lang : I see people putting things in /etc/rsyslog.d besides configs, so locking down /etc may trip them up. ProtectSystem=full will make /etc read-only. Do you have a use case in mind where rsyslog

Re: [rsyslog] [RFC] locking down rsyslog.service on Debian

2023-10-10 Thread David Lang via rsyslog
I see people putting things in /etc/rsyslog.d besides configs, so locking down /etc may trip them up. looking for workdir in the config will identify the directory that rsyslog needs to be able to write to for state and similar. It's also common for people to have rsyslog write to locations

Re: [rsyslog] [RFC] locking down rsyslog.service on Debian

2023-10-10 Thread David Lang via rsyslog
what directories will rsyslog be able to access (both read and write) with this config? David Lang On Tue, 10 Oct 2023, Michael Biebl via rsyslog wrote: Date: Tue, 10 Oct 2023 20:20:14 +0200 From: Michael Biebl via rsyslog To: rsyslog-users Cc: Michael Biebl Subject: [rsyslog] [RFC]

Re: [rsyslog] Input module for custom log files and JSON format

2023-10-01 Thread David Lang via rsyslog
look at mmnormalize for ways to parse the message into various fields under $! that you can then use in templates. There is a very simple json parsing option as part of this. David Lang On Mon, 2 Oct 2023, Karsten Ohme via rsyslog wrote: Hi all, I'm looking for an input plugin support

Re: [rsyslog] Templates and Separating Logs

2023-09-28 Thread David Lang via rsyslog
the first thing I would suggest is to stop mixing sytax types (I may have been incorrect here in piecing the config back together from the mangling in the mail) if $fromhost contains 'a8-ansi-d00' then { authpriv.* -?SECU *.info;mail.none;authpriv.none;cron.none-?MESG

Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx

2023-09-21 Thread David Lang via rsyslog
hmm, dlang@dlang-mobile:~$ nslookup 66.167.227.145 8.8.8.8 145.227.167.66.in-addr.arpa name = mail.lang.hm. Authoritative answers can be found from: dlang@dlang-mobile:~$ nslookup mail.lang.hm 8.8.8.8 Server: 8.8.8.8 Address:8.8.8.8#53 Non-authoritative answer: Name:

Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx

2023-09-21 Thread David Lang via rsyslog
That is a queue on the output, but the incoming message still goes to the main queue. create a ruleset for the input and put a queue on that ruleset to avoid the message going into the main queue. when you say you aren't useing journald, and are just sending the logs to systemd, you aren't

Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx

2023-09-21 Thread David Lang via rsyslog
On Thu, 21 Sep 2023, TG Servers wrote:  I did not get a single message from you David regarding that, that confused me quite a bit as Rainer mentioned you already before, now I know why : 450 4.7.25 Client host rejected: cannot find your hostname, [66.167.xxx.xxx]; from= to= proto=ESMTP helo=

Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx

2023-09-21 Thread David Lang via rsyslog
if you are sending logs to journald and having journald send logs to syslog, you are using journald as a queue for the delivery when you were delivering directly to rsyslog, what was probably happening (we don't know because you never enabled impstats to see) is that the logs were arriving,

Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx

2023-09-21 Thread David Lang via rsyslog
depends on the journald config. It can be configured to queue to disk, with limits on disk size. David Lang On Thu, 21 Sep 2023, Rainer Gerhards wrote: I guess it works because journal always throws messages away if it cannot deliver them quickly. Luke a very short timeout+drop queue config

Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx

2023-09-21 Thread David Lang via rsyslog
now you have journald acting as a queue, so all messages from journald will end up delayed when your script cannot keep up. You haven't solved the problem of the slow script, you've just added another layer of buffer to fill up before you notice. with rsyslog you can set the queue size to

  1   2   3   4   5   6   7   8   >