> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Friday, May 20, 2011 11:49 AM > To: [email protected] > Subject: [rsyslog] Heavy stability problems when using TLS > > Hi Rainer, > > thank you for your reply, unfortunately my options are quite limited, here... > > But, first things first: > What I am trying to achieve is this: > > ... use rsyslog for a centralized log server and log transmission > > should be as safe and secure as possible. In our case, that means we > > want to use syslog via TCP/TLS wherever possible and fall back to TCP > > or UDP for those nodes that don't support TLS (or even TCP). > > In more detail, our plan is to have rsyslog listening on 3 ports: > * 514 for syslog/UDP > * 10514 for syslog/plain TCP > * 10515 for syslog/TLS/TCP > > UDP seems fine (in order to debug the crashes, I disabled it recently, but > until then it worked well). > For the TCP stuff, I tried to first start a listener on port 10514 (plain TCP), set > all the TLS parameters and then start the TLS listener on port 10515. That > resulted in port 10515 accepting plain TCP messages, but no TLS. Not starting > the listener on port 10514 in the first place made port 10515 run as expected > (TLS). Since, all nodes that were configured to forward logs to the central > server so far support TLS, I decided to "postpone" this problem for now - I > guess it is some misconfiguration. To be more specific the relevant part of > the configuration was: > > $ModLoad immark # provides --MARK-- message capability #$ModLoad > imudp # provides UDP syslog reception $ModLoad imtcp # provides TCP > syslog reception and GSS-API (if compiled to support it) $ModLoad imsolaris > # Reads local Solaris log messages including the kernel log. > > ############################# > # plain TCP # > ############################# > > #$InputTCPServerRun 10514 > > ############################# > # plain UDP # > ############################# > > #$UDPServerRun 514 > > ############################# > # TCP/TLS # > ############################# > > # make gtls driver the default > $DefaultNetstreamDriver gtls > > # certificate files > $DefaultNetstreamDriverCAFile /etc/rsyslog/CA/ca-cert.pem > $DefaultNetstreamDriverCertFile > /etc/rsyslog/machine_certificates/machine-cert.pem > $DefaultNetstreamDriverKeyFile > /etc/rsyslog/machine_certificates/machine-key.pem > > # Hosts > $InputTCPServerStreamDriverPermittedPeer host1 > $InputTCPServerStreamDriverPermittedPeer host2 > (...) > > $InputTCPServerStreamDriverAuthMode x509/name > $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > $InputTCPServerRun 10515 # start up listener at port 10515 > > If I can not get it working like this, I could still imagine to start more instances > of rsyslog, where each run another port, or something like that - I am not > really concerned about this topic, so far.
I think imtcp has the plumbing to use TLS and plain concurrently, but needs some code changes to actually do that. The obvious option is to use imptcp for plain tcp, which offers better performance as well. But it is a Linux-centric module, I am not sure if it will run at all on Solaris (this module is strictly targeted towards platforms that support the required APIs, in order to gain speed). > > > About the crashes: > This problem cannot be postponed, as it has the potential to become a show- > stopper for rsyslog in our environment. > I am working in a quite big environment and while I have the server in my > hands (however, can't change it to another OS due to company policies - > solaris would not have been my first choice!), the clients are managed by a > vendor. Since logging is not their main business, they don't support any > version changes or alike. > > At the bottom line: > * I can change the rsyslog installation/configuration on the server. > * I can collect any logs etc. on the server. > * I can set up a Linux box, but can not put it in the real network. > * I can not change anything on the clients, but I can request configuration > changes > * I might be able to have the vendor collect logs on the client(s). > > I know (very well!!), that this makes it quite difficult to identify the problem, > but I can't help it... :-( This makes finding a solution really hard. I have worked in similar environments on custom contracts, but even then it is pretty hard to find the problem if we can't reproduce it with sufficient effort. > > What would you suggest me to do in order to find the problem under this > conditions? Well, I think the key thing is try to setup some lab where the crashes happen as well. I guess it is either message-induced or induced by some issues in Solaris threading. Maybe it helps if you record traffic (from sufficient sources) and replay it against a Linux box. If that one crashes, we have more options. If it does not crash, this does not directly point us to something, unfortunately... Rainer > > Best regards > Ole > > > -----Ursprüngliche Nachricht----- > Von: [email protected] [mailto:rsyslog- > [email protected]] Im Auftrag von Rainer Gerhards > Gesendet: Freitag, 20. Mai 2011 09:30 > An: rsyslog-users > Betreff: Re: [rsyslog] Heavy stability problems when using TLS > > Consolidated from both messages (so read in full ;)) > > > -----Original Message----- > > From: [email protected] [mailto:rsyslog- > > [email protected]] On Behalf Of [email protected] > > Sent: Friday, May 20, 2011 9:12 AM > > To: [email protected] > > Subject: Re: [rsyslog] Heavy stability problems when using TLS > > > > ... still crashes with 5.8.1 ... > > > > In the core dumps (as far as I can read something in there - had to > > switch them to minimal content, otherwise they rapidly fill up my HD, > > unfortunately that makes them unreadable to pstack), I see messages > > like: > > * errno=50: Cannot assign requested address. socket: 3 > > * Urign requested address. socket (...binary rubbish...) Bind error, > > errno=50: Cannot assign requested address. socket: 3 > > > > > > No luck in capturing a crash, when running rsyslog in debug mode, so > > far, will keep on trying... > > > > Any advice? > > > > We aim to use rsyslog for a centralized log server and log > > transmission should be as safe and secure as possible. In our case, > > that means we want to use syslog via TCP/TLS wherever possible and > > fall back to TCP or UDP for those nodes that don't support TLS (or even > TCP). > > > > I sent a post in April, already, in order to find out, if it is > > possible to run plain TCP and TCP/TLS on the same server instance > > (seems it does not work, so far). > > Can you elaborate on what you exactly desire. I think it is possible even > without modification. > > > At this time, we managed to connect a > > few nodes via TCP/TLS, but the server and apparently even the clients > > keep crashing several times a day, causing a loss messages. > > > > For obvious reasons, this is not what we want, at all. > > > > I already found a bug report, that seems to point in the right > > direction, but I don't see a solution even though the status is > > "resolved": > > > > * http://bugzilla.adiscon.com/show_bug.cgi?id=194 > > > > * and its spin off > > http://bugzilla.adiscon.com/show_bug.cgi?id=222 > > > > What was the solution? > You need to read the tracker in full. Unfortunately, some new bug was > mangled into it when the real one was fixed. Bug 222 handles this new > problem. 222 has solution information for it towards the end. > > From what I read, I think your problem is not related to these two trackers > (except that it seems to occur with TLS as well, which is not really a > relationship ;)). > > > What can be done to prevent the crashes? > > > > I can provide whatever information you want (at least in regard to the > > server), but I would rather not post our config to the public. > > > > The server is using rsyslog 5.8.0 on Solaris and the clients that are > > currently connected run version 4.4.2 on Red Hat. > > I'd suggest to upgrade the clients first, and see if they continue crashing. > Based on the results, let's look further. Alternatively, move the server to a > Linux box and check if it keep crashing. Solaris is very different in regard to > threading, so this may give us a clue. > > HTH > Rainer > > > > Best regards > > Ole > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
