Re: [rsyslog] AMQP as log destination?
On Fri, 7 Dec 2012, Fabio Sangiovanni wrote: acronym alert, what does AMQP stand for? It's a standard protocol to communicate with message queueing systems. "the nice thing about standards is that there are so many to choose from" and http://xkcd.com/927/ http://www.amqp.org/ it's vision sounds nice "To become the standard protocol for interoperability between all messaging middleware" but it's "yet another standard" to compete with all the others http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol Message queueing system implementations that support AMQP: RabbitMQ, Apache QPid It would not be that hard to get AMQP added to rsyslog, you would just need to have someone either write input and output modules (probably adapted from the 0mq modules to start with), or sponsor development through Adiscon professional services (e-mail Rainer directly to get a quote, it's usually surprisingly cheap) persistent on-disk queues are already an option. Yes, and I'd be happy to use them together with an AMQP output plugin for reliable massive log processing. rsyslog already supports Reliable Event Logging Protocol (RELP), and 0MQ options for reliable massive log processing. This would just be one additional option (and there's nothing wrong with supporting lots of options) My logging system processed 18B lines of logs in October, it's handled 93K lines of logs in a single second, and I've tested it up to 380K lines of logs per second (effectivly gig-E wire speed), and others have used rsyslog in environments where they have tested it to 1M lines of logs per second. However, "reliable" is a relative term. reliability and performance tend to be opposed and you have to make tradeoffs between the two. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Multiple Apache vhosts, keep individual log files AND send to remote logstash
On Fri, 7 Dec 2012, Ben Bradley wrote: Hi everyone I've just started investigating centralised logging and I'm gradually building up a plan of action. I'd like to store the logs on a central server running logstash/ElasticSearch so they can be searched and monitored using Kibana. With rsyslog sending the logs over the network to a logstash server. I don't want to run logstash as the log "sender" on each server, I'd prefer to keep the servers (log "clients") as lean and simple possible. So that means either using syslog, syslog-ng or the one I'm testing now, rsyslog. 1) Should I have rsyslog sending to logstash over the network? Or should I be running another rsyslog on the collector server, which then sends to logstash for processing? This is up to you, there are advantages in each direction. Using rsyslog for all network transport and having it deliver locally to logstash/elasticsearch/other for processing means that you you can take advantage of all rsyslog features for your transport. In a centralized environment your traffic volumes can be high, rsyslog can handle very high traffic levels, can your other software? If you are really comforatable with logstash, you may want to eliminate the need to run one more daemon, but when you hire new people and hand the system over, should they need to be as comfortable with logstash? They will have to be comforatable with rsyslog in any case. At that point which is the 'extra' thing to deal with, rsyslog or logstash? For Apache, I would like to have separate vhost log files on the web server, in addition to these logs being sent to a remote log collector. I've tested rsyslog using the imfile module to watch each Apache log files, but this means I have to hard-code each vhost log file into my rsyslog.conf. This is not ideal as people will invariably forget when they add/remove sites on the server. 2) What's the best way to log to both vhost-specific log files on the web server and to send these logs over the network, without using imfile and manually watching tens of individual log files? Get Apache to log to rsyslog, then have rsyslog split the log to both a file and over the network to logstash? Are there big performance implications for logging both locally and over the network? I could change my Apache config to log to a single access/error log for all vhosts, then watch these main log files with imfile. So long as rsyslog is then able to produce vhost-specific log files somewhere on the web server machine. it depends on how you format the log file. If you have the logfile start with the vhost name, then rsyslog can easily produce per-host files (look in the rsyslog documentation for the dynafile templates. another approach you can do is have apache log to a local named pipe and have a process listen on that named pipe and tagging/reformatting the log file and pass it to your syslog server. David Lang Any comments/suggestions? I am sure others have had a similar need. I just don't want to ditch local log files until we fully know how well the centralised log server performs. Thanks in advance! Cheers, Ben ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] rsyslog 7.2.4 (v7-stable) released
Hi all, we have just released 7.2.4 of the v7 stable branch. This is a pure bug-fixing release. More information on the changes can be found in the ChangeLog. ChangeLog: http://www.rsyslog.com/changelog-for-7-2-4-v7-stable/ Download: http://www.rsyslog.com/rsyslog-7-2-4-v7-stable/ As always, feedback is appreciated. Best regards, Tim Eifler ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Multiple Apache vhosts, keep individual log files AND send to remote logstash
Hi Ben, 2012/12/7 Ben Bradley > Hi everyone > > I've just started investigating centralised logging and I'm gradually > building up a plan of action. > > I'd like to store the logs on a central server running > logstash/ElasticSearch so they can be searched and monitored using Kibana. > With rsyslog sending the logs over the network to a logstash server. I > don't want to run logstash as the log "sender" on each server, I'd prefer > to keep the servers (log "clients") as lean and simple possible. So that > means either using syslog, syslog-ng or the one I'm testing now, rsyslog. > > 1) Should I have rsyslog sending to logstash over the network? Or should I > be running another rsyslog on the collector server, which then sends to > logstash for processing? > Yes, I think it's better to send logs directly to logstash, so you won't have to maintain yet another rsyslog daemon. > > > For Apache, I would like to have separate vhost log files on the web > server, in addition to these logs being sent to a remote log collector. > > I've tested rsyslog using the imfile module to watch each Apache log > files, but this means I have to hard-code each vhost log file into my > rsyslog.conf. This is not ideal as people will invariably forget when they > add/remove sites on the server. > > 2) What's the best way to log to both vhost-specific log files on the web > server and to send these logs over the network, without using imfile and > manually watching tens of individual log files? > Just a quick note: I assume imfile can be changed to support wildcards and/or templates. > Get Apache to log to rsyslog, then have rsyslog split the log to both a > file and over the network to logstash? > Yeah, if you can do that it seems like the best solution. I don't know how you can do it, since I have little experience with Apache. > Are there big performance implications for logging both locally and over > the network? > I would assume rsyslog won't break a sweat, if that's what you're thinking about. I would only consider the storage as a possible issue (eg: you might want to make sure you logrotate&compress logs that you store locally) > > I could change my Apache config to log to a single access/error log for > all vhosts, then watch these main log files with imfile. So long as rsyslog > is then able to produce vhost-specific log files somewhere on the web > server machine. > Although I've never done it, I think you can use mmnormalize to make rsyslog parse your Apache logs: http://www.rsyslog.com/using-rsyslog-mmnormalize-module-effectively-with-adiscon-loganalyzer/ > > > Any comments/suggestions? > I am sure others have had a similar need. I just don't want to ditch local > log files until we fully know how well the centralised log server performs. > That sounds reasonable :) One suggestion: if you have lots of logs, you may want to skip using logstash, which is another moving piece which can also be a bottleneck. You can get your rsyslog to output directly to Elasticsearch, and still use Kibana on top of that. Of course, this is applicable only if you don't need a logstash-specific feature, such as grok. For rsyslog, you'll need omelasticsearch, here's a (little old) tutorial on using it: http://wiki.rsyslog.com/index.php/HOWTO:_rsyslog_%2B_elasticsearch and some more on queues (for performance and reliability): http://wiki.rsyslog.com/index.php/Queues_on_v6_with_omelasticsearch All you have to do in addition to that is to make sure your timestamp goes in a field called "@timestamp" in ES, because that's hardcoded in Kibana so it can sort your logs. Then, for a field to be shown in Kibana, it also needs to begin with an "@" as far as I know. Best regards, Radu ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Multiple Apache vhosts, keep individual log files AND send to remote logstash
Hi Ben, you can use a syslogtag to separate one vHost from another in your vHost log directive. You have to cretae a template on your logserver which stores your logfiles separated by your syslogtag rgds Axel Am 07.12.2012 12:45, schrieb Ben Bradley: 2) What's the best way to log to both vhost-specific log files on the web server and to send these logs over the network, without using imfile and manually watching tens of individual log files? Get Apache to log to rsyslog, then have rsyslog split the log to both a file and over the network to logstash? Are there big performance implications for logging both locally and over the network? I could change my Apache config to log to a single access/error log for all vhosts, then watch these main log files with imfile. So long as rsyslog is then able to produce vhost-specific log files somewhere on the web server machine. Any comments/suggestions? I am sure others have had a similar need. I just don't want to ditch local log files until we fully know how well the centralised log server performs. Thanks in advance! Cheers, Ben ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Never argue with an idiot; people watching may not tell the difference ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] Multiple Apache vhosts, keep individual log files AND send to remote logstash
Hi everyone I've just started investigating centralised logging and I'm gradually building up a plan of action. I'd like to store the logs on a central server running logstash/ElasticSearch so they can be searched and monitored using Kibana. With rsyslog sending the logs over the network to a logstash server. I don't want to run logstash as the log "sender" on each server, I'd prefer to keep the servers (log "clients") as lean and simple possible. So that means either using syslog, syslog-ng or the one I'm testing now, rsyslog. 1) Should I have rsyslog sending to logstash over the network? Or should I be running another rsyslog on the collector server, which then sends to logstash for processing? For Apache, I would like to have separate vhost log files on the web server, in addition to these logs being sent to a remote log collector. I've tested rsyslog using the imfile module to watch each Apache log files, but this means I have to hard-code each vhost log file into my rsyslog.conf. This is not ideal as people will invariably forget when they add/remove sites on the server. 2) What's the best way to log to both vhost-specific log files on the web server and to send these logs over the network, without using imfile and manually watching tens of individual log files? Get Apache to log to rsyslog, then have rsyslog split the log to both a file and over the network to logstash? Are there big performance implications for logging both locally and over the network? I could change my Apache config to log to a single access/error log for all vhosts, then watch these main log files with imfile. So long as rsyslog is then able to produce vhost-specific log files somewhere on the web server machine. Any comments/suggestions? I am sure others have had a similar need. I just don't want to ditch local log files until we fully know how well the centralised log server performs. Thanks in advance! Cheers, Ben ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] AMQP as log destination?
Il giorno 06/dic/2012, alle ore 23:36, Radu Gheorghe ha scritto: > Hi Fabio, > > If you need AMPQ for integration with other apps, you can try Logstash with > syslog input and AMPQ output. > > That said, Logstash needs AMPQ to have a persistency layer when shipping > logs, since it has very limited queueing. As David pointed out, rsyslog has > on disk and in memory queues which offer that out of the box. So if you > need AMPQ for mass log shipping, I think you should consider using rsyslog > directly. You can couple it directly to quite a lot of stuff, like > Elasticsearch or MongoDB. > I know about logstash, but AMQP protocol is marked as "unsupported"; other than that, I'd really prefer not to use other software as relay, to keep the infrastructure as simple as possible. > And if you're looking for is missing (like AMPQ is), you can always develop > input/output plugins or get some custom development from Adiscon to do that > for you. I would assume this would be the good, clean solution in the long > run. > That would be nice, I'll contact them :) > Otherwise, like Jerome suggested, you can always hack a little script to do > what you want and use omprog to pipe all logs to that script. > > Best regards, > Radu > Fabio ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Ubuntu 12 (Precise) v7-devel packages
Thank you very much Andre! I will test it and report the results. Sorry for the delay in replying, I was hoping to come back with the feedback directly, but something has come up and my tests will be delayed for a while. Best regards, Radu 2012/12/6 Andre Lorbach > Hi, > > thanks for sending me your changed files. I merged them into my build > system and updated our Ubuntu Packages to "rsyslog_7.3.4-1adiscon2". > Maybe you can run some tests with the new packages (rsyslog-elasticsearch, > rsyslog-imptcp, rsyslog-mmjsonparse and rsyslog-mongodb) on your system > using the Adiscon Ubuntu Repository? > > Best regards, > Andre Lorbach > > > -Original Message- > > From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog- > > boun...@lists.adiscon.com] On Behalf Of Radu Gheorghe > > Sent: Mittwoch, 5. Dezember 2012 16:28 > > To: rsyslog-users > > Subject: Re: [rsyslog] Ubuntu 12 (Precise) v7-devel packages > > > > Hi, > > > > As you might have figured, we need Ubuntu packages as well. Especially > for > > Precise. > > > > The main question I have is where to contribute, and are contributions > > needed? I think it would be nice to join forces somehow, for making > recent > > rsyslog packages available. > > > > To be more specific, we need rsyslog and rsyslog-relp, with > omelasticsearch, > > imptcp and mmjsonparse on top of them. So I took the source package from > > Todd's PPA (I hope you don't mind, Todd!), and from it I've built: > > - rsyslog-elasticsearch > > - rsyslog-imptcp > > - rsyslog-mmjsonparse > > > > I also did a minor fix by taking out the "-c5" parameter from > /etc/default and > > the init script. All in all it seems to work fine - although there's > quite some > > more work ahead, like making it available for 32-bit and for other > versions of > > Ubuntu. > > > > Now to put the question into context, for me (and others interested in > the > > components mentioned above), I see the following options of publishing > the > > packages I've done: > > 1. my own PPA/repo > > 2. Adiscon's repo. Or PPA if you guys want to make one 3. Todd's PPA 4. > > Debian Experimental repo > > > > Of course that, except for option 1, the maintainers would have to agree > first > > :) > > > > What do you think or suggest? > > > > Best regards, > > Radu > > > > 2012/12/1 Andre Lorbach > > > > > > thanks for that effort. Without having had a closer look at the > > > > package > > > itself, I > > > > just wondered if you based it on the latest Ubuntu or Debian package? > > > > > > It is based on the latest Ubuntu RSyslog package I could install on > > > Ubuntu 12.04. > > > What I basically did was taking the package source, modifying, adding > > > and updating dependencies like libee, libestr, librelp. > > > Then I created a local repository using mini-dinstall and dput, and > > > added all these packages to it. > > > After initially and successful testing, I uploaded the local > > > repository to our webserver. > > > > > > > Also, an observation while skimming through the repo: The 0ubuntu? > > > > versioning scheme is usually reserved for official Ubuntu packages. > > > > You could use 0adisconX or something like that. This would have the > > > > additional benefit, that once there is an official Ubuntu package > > > available, it > > > > would supersede your version as XXX-0ubuntuX > XXX-0adisconX > > > > > > Thanks for the hint, I wasn't aware of this versioning fact. Your > > > recommendation sounds reasonable, I will change this with the next > > > package update. > > > > > > Best regards, > > > Andre Lorbach > > > ___ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > ___ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond > > our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-se
Re: [rsyslog] AMQP as log destination?
Il giorno 06/dic/2012, alle ore 20:47, Jerome Renard ha scritto: > Not AMQP, but maybe you will find omzmq3 useful [1] > > You can also use the omprog module [2] and from your program send logs > to a RabbitMQ server (or anything else that support AMQP) > > 'Hope that help :) > > 1. > http://git.adiscon.com/?p=rsyslog.git;a=tree;f=plugins/omzmq3;h=6c9f8763a462af4756a6c4579dc3b27c82722b19;hb=HEAD > 2. http://www.rsyslog.com/doc/rsyslog_conf_modules.html/omprog.html > Hi, unfortunately those are not options. We need to integrate with an AMQP system, and omprog module would mean too much perfomance loss and one more possibile point of failure. Fabio > -- > Jérôme > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] AMQP as log destination?
acronym alert, what does AMQP stand for? It's a standard protocol to communicate with message queueing systems. http://www.amqp.org/ http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol Message queueing system implementations that support AMQP: RabbitMQ, Apache QPid persistent on-disk queues are already an option. Yes, and I'd be happy to use them together with an AMQP output plugin for reliable massive log processing. Fabio ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
