Re: [rsyslog] AMQP as log destination?
acronym alert, what does AMQP stand for? It's a standard protocol to communicate with message queueing systems. http://www.amqp.org/ http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol Message queueing system implementations that support AMQP: RabbitMQ, Apache QPid persistent on-disk queues are already an option. Yes, and I'd be happy to use them together with an AMQP output plugin for reliable massive log processing. Fabio ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] AMQP as log destination?
Il giorno 06/dic/2012, alle ore 20:47, Jerome Renard jerome.ren...@gmail.com ha scritto: Not AMQP, but maybe you will find omzmq3 useful [1] You can also use the omprog module [2] and from your program send logs to a RabbitMQ server (or anything else that support AMQP) 'Hope that help :) 1. http://git.adiscon.com/?p=rsyslog.git;a=tree;f=plugins/omzmq3;h=6c9f8763a462af4756a6c4579dc3b27c82722b19;hb=HEAD 2. http://www.rsyslog.com/doc/rsyslog_conf_modules.html/omprog.html Hi, unfortunately those are not options. We need to integrate with an AMQP system, and omprog module would mean too much perfomance loss and one more possibile point of failure. Fabio -- Jérôme ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Ubuntu 12 (Precise) v7-devel packages
Thank you very much Andre! I will test it and report the results. Sorry for the delay in replying, I was hoping to come back with the feedback directly, but something has come up and my tests will be delayed for a while. Best regards, Radu 2012/12/6 Andre Lorbach alorb...@ro1.adiscon.com Hi, thanks for sending me your changed files. I merged them into my build system and updated our Ubuntu Packages to rsyslog_7.3.4-1adiscon2. Maybe you can run some tests with the new packages (rsyslog-elasticsearch, rsyslog-imptcp, rsyslog-mmjsonparse and rsyslog-mongodb) on your system using the Adiscon Ubuntu Repository? Best regards, Andre Lorbach -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog- boun...@lists.adiscon.com] On Behalf Of Radu Gheorghe Sent: Mittwoch, 5. Dezember 2012 16:28 To: rsyslog-users Subject: Re: [rsyslog] Ubuntu 12 (Precise) v7-devel packages Hi, As you might have figured, we need Ubuntu packages as well. Especially for Precise. The main question I have is where to contribute, and are contributions needed? I think it would be nice to join forces somehow, for making recent rsyslog packages available. To be more specific, we need rsyslog and rsyslog-relp, with omelasticsearch, imptcp and mmjsonparse on top of them. So I took the source package from Todd's PPA (I hope you don't mind, Todd!), and from it I've built: - rsyslog-elasticsearch - rsyslog-imptcp - rsyslog-mmjsonparse I also did a minor fix by taking out the -c5 parameter from /etc/default and the init script. All in all it seems to work fine - although there's quite some more work ahead, like making it available for 32-bit and for other versions of Ubuntu. Now to put the question into context, for me (and others interested in the components mentioned above), I see the following options of publishing the packages I've done: 1. my own PPA/repo 2. Adiscon's repo. Or PPA if you guys want to make one 3. Todd's PPA 4. Debian Experimental repo Of course that, except for option 1, the maintainers would have to agree first :) What do you think or suggest? Best regards, Radu 2012/12/1 Andre Lorbach alorb...@ro1.adiscon.com thanks for that effort. Without having had a closer look at the package itself, I just wondered if you based it on the latest Ubuntu or Debian package? It is based on the latest Ubuntu RSyslog package I could install on Ubuntu 12.04. What I basically did was taking the package source, modifying, adding and updating dependencies like libee, libestr, librelp. Then I created a local repository using mini-dinstall and dput, and added all these packages to it. After initially and successful testing, I uploaded the local repository to our webserver. Also, an observation while skimming through the repo: The 0ubuntu? versioning scheme is usually reserved for official Ubuntu packages. You could use 0adisconX or something like that. This would have the additional benefit, that once there is an official Ubuntu package available, it would supersede your version as XXX-0ubuntuX XXX-0adisconX Thanks for the hint, I wasn't aware of this versioning fact. Your recommendation sounds reasonable, I will change this with the next package update. Best regards, Andre Lorbach ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
Re: [rsyslog] AMQP as log destination?
Il giorno 06/dic/2012, alle ore 23:36, Radu Gheorghe radu0gheor...@gmail.com ha scritto: Hi Fabio, If you need AMPQ for integration with other apps, you can try Logstash with syslog input and AMPQ output. That said, Logstash needs AMPQ to have a persistency layer when shipping logs, since it has very limited queueing. As David pointed out, rsyslog has on disk and in memory queues which offer that out of the box. So if you need AMPQ for mass log shipping, I think you should consider using rsyslog directly. You can couple it directly to quite a lot of stuff, like Elasticsearch or MongoDB. I know about logstash, but AMQP protocol is marked as unsupported; other than that, I'd really prefer not to use other software as relay, to keep the infrastructure as simple as possible. And if you're looking for is missing (like AMPQ is), you can always develop input/output plugins or get some custom development from Adiscon to do that for you. I would assume this would be the good, clean solution in the long run. That would be nice, I'll contact them :) Otherwise, like Jerome suggested, you can always hack a little script to do what you want and use omprog to pipe all logs to that script. Best regards, Radu Fabio ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] Multiple Apache vhosts, keep individual log files AND send to remote logstash
Hi everyone I've just started investigating centralised logging and I'm gradually building up a plan of action. I'd like to store the logs on a central server running logstash/ElasticSearch so they can be searched and monitored using Kibana. With rsyslog sending the logs over the network to a logstash server. I don't want to run logstash as the log sender on each server, I'd prefer to keep the servers (log clients) as lean and simple possible. So that means either using syslog, syslog-ng or the one I'm testing now, rsyslog. 1) Should I have rsyslog sending to logstash over the network? Or should I be running another rsyslog on the collector server, which then sends to logstash for processing? For Apache, I would like to have separate vhost log files on the web server, in addition to these logs being sent to a remote log collector. I've tested rsyslog using the imfile module to watch each Apache log files, but this means I have to hard-code each vhost log file into my rsyslog.conf. This is not ideal as people will invariably forget when they add/remove sites on the server. 2) What's the best way to log to both vhost-specific log files on the web server and to send these logs over the network, without using imfile and manually watching tens of individual log files? Get Apache to log to rsyslog, then have rsyslog split the log to both a file and over the network to logstash? Are there big performance implications for logging both locally and over the network? I could change my Apache config to log to a single access/error log for all vhosts, then watch these main log files with imfile. So long as rsyslog is then able to produce vhost-specific log files somewhere on the web server machine. Any comments/suggestions? I am sure others have had a similar need. I just don't want to ditch local log files until we fully know how well the centralised log server performs. Thanks in advance! Cheers, Ben ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] rsyslog 7.2.4 (v7-stable) released
Hi all, we have just released 7.2.4 of the v7 stable branch. This is a pure bug-fixing release. More information on the changes can be found in the ChangeLog. ChangeLog: http://www.rsyslog.com/changelog-for-7-2-4-v7-stable/ Download: http://www.rsyslog.com/rsyslog-7-2-4-v7-stable/ As always, feedback is appreciated. Best regards, Tim Eifler ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Multiple Apache vhosts, keep individual log files AND send to remote logstash
On Fri, 7 Dec 2012, Ben Bradley wrote: Hi everyone I've just started investigating centralised logging and I'm gradually building up a plan of action. I'd like to store the logs on a central server running logstash/ElasticSearch so they can be searched and monitored using Kibana. With rsyslog sending the logs over the network to a logstash server. I don't want to run logstash as the log sender on each server, I'd prefer to keep the servers (log clients) as lean and simple possible. So that means either using syslog, syslog-ng or the one I'm testing now, rsyslog. 1) Should I have rsyslog sending to logstash over the network? Or should I be running another rsyslog on the collector server, which then sends to logstash for processing? This is up to you, there are advantages in each direction. Using rsyslog for all network transport and having it deliver locally to logstash/elasticsearch/other for processing means that you you can take advantage of all rsyslog features for your transport. In a centralized environment your traffic volumes can be high, rsyslog can handle very high traffic levels, can your other software? If you are really comforatable with logstash, you may want to eliminate the need to run one more daemon, but when you hire new people and hand the system over, should they need to be as comfortable with logstash? They will have to be comforatable with rsyslog in any case. At that point which is the 'extra' thing to deal with, rsyslog or logstash? For Apache, I would like to have separate vhost log files on the web server, in addition to these logs being sent to a remote log collector. I've tested rsyslog using the imfile module to watch each Apache log files, but this means I have to hard-code each vhost log file into my rsyslog.conf. This is not ideal as people will invariably forget when they add/remove sites on the server. 2) What's the best way to log to both vhost-specific log files on the web server and to send these logs over the network, without using imfile and manually watching tens of individual log files? Get Apache to log to rsyslog, then have rsyslog split the log to both a file and over the network to logstash? Are there big performance implications for logging both locally and over the network? I could change my Apache config to log to a single access/error log for all vhosts, then watch these main log files with imfile. So long as rsyslog is then able to produce vhost-specific log files somewhere on the web server machine. it depends on how you format the log file. If you have the logfile start with the vhost name, then rsyslog can easily produce per-host files (look in the rsyslog documentation for the dynafile templates. another approach you can do is have apache log to a local named pipe and have a process listen on that named pipe and tagging/reformatting the log file and pass it to your syslog server. David Lang Any comments/suggestions? I am sure others have had a similar need. I just don't want to ditch local log files until we fully know how well the centralised log server performs. Thanks in advance! Cheers, Ben ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] AMQP as log destination?
On Fri, 7 Dec 2012, Fabio Sangiovanni wrote: acronym alert, what does AMQP stand for? It's a standard protocol to communicate with message queueing systems. the nice thing about standards is that there are so many to choose from and http://xkcd.com/927/ http://www.amqp.org/ it's vision sounds nice To become the standard protocol for interoperability between all messaging middleware but it's yet another standard to compete with all the others http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol Message queueing system implementations that support AMQP: RabbitMQ, Apache QPid It would not be that hard to get AMQP added to rsyslog, you would just need to have someone either write input and output modules (probably adapted from the 0mq modules to start with), or sponsor development through Adiscon professional services (e-mail Rainer directly to get a quote, it's usually surprisingly cheap) persistent on-disk queues are already an option. Yes, and I'd be happy to use them together with an AMQP output plugin for reliable massive log processing. rsyslog already supports Reliable Event Logging Protocol (RELP), and 0MQ options for reliable massive log processing. This would just be one additional option (and there's nothing wrong with supporting lots of options) My logging system processed 18B lines of logs in October, it's handled 93K lines of logs in a single second, and I've tested it up to 380K lines of logs per second (effectivly gig-E wire speed), and others have used rsyslog in environments where they have tested it to 1M lines of logs per second. However, reliable is a relative term. reliability and performance tend to be opposed and you have to make tradeoffs between the two. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
