Re: [rsyslog] Capturing messages before RELP connection established.

Hi, Let me see if I get this right: rsyslog doesn't read files from /var/log/DefaultConfig.log ? Is something continuously writing to that file? Because imfile normally tails files, it doesn't read from the beginning. I have a feeling I don't quite understand what you're trying to do (end to end

Re: [rsyslog] JSON parse array question

Hi Gabor, I see that you have two mentions of user_ids in the template: property(outname="user_ids" name="$!user_ids") property(outname="user_ids" name="$!mesg") Maybe that's why? I would normally expect just one instance of the same variable. My usual way of troubleshooting t

Re: [rsyslog] creating separate config file for a log file

Hi Keith, rsyslog.conf is read sequentially, so wherever you have your include directive (more details here https://www.rsyslog.com/doc/master/rainerscript/include.html) is where your new config file(s) will be read. Your included config (and the directories leading to it) need to be readable by

Re: [rsyslog] an experiment: first rsyslog open meeting

ally, even if this is not the case > ;-)). > > Hope that makes sense to you... > > Rainer > > El mar., 29 sept. 2020 a las 9:05, Radu Gheorghe > () escribió: > > > > I love the idea, it’s just that I have a call already scheduled for > then. Maybe if you guys (and gir

Re: [rsyslog] an experiment: first rsyslog open meeting

I love the idea, it’s just that I have a call already scheduled for then. Maybe if you guys (and girls?) are still around at 4PM UTC I can join later? Best regards, Radu -- Sematext Cloud - Full Stack Observability - https://sematext.com Solr and Elasticsearch Consulting, Training and Production

Re: [rsyslog] rsyslog+elasticsearch (and some Kafka and a few others) eBook

the kindles > > I will be looking at this later in the week (my new 13" e-reader is due > tomorrow, I can't wait) > > David Lang > > On Tue, 6 Feb 2018, Radu Gheorghe wrote: > >> Date: Tue, 6 Feb 2018 07:42:36 -0800 >> From: Radu Gheorghe >> Reply-

Re: [rsyslog] rsyslog+elasticsearch (and some Kafka and a few others) eBook

1, 2018 at 9:08 AM, deoren wrote: > On 2/1/2018 6:27 AM, Radu Gheorghe wrote: >> >> Hi, >> >> Today we just published what I hope to be a quite complete eBook about >> centralizing logs with rsyslog. The destination I had in mind was >> Elasticsearch, but I think

Re: [rsyslog] rsyslog+elasticsearch (and some Kafka and a few others) eBook

gt; the end...). I am preparing a logo vote page, because it now really is > time to get to somewhat more decent. > > Rainer > > 2018-02-01 13:27 GMT+01:00 Radu Gheorghe : >> Hi, >> >> Today we just published what I hope to be a quite complete eBook about &g

[rsyslog] rsyslog+elasticsearch (and some Kafka and a few others) eBook

Hi, Today we just published what I hope to be a quite complete eBook about centralizing logs with rsyslog. The destination I had in mind was Elasticsearch, but I think it should apply to many other use-cases. Here's the blog post with more details on what it contains and the download link: https

[rsyslog] Trouble while trying to add tests to testbench

Hello, I'm trying to add a couple of tests for ompgsql, but I got stuck a bit early: trying to make the testbench run in the first place. I watched this nice video about the testbench: https://www.youtube.com/watch?v=GLogMXhB48A But when I run `make check` it doesn't run any test. Also, there wa

Re: [rsyslog] Compile issues on OSX (*whole-archive; duplicate symbols)

Thanks for your replies, Rainer and David! At least a real bug was found, so it wasn't all for nothing :D Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ ___ rsyslog mai

Re: [rsyslog] Compile issues on OSX (*whole-archive; duplicate symbols)

fixes a compile issue. Still, it looks like my issues came with the later versions. -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, Feb 19, 2016 at 11:14 AM, Radu Gheorghe wrote: > Hello, > > I just noticed th

[rsyslog] Compile issues on OSX (*whole-archive; duplicate symbols)

Hello, I just noticed that on OSX (10.11.1 is what I have now), I can simply do `brew install rsyslog` and I'm getting rsyslog 7.4.5. Nice. Then I looked at the formula and I saw it just compiles the thing and it works. Double-nice. So then I thought I can just go ahead and compile my own 8.16 wi

Re: [rsyslog] feedback request on format-changing bugfix

Hi Rainer, No concerns here, it sounds like the right thing to do. I suppose scripts working with the CEE cookie already use the CEE option? At least that was our case (we were parsing with mmjsonparse afterwards). Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics So

[rsyslog] When to merge changes to rsyslog-doc

Hello, I see that we now have a problem with the rsyslog-doc project that is so nice to have :) And I need some help. Some of the PRs that come to https://github.com/rsyslog/rsyslog-doc/pulls are related to new features (neat, ha?). These have their own issues in the main rsyslog repo and, being

Re: [rsyslog] liblognorm full JSON format doesn't work in 8.13.0

Thanks, Rainer! I knew I was missing something basic (i.e. wrong version) :p -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Jan 26, 2016 at 4:16 PM, Rainer Gerhards wrote: > 2016-01-26 15:14 GMT+01:00 Radu G

Re: [rsyslog] liblognorm full JSON format doesn't work in 8.13.0

ecause we don't have a general available release yet, and it's just experimental? Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Oct 7, 2015 at 12:02 PM, Radu Gheorghe wrote: > Hello, &g

Re: [rsyslog] ditch v5-doc on rsyslog.com?

On Mon, Jan 25, 2016 at 9:38 AM, Rainer Gerhards wrote: > Do you know how we could use sphinx to add a warning message to the to of > each page? > Nope, unfortunately, I can't find a way to do that. I can only think of the ugly way of having a small script to touch all RST files. Which will make

Re: [rsyslog] ditch v5-doc on rsyslog.com?

-1 for removing v5 docs. It would help people still having these versions. We could add a note that this is very old and unsupported. I think we should remove the old syntax from v8 docs to make the difference clearer (and keep the docs more maintainable), but that's a different subject :) -- Perf

Re: [rsyslog] IMjournal Package in Yocto Linux

Potentially stupid question: do you have libsystemd-journal installed? If yes, maybe it's in a different directory than it's expected and indeed you'd need to adjust PKG_CONFIG_PATH for compiling rsyslog. Also, I've never used bitbake, but can you compile rsyslog manually on the same system? -- Pe

Re: [rsyslog] elasticsearch 2.0 and field names

On Tue, Dec 8, 2015 at 1:44 PM, Peter Portante wrote: > On Tue, Dec 8, 2015 at 6:38 AM, Brian Knox wrote: > >> As a short term solution I'm working on a small service (in golang) that >> accepts logs over tcp, can replace characters in JSON field names in a @cee >> syslog line, and then forward t

Re: [rsyslog] Time Format

Hello, I think the actual need for this functionality would be outside RFC5424. Or RFC3164 for that matter. It sounds like Vicks (and also Ciprian and I) would need it as a function of mmnormalize/liblognorm so that we can parse logs from files. This different format in the Email is something I o

Re: [rsyslog] Time Format

r & Elasticsearch Support * http://sematext.com/ > > On Mon, Nov 16, 2015 at 2:13 PM, Radu Gheorghe > wrote: > >> Hello, >> >> As far as I know, this isn't a date format that rsyslog (or liblognorm) >> understands. Because it's not an RFC-3339 time

Re: [rsyslog] Specifying TTL in omelasticsearch index

or the links as well. > > On Mon, Nov 23, 2015 at 12:07 AM, Radu Gheorghe > wrote: > >> Hi Alec, >> >> I assume you're looking to remove old data from Elasticsearch >> automatically. If so, then rsyslog is not the tool for the job. But I >> see

Re: [rsyslog] Specifying TTL in omelasticsearch index

Hi Alec, I assume you're looking to remove old data from Elasticsearch automatically. If so, then rsyslog is not the tool for the job. But I see two options: - the bad one: use the _ttl field: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-ttl-field.html you can see that

Re: [rsyslog] Time Format

Hello, As far as I know, this isn't a date format that rsyslog (or liblognorm) understands. Because it's not an RFC-3339 timestamp, nor a traditional RFC-3164-syslog timestamp. So far, we've only hacked our way into parsing this with mmnormalize by getting the two bits (the date and the time) as

Re: [rsyslog] Rsyslog TLS configuration Cookbook

Wow, thanks for sharing! It looks pretty awesome to me! I wouldn't add/substract anything, but maybe I'm missing something. -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, Nov 16, 2015 at 10:15 AM, Vicks Mortal wrote: > He

Re: [rsyslog] issue forwarding over tls connection

Maas wrote: > >> Hi all, >> >> I've verified the situation with the latest 8.14 release and the official >> packages, and i can confirm that the issue is still present. Including the >> segmentationfault when using rsyslogd -N3. I'll file a bug r

Re: [rsyslog] issue forwarding over tls connection

master config > /etc/rsyslog.conf > rsyslogd: invalid or yet-unknown config file command > 'ActionSendStreamDriverPermittedPeers' - have you forgotten to load a > module? [try http://www.rsyslog.com/e/3003 ] > rsyslogd: End of config validation run. Bye. > > [root

Re: [rsyslog] issue forwarding over tls connection

t rsyslogd-2088: error: peer name not > authorized - not permitted to talk to it. Names: CN: > logmanagement.xxx.yy; [try http://www.rsyslog.com/e/2088 ] > > So with this "hybrid" config i can transfer logs over the tls channel. But > unfortunately the system is not really

Re: [rsyslog] issue forwarding over tls connection

Hello, We had this problem at one point when having different versions of rsyslog (and/or gnutls) acting as client and server. Another time when I encountered this was when I didn't set up certificates properly. I hope this helps. Best regards, Radu -- Performance Monitoring * Log Analytics * Se

Re: [rsyslog] Speed up Disk Assisted de-queuing

On Wed, Nov 4, 2015 at 5:19 PM, Joe Blow wrote: > Radu - My checkpoint interval is set at 100k. Are you suggesting this be > lowered? raised? It sounds like the higher the better, but if your problem is on how fast it can read... I think there's not much you can do - that seems to be a setting f

Re: [rsyslog] Speed up Disk Assisted de-queuing

Hi Joe, I think the way to change it is via queue.checkpointinterval (and maybe also set queue.syncqueuefiles to "off"). It's also possible that there are more settings under http://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html Best regards, Radu -- Performance Monitoring * Log An

Re: [rsyslog] gtls with rainerscript syntax?

rmed. > Now I just need to document it on our side so that we can revisit this > issue when upgrading. > > Thanks again! > > Cheers, > Jörgen > > > > On Wed, Nov 4, 2015 at 2:27 PM, Radu Gheorghe > wrote: > >> Hello, >> >> Regarding defaultNe

Re: [rsyslog] gtls with rainerscript syntax?

quot;/etc/rsyslog.d/keys/host-key.pem" > ) > > $DefaultNetstreamDriverCertFile /etc/rsyslog.d/keys/host-cert.pem > > Perhaps the new config option has a different name, couldn't find it though. > > Thanks for all the help so far! > > Kind regards, > Jörgen > &

Re: [rsyslog] gtls with rainerscript syntax?

ith the > rainerscript syntax? > > When stuck on 7.x (EL7) should i revert to using traditional syntax? > > When will traditional syntax be deprecated and removed ? > > Thanks! > > Jörgen > > > On Wed, Nov 4, 2015 at 11:36 AM, Radu Gheorghe > wrote: > >>

Re: [rsyslog] mmpstrucdata doesn't seem to work

Oh, right! It does clarify! Thanks! -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Nov 4, 2015 at 11:58 AM, Rainer Gerhards wrote: > 2015-11-04 8:49 GMT+01:00 Radu Gheorghe : >> Hello and thanks for

Re: [rsyslog] gtls with rainerscript syntax?

1: error: driver mode 1 > not supported by ptcp netstream driver [try http://www.rsyslog.com/e/2081 ] > > Still an issue creating a listener. I suppose the problem is in my > configuration? > > Also, there doesn't seem to be a imjournal module which was available in > CentOS.

Re: [rsyslog] gtls with rainerscript syntax?

og/lmnsd_gtls.so', rsyslog error -2078 > [try http://www.rsyslog.com/e/2068 ] > Nov 4 09:20:21 logmanagement-client rsyslogd-2068: tcpsrv could not create > listener (inputname: 'tcptls') [try http://www.rsyslog.com/e/2068 ] > Nov 4 09:20:21 logmanagement-client rsyslog

Re: [rsyslog] gtls with rainerscript syntax?

Hi Jörgen, You can find the client config in this blog post: http://blog.sematext.com/2014/03/25/encrypting-logs-on-their-way-to-elasticsearch-part-2-tls-syslog/ I suppose you can deduce the server config from that and the linked howtos (which are old-style). If you can't, please let me know and

Re: [rsyslog] mmpstrucdata doesn't seem to work

gt; > Sent from phone, thus brief. > Am 03.11.2015 17:47 schrieb "Radu Gheorghe" : > >> Hi David, >> >> Here's how the debug template writes with a "server" config like the >> one I pasted in the first Email: >> >> Debug line wit

Re: [rsyslog] mmpstrucdata doesn't seem to work

that $! variable. Thanks and best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, Nov 2, 2015 at 7:45 PM, David Lang wrote: > can you show us a same of the rawlog that you are receiving? > > among o

[rsyslog] mmpstrucdata doesn't seem to work

Hello rsysloggers :) I'm having trouble setting up mmpstrucdata (running on 64-bit Ubuntu 14.04 with rsyslog 8.13 installed from the official packages, if it matters). I've followed the docs (http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmpstrucdata.html) and I didn't get anything o

Re: [rsyslog] omhiredis question

t; distros archives. Not sure if that's the case here. A PR would definitely > help. > > Rainer > > Sent from phone, thus brief. > Am 14.10.2015 19:23 schrieb "Radu Gheorghe" : > > > Hi Peter, > > > > That's right, there's no package

Re: [rsyslog] omhiredis question

Hi Peter, That's right, there's no package yet. I can't say anything about the plans, but it would certainly be useful for those interested. If you can/want to add that package yourself, I think a PR here would do: https://github.com/rsyslog/rsyslog-pkg-ubuntu And you can look at other commits/p

[rsyslog] liblognorm full JSON format doesn't work in 8.13.0

Hello, I'm trying to use this Apache Logs pattern in 8.13 on Ubuntu: https://github.com/rsyslog/liblognorm-rulebases/blob/master/rules/v2/apache_combined.rb And I remember testing that exact rule on a hand-compiled master of 8.13 before it was released. Now it doesn't seem to work both rsyslog an

Re: [rsyslog] Accepting and parsing GELF?

Hi, A few more questions from me, this time regarding the transport of GELF, not the message format. Clients seem to send GELF over UDP, TCP (and even HTTP!) and compress messages via GZIP or ZLIB (this is in the GELF specs ). What's more, it also allows c

Re: [rsyslog] Can we have a minimum bulk size for omelasticsearch?

Hi David, This sounds interesting, I especially like the idea of spawning threads based on load. The problem, really, is that an ES cluster would have an "optimal" batch size. If we send batches too small or too large, ingestion will slow down. Fortunately, that optimum isn't really that precise.

Re: [rsyslog] rsyslog v8.12 segfault with queue.maxfilesize="-1"

Hello, It smells like a bug because it shouldn't segfault. I think an issue on GitHub would help: https://github.com/rsyslog/rsyslog Other than that, I think: - action queues are direct by default. You'd need to set queue.type to linkedlist or fixedarray first (to enable the memory part of the qu

Re: [rsyslog] Can we have a minimum bulk size for omelasticsearch?

On Sat, Aug 22, 2015 at 6:26 AM, David Lang wrote: > On Fri, 21 Aug 2015, Otis Gospodnetić wrote: > > Hi, >> >> This sounds like something that should be om-specific. What Radu is >> suggesting would definitely help with ES, but may not be relevant for >> other >> output targets. >> What I think

Re: [rsyslog] Can we have a minimum bulk size for omelasticsearch?

On Fri, Aug 21, 2015 at 1:22 PM, Rainer Gerhards wrote: > 2015-08-21 12:19 GMT+02:00 Otis Gospodnetić : > > Hi, > > > > This sounds like something that should be om-specific. What Radu is > > suggesting would definitely help with ES, but may not be relevant for > other > > output targets. > > Wh

Re: [rsyslog] Can we have a minimum bulk size for omelasticsearch?

Support * http://sematext.com/ On Fri, Aug 21, 2015 at 9:24 AM, David Lang wrote: > > On Fri, 21 Aug 2015, Radu Gheorghe wrote: > > Hello rsyslog users :) >> >> We've seen a problem that is similar to the one reported here: >> http://www.gossamer-threads.com/lists/rsys

[rsyslog] Can we have a minimum bulk size for omelasticsearch?

Hello rsyslog users :) We've seen a problem that is similar to the one reported here: http://www.gossamer-threads.com/lists/rsyslog/users/17550 While that looks like a bug, ours seems like a design issue. Basically we see bulks of one document all over the place. Not 100% what's the root cause, b

Re: [rsyslog] YUM repo doesn't work on Amazon Linux out of the box

nging the priority lower than 10 fixes the issue >> Radu described. >> >> Otis >> -- >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >> Solr & Elasticsearch Support * http://sematext.com/ >> >> >> On Thu, Aug

[rsyslog] YUM repo doesn't work on Amazon Linux out of the box

Hello, I've meant to write this some time ago but somehow forgot :( The thing is, Amazon Linux is RPM-based, so most YUM repos out there work. Not the one for rsyslog, for two reasons: 1) Amazon Linux reports "latest" for $releasever, but that is fixed by following step 4 here: http://www.rsyslog

Re: [rsyslog] Question on "contains"

Hi Nick, I don't know if the array approach would work (I guess not, but you can try). I would assume that "contains" would be faster than the regex approach, even with more IPs, because your regex would also be complicated. If you have a really long list of IPs, then it might be worth parsing th

Re: [rsyslog] V5 Disk Assisted Queue is stucked when MaxDiskSpace is reached

d is acceptable for us, but being > forced to manually delete and restart is more complicated. > > I hoped the problem was some sort of misconfiguration on my side, or maybe > a know issue using omrelp with logstash relp input. > > > > > On 30 June 2015 at 09:55, Radu Gh

Re: [rsyslog] V5 Disk Assisted Queue is stucked when MaxDiskSpace is reached

Hi Nicolas, I have some vague memories about nasty bugs in disk-assisted queues that were fixed in the last few years. RELP modules surely have changed as well. Can you try with the latest stable (8.10 I think) and see if it helps? Even if it doesn't, I'm pretty sure the fix will come in the 8.x b

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

David Lang wrote: > I seem to remember seeing that there is a different variable for > omelasticsearch to set the max bulk size for the ES insert as opposed to > the batch size used internally by rsyslog. I don't remember what it is. > > > David Lang > > On Wed, 17

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

; queue[DA]","origin":"core.queue","size":27838434,"enqueued":9,"full":735,"discarded.full":9," > > discarded.nf":0,"maxqsize":28153530} > > 2015-06-17T12:17:48.708370+08:00 localhost rsyslogd-

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

rds > ) > > David Lang > > On Wed, 17 Jun 2015, Radu Gheorghe wrote: > > This sounds interesting, David. I guess it's possible to renice just some >> threads from an app and make it "nicer", right? Googling a bit it seems it >> is possible. >&

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

stest insert speed (even if > less optimized than if there were larger batches) But if anything else on > the system need the resources, the indexing threads work slower, which will > result in larger batches. > > all self tuning. > > David Lang > > > > On Wed, 17 Jun 20

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

Maybe this went overlooked, but David suggested earlier that you can slowdown the queue to let more messages arrive before sending a bulk. queue.dequeueslowdown is the option and it's in microseconds. I think you have a vali

Re: [rsyslog] Rsyslog Remote Logging Timestamp Consistency

y issue. > > > > On Mon, Jun 15, 2015 at 3:24 PM, Radu Gheorghe > > wrote: > > > Hi Gururaj, > > > > RSYSLOG_FileFormat is what the server would use when writing logs to > files > > (so that you'll have the high-precision timestamp in th

Re: [rsyslog] Rsyslog Remote Logging Timestamp Consistency

Hi Gururaj, RSYSLOG_FileFormat is what the server would use when writing logs to files (so that you'll have the high-precision timestamp in the file). On the client, however, you should use RSYSLOG_ForwardFormat or RSYSLOG_SyslogProtocol23Format to make sure the high-precision timestamp is preserv

Re: [rsyslog] Template Mystery for me

Hello, template(name="lumberjack" type="list") { property(name="$!all-json") } then with: module(load="mmjsonparse") action(type="mmjsonparse") You should have your fields in the "lumberjack" template that you can use in the omelasticsearch action. The only trouble you may find is with the ti

Re: [rsyslog] parse logs to elasticserach

Hi Muhammad, >From what I understand, rsyslog parsed all the JSON, so you can pass on all this JSON with a template that looks like this: template(name="allTheJson" type="list") { property(name="$!all-json") } -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch

Re: [rsyslog] parse logs to elasticserach

Hi Muhammad, If you only want to include properties from the JSONs you're parsing, the template looks very nice: template(name="messageToES" type="list") { property(name="$!all-json") } If you want to include other properties (e.g. severity from the syslog message), there's a well commented

Re: [rsyslog] parse logs to elasticserach

Hi Muhammad, You can use mmjsonparse to parse those JSONs in rsyslog (similarly to using the JSON codec or filter in Logstash). There's a blog post that explains how it works and how to write those JSONs to Elasticsearch: http://blog.sematext.com/2013/05/28/structured-logging-with-rsyslog-and-elas

Re: [rsyslog] Rsyslog rocket logo (reloaded)

you can since he could pull apart the layers for example to use > > just the rocket as a logo elsewhere on the web site, etc. > > > > -- > > Dave Caplinger, Director of Architecture | Ph: (402) 361-3063 | > > Solutionary — An NTT Group Security Company > > > &g

Re: [rsyslog] Best practice for an application to get structured data to rsyslog

On Wed, Apr 15, 2015 at 6:25 PM, Dave Caplinger < davecaplin...@solutionary.com> wrote: > On Apr 14, 2015, at 11:43 PM, David Lang wrote: > > > > On Wed, 15 Apr 2015, Ezell, Matthew A. wrote: > > > [...] > > what I do is to take whatever message was output and then run mmjsonparse > > against it

Re: [rsyslog] action queues vs message modifier modules vs dequeuebatchsize

t 1:51 PM, Rainer Gerhards wrote: > 2015-04-08 12:43 GMT+02:00 Radu Gheorghe : > > Thanks Rainer! So let me see if I get this straight. Say I want to > > mmnormalize some logs and then omelasticsearch them. I would (please > > correct me where I'm wrong): > > > >

Re: [rsyslog] action queues vs message modifier modules vs dequeuebatchsize

t.com/ On Wed, Apr 8, 2015 at 12:28 PM, Rainer Gerhards wrote: > 2015-04-08 9:34 GMT+02:00 Radu Gheorghe : > > Hello, > > > > I have three questions about action queues: > > 1) if I have a message modifier action (e.g. mmnormalize), should I add > an > > action qu

[rsyslog] action queues vs message modifier modules vs dequeuebatchsize

Hello, I have three questions about action queues: 1) if I have a message modifier action (e.g. mmnormalize), should I add an action queue for better performance? 2) what's the flow of messages if I have both mmnormalize and, say, omelasticsearch? Messages flow from the main queue to mmnormalize's

Re: [rsyslog] Integer type property not working in omelasticsearch

Hi Luis, Try removing the quotes around the value, like: constant(value="\",\"bytes\":") property(name="$!bytes" type="int") # next constant will probably begin with a (escaped) quote. Remove it, too. Alternatively, you can define the mapping for bytes in Elasticsearch upfront.

Re: [rsyslog] Getting ESXi logs into graylog2 via rsyslog

> > David, btw, has the ultimate cure: use UTC consistently on all machines. I also try to stick with a "if TZ<>UTC then complaints > /dev/null" policy. But sometimes you just can't do that, unfortunately. ___ rsyslog mailing list http://lists.adiscon.n

Re: [rsyslog] Getting ESXi logs into graylog2 via rsyslog

Thanks for explaining, Rainer. So things are as I suspected - no easy way of saying "this is a Paris timestamp with no TZ info, give me the RFC-3339 equivalent". I will say something if I have an idea other than "I hate DST". ___ rsyslog mailing list htt

Re: [rsyslog] Getting ESXi logs into graylog2 via rsyslog

On Thu, Jan 29, 2015 at 9:48 AM, Rainer Gerhards wrote: > 2015-01-29 8:46 GMT+01:00 Radu Gheorghe : > [...] > > 2015-01-27T16:17:57Z > > > > And you can output everything except that Z (or it may be +00:00, I don't > > remember) and append a hardcoded

Re: [rsyslog] Getting ESXi logs into graylog2 via rsyslog

Hi Brandon, I haven't used graylog2 in years, so I might be completely off, but here here are two ideas that might help. AFAIK, graylog2 uses Elasticsearch as the backend for storing logs, so if you figure out how data is normally written, you could hook rsyslog directly to ES via omelasticsearch

Re: [rsyslog] plans for rsyslog 8.8

On Thu, Jan 15, 2015 at 9:28 PM, David Lang wrote: > I'm missing something here. If rsyslog has a queue for the destination, > and the delivery to the destination is via TCP, how is a pull any better > than a push? if the destination accepts data at a faster pace than it can > really handle, why

Re: [rsyslog] plans for rsyslog 8.8

On Thu, Jan 15, 2015 at 7:12 PM, Rainer Gerhards wrote: [...] > 2. provide a general infrastructure for pull models, whatever this is to be > used for > > [...] > Use cases for 2 exists, but I don't know the specifics. They surface every > now and then on the ML when someone ask for pull integra

Re: [rsyslog] Question about action.resumeRetryCount

Yes. That's my understanding at least (and my experience from testing it). -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Dec 9, 2014 at 5:28 PM, Boylan, James wrote: > This configuration defaults to 0. Doesn't this mean

Re: [rsyslog] Feedback Request: do we still need -devel versions?

+1 for master-only and killing off -devel if few (if any) people are using -devel +1 for David's idea of making exceptions when there are big changes (like a beta). As the community grows (and it looks like it slowly does) people will want to try that devel/beta out, especially if there are some ju

Re: [rsyslog] not forwarding when imfile has readmode=2

terations happen more frequently in inotify mode. > But again, it's just a guess, I may be totally wrong. > > That said, there is no fix yet for the issue. > > Rainer > > > > David Lang > > > > > > On Wed, 29 Oct 2014, Radu Gheorghe wrote: > > >

Re: [rsyslog] not forwarding when imfile has readmode=2

s * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Oct 18, 2014 at 8:06 PM, Radu Gheorghe wrote: > Hi David, > > Thanks for commenting - I'm aware of this (normal) behavior. > > It's just a dummy file, really, I do `echo blabla >> /var/log/

Re: [rsyslog] Don't Store Logs Locally After Forwarding Them To A Cent. Rsyslog

Hi Tarkan, It depends on which rsyslog version you have and what your configuration looks like. If you have a pre-v7 rsyslog with something like: *.* :omrelp::; # and later *.* /var/log/messages You can change it to: *.* :omrelp::; &~ #this stops processing those events processed by the acti

Re: [rsyslog] Dealing with ancient rsyslog versions on new Linux distros

> > I'll also review the other instructions within this week. At least Ubuntu > should really be the same. Thanks! For RHEL and friends it should be yum install rsyslog instead of yum update. > Just want to finish imfile first :-) > > Heh :) Many thanks for that, Rainer! __

Re: [rsyslog] Dealing with ancient rsyslog versions on new Linux distros

+1 to replacing *apt-get upgrade* with *apt-get install rsyslog*. I do that 9 times out of 10 when I upgrade rsyslog, because: - if I'm on a client's machine, I want to avoid breaking any currently working stuff - even if the machine is mine, sometimes I just want to quickly upgrade rsyslog do take

Re: [rsyslog] not forwarding when imfile has readmode=2

g message before it can be sure that the prior one has finished. So > you have to have it reading a file with more than one message in it. > > What does the file you are realing look like? > > David Lang > > On Fri, 17 Oct 2014, Radu Gheorghe wrote: > > Date: Fri,

Re: [rsyslog] not forwarding when imfile has readmode=2

On Fri, Oct 17, 2014 at 6:50 PM, Rainer Gerhards wrote: > 2014-10-17 17:28 GMT+02:00 Radu Gheorghe : > > > Hi Rainer, > > > > Thanks, you're right. With polling more it works well. I'd still prefer > > inotify, though, because it plays nicely with lo

Re: [rsyslog] not forwarding when imfile has readmode=2

ct 17, 2014 at 3:44 PM, Rainer Gerhards wrote: > could you check if that's an incompatibility with inotify mode. To do so, > just add > >mode="polling" > > to the module definition. > > Rainer > > 2014-10-17 13:32 GMT+02:00 Radu Gheorghe : > &

[rsyslog] not forwarding when imfile has readmode=2

Hello, I was just testing an rsyslog 8.4.2 with our Logsene and I didn't understand why I couldn't see no logs from the tailed files. The problem seems to be that, with ReadMode set to 2, imfile doesn't pass events on no matter what you put in the file. I always see this in the debug log and noth

Re: [rsyslog] mmjsonparser issue

Hi Muhammad, mmjsonparse is for parsing stuff that is already JSON. To parse your squid logs and make them JSON, you'll need mmnormalize . An end-to-end usecase is described here: https://developer.rackspace.com/blog/rsyslog

Re: [rsyslog] Feedback request: json template - which is easier to understand

Hi Rainer, Your suggested configuration looks a bit better, but it still requires one to put commas - so it's still on the ugly side. But better. The "vision" looks much better still, because it's easy to add/remove properties without breaking the syntax. Regarding priorities, I really don't know

Re: [rsyslog] json files directly to ES

If you're looking for a grok equivalent, have a look at mmnormalize: http://www.rsyslog.com/doc/master/configuration/modules/mmnormalize.html It's not as flexible as grok is by using regular expressions, but it should be a lot faster. You would have to come up with your own patterns, though, and y

Re: [rsyslog] json files directly to ES

Hi, I'm not sure where Logstash fits in this picture - I thought you'd get JSONs from a file and send them to Elasticsearch. ES version 1.1.1 should be OK, I'm not sure which rsyslog version you're on but all recent ones should work fine. You seem to have stuff in ES, but all the documents are e

Re: [rsyslog] json files directly to ES

2573:7f38231c4700: omelasticsearch: no local error logger >> defined - ignoring ES error information >> 7794.801797633:7f38231c4700: omelasticsearch: result doAction: 0 >> (bulkmode 0) >> 7794.801801664:7f38231c4700: Action 0x7f382d70c650 transitioned to state: >> rdy &

Re: [rsyslog] json files directly to ES

Two more points from me that will hopefully help: - if you're not sure where something breaks, try to isolate the problem by reducing the config to the bare minimum and building up on it once it works. For example, I wouldn't bother with rulesets if no logs can get to ES in the first place. Just ma

Re: [rsyslog] about rsyslog and functionalities

..@gmail.com> wrote: > *Radu Gheorghe wo ... just wo.I'm right now installing my lab to > begin testing all this stuff. Thanks a lot. I'm gonna keep you guys up to > date about everything. Thanks again ...* > > On Sat, Oct 4, 2014 at 12:29 PM, Radu Gheorghe >

Re: [rsyslog] Confused by timegenerated vs timereported

Hi Earl, If you send your messages via /dev/log, the problem might be caused by imuxsock , which ignores the provided timestamp by default. At least this is what I understand from the doc: *SysSock.IgnoreTimestamp* [*on*/off]

  1   2   3   4   5   6   >