Re: [rsyslog] Fwd: Re: RES: Client/server architecture with omrelp/imrelp - Lost of messages in case of server side failure

it because I belive It is using the default value (60 seconds). Thank you for your feedback. Regards, Jether 2017-10-30 9:46 GMT-02:00 mostolog--- via rsyslog <rsyslog@lists.adiscon.com : Hi You are missing retryInterval, and I would say it's needed for proper retry after a queue lock. In

[rsyslog] Fwd: Re: RES: Client/server architecture with omrelp/imrelp - Lost of messages in case of server side failure

Hi You are missing retryInterval, and I would say it's needed for proper retry after a queue lock. In our tests, once relp queue was full, it never resumed (although resumeretrycount=-1 was set) until we added interval setting.  On 26/10/17 15:52, Jether B. Santos via rsyslog wrote: Hi, In

Re: [rsyslog] RES: Client/server architecture with omrelp/imrelp - Lost of messages in case of server side failure

Hi You are missing retryInterval, and I would say it's needed for proper retry after a queue lock. In our tests, once relp queue was full, it never resumed (although resumeretrycount=-1 was set) until we added interval setting. On 26/10/17 15:52, Jether B. Santos via rsyslog wrote: Hi,

[rsyslog] Truncated UDP message error is shown on debug?

Hi As we were not receiving messages from one server while doing some tests, we run:     rsyslogd -dn but we weren't able to see anything wrong. However, running:     rsyslog -n showed the following message:     rsyslogd: omfwd/udp: message is 113152 bytes long, but UDP can send at most

Re: [rsyslog] Exclude patterns for imfile whith different multiline format

schrieb "mostolog--- via rsyslog" <rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>: Hi One of our relay rsyslog servers has 2 kind of logs: apache and java. Sadly, *not all java applications share the same log format.* We have solved th

Re: [rsyslog] Exclude patterns for imfile whith different multiline format

phone, thus brief. Am 17.10.2017 15:23 schrieb "mostolog--- via rsyslog" <rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>: Hi One of our relay rsyslog servers has 2 kind of logs: apache and java. Sadly, *not all java applications share the same log

Re: [rsyslog] Exclude patterns for imfile whith different multiline format

If Mohammed doesn't go to the mountain, the mountain must go to Mohammed. AKA: That's not possible. It would be easier to make them change the format (and, as you know, that's already impossible.) On 17/10/17 16:43, David Lang wrote: have the java apps put the different types of logs in

[rsyslog] Exclude patterns for imfile whith different multiline format

Hi One of our relay rsyslog servers has 2 kind of logs: apache and java. Sadly, *not all java applications share the same log format.* We have solved this using a bash script to generate our config which has one input for each file. To avoid having to reconfigure rsyslog each time a log

Re: [rsyslog] Does imfile-to-relp need DA queues?

Clear as water. The idea which was in our minds due to rsyslog issues with RELP (mention on another thread) is, by these and Rainer words completely banned since now. Thanks a lot On 04/10/17 10:23, David Lang wrote: On Wed, 4 Oct 2017, mostolog--- via rsyslog wrote: About

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

dd3b03346/runtime/queue.c#L2373 On 03/10/17 10:50, Rainer Gerhards wrote: What exactly do you mean? Rainer Sent from phone, thus brief. Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog" <rsyslog@lists.adiscon.com>: Hmm...wondering if theres is a bug here a

Re: [rsyslog] Does imfile-to-relp need DA queues?

is, but to do so it maintains a separate file of yet-unacked messages. Rainer On 04/10/17 09:31, Rainer Gerhards wrote: 2017-10-04 9:29 GMT+02:00 mostolog--- via rsyslog <rsyslog@lists.adiscon.com>: Thanks Rainer and David About this: is there any option/planned feature to ensure message has

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

Sent from phone, thus brief. Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog" <rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>: Hmm...wondering if theres is a bug here as: main_queue(     queue.filename="main"

Re: [rsyslog] Does imfile-to-relp need DA queues?

-04 9:29 GMT+02:00 mostolog--- via rsyslog <rsyslog@lists.adiscon.com>: Thanks Rainer and David About this: is there any option/planned feature to ensure message has been delivered before marking it as processed? eg: tell source message has been acknowledged when target has stored the

Re: [rsyslog] Does imfile-to-relp need DA queues?

Thanks Rainer and David About this: is there any option/planned feature to ensure message has been delivered before marking it as processed? eg: tell source message has been acknowledged when target has stored the event     source--->relp--->rsyslog--->kafka This would be "as easy as" not

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

, without apparent/documented reason https://github.com/rsyslog/rsyslog/blob/a6ae18ee4cb4530e4142a2e98aed404dd3b03346/runtime/queue.c#L2373 On 03/10/17 10:50, Rainer Gerhards wrote: What exactly do you mean? Rainer Sent from phone, thus brief. Am 03.10.2017 10:27 schrieb "mostolog--

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

. Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog" <rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>: Hmm...wondering if theres is a bug here as: main_queue(     queue.filename="main"     queue.maxdiskspace=&

[rsyslog] Does imfile-to-relp need DA queues?

Hi We're trying to confirm if *imfile-to-relp* is already 100% reliable and doesn't need any queue/DA, or if -*in the event of a failure*- messages being processed (in memory) will be lost/skipped. Would a configuration like this (without DA queue in ruleset) ensure all file contents are

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

Hmm...wondering if theres is a bug here as: main_queue(     queue.filename="main"     queue.maxdiskspace="4G"     queue.saveonshutdown="on"     queue.lowwatermark="3"     queue.highwatermark="6"     queue.size="10"     queue.type="LinkedList"     queue.fulldelaymark="9"    

[rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

Hi Some time ago we reported issues with RELP /hanging/ and not forwarding messages until restart. Last week we were able to replicate and still figuring out *how rsyslog delay system works*. According to documentation, /queue.fulldelaymark// //Number of messages when the queue

Re: [rsyslog] feedback request: debug log improvements

My 2 cents: As Rsyslog is used to handle several log formats, and encourage to use them, it would be great it also use a well-known format. Adding timestamps, caller (which component produces the log), thread-id or an identifier to "trace" an event going from input to output (print message

Re: [rsyslog] Imfile module parameters not known

you have two ")" at the end.. also 2 StateFile...review your config On 21/07/17 15:06, Luv via rsyslog wrote: This is the way I am reading logs from file, input(type="imfile" File="/var/log/nginx/infotrack_access.log.5.gz" StateFile="/var/spool/rsyslog/statefile1" ruleset="nginxoldruleset"

Re: [rsyslog] If a ruleset is bound to a specific input, will rsyslog check a message arriving on that input against a different ruleset?

It's implicit, AFAIK On 11/07/17 07:48, deoren wrote: Here is some pseducode based off of another recent thread: ruleset(name="remote-rules"){ action( ... ) action( ... ) stop } input(type="imudp" port="1514"

Re: [rsyslog] How to downgrade rsyslog from 2.7 to 2.4

This Dockerfile may help you building whatever rsyslog you need. FROM ubuntu:16.04 ARG DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y curl dnsutils vim && apt-get clean && rm -rf /var/lib/apt/lists/* RUN apt-get update && apt-get install -y software-properties-common &&

Re: [rsyslog] rsyslog going down frequently

impstats will show yo how many queues you have, and you can figure it out how much memory they are taking. Each queue you use, increases the memory footprint. On 29/06/17 11:12, Luv via rsyslog wrote: I am sending data to elasticsearch via rsyslog. I have been successful in that. But from

Re: [rsyslog] RELP does not resume when target goes back after a few hours down

According to http://www.rsyslog.com/doc/v8-stable/configuration/actions.html /If multiple retires fail, the interval is automatically extended. This is to prevent excessive ressource use for retires. After each 10 retries, the interval is extended by itself. To be precise, the actual interval

[rsyslog] RELP does not resume when target goes back after a few hours down

Hi We have been running a RELP->KAFKA infrastructure for a few weeks now, and this tests allowed us to detect issues and problems on our processing pipeline. Before planning imkafka tests and deployment, we have been making some fault-tolerant tests and we have observed the same undesired

Re: [rsyslog] Configuration for rsyslogd to send logs to remote server when it will have internet and after system reboot

Hi Ruben, this is more or less what I would do: * Each device logs into a file using omfile (this could be avoided if using /var/log/messages) * Theres a imfile which always retry and uses ruleset to forward (using omrelp) * My omrelp would just read and retry without a queue (or queue

Re: [rsyslog] The long way to OC questions

Sorry, wrong list :S On 06/06/17 14:07, mosto...@gmail.com wrote: Hi. Straight to the topic. Thanks in advance. * Does Opencast ingest (or Galicaster) splits a video into several segments, in order to process each part by different workers, hence increasing process speed?

[rsyslog] Installing 8.27 from repo in debian stretch

Following http://www.rsyslog.com/debian-repository/ results in: ... Executing: /tmp/apt-key-gpghome.zwvA0EHp4N/gpg.1.sh --recv-keys --keyserver keys.gnupg.net AEF0CF8E gpg: key 894ECF17AEF0CF8E: public key "Andre Lorbach (Key for signing deb packages) " imported gpg:

Re: [rsyslog] drop messages without timestamp

yslog-boun...@lists.adiscon.com] On Behalf Of mostolog--- via rsyslog Sent: Monday, April 24, 2017 10:05 AM To: rsyslog-users <rsyslog@lists.adiscon.com> Cc: mosto...@gmail.com Subject: Re: [rsyslog] drop messages without timestamp FYI: Java stack traces tend to be quite long, and a few weeks ago we

Re: [rsyslog] drop messages without timestamp

FYI: Java stack traces tend to be quite long, and a few weeks ago we had to increase maxmessagesize to 64KB. Would that be enough for your needs? El 21/04/17 a las 09:18, David Lang escribió: unless it's a massive log message, the best thing to do is probably increase maxmessagesize on the

Re: [rsyslog] Send logs in JSON format to Elasticsearch

I dont know if this is what you are talking http://www.rsyslog.com/doc/master/configuration/modules/mmjsonparse.html BTW: whats your rsyslog omelasticsearch index rate? Last test I made (with a basic configuration) we got ~10k/min only with latest ES version (perhaps its not fully compatible).

Re: [rsyslog] RELP suspendDuration went far too large?

El 14/02/17 a las 16:00, Rainer Gerhards escribió: Search on "debug on demand" should get you useful results. Great! Doing it as we speak! I'd also suggest to review the action () parameters. Anything in particular? there is a setting to have it log suspend messages, that gives you the

Re: [rsyslog] Am I missing something?

Isn't that configuration option used to allow "slash in syslogtag/appname/programname field"? why is then called ~hostname? El 14/02/17 a las 17:33, Rainer Gerhards escribió: I don't understand the question. RG 2017-02-14 16:46 GMT+01:00 mostolog--- via rsyslog <rsyslog@list

[rsyslog] Am I missing something?

Why permitSlashInHostname is not permitSlashInTag instead? From: http://www.rsyslog.com/doc/master/configuration/properties.html ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/

Re: [rsyslog] RELP suspendDuration went far too large?

This *could* also be related to https://github.com/rsyslog/librelp/pull/54 which I am about to merge soon and which will be released alongside rsyslog 8.25. Will test 8.25 and let you know if it's solved. there is a setting to have it log suspend messages, that gives you the duration. It's

[rsyslog] RELP suspendDuration went far too large?

Hi While using RELP to forward messages to a central rsyslog, we had a full disk issue on target. After cleaning the disk and recovering space, relay rsyslog seems is not forwarding traffic as it should (full speed!). Messages with priority 12 still arrive, and impstats shows: Feb 14

Re: [rsyslog] Using liblognorm rule tags

Seem that works. Thanks El 09/02/17 a las 15:32, David Lang escribió: On Thu, 9 Feb 2017, mostolog--- via rsyslog wrote: Hi While working with liblognorm we have found the /need/ of *using rule tags*. However setting */rule=aa:%.:@syslog% Foo%message:rest%/* doesn't seem to add a /tags

[rsyslog] Using liblognorm rule tags

Hi While working with liblognorm we have found the /need/ of *using rule tags*. However setting */rule=aa:%.:@syslog% Foo%message:rest%/* doesn't seem to add a /tags/ field to the message: <12>2017-02-09T13:32:34.884+01:00 computer tag: FooWhateverFollowsREDACTED although it seems to

Re: [rsyslog] Rsyslog imkafka

Ping! El 20/01/17 a las 15:19, mosto...@gmail.com escribió: Hi. Just started deploying kafka for testing with omkafka. Have anyone considered doing an *imkafka* module? So far, we have: * RELP->rsyslog->omelasticsearch->elasticsearch * RELP->rsyslog->omkafka Otherwise, I'll go for

Re: [rsyslog] working example of omkafka?

Finally managed to get it working, although not fully working :( First: rsyslog wasn't adding topics to kafka, cause I was using "@" within topic names, and that's an unsupported character, and maybe because I omitted :9092 Second: logstash is still not able to connect with kafka, cause:

Re: [rsyslog] working example of omkafka?

El 30/01/17 a las 19:25, Andrew Griffin escribió: I have a rsyslog -> kafka -> splunk stack working pretty well, I could probably answer a few of your questions - You can list topics (and a lot of other stuff) on the kafka brokers themselves using the kafka-topics.sh script included with

[rsyslog] working example of omkafka?

Hi *Do any of you have a working example of rsyslog (omkafka) -> kafka <- losgstash ?* So far I have been able to deploy a zookeper cluster using docker, deploy a kafka cluster (to be honest, I don't know how to test if it's working as a cluster or if I failed miserably), rsyslog as

Re: [rsyslog] Out of memory

davidelang: did you find something interesting in the 50K@100m/s vs. 50K@2000m/s ? Perhaps we should try after leak being fixed? El 19/01/17 a las 13:55, mosto...@gmail.com escribió: Attached 2 files. 1 for 100m/s and the other for 2000m/s. Let me know if you need longer traces. I'll try

Re: [rsyslog] Out of memory

Considering the below explanation, an idle (not going to receive more messages the next 7 days) imrelp rsyslog...shouldn't memory be freed from current 512MB usage in the near future? For the scenario I have described: no, as long as the high address memory block is allocated,

[rsyslog] Rsyslog imkafka

Hi. Just started deploying kafka for testing with omkafka. Have anyone considered doing an *imkafka* module? So far, we have: * RELP->rsyslog->omelasticsearch->elasticsearch * RELP->rsyslog->omkafka Otherwise, I'll go for logstash. Regards ___

Re: [rsyslog] Out of memory

Attached stats and valgrind logs. Let me know if I can do anything else. rsyslog was built with a few modules, debug and rtinst. Run: valgrind --tool=memcheck --log-file=/data/valgrind.log --show-possibly-lost=yes --track-origins=yes --trace-children=yes --leak-check=full

Re: [rsyslog] Out of memory

messages, but that's never gonna happen if we are still at 20k, and no more messages coming. Regards El 19/01/17 a las 11:16, Rainer Gerhards escribió: 2017-01-19 10:19 GMT+01:00 mostolog--- via rsyslog <rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>: w

Re: [rsyslog] How to know which modules rsyslog was compiled with

good point ;) El 19/01/17 a las 16:16, Rainer Gerhards escribió: 2017-01-19 16:12 GMT+01:00 mostolog--- via rsyslog <rsyslog@lists.adiscon.com>: Is there a simple way to know which modules rsyslog was compiled with? this is a serious answer: check, which module binaries were created.

[rsyslog] How to know which modules rsyslog was compiled with

Is there a simple way to know which modules rsyslog was compiled with? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards

Re: [rsyslog] Out of memory

Attached 2 files. 1 for 100m/s and the other for 2000m/s. Let me know if you need longer traces. I'll try valgrind memcheck later. Thu Jan 19 13:46:38 2017: global: origin=dynstats Thu Jan 19 13:46:38 2017: tofile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0

Re: [rsyslog] Out of memory

El 18/01/17 a las 17:05, Rainer Gerhards escribió: If this is a test environment, I would strongly recommend to run it in a terminal session under valgrind control. If it is an actual leak, valgrind can provide very good diagnostic information. That would be very helpful and clarify what is

Re: [rsyslog] Out of memory

Below an impstats trace showing a few tests: * receiving some idle messages * 1 message per second * 10m/sec * 100m/s * 1000m/s * 2000m/s None of the above created any queue files when using omfile as output. *Still, memory was slowly increasing (RGROW=1056K with 100m/s input)

Re: [rsyslog] Out of memory

when you give us your config, please include the name= portion so we can see what actions match up with what in impstats Pasted at the end, but they are named according to what they do. at 10:37, the queue is small and maxrss is still low (8092) at 10:42, you got a huge burst of new traffic

Re: [rsyslog] Out of memory

MaxMessageSize="32k" just a note that that's a rather large message size. Most messages are ~1KB, but some Java stack traces take up to 31KB. Setting this for 1% messages have any impact on the other 99%? you are obviously not showing us the full config, because nothing loads the

Re: [rsyslog] Out of memory

*Update* In the log below you may notice suddenly relp queue grow to 35k events (that's when the forwarding relay was restarted). And -at the end- it drops to 0, as the relay stop forwarding. Why rsyslog has't been able to reduce queue size from ~35k for quite a long time? Is 10k the aprox.

Re: [rsyslog] Out of memory

Hi This might be not enough yet to diagnose, but here's an spoiler. We have managed to review configuration and improve memory footprint, but still growing over time and ends being killed by oom: module(load="impstats" log.file="/data/stats.log") syslog.=debug /data/rsyslog-stats

Re: [rsyslog] Parsing very non standard input

have you tried mmnormalize? El 18/01/17 a las 09:58, Benoit DOLEZ escribió: Hi, I don't find how to properly parse a log from tcp/udp input that do not respect standard protocol. The line received has the format : -MM-DD HH:MM:SS HOSTNAME SEVERITY ID MESSAGE sample: 2016-11-12

[rsyslog] Out of memory

Hi At this moment, our testing rsyslog is receiving messages from relay through RELP in RFC3164 format with message in JSON like: DATE RELAYHOST TAG: {"field":"foo",..."data":"original_forwarded_message"} Once received, mmjsonparse is executed over MSG to check if incoming messages

Re: [rsyslog] Omkafka error log json addition

Hello all, I am working on a patch that will add intended topic name to the errorFile produced by the omkafka module in the writeDataError function called from the deliveryCallback. Thanks for the contribution. We are actually moving to kafka, so will benefit for that :) The question I

Re: [rsyslog] segfault in libfastjson.so.4.0.0 (with workaround)

Could you create an issue for this on github? could you also paste some related lines from /opt/secrets/log/8.8.8.8.log ? El 03/01/17 a las 13:54, Lennard Klein escribió: Hi list, I've ran into a segfault in libfastjson. First some details: RHEL7 using adiscon repositories:

Re: [rsyslog] Handling "log explosions"

If you know that a log message is uninteresting, then you want to throw it away, but count how many times it happened because the number of times that an uninteresting log happens can be interesting. That's exactly what I'm looking for 1.rsyslog gets message 2.if it's a notifiable error

Re: [rsyslog] Wildcard Support to log all the file logs of a directory

imfile module use state files to save each file "reading position", among other things. maybe you should delete them? Have a look at: https://raw.githubusercontent.com/mostolog/rsyslog-doc/patch-1/source/overview.rst

Re: [rsyslog] Wildcard Support to log all the file logs of a directory

Latest rsyslog version is 8.23 http://www.rsyslog.com/downloads/download-v8-stable/ AFAIK, in order to use imfile, you should run ./configure --enable-imfile before building source IIRC, inotify is set by default on recent versions El 30/12/16 a las 11:03, Shweta escribió: Yes, I am using

Re: [rsyslog] Handling "log explosions"

Maybe that could work. Thanks El 29/12/16 a las 13:02, Benoit DOLEZ escribió: Hi, Does this documentation answer your needs ? http://www.rsyslog.com/doc/v8-stable/configuration/dyn_stats.html Regards Benoit Le 29/12/2016 à 12:27, mostolog--- via rsyslog a écrit : impstats Ok. I'll

Re: [rsyslog] Handling "log explosions"

impstats Ok. I'll have a look define 'too fast'. And how do you tell the difference between your logging system having a problem and generating so many messages and the systems you are collecting logs from generating the messages? If you setup thresholds and start throwing away messages

Re: [rsyslog] Handling "log explosions"

El 28/12/16 a las 22:41, David Lang escribió: On Wed, 28 Dec 2016, mostolog--- via rsyslog wrote: While testing our current infrastructure we have suffered a /log explosion/, ie: errors when processing logs caused error logs on the machine that also caused errors when processed...and finally

Re: [rsyslog] Does it make sense to have a queue with imfile+omrelp?

Thanks for your clarifying answers. El 28/12/16 a las 22:38, David Lang escribió: On Wed, 28 Dec 2016, mostolog--- via rsyslog wrote: Even more: does it make sense to have queues when using omfile? usually not, it's usually less effort to write the data to the file than to move

[rsyslog] Handling "log explosions"

Hello While testing our current infrastructure we have suffered a /log explosion/, ie: errors when processing logs caused error logs on the machine that also caused errors when processed...and finally, disk became full and everything died. I'm wondering if worrying about this is useful, or

Re: [rsyslog] Does it make sense to have a queue with imfile+omrelp?

Even more: does it make sense to have queues when using omfile? El 28/12/16 a las 15:52, mosto...@gmail.com escribió: Hi Does it make any sense to use queues when reading a file (imfile) and forwarding to a central location using RELP (omrelp) ? We would like to read & send events in a

[rsyslog] Does it make sense to have a queue with imfile+omrelp?

Hi Does it make any sense to use queues when reading a file (imfile) and forwarding to a central location using RELP (omrelp) ? We would like to read & send events in a reliable way (reading offset/position is only updated if successfully received on server). but I don't understand why

Re: [rsyslog] Deleting already "processed" files

well, if logs are redirected to the host syslog, you don't have the logs in files for imfile to have to read, you have them in rsyslog (or the systemd journal, but that's your problem if you do :-) if you have logs that are written to disk, then you have two choices 1. have something tail

Re: [rsyslog] Deleting already "processed" files

hing worth an enhancement PR. But it won't be prioprity from my side. Rainer 2016-12-21 19:34 GMT+01:00 mostolog--- via rsyslog <rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>: Hi To contextualize: A server is generating multiple log files and sending them using

Re: [rsyslog] Liblognorm - Usage of field type iptables

Sorry. Just awake and didnt look for it before asking... On Thursday, December 22, 2016, mostolog wrote: > Rainer: is liblognorm doc also at github using rst format? > > On Wednesday, December 21, 2016, matthew.gaetano

Re: [rsyslog] Liblognorm - Usage of field type iptables

Rainer: is liblognorm doc also at github using rst format? On Wednesday, December 21, 2016, matthew.gaetano wrote: > Alright! Almost out the door for the holiday's and swore i would come back > to > this before I left in the event anyone stumbled across it. Note that

Re: [rsyslog] Deleting already "processed" files

On Wednesday, December 21, 2016, David Lang <da...@lang.hm> wrote: > On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote: > > To contextualize: >> >> A server is generating multiple log files and sending them using RELP to >> a Rsyslog relay. I would love to auto

[rsyslog] Deleting already "processed" files

Hi To contextualize: A server is generating multiple log files and sending them using RELP to a Rsyslog relay. I would love to automatically delete older logs files if they are already sent/acknowledged. Is there a way to make Rsyslog delete a file after successfully sending it via RELP?

Re: [rsyslog] About reloading configuration

n most case I think we need a simple rainer-script if/then/else using a liblognorm app specific rule file. Then adding a new parser implies reloading rsyslog... Thank you for responses. I'm continuing to analyse source... Regards Benoit Le 21/12/2016 à 10:33, mostolog--- via rsyslog a écrit : Y

Re: [rsyslog] saving liblognorm literals

El 21/12/16 a las 17:46, Rainer Gerhards escribió: see https://github.com/rsyslog/liblognorm/pull/238 wow! that was fast! 2016-12-21 16:54 GMT+01:00 David Lang : Can you explain your ruleset where you need to store literal as a value in the json? I think the original

Re: [rsyslog] omelasticsearch index warning

Thanks for your answers. I'll also try to have a look on this. El 21/12/16 a las 17:48, matthew.gaetano escribió: Though the examples given are specific modifications for aspects of Elasticsearch 5.X i believe the modification method would be valid for any version of Elasticsearch supported by

Re: [rsyslog] omelasticsearch index warning

ping! El 15/12/16 a las 12:37, mosto...@gmail.com escribió: Hi Indexing on elasticsearch 5.1.1 we're getting: [2016-12-15T12:35:27,844][WARN ][o.e.d.r.a.a.i.RestTypesExistsAction] [HEAD /{index}/{type}] is deprecated! Use [HEAD /{index}/_mapping/{type}] Is that a omelasticsearch issue or

[rsyslog] saving liblognorm literals

Hi (Moving from grok to liblognorm) We are trying to store a literal as a variable, as we were doing using grok. eg: (?ACCEPT) We aren't sure if this is correct: rule=:%{"type":"literal", "text":"a", "name":"var"}% As stated in

Re: [rsyslog] About reloading configuration

You may also find interesting to be able to reload liblognorm rules, which might be an smaller change. The question here is the needed amount of work. El 21/12/16 a las 10:29, Rainer Gerhards escribió: Possible of course, but considerable work (gut feeling: 6+ strong weeks). Rainer Sent

Re: [rsyslog] Assert Failure in imtcp.c?

It will be great if you provide a replication use case: * rsyslog version (may be 8.23) * configuration file * log example to replicate the issue. If you are able to replicate the issue with a "not too old" syslog version, please file an issue on github El 20/12/16 a las 21:14, Andrew

[rsyslog] omfile.rst documentation

Hi Reviewing omfile documentation https://github.com/rsyslog/rsyslog-doc/blob/master/source/configuration/modules/omfile.rst It states: - **$F$OMFileForceCHOwn** equivalent to the "ForceChOwn" parameter - **$ActionFileEnableSync** equivalent to the "enableSync" parameter There's a typo

Re: [rsyslog] liblognorm vs grok

Just created https://github.com/rsyslog/liblognorm/issues/236 El 20/12/16 a las 11:58, mosto...@gmail.com escribió: El 20/12/16 a las 11:55, Rainer Gerhards escribió: 2016-12-20 11:54 GMT+01:00 mostolog--- via rsyslog<rsyslog@lists.adiscon.com>: Must first line be... "ve

Re: [rsyslog] liblognorm segfault ?

ping? Hi Having more problems with liblognorm. Let me now if I should open an issue. echo "a" | /usr/lib/lognorm/lognormalizer -r a.rb Segmentation fault (core dumped) File: version=2 #foo type=@rfc3164pri:<%priority:number%>

Re: [rsyslog] liblognorm vs grok

Must first line be... "version=2" (v lowercase) or "Version=2" (V uppercase) ? El 14/12/16 a las 10:44, mosto...@gmail.com escribió: El 07/12/16 a las 21:00, Rainer Gerhards escribió: I'm getting /invalid field type 'alternative'/ when using it. Any ideas? rule=test:%[

Re: [rsyslog] Are we building an ERK stack?

This is exactly why we have $. variables as well as $! variables. They work exactly the same, but by convention, $! variables are where you put things that you are going to want to send elsewhere, and $. variables are where you put things that you need to create for your internal logic,

[rsyslog] liblognorm segfault ?

Hi Having more problems with liblognorm. Let me now if I should open an issue. echo "a" | /usr/lib/lognorm/lognormalizer -r a.rb Segmentation fault (core dumped) File: version=2 #foo type=@rfc3164pri:<%priority:number%> type=@rfc3164header:%date:date-rfc3164%

Re: [rsyslog] Are we building an ERK stack?

Solved using json template (code blindness). Is there any way to set fields and use them (like @timestamp) but not indexing them on elastic? (hidden fields) Just tried with @timestamp, but it's being indexed :( El 15/12/16 a las 12:32, mosto...@gmail.com escribió: Hi At this moment we

[rsyslog] omelasticsearch index warning

Hi Indexing on elasticsearch 5.1.1 we're getting: [2016-12-15T12:35:27,844][WARN ][o.e.d.r.a.a.i.RestTypesExistsAction] [HEAD /{index}/{type}] is deprecated! Use [HEAD /{index}/_mapping/{type}] Is that a omelasticsearch issue or are we missing something?

Re: [rsyslog] Are we building an ERK stack?

Hi At this moment we are frowarding RELP messages to Elasticsearch using omelasticsearch plugin, but sadly message appears as json instead of storing each properties. eg: message is { "app": "app1"... instead of indexed document having a app property. Should we specify an especial param on

Re: [rsyslog] global queue configuration?

Probably, carrying multiple conversations at the same time didn't help clarifying our concerns. My fault on that. Really sorry. We know 0% loss is "impossible", but starting from there we'll like to have the /best/ system possible. Considering we found some issues with RELP+full queues we

Re: [rsyslog] rsyslog-doc: dot graphs in doc

To solve this hopefully once and forever, I googled again for "sphinx dot" and the first result seems to be fully appropriate: http://www.sphinx-doc.org/en/1.4.8/ext/graphviz.html As I said two weeks or so ago, we need to check if we have sufficiently current sphinx on the involved systems,

Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic

set $!data=$msg; you are getting confused over the difference between a string that looks like json and an actual json structure. Sorry, I was. action(type="mmnormalize" path="data") this would populate $!data with the structure parsed out of $msg. mmjsonparse puts the parsed data

Re: [rsyslog] global queue configuration?

On Wed, 14 Dec 2016, David Lang wrote: this will slow rsyslog down by a factor of >1000x. I ran tests on this using a very high end PCI based SSD and had a 2000x - 8000x slowdown, and this was with v5, the added improvements in performance since wukk cause an even bigger slowdown as they

Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic

Here you go. This is what we have so far. global( MaxMessageSize="32k" workDirectory="/data" parser.escapeControlCharactersOnReceive="off" ) module(load="imrelp") input( port="20514" type="imrelp" name="imrelp" ruleset="json"

Re: [rsyslog] rsyslog-doc: dot graphs in doc

Trying to keep the discussion alive and separated from quite long-long-long github issue IMHO it's better to have embedded graphs in doc, rather than separate images files, cause it will ease doc maintenance. Said so, I agree with you using a third-party web provider is not a valid choice.

Re: [rsyslog] global queue configuration?

the parameters you listed are all viable. But listing them in global() would have no effect. You need to specify them in an action(), ruleset() or main-queue() so they are used there. Disk queues always use the disk and do not buffer anything in memory, which means they are ultra-reliable,

  1   2   >