[rsyslog] scalability of mmnormalize

[Micah Yoder]

Hi,

Couple questions...

Does mmnormalize use multiple cores? I see in the liblognorm
documentation that it does not internally, but it can be called from
multiple threads.

Any ideas as to how many messages it should be able to process per
second on a beefy server with, say, 12 cores? I'm sure it depends on the
number of rules. With, say, 50 rules, think it could do 50 messages a
second?

We had a logging bottleneck with I think fewer messages than this with
fewer rules, but CPU load was low - hence it didn't look like it was
utilizing all the cores. Then we disabled normalization for now. That
was with rsyslog 8.2.2. Has performance improved in 8.7?

Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] scalability of mmnormalize

[Micah Yoder]

Thank you, good to know!

On 1/28/15 5:30 PM, David Lang wrote:
> By default you will only use one core for mmnormalize, but you can
> configure it to use more (although doing so when you don't need to ends
> up slowing you down due to contention on the queue)
> 
> I don't have info on how fast to expect it to be, it would vary by quite
> a bit depending on the speed of your cpu/ram/cache.
> 
> the number of rules does not significantly affect the speed of
> mmnormalize, the way that it compiles the rules when loading the ruleset
> means that the length of the log message (and how complex the rule is
> that match it) matter more than how many rules there are.
> 
> If you are running into bottlenecks at 10s of logs/sec then something is
> very wrong. Normally rsyslog can do hundreds of thousands of logs/sec,
> so I would still expect thousands of logs/sec with mmnormalize (as long
> as you don't use regex matches)
> 
> If you think that it's being limited by only using one core, use the 'H'
> option in top to show the individual threads, see if any of the rsyslog
> threads show that they are using 100% cpu. If not then you have some
> other problem.
> 
> David Lang
> 
> On Wed, 28 Jan 2015, Micah Yoder wrote:
> 
>> Hi,
>>
>> Couple questions...
>>
>> Does mmnormalize use multiple cores? I see in the liblognorm
>> documentation that it does not internally, but it can be called from
>> multiple threads.
>>
>> Any ideas as to how many messages it should be able to process per
>> second on a beefy server with, say, 12 cores? I'm sure it depends on the
>> number of rules. With, say, 50 rules, think it could do 50 messages a
>> second?
>>
>> We had a logging bottleneck with I think fewer messages than this with
>> fewer rules, but CPU load was low - hence it didn't look like it was
>> utilizing all the cores. Then we disabled normalization for now. That
>> was with rsyslog 8.2.2. Has performance improved in 8.7?
>>
>> Thanks!
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
>>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] mapping action numbers in impstats to config file lines

[Micah Yoder]

Hi,

There isn't currently a good way to find out which action number
corresponds to which action in the config, is there?  Do you guys just
count off the action lines, in the order the files are included, to
figure out which action in impstats goes to which line?

We have a fair number of included files, generated by snippets in
Puppet. That makes such a prospect non-trivial, and it can change action
numbers when we add or remove parts of our configuration.

Seems like it would be good to either:

1) Allow naming of actions and have impstats print the action name
instead of number
 - or -
2) When rsyslog starts, just dump a list of the action numbers and the
file:line# that defines them

The former would be nicest but the latter probably easier to implement.

If nothing like this is currently possible, it seems as though the
second would not be terribly difficult to code. I may be open to trying.
(I pretty much know C, but rarely use it and would need someone to
bounce questions off of regarding the right way to do it in the codebase.)

Thanks!


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mapping action numbers in impstats to config file lines

[Micah Yoder]

On 6/9/15 2:05 PM, David Lang wrote:

> This is already available. Just add name="name" in the action() statement
> 
> I name the things that it makes sense to put in action() statements, and
> put up with numbers for the items between them.

sweet, not sure how I missed that!
Thanks


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] segfault problems

[Micah Yoder]

Hi,

So we have a somewhat complex environment involving rules for various
application logs and elasticsearch output generated by Puppet, and some
other parts involving regex extractions.

Our staging environment runs 8.10, and there have been a small number of
segfaults. Maybe one a month.

In the dev environment, I made some changes and now get a segfault
usually within a few seconds of starting rsyslog, even when upgrading to
8.13.

Enabling debug logging and capturing the last part of the output before
the crash, here's what I got:

1205.991657460:main Q:Reg/w0  : eval expr 0x7fccf4f6dfc0, return
datatype 'S'
1205.991665047:main Q:Reg/w0  : SET !vip_name =
1205.991673224:main Q:Reg/w0  :   function 'exec_template' (id:12,
params:1)
1205.991685453:main Q:Reg/w0  : string 'extractvip'
1205.991699401:main Q:Reg/w0  : END SET
1205.991707006:main Q:Reg/w0  : eval expr 0x7fccf4f6e190, type 'F[70]'
1205.991710003:main Q:Reg/w0  : rainerscript: executing function id 12

I need to be careful about how much of our environment details I post
here, but here's the code where this comes from:

template(name="extractvip" type="string" string="%syslogtag:14:$%")
set $!vip_name = exec_template("extractvip");

Note that this is code that did NOT change.  It's been in our
environment for probably 2 years.  I repeated it 3 times, with the same
result.  rsyslog crashed with a message like this being at the end of
the line.

I then removed the file with this command (it was included) and ran it
again twice, getting this both times:

2271.865308042:main Q:Reg/w0  : eval expr 0x7f9fe74869d0, return
datatype 'N'
2271.865310862:main Q:Reg/w0  : if condition result is 1
2271.865313611:main Q:Reg/w0  : SET .endpoint =
2271.865320618:main Q:Reg/w0  :   function 're_extract' (id:8, params:5)
2271.865332270:main Q:Reg/w0  : var 'msg'
2271.865343109:main Q:Reg/w0  : string '[redacted, VIP F5
partition name]\/([a-zA-Z0-9.]+)'
2271.865492825:main Q:Reg/w0  : 0
2271.865505204:main Q:Reg/w0  : 1
2271.865516558:main Q:Reg/w0  : string ''
2271.865530362:main Q:Reg/w0  : END SET
2271.865538150:main Q:Reg/w0  : eval expr 0x7f9fe74067d0, type 'F[70]'
2271.865541189:main Q:Reg/w0  : rainerscript: executing function id 8
2271.865544581:main Q:Reg/w0  : eval expr 0x7f9fe7408da0, type 'V[86]'
2271.865548299:main Q:Reg/w0  : rainerscript: var 1: '[redacted, access
log line for a VIP]'
2271.865551401:main Q:Reg/w0  : eval expr 0x7f9fe7408da0, return
datatype 'S'
2271.865554481:main Q:Reg/w0  : eval expr 0x7f9fe7408e20, type 'N[78]'
2271.865561495:main Q:Reg/w0  : eval expr 0x7f9fe7408e20, return
datatype 'N'
2271.865564783:main Q:Reg/w0  : eval expr 0x7f9fe7409010, type 'N[78]'
2271.865567815:main Q:Reg/w0  : eval expr 0x7f9fe7409010, return
datatype 'N'

The line that does that is just setting $.endpoint to an re_extract of
$msg.  This also did not change.

Both of the things though are included *after* things that did change.

I removed this file also and rsyslog has been running ever since.

What did change is some elasticsearch queues, from direct to disk
assisted and some other minor structure changes.

Any pointers to finding the source of this segfault?  I can try to run
it under gdb if that would help, but I'm not great with that.

Thanks!


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault problems

[Micah Yoder]

On 10/7/15 12:11 PM, David Lang wrote:
> It looks like json-c is not really thread-safe. specifically when using
> anything that outputs a json structure. Running into problems requires
> that you have multiple threads working on the same message, but if you
> have an action queue for your elasticsearch output, that would create
> the conditions that can trigger this.

That may be part of it, thanks.  I get this in dmesg on the staging
server (8.10):

rs:action-app_e[45178] general protection ip:7fcc56c91633
sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000]
rs:action-app_e[3213]: segfault at 0 ip 7f538356e633 sp
7f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000]

With the changes I made in the dev server (8.13), here are the dmesg
outputs:

rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 7f0d9a667f2c sp
7f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000]
rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 7fccf3c67f2c sp
7fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000]
rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 7f473c5e2f2c sp
7f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000]
rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 7fa5a79c1f2c sp
7fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000]
rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 7f53b11f73d8 error 14
rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 7f9fd4df63d8 error 14

No json-c there.

Maybe we should just update to 8.13 in staging, putting my other changes
off for a bit.  We do kind of need them though - currently messages are
being dropped, we think because the ElasticSearch actions are direct
queues instead of linked lists, and they are causing bottlenecks
elsewhere in the system.

Will discuss with my team, thanks!


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault problems

[Micah Yoder]

well.  I updated our spare staging server, which gets virtually no
logs but has the same config that's been working fairly well for a
while, from 8.10 to 8.13 and it's been crashing all night, being
restarted by Puppet.

rs:main Q:Reg[14469]: segfault at 7fc31c023000 ip 7fc34d85cf2c sp
7fc3418412d0 error 6 in rsyslogd[7fc34d838000+8d000]

Error 6 seems to be no such device or address.

I think we have a problem here .

On 10/7/15 1:29 PM, Ciprian Hacman wrote:
> I strongly recommend using 8.13 if you push logs to ES. There are many
> useful patches since 8.10 (some of them might not be in the changelog
> though).
> 
> Ciprian
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
> 
> On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder 
> wrote:
> 
>> On 10/7/15 12:11 PM, David Lang wrote:
>>> It looks like json-c is not really thread-safe. specifically when using
>>> anything that outputs a json structure. Running into problems requires
>>> that you have multiple threads working on the same message, but if you
>>> have an action queue for your elasticsearch output, that would create
>>> the conditions that can trigger this.
>>
>> That may be part of it, thanks.  I get this in dmesg on the staging
>> server (8.10):
>>
>> rs:action-app_e[45178] general protection ip:7fcc56c91633
>> sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000]
>> rs:action-app_e[3213]: segfault at 0 ip 7f538356e633 sp
>> 7f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000]
>>
>> With the changes I made in the dev server (8.13), here are the dmesg
>> outputs:
>>
>> rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 7f0d9a667f2c sp
>> 7f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000]
>> rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 7fccf3c67f2c sp
>> 7fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000]
>> rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 7f473c5e2f2c sp
>> 7f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000]
>> rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 7fa5a79c1f2c sp
>> 7fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000]
>> rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 7f53b11f73d8 error 14
>> rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 7f9fd4df63d8 error 14
>>
>> No json-c there.
>>
>> Maybe we should just update to 8.13 in staging, putting my other changes
>> off for a bit.  We do kind of need them though - currently messages are
>> being dropped, we think because the ElasticSearch actions are direct
>> queues instead of linked lists, and they are causing bottlenecks
>> elsewhere in the system.
>>
>> Will discuss with my team, thanks!
>>
>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
> 

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault problems

[Micah Yoder]

I have some gdb output following the instructions at the bottom of
http://www.rsyslog.com/doc/v8-stable/troubleshooting/troubleshoot.html

Is there someone I can send this to?

Thanks!

On 10/8/15 9:22 AM, Micah Yoder wrote:
> well.  I updated our spare staging server, which gets virtually no
> logs but has the same config that's been working fairly well for a
> while, from 8.10 to 8.13 and it's been crashing all night, being
> restarted by Puppet.
> 
> rs:main Q:Reg[14469]: segfault at 7fc31c023000 ip 7fc34d85cf2c sp
> 7fc3418412d0 error 6 in rsyslogd[7fc34d838000+8d000]
> 
> Error 6 seems to be no such device or address.
> 
> I think we have a problem here .
> 
> On 10/7/15 1:29 PM, Ciprian Hacman wrote:
>> I strongly recommend using 8.13 if you push logs to ES. There are many
>> useful patches since 8.10 (some of them might not be in the changelog
>> though).
>>
>> Ciprian
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>> On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder 
>> wrote:
>>
>>> On 10/7/15 12:11 PM, David Lang wrote:
>>>> It looks like json-c is not really thread-safe. specifically when using
>>>> anything that outputs a json structure. Running into problems requires
>>>> that you have multiple threads working on the same message, but if you
>>>> have an action queue for your elasticsearch output, that would create
>>>> the conditions that can trigger this.
>>>
>>> That may be part of it, thanks.  I get this in dmesg on the staging
>>> server (8.10):
>>>
>>> rs:action-app_e[45178] general protection ip:7fcc56c91633
>>> sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000]
>>> rs:action-app_e[3213]: segfault at 0 ip 7f538356e633 sp
>>> 7f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000]
>>>
>>> With the changes I made in the dev server (8.13), here are the dmesg
>>> outputs:
>>>
>>> rs:main Q:Reg[26553]: segfault at 7f0d7009a000 ip 7f0d9a667f2c sp
>>> 7f0d7fbfd2d0 error 6 in rsyslogd[7f0d9a643000+8d000]
>>> rs:main Q:Reg[26835]: segfault at 7fccc4073000 ip 7fccf3c67f2c sp
>>> 7fcce1bf82d0 error 6 in rsyslogd[7fccf3c43000+8d000]
>>> rs:main Q:Reg[27566]: segfault at 7f471404a000 ip 7f473c5e2f2c sp
>>> 7f472b1fc2d0 error 6 in rsyslogd[7f473c5be000+8d000]
>>> rs:main Q:Reg[27855]: segfault at 7fa58002d000 ip 7fa5a79c1f2c sp
>>> 7fa5957f72d0 error 6 in rsyslogd[7fa5a799d000+8d000]
>>> rs:main Q:Reg[28169]: segfault at 0 ip (null) sp 7f53b11f73d8 error 14
>>> rs:main Q:Reg[31317]: segfault at 0 ip (null) sp 7f9fd4df63d8 error 14
>>>
>>> No json-c there.
>>>
>>> Maybe we should just update to 8.13 in staging, putting my other changes
>>> off for a bit.  We do kind of need them though - currently messages are
>>> being dropped, we think because the ElasticSearch actions are direct
>>> queues instead of linked lists, and they are causing bottlenecks
>>> elsewhere in the system.
>>>
>>> Will discuss with my team, thanks!
>>>
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
>>
> 
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
> 

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault problems

[Micah Yoder]

Just did. https://github.com/rsyslog/rsyslog/issues/550
Pastebin with gdb info linked therein.
Unfortunately I'm not sure I can get it to a small reproducible config.

David, re: config:
It is a hierarchy of files generated by Puppet to put in several
different elasticsearch rules and application log rules.  If you need
the whole thing, I'll have to discuss that with folks here first.
(There could be a case for hiring Adiscon here)

Here is one of our elastic files:

### Send ssh logs to ElasticSearch!

template(name="idxperday_ssh" type="list") {
constant(value="ssh-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
if $!event.tags[0] == "ssh" then {
  action(type="omelasticsearch" name="action-elastic_ssh"
server="[hostname redacted]" searchIndex="idxperday_ssh"
dynSearchIndex="on" searchType="events" bulkmode="on"
queue.type="linkedlist" queue.filename="ssh" queue.size="20"
queue.spoolDirectory="/var/log-queues" queue.maxdiskspace="100M"
action.resumeretrycount="-1" queue.timeoutenqueue="0"
queue.dequeuebatchsize="1000" template="all")
}

There are several others like this, and several other things.

You had mentioned separating the templates from the actions, but this
current Puppet generated hierarchy makes that somewhat difficult.  In
any case I think that was for the json-c bug, which is something
different I think.

Let me know if the pastebin gdb info makes any sense.

Thanks!



On 10/8/15 12:10 PM, Ciprian Hacman wrote:
> Can you create an issue here?
> https://github.com/rsyslog/rsyslog/issues
> 
> If you have some easier way to reproduce, it would help also.
> 
> Ciprian
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
> 
> On Thu, Oct 8, 2015 at 8:07 PM, Micah Yoder 
> wrote:
> 
>> I have some gdb output following the instructions at the bottom of
>> http://www.rsyslog.com/doc/v8-stable/troubleshooting/troubleshoot.html
>>
>> Is there someone I can send this to?
>>
>> Thanks!
>>
>> On 10/8/15 9:22 AM, Micah Yoder wrote:
>>> well.  I updated our spare staging server, which gets virtually no
>>> logs but has the same config that's been working fairly well for a
>>> while, from 8.10 to 8.13 and it's been crashing all night, being
>>> restarted by Puppet.
>>>
>>> rs:main Q:Reg[14469]: segfault at 7fc31c023000 ip 7fc34d85cf2c sp
>>> 7fc3418412d0 error 6 in rsyslogd[7fc34d838000+8d000]
>>>
>>> Error 6 seems to be no such device or address.
>>>
>>> I think we have a problem here .
>>>
>>> On 10/7/15 1:29 PM, Ciprian Hacman wrote:
>>>> I strongly recommend using 8.13 if you push logs to ES. There are many
>>>> useful patches since 8.10 (some of them might not be in the changelog
>>>> though).
>>>>
>>>> Ciprian
>>>> --
>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>> On Wed, Oct 7, 2015 at 9:17 PM, Micah Yoder 
>>>> wrote:
>>>>
>>>>> On 10/7/15 12:11 PM, David Lang wrote:
>>>>>> It looks like json-c is not really thread-safe. specifically when
>> using
>>>>>> anything that outputs a json structure. Running into problems requires
>>>>>> that you have multiple threads working on the same message, but if you
>>>>>> have an action queue for your elasticsearch output, that would create
>>>>>> the conditions that can trigger this.
>>>>>
>>>>> That may be part of it, thanks.  I get this in dmesg on the staging
>>>>> server (8.10):
>>>>>
>>>>> rs:action-app_e[45178] general protection ip:7fcc56c91633
>>>>> sp:7fcc45bf89d8 error:0 in libjson-c.so.2.0.1[7fcc56c8a000+a000]
>>>>> rs:action-app_e[3213]: segfault at 0 ip 7f538356e633 sp
>>>>> 7f53715f19d8 error 6 in libjson-c.so.2.0.1[7f5383567000+a000]
>>>>>
>>>>> With t

Re: [rsyslog] segfault problems

[Micah Yoder]

On 10/8/15 1:25 PM, David Lang wrote:

> even using puppet, I think you can separate out the template generation

Oh sure i can, it would just be a bit of a re-organization.

> worst case, create a separate include directory that only includes the
> templates and the code that evaluates the templates into variables, then
> your actions in the main include directory can use those.

Well I just took our config and did that manually outside of puppet on
the dev server, in a bit of a hack job.  Unfortunately the crashes
persist with the same regularity.  I am now getting error 6 in the
segfault whereas it was 15 before - though I've also had error 6 before
and also error 14.

However, the actions still called the templates via the dynamic index
per day name so maybe it wasn't separated enough.  But is there even a
way to make it not call the template in that case?

So is it only really important to separate out the templates that set up
the JSON, or all templates?

> But if your various ES things mostly use the template "all" (which is
> what I would expect), then you may be able to do something much simpler
> than you are thinking. In my case it turned out that I only had 3
> templates that ended up getting used in all the different rules.

I use two templates per omelasticsearch - one to define the index name.
 I currently use a separate template per name -
"es_index_name-.mm.dd" but now I see you're right, I could stick the
name into a variable $.es_index_name and use one template for that.

There are two content templates - one "all" and one other type.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault problems

[Micah Yoder]

Well, with that exact same configuration (with the templates all moved to the 
front of the include chain), with it crashing in a few minutes under 8.13, I 
downgraded to 8.10, and it hasn't crashed yet.  It's been a couple hours.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault problems

[Micah Yoder]

On 10/8/15 6:54 PM, Micah Yoder wrote:
> Well, with that exact same configuration (with the templates all moved to the 
> front of the include chain), with it crashing in a few minutes under 8.13, I 
> downgraded to 8.10, and it hasn't crashed yet.  It's been a couple hours.

Bah. Ended up crashing in the middle of the night. :(
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
> 

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Setting TLS cipher suite / disabling RC4

[Micah Yoder]

Hi,

We have a PCI requirement to disable the RC4 cipher on our rsyslog TLS 
setup.  I for the life of me can not find a configuration option to set 
the cipher suite.  What am I missing?


Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Setting TLS cipher suite / disabling RC4

[Micah Yoder]

Thanks for the info and, yes, that's a good point. These certainly are 
not Internet facing and we control both ends. This is probably a good 
case for an exception.  Otherwise I'll look up controling gnutls via 
environment variables and/or sending it through autossh tunnels.


On 05/25/2016 12:22 AM, David Lang wrote:

By the way, and I know this probably doesn't help your audit
requirement, but SSL is going to negotiate the strongest mode that the
sender and receiver have in common, so it is never going to use RC4,
even if you don't explicitly disable it.

When you have random folks on the Internet connecting, the negotiation
of the 'strongest cipher in common' gets interesting enough that you
really do want to disable it. But when you are controlling both sides,
and they are even reasonably up to date and both the same
implementation, the real risk is very low.

But as I say, audits usually don't care about real risk, they have their
checkboxes...

David Lang


On Tue, 24 May 2016, David Lang wrote:


Date: Tue, 24 May 2016 16:16:22 -0700 (PDT)
From: David Lang 
To: rsyslog-users 
Subject: Re: [rsyslog] Setting TLS cipher suite / disabling RC4

On Tue, 24 May 2016, Micah Yoder wrote:


We have a PCI requirement to disable the RC4 cipher on our rsyslog
TLS setup. I for the life of me can not find a configuration option
to set the cipher suite.  What am I missing?


Unfortunantly, rsyslog's use of gnutls is very basic. It has very few
options. If there is anyone who is a guru in this area, we could use a
lot of knowlegeable help.

Rsyslog trats the tls config as a black box providing the minimum
config items needed to make things work.

It's possible that the library honors environment veriables for some
of these settings, if so you can work around the limits that way.

Before rsyslog starting using gnutls, the work-around was to use
stunnel and run the logging traffic through stunnel. This still works.

Patches to improve the control over gnutls would be very much welcome,
but the trouble if that there is already far too much confusion over
getting it to work, so just adding all the possible config options
with good explinations over what's what and when it should be used
would only increase the confusion.

Someone who really knows this library could probably identify a
smallish subset of the options that we really should support and
provide some sort of explination as to what they mean pretty easily.


Unfortunatly this is why so many TLS related questions go unanswered
for a while here on the list.

David Lang


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Sortable timestamp in a CEE variable

[Micah Yoder]

Hi,

So last April I asked a question on serverfault about a problem I was
having - the CEE variables, which were being passed to ElasticSearch -
did not contain a sortable timestamp field, and I wasn't able to get one
via templates.

http://serverfault.com/questions/496527/rsyslog-assigning-a-modified-property-to-a-cee-variable

Rainer answered (I assume it is Rainer anyway!) that this wasn't
possible because templates can not be assigned to a CEE variable.  I
still wonder why this is, and if anything is being done to change this?
 Is it not reasonable to want to assign a template to a CEE variable?

In the mean time I have solved the problem by creating a timestamp
mapping within ElasticSearch, but I think it would be somewhat
preferrable to actually use the timestamp in the original log message.

Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Sortable timestamp in a CEE variable

[Micah Yoder]

David:
Great to hear that!  I looked over some of the improvements in v8 and
was already planning to upgrade to it on release, but it wasn't entirely
clear if it would have this capability.

James:
Well I'm using liblognorm also, and a subtree template. Yeah I suppose
constructing it in JSON would work in simple cases.

In any case if it will be in v8 I'm happy with that.  No all-fire rush,
I can wait.

Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] JSON + rsyslog + Elasticsearch = Love. Now on rsyslog.com

[Micah Yoder]

Back in August I actually wrote an article that covers some similar
content, because I spent a fair amount of time struggling with the
rsyslog documentation for this stuff.  I guess someone thought it was
useful, because it was reposted on the Puppet Labs devops blog. :)

http://developer.rackspace.com/blog/rsyslog-and-elasticsearch.html

It probably could/should be updated and/or clarified at this point,
especially after v8 comes out.

On 1/13/14 7:06 AM, Radu Gheorghe wrote:
> Hi,
> 
> I just reposted on rsyslog.com an
> articlethat
> was written on
> blog.sematext.com about handling CEE-formatted messages with mmnormalize
> and indexing them via omelasticseearch:
> http://www.rsyslog.com/json-elasticsearch/
> 
> Positive feedback and constructive criticism will be equally and deeply
> appreciated :)
> 
> I'm planning to keep such articles flowing, unless there's a good reason
> not to.
> 
> Best regards,
> Radu
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
> 

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] JSON + rsyslog + Elasticsearch = Love. Now on rsyslog.com

[Micah Yoder]

On 1/14/14 4:06 AM, Rainer Gerhards wrote:
> On Mon, Jan 13, 2014 at 10:49 PM, Micah Yoder 
> wrote:
> 
>> Back in August I actually wrote an article that covers some similar
>> content, because I spent a fair amount of time struggling with the
>> rsyslog documentation for this stuff.  I guess someone thought it was
>> useful, because it was reposted on the Puppet Labs devops blog. :)
>>
>> http://developer.rackspace.com/blog/rsyslog-and-elasticsearch.html
>>
>> It probably could/should be updated and/or clarified at this point,
>> especially after v8 comes out.
>>
>>
> do you mind if we pull it from there, update it and place it on rsyslog.com?
> 
> Rainer

No, not at all, go ahead!

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] more documentation stuff was Re: name games...

[Micah Yoder]

On 1/30/14 10:50 AM, David Lang wrote:
> I think the biggest weakness right now is the documentation and it's
> organization.
> 
> What's on your coding todo list? While I suspect that there are things
> on it that are very important, I also suspect that there are going to be
> other things that may be better off pushed down below additional
> documentation.
> 
> We need to get v8 stable and keep fixing bugs, but when it comes to
> adding features, it is probably appropriate to slow down a bit and do
> some documentation first. This may actually help getting external
> developers to contribute more ;-)

As someone who has dug into rsyslog over the last few months for a big
deployment here, I definitely agree with this.  I find a lot of the
documentation quite confusing.

I may be able/willing to contribute some to this. After I understand
something I think I'm able to document it pretty well. (I'm the one who
did the elasticsearch doc.)

I might also be willing to get into the code, if I find something we
need that it can not yet do.  I wouldn't call myself a guru C programmer
but can dig around it all right and if there's someone I can question I
should be able to get something done.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] templates using local variables as a property

[Micah Yoder]

So is something like this possible? I am writing a template with a few
parts - a base directory name, the hostname of the originating server,
and then the log file name.  After defining the template, some
conditions for the different files.  I just want a different file based
on different conditions, without redefining the template.  So, something
like:

template(name="tpltest" type="list") {
constant(value="/var/log/")
property(name="hostname")
constant(value="/")
property(name="$.filename")
}

if $programname == "micah" and $syslogfacility-text == 'local1' then {
set $.filename = "loggerstuff"
action(type="omfile" Template="tpltest" DirOwner="root"
DirGroup="root" FileOwner="root" FileGroup="root" DirCreateMode="0755"
FileCreateMode="0644")
}

if $programname == "puppet-agent" and $syslogfacility-text ==
'local1' then {
set $.filename = "puppet"
action(type="omfile" Template="tpltest" DirOwner="root"
DirGroup="root" FileOwner="root" FileGroup="root" DirCreateMode="0755"
FileCreateMode="0644")
}


It doesn't seem to be parsing in 7.4.4 which I have installed, but I
might be doing something wrong.  If it isn't possible in this version,
is it in 7.5.x or 8.1.x?

Thanks much!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] templates using local variables as a property

[Micah Yoder]

On 2/11/14 1:16 AM, Rainer Gerhards wrote:
>> This is exactly the type of thing that local variables were designed for.
>> But just to check, try doing $! instead of $. the variable will show up as
>> part of a JSON output of $! if you do this, but it will let you test if
>> this is the problem
>>
>>
> local vars are not in 7.4., but I think the actual problem is a different
> one. In the config is:
> 
> set $.filename = "loggerstuff"
> 
> but it must be
> 
> set $.filename = "loggerstuff";
> 
> Note the semicolon at the end! It's a quirck, but we couldn't get away
> without adding the need for it.

Yay got it

In the template...
 property(name="$!test")

if whatever {
  set $!test = "whatever";
  action(omfile... Dynafile... etc)
}

Seems to work!  Thanks.

Local variables will be better than the $! JSON tree because I will
actually use JSON in this (a different part of the config) also.

Would it be recommended to use the latest 7.5 in production at this
point?  It will be stable "soon", correct?  Fortunately, we're not fully
relying on this setup yet, but we want to be ASAP.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Features Available for Sponsorship 404

[Micah Yoder]

Page not found http://www.rsyslog.com/article312/
Link from http://www.rsyslog.com/sponsors/
Thought you might want to know!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] umask in RPM init script

[Micah Yoder]

Hi,

The init script that comes in the adiscon rsyslog RPM sets the umask to
077 before starting rsyslog.

Thus, when I create files with omfile using FileCreateMode and
DirCreateMode, it will restrict that further.  That was bugging me!  I
want to create log files readable by users (with permissions carefully
controlled).

Any chance of either getting that umask line removed from the init
script, or at least allowing it to be overridden with
/etc/sysconfig/rsyslog?

Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] umask in RPM init script

[Micah Yoder]

Submitted pull request. Comments? Who's in charge of this repo? Thanks!

https://github.com/rsyslog/rsyslog-pkg-rhel-centos/pull/5/files

On 3/3/14 4:58 PM, Micah Yoder wrote:
> Hi,
> 
> The init script that comes in the adiscon rsyslog RPM sets the umask to
> 077 before starting rsyslog.
> 
> Thus, when I create files with omfile using FileCreateMode and
> DirCreateMode, it will restrict that further.  That was bugging me!  I
> want to create log files readable by users (with permissions carefully
> controlled).
> 
> Any chance of either getting that umask line removed from the init
> script, or at least allowing it to be overridden with
> /etc/sysconfig/rsyslog?
> 
> Thanks!
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
> 

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] umask in RPM init script

[Micah Yoder]

On 3/4/14 1:51 PM, Rainer Gerhards wrote:
> Also, I have concerns if this is the right path to take, but I will
comment
> on the PR.

Thanks. As noted in the PR, you're right.  For the list:

$umask 0022

in rsyslog.conf works just fine for me.  I had thought that umask(2)
could only make a more restrictive umask, not a looser one, but
apparently I was mistaken!

Thanks.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] epoll error on TLS config

[Micah Yoder]

Hi,

So I'm getting this on rsyslog startup (8.2.1 on CentOS 6.5):

2014-05-12T18:09:42.454656+00:00 micahsyslog2 rsyslogd-2163: epoll_ctl
failed on fd -185298336, id 1/0x7f849285c520, op 1 with Bad file
descriptor : Bad file descriptor [try http://www.rsyslog.com/e/2163 ]

According to that URL, this should be reported as it is an abnormal error.

It only appears with this in my rsyslog.conf:

# Provides TLS-encrypted TCP syslog reception
module(load="imtcp" StreamDriver.name="gtls" MaxSessions="500")
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog.d/server.crt
$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/server.key
input(type="imtcp" port="1054")

Anything obviously wrong here?  I *think* I generated the certs correctly.

It can not receive TLS messages.

Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] epoll error on TLS config

[Micah Yoder]

On 5/12/14, 2:06 PM, Nick Syslog wrote:>
> In my case I was running on a VM and 

That was on a VM (running in OpenStack) but I just tried on a physcal
server and got the same error.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] epoll error on TLS config

[Micah Yoder]

Anything more on this?

Is anyone using TCP+TLS on 8.2.1?  If so, how does your config differ
from this?

# Provides TLS-encrypted TCP syslog reception
module(load="imtcp" StreamDriver.name="gtls" MaxSessions="500")
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog.d/server.crt
$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/server.key
input(type="imtcp" port="1054")

BTW all the TLS examples online are fairly old, using the old syntax.
Having a reference using RainerScript would be pretty useful!

Also, is it completely necessary to use your own CA, and generating
client certificates? I think ideally we would like to just purchase
"normal" certs from an external CA and not use client certs.

On 5/13/14, 11:05 AM, Micah Yoder wrote:
> Sure, here it is, thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] epoll error on TLS config

[Micah Yoder]

yes I can try a special tarball. Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] development info

[Micah Yoder]

Hi again,

Just wondering if there is any documentation for getting started to
hacking on the rsyslog code - structure descriptions, etc.  There's some
in the code comments, but maybe a higher level description of key parts
would be nice.

Also, I'm having a hard time even finding the main() entry point ... :/

Not sure how much I'll do of this, but I'd like to have a better
understanding of how everything works, and be able to dig more into
issues like the TLS epoll thing I brought up a couple weeks ago.

(I'm conversant in C but wouldn't call myself an expert... prefer C++ :) )
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] development info

[Micah Yoder]

[...]
Thanks, that's a useful start!
My one question right now is where the main() entry point is. Not having
too much luck with grep or eyeballing the code.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] epoll error on TLS config

[Micah Yoder]

Rainer and anyone else who might be following this:
I just built the rsyslog head from git, and the epoll error does NOT
show up.

Still not getting messages over TLS, but that's another issue... still
looking into it.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] TLS still not receiving messages

[Micah Yoder]

Hi,

I posted this earlier.  I noted that I had got a weird epoll error with
the 8.2.1 RPM and that I had not received it with the git master branch,
but that messages were still not coming through.

I just refreshed the git tree and switched to v8-stable. No epoll error
but I'm not getting anything through.

I *think* my config is right.

Server:

# Provides TLS-encrypted TCP syslog reception
module(load="imtcp" StreamDriver.name="gtls" MaxSessions="500")
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog.d/server.crt
$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/server.key
input(type="imtcp" port="1054")

Client:

$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog.d/client1.crt
$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/client1.crt

action(type="omfwd"
   Target=""
   Port="1054"
   Protocol="tcp"
   StreamDriverAuthMode="x509/name"
   StreamDriverMode="1"
  )

tcpdump does show packets getting to the server, but of course I can't
tell what's in them.

Anything obviously wrong with the config?

Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] TLS still not receiving messages

[Micah Yoder]

Thanks for the response. Kind of gets weirder.  First, I added the line
as you suggested to the server config; no difference.  I tried s_client
and I *do* get some stuff through to the server log, but it is
gibberish.  Still nothing from rsyslog client itself.  s_client does
show just CONNECT with nothing else, even when I specify the client
cert/key that is given to rsyslog.  Both that and the server cert were
generated from the same CA, though I wonder if I messed something up
somewhere with that.

On 6/27/14, 10:39 AM, Masuda, Bond wrote:
> Micah:
> 
> First, to test TLS connection and verify you've got it setup correctly, use 
> the "s_client" in openssl to connect to your TLS port. If it is working as 
> expected, you should see your certificate on the screen; if not, you'll just 
> see "CONNECTED" and not much else. If it isn't working as expected, then 
> troubleshoot further.
> 
> Looking over my own configurations, I recall that the only way I got it to 
> work was to still use the old configuration style, and include the following 
> statements:
> 
> $DefaultNetstreamDriver gtls
> 
> This was even though I had this for imtcp:
> 
> module( load="imtcp"
> MaxSessions="1000"
> StreamDriver.Name="gtls"
> StreamDriver.Mode="1"
> StreamDriver.AuthMode="anon")
> 
> Try including the $DefaultNetstreamerDriver statement and see if that works 
> (after you verify with openssl s_client that it isn't).
> 
> -Bond
> 
> --- you wrote: ---
> Hi,
> 
> I posted this earlier.  I noted that I had got a weird epoll error with
> the 8.2.1 RPM and that I had not received it with the git master branch,
> but that messages were still not coming through.
> 
> I just refreshed the git tree and switched to v8-stable. No epoll error
> but I'm not getting anything through.
> 
> I *think* my config is right.
> 
> Server:
> 
> # Provides TLS-encrypted TCP syslog reception
> module(load="imtcp" StreamDriver.name="gtls" MaxSessions="500")
> $DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.pem
> $DefaultNetstreamDriverCertFile /etc/rsyslog.d/server.crt
> $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/server.key
> input(type="imtcp" port="1054")
> 
> Client:
> 
> $DefaultNetstreamDriver gtls
> $DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.pem
> $DefaultNetstreamDriverCertFile /etc/rsyslog.d/client1.crt
> $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/client1.crt
> 
> action(type="omfwd"
>Target=""
>Port="1054"
>Protocol="tcp"
>StreamDriverAuthMode="x509/name"
>StreamDriverMode="1"
>   )
> 
> tcpdump does show packets getting to the server, but of course I can't
> tell what's in them.
> 
> Anything obviously wrong with the config?
> 
> Thanks!
> 
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
> 

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog memory leaks

[Micah Yoder]

Hi,

We're seeing signifcant memory leakage in rsyslogd in our production
environment.  Verified with 8.2.1 and after the upgrade to 8.2.2.

To give you an idea, rsyslogd now uses nearly 40GB of RSS memory with
about 20 hours of uptime.

This is using the package from the RPM repo.

Unfortunately I'm not sure how much detail I can go into on our
configuration on the public list.  It's moderately complex, but I'm sure
there are more complex systems around.

I ran some impstats output through the online stats analyzer and got a
lot of this:

- 33% File Requests were evicted ('10885927' requests, '3594792'
evicted) between line 253 and 255 (Startdate: '2014/Jul/22 07:16:42'
Enddate: '2014/Jul/22 07:16:42')
- 7% File Requests were evicted ('1982155' requests, '141794' evicted)
between line 255 and 301 (Startdate: '2014/Jul/22 07:16:42' Enddate:
'2014/Jul/22 07:17:42')
- 32% File Requests were evicted ('10907196' requests, '3598535'
evicted) between line 303 and 305 (Startdate: '2014/Jul/22 07:17:42'
Enddate: '2014/Jul/22 07:17:42')

This appears on dynafile lines - could this have something to do with it?

Has anyone else seen significant memory leakage?

I/O wait is pretty much zero, as is user and system CPU usage.  The
system is virtually idle, with very low load, even for quite a high
traffic of logs coming in.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog memory leaks

[Micah Yoder]

On 7/22/14, 3:21 PM, David Lang wrote:

> Well, that indicates that you have a problem with your dynafile
> configuration, you don't allow enough files to be open, lookup
> dynafilecachesize and increase it to be much larger
> 
> With this sort of problem, your performance is going to be horrid, and
> as such, your queues will grow. Not seeing your impstats output, but
> I'll bet the size of the queues is large, and if you mutiply those sizes
> by the maxmessagesize, you will probably end up accounting for a lot of
> the memory that you think is 'leaking'

Thanks, I will check that.  The "main queue" has size=2, but this one is
rather different:

Tue Jul 22 09:57:51 2014: action 16 queue: size=438266 enqueued=438268
full=0 discarded.full=0 discarded.nf=0 maxqsize=438266

Any idea how one goes about finding which action line this corresponds to?

Also, if this accounted for the majority of the memory, each item would
have to consume 5-10k. Does that sound right?  (The vast majority of our
log lines are much less than that, but it's possible that something
sends a stream of logs that large.)



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog memory leaks

[Micah Yoder]

Update: Got the increased dynafile queue size into prod yesterday.
Memory leak still exists. However, that action 16, which has the large
size queue, seems to be (if I'm counting right) going to an unreachable
elasticsearch server. Looks like that may be the issue  :/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog memory leaks

[Micah Yoder]

Action disabled and that appears to have solved it.  I also agree that
this means it isn't a leak, but it did look like one at first!

Sorry for any confusion.

On 7/24/14, 10:47 AM, David Lang wrote:
> disable that action and see if it solves your memory usage.
> 
> for what it's worth, the term "memory leak" doesn't mean any time a
> program starts using lots of memory, it's when the program allocates
> memory for something and looses track of it so it can never free it.
> 
> In this case, if it is the queue for this action, this is not a leak,
> it's rsyslog doing exactly what you told it to and keeping track of
> these messages. If the server you are trying to deliver the messages to
> comes online, all that memory would end up being freed as the messages
> were delivered, showing that it's not a leak.
> 
> David Lang
> 
>  On Thu, 24 Jul 2014, Micah Yoder wrote:
> 
>> Update: Got the increased dynafile queue size into prod yesterday.
>> Memory leak still exists. However, that action 16, which has the large
>> size queue, seems to be (if I'm counting right) going to an unreachable
>> elasticsearch server. Looks like that may be the issue  :/
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] forwarding to central server help

[Micah Yoder]

What we do is have separate templates for different types of logs (auth,
"normal", cron, etc) and check the $syslogfacility-text and use a
different template accordingly.

On 8/12/14, 11:26 AM, Kevin McGillicuddy wrote:
> So for my network I have 3 servers and want to forward all logs to one sevrer
> 
> On the client servers I have everything setup good
> *.*  
> :omrelp:ipaddress:20514
> 
> I can see logs from that server coming into my central server, the issue is 
> that I want to break certain logs into certain files on the central server - 
> so on client 1 I would want my rootsh logs and my secure logs into 2 separate 
> files
> 
> Here is my central server configuration
> 
> # provides UDP syslog reception
> $ModLoad imudp
> $UDPServerRun 514
> 
> # provides TCP syslog reception
> $ModLoad imtcp
> $InputTCPServerRun 10514
> 
> # provides RELP syslog reception
> $ModLoad imrelp
> $InputRELPServerRun 20514
> 
> $template root_perhoste,"/var/log/hosts/%HOSTNAME%/rootsh.log"
> $template syslog_perhost,"/var/log/hosts/%HOSTNAME%/syslog.log"
> 
> rootsh.log ?root_perhost
> secure ?syslog_perhost
> 
> 
> I also have the last part
> (rootsh.log ?root_perhost
> secure ?syslog_perhost)
> 
> Repeated in this file /etc/rsyslog.d/50-default-.conf --because I am running 
> central server on Ubuntu 14.04
> 
> So all the logs come over fine and I can see them all - but they all get 
> dumped into syslog.log - so when I ssh into the client server I see in the 
> syslog.log on the central server that an ssh connection was open but when I 
> switch user to root and run commands as root or have any other logs they also 
> show up in syslog.log and nothing ever logs to the rootsh.log - however I 
> know the logs are coming to the central server that are meant for that file 
> because they show up in syslog.log
> 
> Also rootsh.log is not a standard log file
> 
> Any thoughts?
> 
> 
> Thanks,
> Kevin McGillicuddy
> Server Administrator
> Sight & Sound Theatres
> 717-687-4220 x2317
> kevin.mcgillicu...@sight-sound.com
> 
> [http://www.sight-sound.com/StaticContent/images/signature.gif]
> [http://www.sight-sound.com/StaticContent/images/youtube.gif][http://www.sight-sound.com/StaticContent/images/facebook.gif]
> 
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
> 

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslogd: gnutls returned error on handshake: A TLS packet with unexpected length was received.

[Micah Yoder]

We have the same log messages and are using it in the same way.  We have 
a story in the backlog to investigate, but haven't got to it yet.  It 
doesn't seem to be losing messages.


On 12/13/2016 03:31 PM, yingchun cai via rsyslog wrote:


Hi,  All
I use rsyslog-gnutls-8.23.0-1.el6.x86_64rsyslog-8.23.0-1.el6.x86_64and configured with 
TLS connection. I configured with imptcp as non-tls connection for one port and configure 
imtcp as tls connection on another port.  However,  I do get this error message in the 
log: "rsyslogd: gnutls returned error on handshake: A TLS packet with unexpected 
length was received.  [v8.23.0 try http://www.rsyslog.com/e/2083 ]".
Here is the configuration I have for rsyslog:module(load="imudp")input(type="imudp" 
port="514")module(load="imptcp")input(type="imptcp" port="514")
# Provides TCP syslog reception# for parameters see 
http://www.rsyslog.com/doc/imtcp.htmlmodule(load="imtcp"streamdriver.mode="1"streamdriver.authmode="x509/name"PermittedPeer="*")input(type="imtcp"
 port="2514" name="tcp-tls")
$DefaultNetstreamDriver gtls$DefaultNetstreamDriverCAFile 
/opt/sec/certs/$DefaultNetstreamDriverCertFile 
/opt/sec/certs/app_cert.pem$DefaultNetstreamDriverKeyFile 
/opt/sec/keys/app.key$ActionSendStreamDriverAuthMode 
x509/name$ActionSendStreamDriverPermittedPeer *
Anyone has any idea why I got this error?
thanks
Yingchun
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Logstash vs. omelasticsearch

[Micah Yoder]

On 11/21/2016 05:21 PM, David Lang wrote:

On Mon, 21 Nov 2016, Micah Yoder wrote:


The other reason I preferred Logstash was the configuration format was
a bit more user-friendly than some of the equivalent rsyslog rules.


can you provide some more info about the issues you had?


Hi David, sorry I was going to reply but didn't right away and got behind!

Actually it's been over a year and I don't remember all the specifics. 
Part of it (most of it probably) were segfaults with mmnormalize and/or 
mmjsonparse.  And they've probably been fixed by now.  But since we want 
maximum stability for our other log messages, we knew then that we 
wanted to separate this out from our main rsyslog process.  The 
alternatives would have been a secondary rsyslog process or logstash.  I 
just liked the way logstash config file works a bit more than how you 
set up rsyslog for this sort of thing.  There were some performance 
concerns, but logstash is keeping up fine and server load is low.


Would I switch back to rsyslog for this processing?  In this particular 
application probably not, because we don't really want to touch it 
again! :p  Would I consider rsyslog in the future for something similar? 
 Probably.  Looks like it's come a long way.  Especially with the ERK 
conversations.  I like what I'm seeing.  Main things are great 
documentation and easy to read config files.  Progress could be made on 
both


Maybe I could jump in on some of the documentation at some point.  I 
once wrote an rsyslog+elasticsearch tutorial that got reposted a couple 
places (Rackspace dev blog and Puppet blog).  It's ancient now though.


I might consider jumping in the code if it were written in modern C++ 
instead of C.  I'm a bit baffled why C is still used, but that will 
probably get me flamed to a crisp here! :p



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.