Re: [rsyslog] TCP Stops Local Logging

2017-03-24 Thread David Lang 

On Fri, 24 Mar 2017, Chris wrote:


On RH 6 systems running rsyslog 5.8.10 we noticed that if we setup a
client system to use TCP to log to a remote server:
*.*   @@192.168.1.2

If the remote log server is not reachable for some reason no logging
takes place, not even local logging to the local system log files.
When the log server is available and rsyslog is restarted  both local
logging and remote logging work.  Is this a known issue or is there some
way to ensure that local logging still occurs when  the TCP remote
server is down?


This is working as designed (for the config you specified), if a message cannot 
be delivered to one destination, and you don't have rsyslog configured to throw 
it away, it is not able to finish processing that log message and start work on 
the next one.


You can create an action queue for the delivery to a remote system, and until 
that queue fills up, other log processing will continue.


You really should move to at least v7, if not v8, a lot of things have changed, 
especially the available syntax for specifying queues.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] TCP Stops Local Logging

2017-03-24 Thread Chris 
On RH 6 systems running rsyslog 5.8.10 we noticed that if we setup a
client system to use TCP to log to a remote server:
*.*   @@192.168.1.2

If the remote log server is not reachable for some reason no logging
takes place, not even local logging to the local system log files.  
When the log server is available and rsyslog is restarted  both local
logging and remote logging work.  Is this a known issue or is there some
way to ensure that local logging still occurs when  the TCP remote
server is down?
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog with queue files won't start

2017-03-24 Thread David Lang 
There is the tool that recreates the queue index, you can use that (I'm blanking 
on the name, something.qi)


On Fri, 24 Mar 2017, Peter Viskup wrote:


Not sure whether we will be able to upgrade in near future.
It would be great to be possible to check the queue file consistency
(at least). Is there any description of the format to check against?
Or any tool available out there?

On Thu, Mar 23, 2017 at 8:43 AM, Rainer Gerhards
 wrote:

You should move up to 8.25, chances are very good the issue is fixed
there. If not, we should see a better diagnostic (8.26 will have even
better diagnostics).

Rainer

2017-03-23 8:40 GMT+01:00 Peter Viskup via rsyslog :

Just experienced the same issue again.
Rsyslog version 8.15.0 crashed after the FS for queues got full.

Mar 23 08:29:49 server kernel: [23586716.165832] rs:STRMEP2
queu[31413]: segfault at 1 ip 7f90dc66b34c sp 7f90d99daa40
error 4 in rsyslogd[7f90dc645000+9]

Recovered most of the queue by removal of the oldest queue file.
Removal of the queue file from the crash (the newest one) didn't help.

--
Peter

On Fri, Feb 17, 2017 at 9:02 AM, Peter Viskup  wrote:

Sorry for not to mentioned.
The qi file was rebuilt. Once I moved the files out, the rsyslog
started just fine. I have a copy of the whole queue, thus can be used
for further investigation.
Are we able to check the consistency of queue files somehow?

--
Peter

On Thu, Feb 16, 2017 at 3:44 PM, David Lang  wrote:

odds are that the queue files have been corrupted. you need to rebuild the
.qi file and then it should be able to startup.

David Lang

On Thu, 16 Feb 2017, Peter Viskup via rsyslog wrote:


Date: Thu, 16 Feb 2017 13:50:37 +0100
From: Peter Viskup via rsyslog 
To: rsyslog-users 
Cc: Peter Viskup 
Subject: [rsyslog] Rsyslog with queue files won't start


Just experienced issue with rsyslog with DA queue files.
The process just died without any error.
We do run rsyslog version 8.15.0
These are the last lines from debug output

8488.225603049:STRMEP2 queue[DA]:Reg/w0: omfwd: beginTransaction
8488.225606660:STRMEP2 queue[DA]:Reg/w0:  10.1.25.181
8488.225610889:STRMEP2 queue[DA]:Reg/w0: Action 4 transitioned to state:
itx
8488.225614575:STRMEP2 queue[DA]:Reg/w0: Action 4 transitioned to state:
rdy
8488.225618011:STRMEP2 queue[DA]:Reg/w0: actionCommit, in retry loop, iRet
0
8488.225622069:STRMEP2 queue[DA]:Reg/w0: regular consumer finished,
iret=0, szlog 128708 sz phys 128716
8488.225631031:STRMEP2 queue[DA]:Reg/w0: DeleteProcessedBatch: we
deleted 8 objects and enqueued 0 objects
8488.225635210:STRMEP2 queue[DA]:Reg/w0: doDeleteBatch: delete batch
from store, new sizes: log 128708, phys 128708
8488.225672091:STRMEP2 queue[DA]:Reg/w0: strm 0x7f0761c13b40: file 11
read 4096 bytes
8488.225760644:STRMEP2 queue[DA]:Reg/w0: strm 0x7f0761c13b40: file 11
read 4096 bytes
8488.225864111:STRMEP2 queue[DA]:Reg/w0: strm 0x7f0761c13b40: file 11
read 4096 bytes
8488.225966025:STRMEP2 queue[DA]:Reg/w0: strm 0x7f0761c13b40: file 11
read 4096 bytes
8488.225977652:STRMEP2 queue[DA]:Reg/w0: error -2059 deserializing
property name, offset 208904, step 2
8488.225984716:STRMEP2 queue[DA]:Reg/w0: error property name: 'msg

Re: [rsyslog] $DirCreateMode ignored?

2017-03-24 Thread Rainer Gerhards 
2017-03-24 14:40 GMT+01:00 Tomasz Chmielewski via rsyslog
:
> I'm running rsyslog 8.16.0-1ubuntu3 on Ubuntu 16.04.
>
> I have $DirCreateMode set to 0755:
>
> root@log01:/# grep -i dircre /etc/rsyslog.conf
> $DirCreateMode 0755
>
> root@log01:/# grep -i dircre /etc/rsyslog.d/*
> /etc/rsyslog.d/05-accept-remote.conf:$DirCreateMode 0755
>
>
> Unfortunately, directories automatically created (for data received from
> remote hosts) by rsyslogd are still 0700.
>
> Did I use "$DirCreateMode 0755" in a wrong way? If not - how do I debug
> this?

Old-Style directives ($DirCreateMode) by design only affect old-style
constructs (/path/to/file). So the action() statment needs to have the
mode setting.

>
>
> If it matters - my template for accepting remote data is as below:
>
> template (name="RemoteMessage" type="string" string="%msg:2:$%\n")
> template (name="mypath" type="string"
> string="/var/log/remote/%$year%-%$month%-%$day%/%hostname%/%programname%")
> if ($inputname == 'imtcp' or $inputname == 'imudp') then {
> action(type="omfile" dynafile="mypath" template="RemoteMessage")
> stop
> }
>
>
> Also tried adding dirCreateMode="0755" to the template, but it didn't help
> and directories are still created with 0700 permissions:
>
> template (name="RemoteMessage" type="string" string="%msg:2:$%\n")
> template (name="mypath" type="string" dirCreateMode="0755"
> string="/var/log/remote/%$year%-%$month%-%$day%/%hostname%/%programname%")

You should also get error messages from rsyslog, because this is
nothing you can do inside a template. A template just creates a
string, and that's it.

So you need to apply the setting to the action in question...
> if ($inputname == 'imtcp' or $inputname == 'imudp') then {
> action(type="omfile" dynafile="mypath" template="RemoteMessage")

 action(type="omfile" dynafile="mypath" template="RemoteMessage"
  dirCreateMode="0755")

> stop
> }
>

see the example in the doc:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfile.html

HTH
Rainer
>
> Tomasz Chmielewski
> https://lxadm.com
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] $DirCreateMode ignored?

2017-03-24 Thread Tomasz Chmielewski via rsyslog 

I'm running rsyslog 8.16.0-1ubuntu3 on Ubuntu 16.04.

I have $DirCreateMode set to 0755:

root@log01:/# grep -i dircre /etc/rsyslog.conf
$DirCreateMode 0755

root@log01:/# grep -i dircre /etc/rsyslog.d/*
/etc/rsyslog.d/05-accept-remote.conf:$DirCreateMode 0755


Unfortunately, directories automatically created (for data received from 
remote hosts) by rsyslogd are still 0700.


Did I use "$DirCreateMode 0755" in a wrong way? If not - how do I debug 
this?



If it matters - my template for accepting remote data is as below:

template (name="RemoteMessage" type="string" string="%msg:2:$%\n")
template (name="mypath" type="string" 
string="/var/log/remote/%$year%-%$month%-%$day%/%hostname%/%programname%")

if ($inputname == 'imtcp' or $inputname == 'imudp') then {
action(type="omfile" dynafile="mypath" template="RemoteMessage")
stop
}


Also tried adding dirCreateMode="0755" to the template, but it didn't 
help and directories are still created with 0700 permissions:


template (name="RemoteMessage" type="string" string="%msg:2:$%\n")
template (name="mypath" type="string" dirCreateMode="0755" 
string="/var/log/remote/%$year%-%$month%-%$day%/%hostname%/%programname%")

if ($inputname == 'imtcp' or $inputname == 'imudp') then {
action(type="omfile" dynafile="mypath" template="RemoteMessage")
stop
}


Tomasz Chmielewski
https://lxadm.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] REQUEST FOR COMMENTS: rsyslog code style

2017-03-24 Thread Rainer Gerhards 
Hi all,

I have finally made a shot at specifying a standardized code style for
the rsyslog projects. It took quite a while, as I wanted to suggest
something

a) that is pretty common,
b) where current code can be mostly automatic adjusted,
c) can be automatically enforced in the future.

I have finally found a way and am now looking for feedback. If there
is no objection, I'll start to apply by April, 6th (or later). Note
that once we made this big change, I will not accept style changes
except of extremely important reasons.

Note that I have not just taken my personal preference and turned that
into a proposal. While there is a taste of it, I have also required
some styles that I usually do not use.

Style, samples, more info can be found in this PR:

https://github.com/rsyslog/rsyslog/pull/1479

I would also appreciate if comments go there (and not the mailing
list!) as I would like to keep this as single source of the decision
process.

Thanks,
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] buildbot PR failures

2017-03-24 Thread Rainer Gerhards 
Hi all,

note that we are currently working on integrating back Solaris support
into the build environment. For this reason, all PRs currently fail
with the 4 Solaris tests. Please don't be concerned about this, I'll
handle the situation. Typically a merge is without problems if just
the Solaris ones fail.

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm 2.0.3 released

2017-03-24 Thread Brian Knox via rsyslog 
The load rulebase from a string is nice! Will add that to my wrapper I'm
using in normz ( https://github.com/taotetek/normz ).

On Thu, Mar 23, 2017 at 1:04 PM Florian Riedl  wrote:

> Hi all,
>
> We have just released liblognorm 2.0.3. This new version provides some
> fixes for the the annotate function and adds a test for it. A few
> different issues have also been fixed. See the Changelog for details.
>
> Changelog:Version 2.0.3, 2017-03-22
>
> - add ability to load rulebase from a string
> introduces new API:
> int ln_loadSamplesFromString(ln_ctx ctx, const char *string);
> closes https://github.com/rsyslog/liblognorm/issues/239
> - bugfix: string parser did not correctly parse word at end of line
> - bugfix: literal parser does not always store value if name is specified
> if
> rule=:%{"type":"literal", "text":"a", "name":"var"}%
> is used and matching message is provided, variable var ist not persisted.
> see also
> http://lists.adiscon.net/pipermail/rsyslog/2016-December/043985.html
>
> Download:
> http://www.liblognorm.com/download/liblognorm-2-0-3/
>
> As always, feedback is appreciated.
>
> Best regards,
> Florian Riedl
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog with queue files won't start

2017-03-24 Thread Peter Viskup via rsyslog 
Not sure whether we will be able to upgrade in near future.
It would be great to be possible to check the queue file consistency
(at least). Is there any description of the format to check against?
Or any tool available out there?

On Thu, Mar 23, 2017 at 8:43 AM, Rainer Gerhards
 wrote:
> You should move up to 8.25, chances are very good the issue is fixed
> there. If not, we should see a better diagnostic (8.26 will have even
> better diagnostics).
>
> Rainer
>
> 2017-03-23 8:40 GMT+01:00 Peter Viskup via rsyslog 
> :
>> Just experienced the same issue again.
>> Rsyslog version 8.15.0 crashed after the FS for queues got full.
>>
>> Mar 23 08:29:49 server kernel: [23586716.165832] rs:STRMEP2
>> queu[31413]: segfault at 1 ip 7f90dc66b34c sp 7f90d99daa40
>> error 4 in rsyslogd[7f90dc645000+9]
>>
>> Recovered most of the queue by removal of the oldest queue file.
>> Removal of the queue file from the crash (the newest one) didn't help.
>>
>> --
>> Peter
>>
>> On Fri, Feb 17, 2017 at 9:02 AM, Peter Viskup  wrote:
>>> Sorry for not to mentioned.
>>> The qi file was rebuilt. Once I moved the files out, the rsyslog
>>> started just fine. I have a copy of the whole queue, thus can be used
>>> for further investigation.
>>> Are we able to check the consistency of queue files somehow?
>>>
>>> --
>>> Peter
>>>
>>> On Thu, Feb 16, 2017 at 3:44 PM, David Lang  wrote:
 odds are that the queue files have been corrupted. you need to rebuild the
 .qi file and then it should be able to startup.

 David Lang

 On Thu, 16 Feb 2017, Peter Viskup via rsyslog wrote:

> Date: Thu, 16 Feb 2017 13:50:37 +0100
> From: Peter Viskup via rsyslog 
> To: rsyslog-users 
> Cc: Peter Viskup 
> Subject: [rsyslog] Rsyslog with queue files won't start
>
>
> Just experienced issue with rsyslog with DA queue files.
> The process just died without any error.
> We do run rsyslog version 8.15.0
> These are the last lines from debug output
>
> 8488.225603049:STRMEP2 queue[DA]:Reg/w0: omfwd: beginTransaction
> 8488.225606660:STRMEP2 queue[DA]:Reg/w0:  10.1.25.181
> 8488.225610889:STRMEP2 queue[DA]:Reg/w0: Action 4 transitioned to state:
> itx
> 8488.225614575:STRMEP2 queue[DA]:Reg/w0: Action 4 transitioned to state:
> rdy
> 8488.225618011:STRMEP2 queue[DA]:Reg/w0: actionCommit, in retry loop, iRet
> 0
> 8488.225622069:STRMEP2 queue[DA]:Reg/w0: regular consumer finished,
> iret=0, szlog 128708 sz phys 128716
> 8488.225631031:STRMEP2 queue[DA]:Reg/w0: DeleteProcessedBatch: we
> deleted 8 objects and enqueued 0 objects
> 8488.225635210:STRMEP2 queue[DA]:Reg/w0: doDeleteBatch: delete batch
> from store, new sizes: log 128708, phys 128708
> 8488.225672091:STRMEP2 queue[DA]:Reg/w0: strm 0x7f0761c13b40: file 11
> read 4096 bytes
> 8488.225760644:STRMEP2 queue[DA]:Reg/w0: strm 0x7f0761c13b40: file 11
> read 4096 bytes
> 8488.225864111:STRMEP2 queue[DA]:Reg/w0: strm 0x7f0761c13b40: file 11
> read 4096 bytes
> 8488.225966025:STRMEP2 queue[DA]:Reg/w0: strm 0x7f0761c13b40: file 11
> read 4096 bytes
> 8488.225977652:STRMEP2 queue[DA]:Reg/w0: error -2059 deserializing
> property name, offset 208904, step 2
> 8488.225984716:STRMEP2 queue[DA]:Reg/w0: error property name: 'msg 8488.225991530:STRMEP2 queue[DA]:Reg/w0: error var type: '1'
>
> What's going wrong here?
>
>

>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.