date:20160204

Re: [rsyslog] rsyslog running well on debug mode but not in service mode

2016-02-04 Thread David Lang


On Fri, 5 Feb 2016, chika tambun wrote:


Dear all,


Rsyslog won't send syslog to server on production mode, but it will do
transmit the syslog when i activated debug mode using 'rsyslogd -dn'
Yes, i did create the WorkDirectory
*[and]*
i can't find error message on the debug log because the server did received
the data transmitted according to my request. But when i switch back to non
debug mode 'service rsyslog restart' this message log appear & the server
wont receive anything


what happens if you run it from the command line without the -dn (as opposed to 
letting init start it)


I've seen this sort of thing before when SELinux permissions didn't allow you to 
access the workdir


do ls -lZ /var/spool /var/log

I'll bet that the SELinux tags on /var/spool/rsyslog aren't appropriate.

David Lang


# cat /var/log/messages
2016-02-05T10:51:25.952736+03:00 freeradius rsyslogd: [origin
software="rsyslogd" swVersion="8.15.0" x-pid="30558" x-info="
http://www.rsyslog.com";] exiting on signal 15.
2016-02-05T10:51:26.039020+03:00 freeradius rsyslogd: [origin
software="rsyslogd" swVersion="8.15.0" x-pid="30589" x-info="
http://www.rsyslog.com";] start
2016-02-05T10:51:26.039958+03:00 freeradius rsyslogd-2007: action 'action
9' suspended, next retry is Fri Feb  5 10:51:56 2016 [v8.15.0 try
http://www.rsyslog.com/e/2007 ]
2016-02-05T10:51:26.040045+03:00 freeradius rsyslogd-2007: action 'action
10' suspended, next retry is Fri Feb  5 10:51:56 2016 [v8.15.0 try
http://www.rsyslog.com/e/2007 ]
2016-02-05T10:51:26.043847+03:00 freeradius rsyslogd-2007: action 'action
0' suspended, next retry is Fri Feb  5 10:51:56 2016 [v8.15.0 try
http://www.rsyslog.com/e/2007 ]
2016-02-05T10:51:26.043978+03:00 freeradius rsyslogd-2007: action 'action
1' suspended, next retry is Fri Feb  5 10:51:56 2016 [v8.15.0 try
http://www.rsyslog.com/e/2007 ]
2016-02-05T10:52:41.596501+03:00 freeradius rsyslogd-2027: imfile: could
not persist state file imfile-state:-var-log-radius-radacct-detail - data
may be repeated on next startup. Is WorkDirectory set? [v8.15.0 try
http://www.rsyslog.com/e/2027 ]
2016-02-05T10:52:41.596612+03:00 freeradius rsyslogd-2027: imfile: could
not persist state file imfile-state:-var-log-radius-radius.log - data may
be repeated on next startup. Is WorkDirectory set? [v8.15.0 try
http://www.rsyslog.com/e/2027 ]
2016-02-05T10:52:41.596690+03:00 freeradius rsyslogd-2027: imfile: could
not persist state file imfile-state:-var-log-radius-radacct-reply-detail -
data may be repeated on next startup. Is WorkDirectory set? [v8.15.0 try
http://www.rsyslog.com/e/2027 ]
2016-02-05T10:52:41.596736+03:00 freeradius rsyslogd-2027: imfile: could
not persist state file imfile-state:-var-log-radius-radacct-auth-detail -
data may be repeated on next startup. Is WorkDirectory set? [v8.15.0 try
http://www.rsyslog.com/e/2027 ]
2016-02-05T10:52:41.596750+03:00 freeradius rsyslogd: [origin
software="rsyslogd" swVersion="8.15.0" x-pid="30589" x-info="
http://www.rsyslog.com";] exiting on signal 15.
2016-02-05T10:52:54.441600+03:00 freeradius rsyslogd: [origin
software="rsyslogd" swVersion="8.15.0" x-pid="30824" x-info="
http://www.rsyslog.com";] start

# cat /etc/rsyslog.conf
module(load="imuxsock")
module(load="imklog")
module(load="omrelp")
module(load="imfile" PollingInterval="12")
$RepeatedMsgReduction on
$FileOwner root
$FileGroup root
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser root
$PrivDropToGroup root
$WorkDirectory /var/spool/rsyslog
$IncludeConfig /etc/rsyslog.d/*.conf

action(type="omrelp" target="10.10.10.5" port="15514")
action(type="omrelp" target="10.10.10.4" port="15514")

# rsyslogd -N 1
rsyslogd: version 8.15.0, config validation run (level 1), master config
/etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

# cat /etc/redhat-release
CentOS release 6.6 (Final)

# uname -a
Linux freeradius 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux

# rsyslogd -version
rsyslogd 8.15.0, compiled with:
   PLATFORM:   x86_64-redhat-linux-gnu
   PLATFORM (lsb_release -d):
   FEATURE_REGEXP: Yes
   GSSAPI Kerberos 5 support:  No
   FEATURE_DEBUG (debug build, slow code): No
   32bit Atomic operations supported:  Yes
   64bit Atomic operations supported:  Yes
   memory allocator:   system default
   Runtime Instrumentation (slow code):No
   uuid support:   Yes
   Number of Bits in RainerScript integers: 64

See http://www.rsyslog.com for more information.



*Best regards,*
*chika.tambun*

*"Winning loves preparation"*
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHI

[rsyslog] rsyslog running well on debug mode but not in service mode

2016-02-04 Thread chika tambun

Dear all,


Rsyslog won't send syslog to server on production mode, but it will do
transmit the syslog when i activated debug mode using 'rsyslogd -dn'
Yes, i did create the WorkDirectory
*[and]*
i can't find error message on the debug log because the server did received
the data transmitted according to my request. But when i switch back to non
debug mode 'service rsyslog restart' this message log appear & the server
wont receive anything

# cat /var/log/messages
2016-02-05T10:51:25.952736+03:00 freeradius rsyslogd: [origin
software="rsyslogd" swVersion="8.15.0" x-pid="30558" x-info="
http://www.rsyslog.com";] exiting on signal 15.
2016-02-05T10:51:26.039020+03:00 freeradius rsyslogd: [origin
software="rsyslogd" swVersion="8.15.0" x-pid="30589" x-info="
http://www.rsyslog.com";] start
2016-02-05T10:51:26.039958+03:00 freeradius rsyslogd-2007: action 'action
9' suspended, next retry is Fri Feb  5 10:51:56 2016 [v8.15.0 try
http://www.rsyslog.com/e/2007 ]
2016-02-05T10:51:26.040045+03:00 freeradius rsyslogd-2007: action 'action
10' suspended, next retry is Fri Feb  5 10:51:56 2016 [v8.15.0 try
http://www.rsyslog.com/e/2007 ]
2016-02-05T10:51:26.043847+03:00 freeradius rsyslogd-2007: action 'action
0' suspended, next retry is Fri Feb  5 10:51:56 2016 [v8.15.0 try
http://www.rsyslog.com/e/2007 ]
2016-02-05T10:51:26.043978+03:00 freeradius rsyslogd-2007: action 'action
1' suspended, next retry is Fri Feb  5 10:51:56 2016 [v8.15.0 try
http://www.rsyslog.com/e/2007 ]
2016-02-05T10:52:41.596501+03:00 freeradius rsyslogd-2027: imfile: could
not persist state file imfile-state:-var-log-radius-radacct-detail - data
may be repeated on next startup. Is WorkDirectory set? [v8.15.0 try
http://www.rsyslog.com/e/2027 ]
2016-02-05T10:52:41.596612+03:00 freeradius rsyslogd-2027: imfile: could
not persist state file imfile-state:-var-log-radius-radius.log - data may
be repeated on next startup. Is WorkDirectory set? [v8.15.0 try
http://www.rsyslog.com/e/2027 ]
2016-02-05T10:52:41.596690+03:00 freeradius rsyslogd-2027: imfile: could
not persist state file imfile-state:-var-log-radius-radacct-reply-detail -
data may be repeated on next startup. Is WorkDirectory set? [v8.15.0 try
http://www.rsyslog.com/e/2027 ]
2016-02-05T10:52:41.596736+03:00 freeradius rsyslogd-2027: imfile: could
not persist state file imfile-state:-var-log-radius-radacct-auth-detail -
data may be repeated on next startup. Is WorkDirectory set? [v8.15.0 try
http://www.rsyslog.com/e/2027 ]
2016-02-05T10:52:41.596750+03:00 freeradius rsyslogd: [origin
software="rsyslogd" swVersion="8.15.0" x-pid="30589" x-info="
http://www.rsyslog.com";] exiting on signal 15.
2016-02-05T10:52:54.441600+03:00 freeradius rsyslogd: [origin
software="rsyslogd" swVersion="8.15.0" x-pid="30824" x-info="
http://www.rsyslog.com";] start

# cat /etc/rsyslog.conf
module(load="imuxsock")
module(load="imklog")
module(load="omrelp")
module(load="imfile" PollingInterval="12")
$RepeatedMsgReduction on
$FileOwner root
$FileGroup root
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser root
$PrivDropToGroup root
$WorkDirectory /var/spool/rsyslog
$IncludeConfig /etc/rsyslog.d/*.conf

action(type="omrelp" target="10.10.10.5" port="15514")
action(type="omrelp" target="10.10.10.4" port="15514")

# rsyslogd -N 1
rsyslogd: version 8.15.0, config validation run (level 1), master config
/etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

# cat /etc/redhat-release
CentOS release 6.6 (Final)

# uname -a
Linux freeradius 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux

# rsyslogd -version
rsyslogd 8.15.0, compiled with:
PLATFORM:   x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support:  No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported:  Yes
64bit Atomic operations supported:  Yes
memory allocator:   system default
Runtime Instrumentation (slow code):No
uuid support:   Yes
Number of Bits in RainerScript integers: 64

See http://www.rsyslog.com for more information.



*Best regards,*
*chika.tambun*

*"Winning loves preparation"*
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Mapping log-level names to severity numbers

2016-02-04 Thread Bob Gregory

That's interesting - dropping the cee cookie gives my beleaguered devs one
less thing to complain about, and I'll probably use mmnormalize anyway for
processing things like nginx and haproxy logs into the journal.

I'll have a play tomorrow and see how it goes - thanks again David.

On 4 February 2016 at 19:11, David Lang  wrote:

> oops, libfastjson
>
> https://github.com/rsyslog/libfastjson
>
> http://blog.gerhards.net/2015/12/rsyslog-and-liblognorm-will-switch-to.html
>
> I think the hard requirement for libfastjson just hit in the last day or
> so. liblognorm will still use json-c (but will use libfastjson if it finds
> it), but as of last night, rsyslog master will not compile without
> libfastjson.
>
> with mmnormalize, you can now have the json data type, so you can have
>
> rule=:%something:word% %foo:json%%bar:word% to deal with json embedded in
> the middle of a log message. Given the performance attention that
> liblognorm has had in the last year, I would not be surprised to find this
> at least as fast as mmjsonparse. mmnormalize also lets you match the raw
> logs, so if you have something (like logstash grumble grumble) that really
> doesn't want to talk traditional syslog, but only json chunks, you can
> parse the raw logs with
>
> rule=:%.:json%
>
> and everything will show up and 'just work' :-)
>
> David Lang
>
>
> On Thu, 4 Feb 2016, Bob Gregory wrote:
>
>
>> 1. I wasn't aware that mmnormalize is now json capable. I might kick the
>> tyres on that, but the lookup works for me thus far.
>> 2. I'm currently building from master because I've got a PR open for
>> version 8.17
>> 3. IIRC master is currently building against json-c - is that true? In
>> either case, where do I find more info on libjsonfast? Google tells me
>> nothing.
>>
>> - B
>>
>> On 4 February 2016 at 17:02, David Lang  wrote:
>>
>> On Thu, 4 Feb 2016, Bob Gregory wrote:
>>>
>>> Hi Dave,
>>>

 It's the latter. Currently docker is just spraying logs out onto disk,
 in
 both plain text and json format, and there's no logrotate. Instead, we
 want
 just the json logs to go through rsyslog. We'll forward INFO level
 application logs to Elasticsearch via Redis, and put a human-readable
 version of logs into the journal.

 Marking the journal entries with the appropriate syslog severity makes
 it
 easy to query and filter.

 The lookup_table functionality actually works better than my proposed
 property replacer, because it's simple to modify the lookup if
 requirements
 evolve.


>>> a couple comments
>>>
>>> 1. using mmnormalize and the latest liblognorm (with the version=2
>>> ruleset), rsyslog can parse raw json, it doesn't need the @cee token any
>>> longer and can parse logs that are a mix of json and non-json data.
>>>
>>> 2. the table_lookup code that is in the released versions of rsyslog is
>>> very limited and has some known bugs. It was a prototype from work that
>>> was
>>> discussed and was going to be sponsored, but the company initiating the
>>> work fell through. Yesterday a full implementation was merged into the
>>> master tree for release in 8.17. You really will want to be using that
>>> version for anything beyond a proof of concept.
>>>
>>> 3. we have found some nasty bugs in the json-c library and as a result
>>> have forked it to libjsonfast, 8.16 will optionally use it if it's
>>> available, 8.17 will require it.
>>>
>>> and 8.17 (or a daily build version of it) will pull in the latest
>>> liblognorm and libjsonfast.
>>>
>>> This is one of those cases where you will really want to be on the very
>>> latest version.
>>>
>>>
>>> David Lang
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>>
>>
>>
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>



-- 



*Bob Gregory*

Application Architect

MADE.COM 

Skype: flinkywistypomm


[image: MADE]



Made.com Design Limited is a company registered in England and Wales.

Registered number: 07101408 | Registered office: 100 Charing Cross Road,
London WC2H 0HG
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog?

Re: [rsyslog] Mapping log-level names to severity numbers

2016-02-04 Thread David Lang


oops, libfastjson

https://github.com/rsyslog/libfastjson

http://blog.gerhards.net/2015/12/rsyslog-and-liblognorm-will-switch-to.html

I think the hard requirement for libfastjson just hit in the last day or so. 
liblognorm will still use json-c (but will use libfastjson if it finds it), but 
as of last night, rsyslog master will not compile without libfastjson.


with mmnormalize, you can now have the json data type, so you can have

rule=:%something:word% %foo:json%%bar:word% to deal with json embedded in the 
middle of a log message. Given the performance attention that liblognorm has had 
in the last year, I would not be surprised to find this at least as fast as 
mmjsonparse. mmnormalize also lets you match the raw logs, so if you have 
something (like logstash grumble grumble) that really doesn't want to talk 
traditional syslog, but only json chunks, you can parse the raw logs with


rule=:%.:json%

and everything will show up and 'just work' :-)

David Lang

On Thu, 4 Feb 2016, Bob Gregory wrote:



1. I wasn't aware that mmnormalize is now json capable. I might kick the
tyres on that, but the lookup works for me thus far.
2. I'm currently building from master because I've got a PR open for
version 8.17
3. IIRC master is currently building against json-c - is that true? In
either case, where do I find more info on libjsonfast? Google tells me
nothing.

- B

On 4 February 2016 at 17:02, David Lang  wrote:


On Thu, 4 Feb 2016, Bob Gregory wrote:

Hi Dave,


It's the latter. Currently docker is just spraying logs out onto disk, in
both plain text and json format, and there's no logrotate. Instead, we
want
just the json logs to go through rsyslog. We'll forward INFO level
application logs to Elasticsearch via Redis, and put a human-readable
version of logs into the journal.

Marking the journal entries with the appropriate syslog severity makes it
easy to query and filter.

The lookup_table functionality actually works better than my proposed
property replacer, because it's simple to modify the lookup if
requirements
evolve.



a couple comments

1. using mmnormalize and the latest liblognorm (with the version=2
ruleset), rsyslog can parse raw json, it doesn't need the @cee token any
longer and can parse logs that are a mix of json and non-json data.

2. the table_lookup code that is in the released versions of rsyslog is
very limited and has some known bugs. It was a prototype from work that was
discussed and was going to be sponsored, but the company initiating the
work fell through. Yesterday a full implementation was merged into the
master tree for release in 8.17. You really will want to be using that
version for anything beyond a proof of concept.

3. we have found some nasty bugs in the json-c library and as a result
have forked it to libjsonfast, 8.16 will optionally use it if it's
available, 8.17 will require it.

and 8.17 (or a daily build version of it) will pull in the latest
liblognorm and libjsonfast.

This is one of those cases where you will really want to be on the very
latest version.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.







___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Mapping log-level names to severity numbers

2016-02-04 Thread Bob Gregory

Thanks Dave,

1. I wasn't aware that mmnormalize is now json capable. I might kick the
tyres on that, but the lookup works for me thus far.
2. I'm currently building from master because I've got a PR open for
version 8.17
3. IIRC master is currently building against json-c - is that true? In
either case, where do I find more info on libjsonfast? Google tells me
nothing.

 - B

On 4 February 2016 at 17:02, David Lang  wrote:

> On Thu, 4 Feb 2016, Bob Gregory wrote:
>
> Hi Dave,
>>
>> It's the latter. Currently docker is just spraying logs out onto disk, in
>> both plain text and json format, and there's no logrotate. Instead, we
>> want
>> just the json logs to go through rsyslog. We'll forward INFO level
>> application logs to Elasticsearch via Redis, and put a human-readable
>> version of logs into the journal.
>>
>> Marking the journal entries with the appropriate syslog severity makes it
>> easy to query and filter.
>>
>> The lookup_table functionality actually works better than my proposed
>> property replacer, because it's simple to modify the lookup if
>> requirements
>> evolve.
>>
>
> a couple comments
>
> 1. using mmnormalize and the latest liblognorm (with the version=2
> ruleset), rsyslog can parse raw json, it doesn't need the @cee token any
> longer and can parse logs that are a mix of json and non-json data.
>
> 2. the table_lookup code that is in the released versions of rsyslog is
> very limited and has some known bugs. It was a prototype from work that was
> discussed and was going to be sponsored, but the company initiating the
> work fell through. Yesterday a full implementation was merged into the
> master tree for release in 8.17. You really will want to be using that
> version for anything beyond a proof of concept.
>
> 3. we have found some nasty bugs in the json-c library and as a result
> have forked it to libjsonfast, 8.16 will optionally use it if it's
> available, 8.17 will require it.
>
> and 8.17 (or a daily build version of it) will pull in the latest
> liblognorm and libjsonfast.
>
> This is one of those cases where you will really want to be on the very
> latest version.
>
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>



-- 



*Bob Gregory*

Application Architect

MADE.COM 

Skype: flinkywistypomm


[image: MADE]



Made.com Design Limited is a company registered in England and Wales.

Registered number: 07101408 | Registered office: 100 Charing Cross Road,
London WC2H 0HG
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Mapping log-level names to severity numbers

2016-02-04 Thread David Lang


On Thu, 4 Feb 2016, Bob Gregory wrote:


Hi Dave,

It's the latter. Currently docker is just spraying logs out onto disk, in
both plain text and json format, and there's no logrotate. Instead, we want
just the json logs to go through rsyslog. We'll forward INFO level
application logs to Elasticsearch via Redis, and put a human-readable
version of logs into the journal.

Marking the journal entries with the appropriate syslog severity makes it
easy to query and filter.

The lookup_table functionality actually works better than my proposed
property replacer, because it's simple to modify the lookup if requirements
evolve.


a couple comments

1. using mmnormalize and the latest liblognorm (with the version=2 ruleset), 
rsyslog can parse raw json, it doesn't need the @cee token any longer and can 
parse logs that are a mix of json and non-json data.


2. the table_lookup code that is in the released versions of rsyslog is very 
limited and has some known bugs. It was a prototype from work that was discussed 
and was going to be sponsored, but the company initiating the work fell through. 
Yesterday a full implementation was merged into the master tree for release in 
8.17. You really will want to be using that version for anything beyond a proof 
of concept.


3. we have found some nasty bugs in the json-c library and as a result have 
forked it to libjsonfast, 8.16 will optionally use it if it's available, 8.17 
will require it.


and 8.17 (or a daily build version of it) will pull in the latest liblognorm and 
libjsonfast.


This is one of those cases where you will really want to be on the very latest 
version.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

2016-02-04 Thread David Lang


On Thu, 4 Feb 2016, Rainer Gerhards wrote:


what platform doesn't offer _some_ random source? Anything Linux based
will have /dev/random and /dev/urandom. urandom may not be very good
quality randomness (by some measurements on some systems), but the kernel
provides the best that is available.



as I have learnt from someone in the know, urandom is actually very good
quality randomness. Can't quote through whom, though. But I think there are
some academic papers on that topic.


Yes, if there is 'enough' entropy, urandom is just as good as random. If entropy 
runs short, urandom still uses as secure a prng as anything available (and 
frequently reviewed, updated, etc). The estimate for entropy is very deliberatly 
conservative, so the result is very good.






so it would only be non-linux systems that could have a problem, right?



non-linux, yes


what non-linux systems? everything should have /dev/random, right?






b) use the c runtime library randon number generator (which, I think, is

*not* crypto-grade).



you still need something to initialize the random number generator



yup.

I'll craft a patch to error out within the next days if I don't hear any
objections.


does this hit too late in the process to treat this like a syntax error? If you 
specify encryption and there is no source of randomness available, treat it as 
if the module didn't exist or something along those lines?


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Mapping log-level names to severity numbers

2016-02-04 Thread Bob Gregory

Hi Dave,

It's the latter. Currently docker is just spraying logs out onto disk, in
both plain text and json format, and there's no logrotate. Instead, we want
just the json logs to go through rsyslog. We'll forward INFO level
application logs to Elasticsearch via Redis, and put a human-readable
version of logs into the journal.

Marking the journal entries with the appropriate syslog severity makes it
easy to query and filter.

The lookup_table functionality actually works better than my proposed
property replacer, because it's simple to modify the lookup if requirements
evolve.

 -- B

On 4 February 2016 at 15:15, Dave Caplinger 
wrote:

> Hi Bob,
>
> I'm curious to better understand your objective:
>
> > [some docker container] logs contain a textual severity level based on
> the log4j levels:
> > DEBUG, INFO, WARN, ERROR, CRITICAL, FATAL.
> >
> > The docker syslog integration dumps all the stdout of a container into
> > syslog with a severity of LOG_INFO, and stderr with LOG_ERR.
> >
> > I'd like to parse the incoming json and map the level names to syslog
> > severity numbers.
>
> Is it correct that you're trying to maintain the distinction between the
> LOG_INFO and LOG_ERR streams coming out of the docker containers?  If
> that's *all* you're trying to achieve, would just adding another property
> to your JSON output to store the info/err value be sufficient?
>
> Or... do you have existing downstream log processing that depends on the
> syslog severity values, so mapping the log4j {DEBUG, INFO, WARN, ERROR,
> CRITICAL, FATAL} text values onto syslog {Debug, Info, Notice, Warning,
> Error, Critical, Alert, Emergency} severities is the critical part?
>
> (Or some other scenario I haven't understood yet ...?)
>
> --
> Dave Caplinger | Director, Technical Product Management
> Solutionary — An NTT Group Security Company
>
> > On Feb 4, 2016, at 7:16 AM, Bob Gregory  wrote:
> >
> > Hi David,
> >
> > We ran logstash-forwarder in a separate container, and shared volumes
> > between app containers and a forwarding container. That's problematic as
> we
> > move toward a clustered environment, because it means running multiple
> > instances of logstash forwarder, or doing something peculiar with user
> > permissions. Instead we'd like to delegate the log routing and filtering
> to
> > the host OS via Docker's log driver.
> >
> >
> > On 4 February 2016 at 11:40, David Lang  wrote:
> >
> >> On Thu, 4 Feb 2016, Bob Gregory wrote:
> >>
> >> can you syslog over the network to localhost? rsyslog can be pretty
> 
> >>> lightweight if you set queue sizes smaller than the default 500K
> messages.
> >>> If you were running logstash-forwarder, rsyslog should be lighter.
> >>>
> >>> That's an interesting idea, but it would mean running a syslog daemon
> in
> >>> each container. Generally, we stick to a single foreground process per
> >>> container, so there's no init system for managing daemonised services,
> but
> >>> that might change in the future.
> >>>
> >>
> >> how were you running the logstash forwarder then?
> >>
> >>
> >> David Lang
> >> ___
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >
> >
> >
> > --
> >
> > 
> >
> > *Bob Gregory*
> >
> > Application Architect
> >
> > MADE.COM 
> >
> > Skype: flinkywistypomm
> >
> >
> > [image: MADE]
> >
> >
> >
> > Made.com Design Limited is a company registered in England and Wales.
> >
> > Registered number: 07101408 | Registered office: 100 Charing Cross Road,
> > London WC2H 0HG
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>



-- 



*Bob Gregory*

Application Architect

MADE.COM 

Skype: flinkywistypomm


[image: MADE]



Made.com Design Limited is a company registered in England and Wales.

Registered number: 07101408 | Registered office: 100 Charing Cross Road,
London WC2H 0HG

Re: [rsyslog] Mapping log-level names to severity numbers

2016-02-04 Thread Dave Caplinger

Hi Bob,

I'm curious to better understand your objective:

> [some docker container] logs contain a textual severity level based on the 
> log4j levels:
> DEBUG, INFO, WARN, ERROR, CRITICAL, FATAL.
> 
> The docker syslog integration dumps all the stdout of a container into
> syslog with a severity of LOG_INFO, and stderr with LOG_ERR.
> 
> I'd like to parse the incoming json and map the level names to syslog
> severity numbers.

Is it correct that you're trying to maintain the distinction between the 
LOG_INFO and LOG_ERR streams coming out of the docker containers?  If that's 
*all* you're trying to achieve, would just adding another property to your JSON 
output to store the info/err value be sufficient?

Or... do you have existing downstream log processing that depends on the syslog 
severity values, so mapping the log4j {DEBUG, INFO, WARN, ERROR, CRITICAL, 
FATAL} text values onto syslog {Debug, Info, Notice, Warning, Error, Critical, 
Alert, Emergency} severities is the critical part?

(Or some other scenario I haven't understood yet ...?)

--
Dave Caplinger | Director, Technical Product Management
Solutionary — An NTT Group Security Company

> On Feb 4, 2016, at 7:16 AM, Bob Gregory  wrote:
> 
> Hi David,
> 
> We ran logstash-forwarder in a separate container, and shared volumes
> between app containers and a forwarding container. That's problematic as we
> move toward a clustered environment, because it means running multiple
> instances of logstash forwarder, or doing something peculiar with user
> permissions. Instead we'd like to delegate the log routing and filtering to
> the host OS via Docker's log driver.
> 
> 
> On 4 February 2016 at 11:40, David Lang  wrote:
> 
>> On Thu, 4 Feb 2016, Bob Gregory wrote:
>> 
>> can you syslog over the network to localhost? rsyslog can be pretty
 
>>> lightweight if you set queue sizes smaller than the default 500K messages.
>>> If you were running logstash-forwarder, rsyslog should be lighter.
>>> 
>>> That's an interesting idea, but it would mean running a syslog daemon in
>>> each container. Generally, we stick to a single foreground process per
>>> container, so there's no init system for managing daemonised services, but
>>> that might change in the future.
>>> 
>> 
>> how were you running the logstash forwarder then?
>> 
>> 
>> David Lang
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>> 
> 
> 
> 
> -- 
> 
> 
> 
> *Bob Gregory*
> 
> Application Architect
> 
> MADE.COM 
> 
> Skype: flinkywistypomm
> 
> 
> [image: MADE]
> 
> 
> 
> Made.com Design Limited is a company registered in England and Wales.
> 
> Registered number: 07101408 | Registered office: 100 Charing Cross Road,
> London WC2H 0HG
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Mapping log-level names to severity numbers

2016-02-04 Thread Bob Gregory

Hi David,

We ran logstash-forwarder in a separate container, and shared volumes
between app containers and a forwarding container. That's problematic as we
move toward a clustered environment, because it means running multiple
instances of logstash forwarder, or doing something peculiar with user
permissions. Instead we'd like to delegate the log routing and filtering to
the host OS via Docker's log driver.

On 4 February 2016 at 11:40, David Lang  wrote:

> On Thu, 4 Feb 2016, Bob Gregory wrote:
>
> can you syslog over the network to localhost? rsyslog can be pretty
>>>
>> lightweight if you set queue sizes smaller than the default 500K messages.
>> If you were running logstash-forwarder, rsyslog should be lighter.
>>
>> That's an interesting idea, but it would mean running a syslog daemon in
>> each container. Generally, we stick to a single foreground process per
>> container, so there's no init system for managing daemonised services, but
>> that might change in the future.
>>
>
> how were you running the logstash forwarder then?
>
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>

-- 

*Bob Gregory*

Application Architect

MADE.COM 

Skype: flinkywistypomm

[image: MADE]

Made.com Design Limited is a company registered in England and Wales.

Registered number: 07101408 | Registered office: 100 Charing Cross Road,
London WC2H 0HG
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

2016-02-04 Thread Peter Viskup

On Thu, Feb 4, 2016 at 12:53 PM, Rainer Gerhards
 wrote:
> 2016-02-04 12:38 GMT+01:00 David Lang :
>
>> On Thu, 4 Feb 2016, Rainer Gerhards wrote:
>>
>> what platform doesn't offer _some_ random source? Anything Linux based
>> will have /dev/random and /dev/urandom. urandom may not be very good
>> quality randomness (by some measurements on some systems), but the kernel
>> provides the best that is available.
>>
>
> as I have learnt from someone in the know, urandom is actually very good
> quality randomness. Can't quote through whom, though. But I think there are
> some academic papers on that topic.
>

At least this one blog seems to provide some clear background about /dev/urandom
http://www.2uo.de/myths-about-urandom/

The quality of randomness is tightly tied to available entropy. Thus
/dev/urandom + haveged is best low-cost choice at these times IMHO.
http://www.issihosts.com/haveged/

-- 
Peter Viskup
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

2016-02-04 Thread Rainer Gerhards

2016-02-04 12:38 GMT+01:00 David Lang :

> On Thu, 4 Feb 2016, Rainer Gerhards wrote:
>
> 2016-02-04 3:06 GMT+01:00 David Lang :
>>
>> 530f91a42307f33c9dd43a7d0c802b3fa469beec
>>>
>>> Author: Rainer Gerhards 
>>> Date:   Tue Feb 2 15:51:52 2016 +0100
>>>
>>> prevent a clang static analyzer warning
>>>
>>> The static analyzer correctly complains about "garbagge
>>> value being used", but this is exactly what we want. The
>>> code in question is a fallback when we cannot obtain any
>>> other source of randomness for cryptography needs.
>>>
>>>
>>> If there is absolutely no source of randomness, cryptography should
>>> abort,
>>> not use whatever value happens to be in ram (which should be 0)
>>>
>>> If urandom isn't available, abort with a clear message that access to it
>>> is required, don't silently use garbage to initialize the cryptography.
>>>
>>>
>>> Just to make things clear: this commit didn't change behaviour. It just
>> addresses the static analyzer warning but keeps everything else as-is. So
>> if I revert that change, the only thing that will change is that the
>> static
>> analyzer will break all builds.
>>
>
> understood
>
> So the real issue is how to work if /dev/urandom is not available. I used
>> per-existing values in memory so far (based on my understanding that a
>> couple of tools do so).
>>
>
> various tools have done so, and been caught generating predicatable keys
> (this was the source of the debian ssh key fiasco a few years back)


actually not: that fiasco was caused by zeroing-out memory that was
supposed to be garbage,


>
>
> If the consensus is that this is a bad idea, we
>> have actually two choices:
>>
>> a) error out (which could potentially completey exclude some platform)
>>
>
> what platform doesn't offer _some_ random source? Anything Linux based
> will have /dev/random and /dev/urandom. urandom may not be very good
> quality randomness (by some measurements on some systems), but the kernel
> provides the best that is available.
>

as I have learnt from someone in the know, urandom is actually very good
quality randomness. Can't quote through whom, though. But I think there are
some academic papers on that topic.


>
> so it would only be non-linux systems that could have a problem, right?


non-linux, yes


>
>
> b) use the c runtime library randon number generator (which, I think, is
>> *not* crypto-grade).
>>
>
> you still need something to initialize the random number generator
>
>
yup.

I'll craft a patch to error out within the next days if I don't hear any
objections.

Rainer

> David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Mapping log-level names to severity numbers

2016-02-04 Thread David Lang


On Thu, 4 Feb 2016, Bob Gregory wrote:


can you syslog over the network to localhost? rsyslog can be pretty

lightweight if you set queue sizes smaller than the default 500K messages.
If you were running logstash-forwarder, rsyslog should be lighter.

That's an interesting idea, but it would mean running a syslog daemon in
each container. Generally, we stick to a single foreground process per
container, so there's no init system for managing daemonised services, but
that might change in the future.


how were you running the logstash forwarder then?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

2016-02-04 Thread David Lang


On Thu, 4 Feb 2016, Rainer Gerhards wrote:


2016-02-04 3:06 GMT+01:00 David Lang :


530f91a42307f33c9dd43a7d0c802b3fa469beec

Author: Rainer Gerhards 
Date:   Tue Feb 2 15:51:52 2016 +0100

prevent a clang static analyzer warning

The static analyzer correctly complains about "garbagge
value being used", but this is exactly what we want. The
code in question is a fallback when we cannot obtain any
other source of randomness for cryptography needs.


If there is absolutely no source of randomness, cryptography should abort,
not use whatever value happens to be in ram (which should be 0)

If urandom isn't available, abort with a clear message that access to it
is required, don't silently use garbage to initialize the cryptography.



Just to make things clear: this commit didn't change behaviour. It just
addresses the static analyzer warning but keeps everything else as-is. So
if I revert that change, the only thing that will change is that the static
analyzer will break all builds.


understood


So the real issue is how to work if /dev/urandom is not available. I used
per-existing values in memory so far (based on my understanding that a
couple of tools do so).


various tools have done so, and been caught generating predicatable keys (this 
was the source of the debian ssh key fiasco a few years back)



If the consensus is that this is a bad idea, we
have actually two choices:

a) error out (which could potentially completey exclude some platform)


what platform doesn't offer _some_ random source? Anything Linux based will have 
/dev/random and /dev/urandom. urandom may not be very good quality randomness 
(by some measurements on some systems), but the kernel provides the best that is 
available.


so it would only be non-linux systems that could have a problem, right?


b) use the c runtime library randon number generator (which, I think, is
*not* crypto-grade).


you still need something to initialize the random number generator

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

2016-02-04 Thread Peter Portante

On Thu, Feb 4, 2016 at 2:15 AM, Rainer Gerhards 
wrote:

> 2016-02-04 3:06 GMT+01:00 David Lang :
>
> > 530f91a42307f33c9dd43a7d0c802b3fa469beec
> >
> > Author: Rainer Gerhards 
> > Date:   Tue Feb 2 15:51:52 2016 +0100
> >
> > prevent a clang static analyzer warning
> >
> > The static analyzer correctly complains about "garbagge
> > value being used", but this is exactly what we want. The
> > code in question is a fallback when we cannot obtain any
> > other source of randomness for cryptography needs.
> >
> >
> > If there is absolutely no source of randomness, cryptography should
> abort,
> > not use whatever value happens to be in ram (which should be 0)
> >
> > If urandom isn't available, abort with a clear message that access to it
> > is required, don't silently use garbage to initialize the cryptography.
> >
> >
> Just to make things clear: this commit didn't change behaviour. It just
> addresses the static analyzer warning but keeps everything else as-is. So
> if I revert that change, the only thing that will change is that the static
> analyzer will break all builds.
>
> So the real issue is how to work if /dev/urandom is not available. I used
> per-existing values in memory so far (based on my understanding that a
> couple of tools do so). If the consensus is that this is a bad idea, we
> have actually two choices:
>
> a) error out (which could potentially completey exclude some platform)
>

Error out.  Thanks!


> b) use the c runtime library randon number generator (which, I think, is
> *not* crypto-grade).
>
> More comments are appreciated.
>
> Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog running well on debug mode but not in service mode

[rsyslog] rsyslog running well on debug mode but not in service mode

Re: [rsyslog] Mapping log-level names to severity numbers

Re: [rsyslog] Mapping log-level names to severity numbers

Re: [rsyslog] Mapping log-level names to severity numbers

Re: [rsyslog] Mapping log-level names to severity numbers

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

Re: [rsyslog] Mapping log-level names to severity numbers

Re: [rsyslog] Mapping log-level names to severity numbers

Re: [rsyslog] Mapping log-level names to severity numbers

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

Re: [rsyslog] Mapping log-level names to severity numbers

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

Re: [rsyslog] encryption handling without /dev/urandom - was: please revert this changeset

15 matches

Site Navigation

Mail list logo

Footer information