Re: [rsyslog] rsyslog dropping logs

2012-11-29 Thread David Lang 
Unfortunantly most of these lines contain 'error' in the data being logged, but 
the last line looks like a problem



8190.749818590:7f47e1f20700: main Q: error -7 persisting queue - data lost!


could you get more of the log around that time?

Also, if you can identify a log message that doesn't get output, data around 
that point would be great (and I do realize how hard that is to do)



try doing

grep -i error |grep -v ' recv('

that will find cases where error has caps in it, and will strip out the lines 
where it's just receiving the message and the message contains 'error'


David Lang

On Thu, 29 Nov 2012, Luke Marrott wrote:


Date: Thu, 29 Nov 2012 13:17:24 -0700
From: Luke Marrott 
Reply-To: rsyslog-users 
To: rsyslog-users 
Subject: Re: [rsyslog] rsyslog dropping logs

My debug was about 200MB but here is most of the lines that contained the
word error. It seems like most of them are actual messages, in which case
I'm still not sure why I'm not getting all my logs.

lmarrottMBP:Downloads lmarrott$ cat ./rsyslogdebug.txt | grep error
8064.318695324:7f47e1f20700: Command 'errormessagestostderr':
8071.284459711:7f47d97fb700: recv(4,257),acl:1,msg:<164>Nov 29 2012
09:54:30: %ASA-4-313005: No matching connection for ICMP error message:
icmp src outside: dst outside: (type 3, code 1) on
outside interface.  Original IP payload: udp src / dst
/.
8073.282005100:7f47d97fb700: recv(4,143),acl:1,msg:<131>HOST:
*ethoipSocketTask: Nov 29 08:54:32.487: %ETHOIP-3-PKT_RECV_ERROR:
ethoip.c:338 ethoipSocketTask: ethoipRecvPkt returned errorember:0.0.0.0.
source member unknown.with invalid SPA(Source IP Address) /TPA(Destination IP Address) KlA==
8075.329472799:7f47d97fb700: recv(4,254),acl:1,msg:<164>Nov 29 2012
09:54:34: %ASA-4-313005: No matching connection for ICMP error message:
icmp src outside: dst outside: (type 3, code 1) on
outside interface.  Original IP payload: udp src /15637 dst
/37.
8075.425051462:7f47d97fb700: recv(4,120),acl:1,msg:<131>HOST: *SISF BT
Process: Nov 29 09:54:38.193: %SISF-3-INTERNAL: sisf_shim_utils.c:316
Internal error, 0 length0::7ec5:37ff:fe76:698d V=499 I=wireless:0 P=0005
M=ST) received with invalid SPA(Source IP Address)
172.24.13.60/TPA(Destination IP Address) 2.24.0.1reless,0x41C04FEC 0x41C057CC 0x41C0583C 0x429735D4 0x4034E898
0x41C03A1C
0x41C02E74?!

???

???
8080.405977456:7f47d97fb700: recv(4,154),acl:1,msg:<170>781802: 754500: Nov
29 16:54:37.388 UTC: %PM_SCP-SP-2-LCP_FW_ERR_INFORM: Module 9 is
experiencing the following error: Bus Asic #0 out of sync error ARP (op ARP
REQUEST) received with invalid SPA(Source IP Address)
172.24.3.123/TPA(Destination IP Address) a

Re: [rsyslog] rsyslog dropping logs

2012-11-29 Thread Luke Marrott 
.413 UTC: %PM_SCP-SP-2-LCP_FW_ERR_INFORM: Module 9 is
experiencing the following error: Bus Asic #0 out of sync error es 71455
TCP FINs
8172.810010498:7f47d97fb700: recv(4,141),acl:1,msg:<172>3926398: 3921147:
Nov 29 16:56:11.292 UTC: %PM-SP-4-ERR_DISABLE: bpduguard error detected on
Gi6/44, putting Gi6/44 in err-disable stateeved file
461_message_2012-11-29_084210.xml
8173.282806737:7f47d97fb700: recv(4,143),acl:1,msg:<131>HOST:
*ethoipSocketTask: Nov 29 08:56:12.488: %ETHOIP-3-PKT_RECV_ERROR:
ethoip.c:338 ethoipSocketTask: ethoipRecvPkt returned error0.4.20, not from
a mobility peer
8188.442442397:7f47d97fb700: recv(4,154),acl:1,msg:<170>781831: 754529: Nov
29 16:56:25.431 UTC: %PM_SCP-SP-2-LCP_FW_ERR_INFORM: Module 9 is
experiencing the following error: Bus Asic #0 out of sync error es 0 TCP
FINs
8190.749818590:7f47e1f20700: main Q: error -7 persisting queue - data lost!


:Luke Marrott



On Thu, Nov 29, 2012 at 11:12 AM, David Lang  wrote:

> any errors :-)
>
> go ahead and past pieces inline.
>
> David Lang
>
>
> On Thu, 29 Nov 2012, Luke Marrott wrote:
>
>  Which parts of the debug would be the most beneficial? Should I attach it
>> or paste pieces inline?
>>
>> :Luke Marrott
>>
>>
>>
>> On Mon, Nov 19, 2012 at 7:11 PM, David Lang  wrote:
>>
>>  On Mon, 19 Nov 2012, Luke Marrott wrote:
>>>
>>>  So I have been trying to figure this out. Went through the config and
>>> got
>>>
>>>> rid of everything that I wasn't using or was commented out from the
>>>> default
>>>> template and it's still not getting as much as Splunk is getting so it
>>>> has
>>>> to be something with my installation or my configuration.
>>>>
>>>> I ran the config check -N 1 and here is the output:
>>>> [root@nwcacti lmarrott]# /usr/local/sbin/rsyslogd -f /etc/rsyslog.conf
>>>> -n
>>>> -N 1
>>>> rsyslogd: version 5.8.10, config validation run (level 1), master config
>>>> /etc/rsyslog.conf
>>>> rsyslogd: WARNING: rsyslogd is running in compatibility mode.
>>>> Automatically
>>>> generated config directives may interfer with your rsyslog.conf
>>>> settings.
>>>> We suggest upgrading your config and adding -c5 as the first rsyslogd
>>>> option.
>>>> rsyslogd: Warning: backward compatibility layer added to following
>>>> directive to rsyslog.conf: ModLoad immark
>>>> rsyslogd: Warning: backward compatibility layer added to following
>>>> directive to rsyslog.conf: MarkMessagePeriod 1200
>>>> rsyslogd: Warning: backward compatibility layer added to following
>>>> directive to rsyslog.conf: ModLoad imuxsock
>>>> rsyslogd: End of config validation run. Bye.
>>>>
>>>>
>>>> How do I upgrade my config?
>>>>
>>>> I also ran a debug and it seems like there are a lot of things it's
>>>> complaining about. But then again maybe it's normal.
>>>>
>>>>
>>> start rsyslog with -c5 to avoid this particular error
>>>
>>> If you can send us the debug log (with the -c5) we can look at the errors
>>> that show up, but I suspect that things will work a LOT better for you
>>> with
>>> the -c5
>>>
>>> David Lang
>>>
>>>
>>>  :Luke Marrott
>>>>
>>>>
>>>>
>>>> On Fri, Nov 9, 2012 at 5:02 PM, David Lang  wrote:
>>>>
>>>>  I'm not sure exactly what will happen, but I suspect that all the logs
>>>>
>>>>> will end up in all the possible destinations. I don't think rsyslog
>>>>> really
>>>>> will process all the local logs to one set of rules and all the remote
>>>>> logs
>>>>> to another set of rules
>>>>>
>>>>>
>>>>> At least, not unless you are using rulesets, which I am not seeing.
>>>>>
>>>>>
>>>>> a couple thousand log messages/sec should not cause any problems.
>>>>>
>>>>>
>>>>> David Lang
>>>>>
>>>>>  On Fri, 9 Nov 2012, Luke Marrott wrote:
>>>>>
>>>>>  Date: Fri, 9 Nov 2012 15:14:32 -0700
>>>>>
>>>>>
>>>>>> From: Luke Marrott 
>>>>>> Reply-To: rsyslog-users 
>>>>>> To: rsyslog-users 
>>>>>> Subject: Re: [rsyslog] rsyslog dropping logs
>>>>>>
>&

Re: [rsyslog] rsyslog dropping logs

2012-11-29 Thread David Lang 

any errors :-)

go ahead and past pieces inline.

David Lang

On Thu, 29 Nov 2012, Luke Marrott wrote:


Which parts of the debug would be the most beneficial? Should I attach it
or paste pieces inline?

:Luke Marrott



On Mon, Nov 19, 2012 at 7:11 PM, David Lang  wrote:


On Mon, 19 Nov 2012, Luke Marrott wrote:

 So I have been trying to figure this out. Went through the config and got

rid of everything that I wasn't using or was commented out from the
default
template and it's still not getting as much as Splunk is getting so it has
to be something with my installation or my configuration.

I ran the config check -N 1 and here is the output:
[root@nwcacti lmarrott]# /usr/local/sbin/rsyslogd -f /etc/rsyslog.conf -n
-N 1
rsyslogd: version 5.8.10, config validation run (level 1), master config
/etc/rsyslog.conf
rsyslogd: WARNING: rsyslogd is running in compatibility mode.
Automatically
generated config directives may interfer with your rsyslog.conf settings.
We suggest upgrading your config and adding -c5 as the first rsyslogd
option.
rsyslogd: Warning: backward compatibility layer added to following
directive to rsyslog.conf: ModLoad immark
rsyslogd: Warning: backward compatibility layer added to following
directive to rsyslog.conf: MarkMessagePeriod 1200
rsyslogd: Warning: backward compatibility layer added to following
directive to rsyslog.conf: ModLoad imuxsock
rsyslogd: End of config validation run. Bye.


How do I upgrade my config?

I also ran a debug and it seems like there are a lot of things it's
complaining about. But then again maybe it's normal.



start rsyslog with -c5 to avoid this particular error

If you can send us the debug log (with the -c5) we can look at the errors
that show up, but I suspect that things will work a LOT better for you with
the -c5

David Lang



:Luke Marrott



On Fri, Nov 9, 2012 at 5:02 PM, David Lang  wrote:

 I'm not sure exactly what will happen, but I suspect that all the logs

will end up in all the possible destinations. I don't think rsyslog
really
will process all the local logs to one set of rules and all the remote
logs
to another set of rules


At least, not unless you are using rulesets, which I am not seeing.


a couple thousand log messages/sec should not cause any problems.


David Lang

 On Fri, 9 Nov 2012, Luke Marrott wrote:

 Date: Fri, 9 Nov 2012 15:14:32 -0700



From: Luke Marrott 
Reply-To: rsyslog-users 
To: rsyslog-users 
Subject: Re: [rsyslog] rsyslog dropping logs

Only one configuration there.

I have all my messages going to directories by host so your method
doesn't
seem to be working.

I did a tcpdump only on port 514 for a few seconds and I had like 2000
messages.

:Luke Marrott



On Fri, Nov 9, 2012 at 2:48 PM, David Lang  wrote:

 are these two different configs (the sender and the receiver)?



a simple way to see the message rate is to do a
cut -f 1 -d ' ' logfiles |sort |uniq -c to look at the timestamps and
see
how many timestamps you have in a second.

David Lang


On Fri, 9 Nov 2012, Luke Marrott wrote:

 Date: Fri, 9 Nov 2012 13:07:02 -0700

 From: Luke Marrott 

Reply-To: rsyslog-users 
To: rsyslog-users 

Subject: Re: [rsyslog] rsyslog dropping logs

Full configuration:
[root@hostname]# cat /etc/rsyslog.conf
# if you experience problems, check
# 
http://www.rsyslog.com/**troubleshoot<http://www.rsyslog.com/troubleshoot>
<http://www.**rsyslog.com/**troubleshoot<http://www.rsyslog.com/**troubleshoot>



<http://www.**rsyslog.com/**troubleshoot<http://rsyslog.com/troubleshoot>
<http://www.**rsyslog.com/troubleshoot<http://www.rsyslog.com/troubleshoot>

for


assistance


# rsyslog v3: load input modules
# If you do not load inputs, nothing happens!
# You may need to set the module load path if modules are not found.

$ModLoad immark   # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g.
via
logger command)
$ModLoad imklog   # kernel logging (formerly provided by rklogd)

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.error;mail.none;authpriv.**none;cron.none


 /var/log/messages

# The authpriv file has restricted access.
authpriv.*
 /var/log/secure

# Log all the mail messages in one place.
mail.*
 -/var/log/maillog


# Log cron stuff
cron.*  -/var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit
 -/var/log/spooler

# Save boot messages also to boot.log
local7.*
 /var/log/boot.log

# Remote Logging (we use TCP for reliable delivery)
# An on-disk queue is created for this action. If t

Re: [rsyslog] rsyslog dropping logs

2012-11-29 Thread Luke Marrott 
Which parts of the debug would be the most beneficial? Should I attach it
or paste pieces inline?

:Luke Marrott



On Mon, Nov 19, 2012 at 7:11 PM, David Lang  wrote:

> On Mon, 19 Nov 2012, Luke Marrott wrote:
>
>  So I have been trying to figure this out. Went through the config and got
>> rid of everything that I wasn't using or was commented out from the
>> default
>> template and it's still not getting as much as Splunk is getting so it has
>> to be something with my installation or my configuration.
>>
>> I ran the config check -N 1 and here is the output:
>> [root@nwcacti lmarrott]# /usr/local/sbin/rsyslogd -f /etc/rsyslog.conf -n
>> -N 1
>> rsyslogd: version 5.8.10, config validation run (level 1), master config
>> /etc/rsyslog.conf
>> rsyslogd: WARNING: rsyslogd is running in compatibility mode.
>> Automatically
>> generated config directives may interfer with your rsyslog.conf settings.
>> We suggest upgrading your config and adding -c5 as the first rsyslogd
>> option.
>> rsyslogd: Warning: backward compatibility layer added to following
>> directive to rsyslog.conf: ModLoad immark
>> rsyslogd: Warning: backward compatibility layer added to following
>> directive to rsyslog.conf: MarkMessagePeriod 1200
>> rsyslogd: Warning: backward compatibility layer added to following
>> directive to rsyslog.conf: ModLoad imuxsock
>> rsyslogd: End of config validation run. Bye.
>>
>>
>> How do I upgrade my config?
>>
>> I also ran a debug and it seems like there are a lot of things it's
>> complaining about. But then again maybe it's normal.
>>
>
> start rsyslog with -c5 to avoid this particular error
>
> If you can send us the debug log (with the -c5) we can look at the errors
> that show up, but I suspect that things will work a LOT better for you with
> the -c5
>
> David Lang
>
>
>> :Luke Marrott
>>
>>
>>
>> On Fri, Nov 9, 2012 at 5:02 PM, David Lang  wrote:
>>
>>  I'm not sure exactly what will happen, but I suspect that all the logs
>>> will end up in all the possible destinations. I don't think rsyslog
>>> really
>>> will process all the local logs to one set of rules and all the remote
>>> logs
>>> to another set of rules
>>>
>>>
>>> At least, not unless you are using rulesets, which I am not seeing.
>>>
>>>
>>> a couple thousand log messages/sec should not cause any problems.
>>>
>>>
>>> David Lang
>>>
>>>  On Fri, 9 Nov 2012, Luke Marrott wrote:
>>>
>>>  Date: Fri, 9 Nov 2012 15:14:32 -0700
>>>
>>>>
>>>> From: Luke Marrott 
>>>> Reply-To: rsyslog-users 
>>>> To: rsyslog-users 
>>>> Subject: Re: [rsyslog] rsyslog dropping logs
>>>>
>>>> Only one configuration there.
>>>>
>>>> I have all my messages going to directories by host so your method
>>>> doesn't
>>>> seem to be working.
>>>>
>>>> I did a tcpdump only on port 514 for a few seconds and I had like 2000
>>>> messages.
>>>>
>>>> :Luke Marrott
>>>>
>>>>
>>>>
>>>> On Fri, Nov 9, 2012 at 2:48 PM, David Lang  wrote:
>>>>
>>>>  are these two different configs (the sender and the receiver)?
>>>>
>>>>>
>>>>> a simple way to see the message rate is to do a
>>>>> cut -f 1 -d ' ' logfiles |sort |uniq -c to look at the timestamps and
>>>>> see
>>>>> how many timestamps you have in a second.
>>>>>
>>>>> David Lang
>>>>>
>>>>>
>>>>> On Fri, 9 Nov 2012, Luke Marrott wrote:
>>>>>
>>>>>  Date: Fri, 9 Nov 2012 13:07:02 -0700
>>>>>
>>>>>  From: Luke Marrott 
>>>>>> Reply-To: rsyslog-users 
>>>>>> To: rsyslog-users 
>>>>>>
>>>>>> Subject: Re: [rsyslog] rsyslog dropping logs
>>>>>>
>>>>>> Full configuration:
>>>>>> [root@hostname]# cat /etc/rsyslog.conf
>>>>>> # if you experience problems, check
>>>>>> # 
>>>>>> http://www.rsyslog.com/**troubleshoot<http://www.rsyslog.com/troubleshoot>
>>>>>> <http://www.**rsyslog.com/**troubleshoot<http://www.rsyslog.com/**troubleshoot>
>>>>>> >

Re: [rsyslog] rsyslog dropping logs

2012-11-19 Thread David Lang 

On Mon, 19 Nov 2012, Luke Marrott wrote:


So I have been trying to figure this out. Went through the config and got
rid of everything that I wasn't using or was commented out from the default
template and it's still not getting as much as Splunk is getting so it has
to be something with my installation or my configuration.

I ran the config check -N 1 and here is the output:
[root@nwcacti lmarrott]# /usr/local/sbin/rsyslogd -f /etc/rsyslog.conf -n
-N 1
rsyslogd: version 5.8.10, config validation run (level 1), master config
/etc/rsyslog.conf
rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically
generated config directives may interfer with your rsyslog.conf settings.
We suggest upgrading your config and adding -c5 as the first rsyslogd
option.
rsyslogd: Warning: backward compatibility layer added to following
directive to rsyslog.conf: ModLoad immark
rsyslogd: Warning: backward compatibility layer added to following
directive to rsyslog.conf: MarkMessagePeriod 1200
rsyslogd: Warning: backward compatibility layer added to following
directive to rsyslog.conf: ModLoad imuxsock
rsyslogd: End of config validation run. Bye.


How do I upgrade my config?

I also ran a debug and it seems like there are a lot of things it's
complaining about. But then again maybe it's normal.


start rsyslog with -c5 to avoid this particular error

If you can send us the debug log (with the -c5) we can look at the errors that 
show up, but I suspect that things will work a LOT better for you with the -c5


David Lang



:Luke Marrott



On Fri, Nov 9, 2012 at 5:02 PM, David Lang  wrote:


I'm not sure exactly what will happen, but I suspect that all the logs
will end up in all the possible destinations. I don't think rsyslog really
will process all the local logs to one set of rules and all the remote logs
to another set of rules


At least, not unless you are using rulesets, which I am not seeing.


a couple thousand log messages/sec should not cause any problems.


David Lang

 On Fri, 9 Nov 2012, Luke Marrott wrote:

 Date: Fri, 9 Nov 2012 15:14:32 -0700


From: Luke Marrott 
Reply-To: rsyslog-users 
To: rsyslog-users 
Subject: Re: [rsyslog] rsyslog dropping logs

Only one configuration there.

I have all my messages going to directories by host so your method doesn't
seem to be working.

I did a tcpdump only on port 514 for a few seconds and I had like 2000
messages.

:Luke Marrott



On Fri, Nov 9, 2012 at 2:48 PM, David Lang  wrote:

 are these two different configs (the sender and the receiver)?


a simple way to see the message rate is to do a
cut -f 1 -d ' ' logfiles |sort |uniq -c to look at the timestamps and see
how many timestamps you have in a second.

David Lang


On Fri, 9 Nov 2012, Luke Marrott wrote:

 Date: Fri, 9 Nov 2012 13:07:02 -0700


From: Luke Marrott 
Reply-To: rsyslog-users 
To: rsyslog-users 

Subject: Re: [rsyslog] rsyslog dropping logs

Full configuration:
[root@hostname]# cat /etc/rsyslog.conf
# if you experience problems, check
# http://www.rsyslog.com/troubleshoot<http://www.rsyslog.com/**troubleshoot>
<http://www.**rsyslog.com/troubleshoot<http://www.rsyslog.com/troubleshoot>>for
assistance


# rsyslog v3: load input modules
# If you do not load inputs, nothing happens!
# You may need to set the module load path if modules are not found.

$ModLoad immark   # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog   # kernel logging (formerly provided by rklogd)

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.error;mail.none;authpriv.none;cron.none

 /var/log/messages

# The authpriv file has restricted access.
authpriv.*  /var/log/secure

# Log all the mail messages in one place.
mail.*
 -/var/log/maillog


# Log cron stuff
cron.*  -/var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit
 -/var/log/spooler

# Save boot messages also to boot.log
local7.*
 /var/log/boot.log

# Remote Logging (we use TCP for reliable delivery)
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /rsyslog/spool # where to place spool files
#$ActionQueueFileName uniqName # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as
possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResume

Re: [rsyslog] rsyslog dropping logs

2012-11-19 Thread Luke Marrott 
So I have been trying to figure this out. Went through the config and got
rid of everything that I wasn't using or was commented out from the default
template and it's still not getting as much as Splunk is getting so it has
to be something with my installation or my configuration.

I ran the config check -N 1 and here is the output:
[root@nwcacti lmarrott]# /usr/local/sbin/rsyslogd -f /etc/rsyslog.conf -n
-N 1
rsyslogd: version 5.8.10, config validation run (level 1), master config
/etc/rsyslog.conf
rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically
generated config directives may interfer with your rsyslog.conf settings.
We suggest upgrading your config and adding -c5 as the first rsyslogd
option.
rsyslogd: Warning: backward compatibility layer added to following
directive to rsyslog.conf: ModLoad immark
rsyslogd: Warning: backward compatibility layer added to following
directive to rsyslog.conf: MarkMessagePeriod 1200
rsyslogd: Warning: backward compatibility layer added to following
directive to rsyslog.conf: ModLoad imuxsock
rsyslogd: End of config validation run. Bye.


How do I upgrade my config?

I also ran a debug and it seems like there are a lot of things it's
complaining about. But then again maybe it's normal.


:Luke Marrott



On Fri, Nov 9, 2012 at 5:02 PM, David Lang  wrote:

> I'm not sure exactly what will happen, but I suspect that all the logs
> will end up in all the possible destinations. I don't think rsyslog really
> will process all the local logs to one set of rules and all the remote logs
> to another set of rules
>
>
> At least, not unless you are using rulesets, which I am not seeing.
>
>
> a couple thousand log messages/sec should not cause any problems.
>
>
> David Lang
>
>  On Fri, 9 Nov 2012, Luke Marrott wrote:
>
>  Date: Fri, 9 Nov 2012 15:14:32 -0700
>>
>> From: Luke Marrott 
>> Reply-To: rsyslog-users 
>> To: rsyslog-users 
>> Subject: Re: [rsyslog] rsyslog dropping logs
>>
>> Only one configuration there.
>>
>> I have all my messages going to directories by host so your method doesn't
>> seem to be working.
>>
>> I did a tcpdump only on port 514 for a few seconds and I had like 2000
>> messages.
>>
>> :Luke Marrott
>>
>>
>>
>> On Fri, Nov 9, 2012 at 2:48 PM, David Lang  wrote:
>>
>>  are these two different configs (the sender and the receiver)?
>>>
>>> a simple way to see the message rate is to do a
>>> cut -f 1 -d ' ' logfiles |sort |uniq -c to look at the timestamps and see
>>> how many timestamps you have in a second.
>>>
>>> David Lang
>>>
>>>
>>> On Fri, 9 Nov 2012, Luke Marrott wrote:
>>>
>>>  Date: Fri, 9 Nov 2012 13:07:02 -0700
>>>
>>>> From: Luke Marrott 
>>>> Reply-To: rsyslog-users 
>>>> To: rsyslog-users 
>>>>
>>>> Subject: Re: [rsyslog] rsyslog dropping logs
>>>>
>>>> Full configuration:
>>>> [root@hostname]# cat /etc/rsyslog.conf
>>>> # if you experience problems, check
>>>> # 
>>>> http://www.rsyslog.com/troubleshoot<http://www.rsyslog.com/**troubleshoot>
>>>> <http://www.**rsyslog.com/troubleshoot<http://www.rsyslog.com/troubleshoot>>for
>>>> assistance
>>>>
>>>>
>>>> # rsyslog v3: load input modules
>>>> # If you do not load inputs, nothing happens!
>>>> # You may need to set the module load path if modules are not found.
>>>>
>>>> $ModLoad immark   # provides --MARK-- message capability
>>>> $ModLoad imuxsock # provides support for local system logging (e.g. via
>>>> logger command)
>>>> $ModLoad imklog   # kernel logging (formerly provided by rklogd)
>>>>
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> #kern.* /dev/console
>>>>
>>>> # Log anything (except mail) of level info or higher.
>>>> # Don't log private authentication messages!
>>>> #*.error;mail.none;authpriv.none;cron.none
>>>>
>>>>  /var/log/messages
>>>>
>>>> # The authpriv file has restricted access.
>>>> authpriv.*  /var/log/secure
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.*
>>>>  -/var/log/maillog
>>>>
>>>>
>>>> # Log cron stuff
>

Re: [rsyslog] rsyslog dropping logs

2012-11-09 Thread David Lang 
I'm not sure exactly what will happen, but I suspect that all the logs will end 
up in all the possible destinations. I don't think rsyslog really will process 
all the local logs to one set of rules and all the remote logs to another set of 
rules



At least, not unless you are using rulesets, which I am not seeing.


a couple thousand log messages/sec should not cause any problems.

David Lang

 On Fri, 9 Nov 2012, Luke Marrott wrote:


Date: Fri, 9 Nov 2012 15:14:32 -0700
From: Luke Marrott 
Reply-To: rsyslog-users 
To: rsyslog-users 
Subject: Re: [rsyslog] rsyslog dropping logs

Only one configuration there.

I have all my messages going to directories by host so your method doesn't
seem to be working.

I did a tcpdump only on port 514 for a few seconds and I had like 2000
messages.

:Luke Marrott



On Fri, Nov 9, 2012 at 2:48 PM, David Lang  wrote:


are these two different configs (the sender and the receiver)?

a simple way to see the message rate is to do a
cut -f 1 -d ' ' logfiles |sort |uniq -c to look at the timestamps and see
how many timestamps you have in a second.

David Lang


On Fri, 9 Nov 2012, Luke Marrott wrote:

 Date: Fri, 9 Nov 2012 13:07:02 -0700

From: Luke Marrott 
Reply-To: rsyslog-users 
To: rsyslog-users 

Subject: Re: [rsyslog] rsyslog dropping logs

Full configuration:
[root@hostname]# cat /etc/rsyslog.conf
# if you experience problems, check
# http://www.rsyslog.com/**troubleshoot<http://www.rsyslog.com/troubleshoot>for 
assistance

# rsyslog v3: load input modules
# If you do not load inputs, nothing happens!
# You may need to set the module load path if modules are not found.

$ModLoad immark   # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog   # kernel logging (formerly provided by rklogd)

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.error;mail.none;authpriv.**none;cron.none
 /var/log/messages

# The authpriv file has restricted access.
authpriv.*  /var/log/secure

# Log all the mail messages in one place.
mail.*  -/var/log/maillog


# Log cron stuff
cron.*  -/var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit  -/var/log/spooler

# Save boot messages also to boot.log
local7.*/var/log/boot.log

# Remote Logging (we use TCP for reliable delivery)
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /rsyslog/spool # where to place spool files
#$ActionQueueFileName uniqName # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1# infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514


# # Receiving Messages from Remote Hosts ##
# TCP Syslog Server:
# provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imtcp.so  # load module
$InputTCPServerRun 514 # start up TCP listener at port 514

# UDP Syslog Server:
$ModLoad imudp.so  # provides UDP syslog reception
$UDPServerRun 514 # start a UDP syslog server at standard port 514


$template Default,"/data/syslog/%**HOSTNAME%/%HOSTNAME%.log"
*.* ?Default


[root@hostname]#


What's a good way to look at message ratE?


:Luke Marrott



On Fri, Nov 9, 2012 at 1:03 PM, David Lang  wrote:

 On Fri, 9 Nov 2012, Luke Marrott wrote:


 Sorry. I wasn't real clear. The server runs on a big VM in another


location
completely. No issues with the server during this time. This has been an
ongoing thing. I'm running Splunk on the same box and if I turn off
rsyslog
and turn splunk on the same port it gets all the messages that don't
seem
to get picked up by rsyslog.

Doesn't appear to be any rate limiting configuration.



Ok, that is a different situation. In my experience, rsyslog is
signicantly better than Splunk at receiving messages. I've testing
rsyslog
up to 380K messages/sec (gige wire speed) and others have tested rsyslog
up
to 1M messages/sec, so it's unlikely to be something fundamental to
rsyslog, but it could easily be some resource contraint you are running
into.

can

Re: [rsyslog] rsyslog dropping logs

2012-11-09 Thread Luke Marrott 
Only one configuration there.

I have all my messages going to directories by host so your method doesn't
seem to be working.

I did a tcpdump only on port 514 for a few seconds and I had like 2000
messages.

:Luke Marrott



On Fri, Nov 9, 2012 at 2:48 PM, David Lang  wrote:

> are these two different configs (the sender and the receiver)?
>
> a simple way to see the message rate is to do a
> cut -f 1 -d ' ' logfiles |sort |uniq -c to look at the timestamps and see
> how many timestamps you have in a second.
>
> David Lang
>
>
> On Fri, 9 Nov 2012, Luke Marrott wrote:
>
>  Date: Fri, 9 Nov 2012 13:07:02 -0700
>> From: Luke Marrott 
>> Reply-To: rsyslog-users 
>> To: rsyslog-users 
>>
>> Subject: Re: [rsyslog] rsyslog dropping logs
>>
>> Full configuration:
>> [root@hostname]# cat /etc/rsyslog.conf
>> # if you experience problems, check
>> # 
>> http://www.rsyslog.com/**troubleshoot<http://www.rsyslog.com/troubleshoot>for
>>  assistance
>>
>> # rsyslog v3: load input modules
>> # If you do not load inputs, nothing happens!
>> # You may need to set the module load path if modules are not found.
>>
>> $ModLoad immark   # provides --MARK-- message capability
>> $ModLoad imuxsock # provides support for local system logging (e.g. via
>> logger command)
>> $ModLoad imklog   # kernel logging (formerly provided by rklogd)
>>
>> # Log all kernel messages to the console.
>> # Logging much else clutters up the screen.
>> #kern.* /dev/console
>>
>> # Log anything (except mail) of level info or higher.
>> # Don't log private authentication messages!
>> #*.error;mail.none;authpriv.**none;cron.none
>>  /var/log/messages
>>
>> # The authpriv file has restricted access.
>> authpriv.*  /var/log/secure
>>
>> # Log all the mail messages in one place.
>> mail.*  -/var/log/maillog
>>
>>
>> # Log cron stuff
>> cron.*  -/var/log/cron
>>
>> # Everybody gets emergency messages
>> *.emerg *
>>
>> # Save news errors of level crit and higher in a special file.
>> uucp,news.crit  -/var/log/spooler
>>
>> # Save boot messages also to boot.log
>> local7.*/var/log/boot.log
>>
>> # Remote Logging (we use TCP for reliable delivery)
>> # An on-disk queue is created for this action. If the remote host is
>> # down, messages are spooled to disk and sent when it is up again.
>> #$WorkDirectory /rsyslog/spool # where to place spool files
>> #$ActionQueueFileName uniqName # unique name prefix for spool files
>> #$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
>> #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
>> #$ActionQueueType LinkedList   # run asynchronously
>> #$ActionResumeRetryCount -1# infinite retries if host is down
>> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
>> #*.* @@remote-host:514
>>
>>
>> # # Receiving Messages from Remote Hosts ##
>> # TCP Syslog Server:
>> # provides TCP syslog reception and GSS-API (if compiled to support it)
>> $ModLoad imtcp.so  # load module
>> $InputTCPServerRun 514 # start up TCP listener at port 514
>>
>> # UDP Syslog Server:
>> $ModLoad imudp.so  # provides UDP syslog reception
>> $UDPServerRun 514 # start a UDP syslog server at standard port 514
>>
>>
>> $template Default,"/data/syslog/%**HOSTNAME%/%HOSTNAME%.log"
>> *.* ?Default
>>
>>
>> [root@hostname]#
>>
>>
>> What's a good way to look at message ratE?
>>
>>
>> :Luke Marrott
>>
>>
>>
>> On Fri, Nov 9, 2012 at 1:03 PM, David Lang  wrote:
>>
>>  On Fri, 9 Nov 2012, Luke Marrott wrote:
>>>
>>>  Sorry. I wasn't real clear. The server runs on a big VM in another
>>>
>>>> location
>>>> completely. No issues with the server during this time. This has been an
>>>> ongoing thing. I'm running Splunk on the same box and if I turn off
>>>> rsyslog
>>>> and turn splunk on the same port it gets all the messages that don't
>>>> seem
>>>> to get picked up by rsyslog.
>>>>
>>>> Doesn't ap

Re: [rsyslog] rsyslog dropping logs

2012-11-09 Thread David Lang 

are these two different configs (the sender and the receiver)?

a simple way to see the message rate is to do a
cut -f 1 -d ' ' logfiles |sort |uniq -c 
to look at the timestamps and see how many timestamps you have in a second.


David Lang

On Fri, 9 Nov 2012, Luke Marrott wrote:


Date: Fri, 9 Nov 2012 13:07:02 -0700
From: Luke Marrott 
Reply-To: rsyslog-users 
To: rsyslog-users 
Subject: Re: [rsyslog] rsyslog dropping logs

Full configuration:
[root@hostname]# cat /etc/rsyslog.conf
# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

# rsyslog v3: load input modules
# If you do not load inputs, nothing happens!
# You may need to set the module load path if modules are not found.

$ModLoad immark   # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog   # kernel logging (formerly provided by rklogd)

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.error;mail.none;authpriv.none;cron.none/var/log/messages

# The authpriv file has restricted access.
authpriv.*  /var/log/secure

# Log all the mail messages in one place.
mail.*  -/var/log/maillog


# Log cron stuff
cron.*  -/var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit  -/var/log/spooler

# Save boot messages also to boot.log
local7.*/var/log/boot.log

# Remote Logging (we use TCP for reliable delivery)
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /rsyslog/spool # where to place spool files
#$ActionQueueFileName uniqName # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1# infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514


# # Receiving Messages from Remote Hosts ##
# TCP Syslog Server:
# provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imtcp.so  # load module
$InputTCPServerRun 514 # start up TCP listener at port 514

# UDP Syslog Server:
$ModLoad imudp.so  # provides UDP syslog reception
$UDPServerRun 514 # start a UDP syslog server at standard port 514


$template Default,"/data/syslog/%HOSTNAME%/%HOSTNAME%.log"
*.* ?Default


[root@hostname]#


What's a good way to look at message ratE?


:Luke Marrott



On Fri, Nov 9, 2012 at 1:03 PM, David Lang  wrote:


On Fri, 9 Nov 2012, Luke Marrott wrote:

 Sorry. I wasn't real clear. The server runs on a big VM in another

location
completely. No issues with the server during this time. This has been an
ongoing thing. I'm running Splunk on the same box and if I turn off
rsyslog
and turn splunk on the same port it gets all the messages that don't seem
to get picked up by rsyslog.

Doesn't appear to be any rate limiting configuration.



Ok, that is a different situation. In my experience, rsyslog is
signicantly better than Splunk at receiving messages. I've testing rsyslog
up to 380K messages/sec (gige wire speed) and others have tested rsyslog up
to 1M messages/sec, so it's unlikely to be something fundamental to
rsyslog, but it could easily be some resource contraint you are running
into.

can you post your full configuration?

what message rate are you seeing?


David Lang
__**_
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad o

Re: [rsyslog] rsyslog dropping logs

2012-11-09 Thread Luke Marrott 
My apologies but I'm not seeing this message get logged. Where would it log
to?

:Luke Marrott



On Fri, Nov 9, 2012 at 1:17 PM, Rick Brown wrote:

> Add:
>
> $ModLoad impstats
>
> to the top of your rsyslog.conf and restart.   Then look for lines like:
>
> Nov  9 14:58:29 scribe1 [syslog.info<46>] rsyslogd-pstats:main Q:
> size=3771664 enqueued=2498832464 full=22295 maxqsize=800
>
> every 5 minutes.   Subtract the smaller enqueued value from the larger,
> divide by 300, and that's your rate per second.
>
> - Original Message -
> > From: "Luke Marrott" 
> > To: "rsyslog-users" 
> > Sent: Friday, November 9, 2012 3:07:02 PM
> > Subject: Re: [rsyslog] rsyslog dropping logs
> >
> > Full configuration:
> > [root@hostname]# cat /etc/rsyslog.conf
> > # if you experience problems, check
> > # http://www.rsyslog.com/troubleshoot for assistance
> >
> > # rsyslog v3: load input modules
> > # If you do not load inputs, nothing happens!
> > # You may need to set the module load path if modules are not found.
> >
> > $ModLoad immark   # provides --MARK-- message capability
> > $ModLoad imuxsock # provides support for local system logging (e.g.
> > via
> > logger command)
> > $ModLoad imklog   # kernel logging (formerly provided by rklogd)
> >
> > # Log all kernel messages to the console.
> > # Logging much else clutters up the screen.
> > #kern.* /dev/console
> >
> > # Log anything (except mail) of level info or higher.
> > # Don't log private authentication messages!
> > #*.error;mail.none;authpriv.none;cron.none
> >/var/log/messages
> >
> > # The authpriv file has restricted access.
> > authpriv.*
> >  /var/log/secure
> >
> > # Log all the mail messages in one place.
> > mail.*
> >  -/var/log/maillog
> >
> >
> > # Log cron stuff
> > cron.*
> >  -/var/log/cron
> >
> > # Everybody gets emergency messages
> > *.emerg *
> >
> > # Save news errors of level crit and higher in a special file.
> > uucp,news.crit
> >  -/var/log/spooler
> >
> > # Save boot messages also to boot.log
> > local7.*
> >/var/log/boot.log
> >
> > # Remote Logging (we use TCP for reliable delivery)
> > # An on-disk queue is created for this action. If the remote host is
> > # down, messages are spooled to disk and sent when it is up again.
> > #$WorkDirectory /rsyslog/spool # where to place spool files
> > #$ActionQueueFileName uniqName # unique name prefix for spool files
> > #$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as
> > possible)
> > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> > #$ActionQueueType LinkedList   # run asynchronously
> > #$ActionResumeRetryCount -1# infinite retries if host is down
> > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
> > #*.* @@remote-host:514
> >
> >
> > # # Receiving Messages from Remote Hosts ##
> > # TCP Syslog Server:
> > # provides TCP syslog reception and GSS-API (if compiled to support
> > it)
> > $ModLoad imtcp.so  # load module
> > $InputTCPServerRun 514 # start up TCP listener at port 514
> >
> > # UDP Syslog Server:
> > $ModLoad imudp.so  # provides UDP syslog reception
> > $UDPServerRun 514 # start a UDP syslog server at standard port 514
> >
> >
> > $template Default,"/data/syslog/%HOSTNAME%/%HOSTNAME%.log"
> > *.* ?Default
> >
> >
> > [root@hostname]#
> >
> >
> > What's a good way to look at message ratE?
> >
> >
> > :Luke Marrott
> >
> >
> >
> > On Fri, Nov 9, 2012 at 1:03 PM, David Lang  wrote:
> >
> > > On Fri, 9 Nov 2012, Luke Marrott wrote:
> > >
> > >  Sorry. I wasn't real clear. The server runs on a big VM in another
> > >> location
> > >> completely. No issues with the server during this time. This has
> > >> been an
> > >> ongoing thing. I'm running Splunk on the same box and if I turn
> > >> off
> > >> rsyslog
> > >> and turn splunk on the same port it gets all the messages that
> > &g

Re: [rsyslog] rsyslog dropping logs

2012-11-09 Thread Rick Brown 
Add: 

$ModLoad impstats

to the top of your rsyslog.conf and restart.   Then look for lines like:

Nov  9 14:58:29 scribe1 [syslog.info<46>] rsyslogd-pstats:main Q: size=3771664 
enqueued=2498832464 full=22295 maxqsize=800 

every 5 minutes.   Subtract the smaller enqueued value from the larger, divide 
by 300, and that's your rate per second. 

- Original Message -
> From: "Luke Marrott" 
> To: "rsyslog-users" 
> Sent: Friday, November 9, 2012 3:07:02 PM
> Subject: Re: [rsyslog] rsyslog dropping logs
> 
> Full configuration:
> [root@hostname]# cat /etc/rsyslog.conf
> # if you experience problems, check
> # http://www.rsyslog.com/troubleshoot for assistance
> 
> # rsyslog v3: load input modules
> # If you do not load inputs, nothing happens!
> # You may need to set the module load path if modules are not found.
> 
> $ModLoad immark   # provides --MARK-- message capability
> $ModLoad imuxsock # provides support for local system logging (e.g.
> via
> logger command)
> $ModLoad imklog   # kernel logging (formerly provided by rklogd)
> 
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
> 
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> #*.error;mail.none;authpriv.none;cron.none
>/var/log/messages
> 
> # The authpriv file has restricted access.
> authpriv.*
>  /var/log/secure
> 
> # Log all the mail messages in one place.
> mail.*
>  -/var/log/maillog
> 
> 
> # Log cron stuff
> cron.*
>  -/var/log/cron
> 
> # Everybody gets emergency messages
> *.emerg *
> 
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit
>  -/var/log/spooler
> 
> # Save boot messages also to boot.log
> local7.*
>/var/log/boot.log
> 
> # Remote Logging (we use TCP for reliable delivery)
> # An on-disk queue is created for this action. If the remote host is
> # down, messages are spooled to disk and sent when it is up again.
> #$WorkDirectory /rsyslog/spool # where to place spool files
> #$ActionQueueFileName uniqName # unique name prefix for spool files
> #$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as
> possible)
> #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> #$ActionQueueType LinkedList   # run asynchronously
> #$ActionResumeRetryCount -1# infinite retries if host is down
> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
> #*.* @@remote-host:514
> 
> 
> # # Receiving Messages from Remote Hosts ##
> # TCP Syslog Server:
> # provides TCP syslog reception and GSS-API (if compiled to support
> it)
> $ModLoad imtcp.so  # load module
> $InputTCPServerRun 514 # start up TCP listener at port 514
> 
> # UDP Syslog Server:
> $ModLoad imudp.so  # provides UDP syslog reception
> $UDPServerRun 514 # start a UDP syslog server at standard port 514
> 
> 
> $template Default,"/data/syslog/%HOSTNAME%/%HOSTNAME%.log"
> *.* ?Default
> 
> 
> [root@hostname]#
> 
> 
> What's a good way to look at message ratE?
> 
> 
> :Luke Marrott
> 
> 
> 
> On Fri, Nov 9, 2012 at 1:03 PM, David Lang  wrote:
> 
> > On Fri, 9 Nov 2012, Luke Marrott wrote:
> >
> >  Sorry. I wasn't real clear. The server runs on a big VM in another
> >> location
> >> completely. No issues with the server during this time. This has
> >> been an
> >> ongoing thing. I'm running Splunk on the same box and if I turn
> >> off
> >> rsyslog
> >> and turn splunk on the same port it gets all the messages that
> >> don't seem
> >> to get picked up by rsyslog.
> >>
> >> Doesn't appear to be any rate limiting configuration.
> >>
> >
> > Ok, that is a different situation. In my experience, rsyslog is
> > signicantly better than Splunk at receiving messages. I've testing
> > rsyslog
> > up to 380K messages/sec (gige wire speed) and others have tested
> > rsyslog up
> > to 1M messages/sec, so it's unlikely to be something fundamental to
> > rsyslog, but it could easily be some resource contraint you are
> > running
> > into.
> >
> > can you post your full configuration?
> >

Re: [rsyslog] rsyslog dropping logs

2012-11-09 Thread Luke Marrott 
Full configuration:
[root@hostname]# cat /etc/rsyslog.conf
# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

# rsyslog v3: load input modules
# If you do not load inputs, nothing happens!
# You may need to set the module load path if modules are not found.

$ModLoad immark   # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog   # kernel logging (formerly provided by rklogd)

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.error;mail.none;authpriv.none;cron.none/var/log/messages

# The authpriv file has restricted access.
authpriv.*  /var/log/secure

# Log all the mail messages in one place.
mail.*  -/var/log/maillog


# Log cron stuff
cron.*  -/var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit  -/var/log/spooler

# Save boot messages also to boot.log
local7.*/var/log/boot.log

# Remote Logging (we use TCP for reliable delivery)
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /rsyslog/spool # where to place spool files
#$ActionQueueFileName uniqName # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1# infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514


# # Receiving Messages from Remote Hosts ##
# TCP Syslog Server:
# provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imtcp.so  # load module
$InputTCPServerRun 514 # start up TCP listener at port 514

# UDP Syslog Server:
$ModLoad imudp.so  # provides UDP syslog reception
$UDPServerRun 514 # start a UDP syslog server at standard port 514


$template Default,"/data/syslog/%HOSTNAME%/%HOSTNAME%.log"
*.* ?Default


[root@hostname]#


What's a good way to look at message ratE?


:Luke Marrott



On Fri, Nov 9, 2012 at 1:03 PM, David Lang  wrote:

> On Fri, 9 Nov 2012, Luke Marrott wrote:
>
>  Sorry. I wasn't real clear. The server runs on a big VM in another
>> location
>> completely. No issues with the server during this time. This has been an
>> ongoing thing. I'm running Splunk on the same box and if I turn off
>> rsyslog
>> and turn splunk on the same port it gets all the messages that don't seem
>> to get picked up by rsyslog.
>>
>> Doesn't appear to be any rate limiting configuration.
>>
>
> Ok, that is a different situation. In my experience, rsyslog is
> signicantly better than Splunk at receiving messages. I've testing rsyslog
> up to 380K messages/sec (gige wire speed) and others have tested rsyslog up
> to 1M messages/sec, so it's unlikely to be something fundamental to
> rsyslog, but it could easily be some resource contraint you are running
> into.
>
> can you post your full configuration?
>
> what message rate are you seeing?
>
>
> David Lang
> __**_
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog
> http://www.rsyslog.com/**professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog dropping logs

2012-11-09 Thread David Lang 

On Fri, 9 Nov 2012, Luke Marrott wrote:


Sorry. I wasn't real clear. The server runs on a big VM in another location
completely. No issues with the server during this time. This has been an
ongoing thing. I'm running Splunk on the same box and if I turn off rsyslog
and turn splunk on the same port it gets all the messages that don't seem
to get picked up by rsyslog.

Doesn't appear to be any rate limiting configuration.


Ok, that is a different situation. In my experience, rsyslog is signicantly 
better than Splunk at receiving messages. I've testing rsyslog up to 380K 
messages/sec (gige wire speed) and others have tested rsyslog up to 1M 
messages/sec, so it's unlikely to be something fundamental to rsyslog, but it 
could easily be some resource contraint you are running into.


can you post your full configuration?

what message rate are you seeing?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog dropping logs

2012-11-09 Thread Luke Marrott 
Sorry. I wasn't real clear. The server runs on a big VM in another location
completely. No issues with the server during this time. This has been an
ongoing thing. I'm running Splunk on the same box and if I turn off rsyslog
and turn splunk on the same port it gets all the messages that don't seem
to get picked up by rsyslog.

Doesn't appear to be any rate limiting configuration.

Thanks!

:Luke Marrott



On Fri, Nov 9, 2012 at 12:54 PM, David Lang  wrote:

> On Fri, 9 Nov 2012, Luke Marrott wrote:
>
>  So I'm wondering if someone can take a look at my configuration and tell
>> me
>> why rsyslog doesn't seem to be logging everything that gets sent to the
>> server?
>>
>> I'm listening on 514 and when I do a TCP dump I see the message come in,
>> but for example this morning we had a power outage and the switch logged
>> 50-60 messages and rsyslog only logged one of them.
>>
>> Here is the specific lines in my config. Let me know if you need more
>> info.
>>
>> $template Default,"/data/syslog/%**HOSTNAME%/%HOSTNAME%.log"
>> *.* ?Default
>>
>
> did the rsyslog box loose power? if so, the problem is probably just that
> rsyslog and the OS didn't get the logs to disk before power was lost.
>
> there is nothing obviously wrong with the snippet of config that you are
> showing. That doesn't mean that something else in the config isn't causing
> the problem.
>
> Do you have any rate limiting enabled for example?
>
> David Lang
> __**_
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog
> http://www.rsyslog.com/**professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog dropping logs

2012-11-09 Thread David Lang 

On Fri, 9 Nov 2012, Luke Marrott wrote:


So I'm wondering if someone can take a look at my configuration and tell me
why rsyslog doesn't seem to be logging everything that gets sent to the
server?

I'm listening on 514 and when I do a TCP dump I see the message come in,
but for example this morning we had a power outage and the switch logged
50-60 messages and rsyslog only logged one of them.

Here is the specific lines in my config. Let me know if you need more info.

$template Default,"/data/syslog/%HOSTNAME%/%HOSTNAME%.log"
*.* ?Default


did the rsyslog box loose power? if so, the problem is probably just that 
rsyslog and the OS didn't get the logs to disk before power was lost.


there is nothing obviously wrong with the snippet of config that you are 
showing. That doesn't mean that something else in the config isn't causing the 
problem.


Do you have any rate limiting enabled for example?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog dropping logs

2008-08-12 Thread Mike McGrath 
On Thu, 24 Jul 2008, Rainer Gerhards wrote:

> (I am not commenting on v2 vs. v3 as this is already done)
>
> First of all, we need to keep in mind that UDP is inherently lossy. Even
> when a frame is seen received by the local stack, it does not mean that
> it will eventually be forwarded to the application. If message bursts
> come in very quickly and the OS scheduler does not schedule the app fast
> enough to receive this messages (or the app is too slow in itself! ;))
> new frames may overwrite frames inside the stack's receive buffers.
>
> So it is always a good idea to avoid UDP if that's possible.
>
> HOWEVER, I, too, find it somewhat unusual that around 90% of all
> incoming frames are discarded before the rsyslog receiver could process
> them. One explanation I have is that you have bursts (or volume in
> general) that outperforms the configured actions. Having seen the config
> file, and seeing it does not include any database writer, it is hard to
> imagine this should happen, assuming reasonable hardware sizing is used.
> A cause could be excessive synchronous writes. Many rules do not put a
> dash in front of the file name and without it (in v2), every write is
> immediately synced. This is very costly. But still, I have never seen
> that this alone outperforms a system.
>
> To dig deeper into what is happening, a debug log would be most useful,
> together with the information which frames have been seen in tcpdump but
> NOT in one of the log files. You can enable debug mode via -dn command
> line switch and is recommended to run rsyslog interactively while doing
> so. Then, you can simply capture its output via stdout redirection.
> Please note that debug mode generates considerable output, and requires
> considerable additional processing time. In any case, though, it should
> show us where the bottleneck is. Please note that I need a consistent
> excerpt from the debug log that shows how things began and how it worked
> during the fault conditions. Usually, this means I need everything ;)
> Debug logs may also reveal sensitive information, even passwords, so you
> should be careful in what you do. I am used to log files around the size
> of 1GB. With reasonable compression, the transfer is usually not a
> problem (but I suggest you place them on a server for me to download).
> Download links and/or smaller logs you can email me privately at
> [EMAIL PROTECTED] (please NOT at my primary, adiscon, email address).
>
> I hope this helps and I am looking forward for the additional
> information.

So after a long hiatus and a new config the problems went away but only
temporarily.  I think because of a rebooted box.  They have returned.  I'm
going to mail the logs your way.  I can send you more.  Its very easy to
reproduce.

-Mike









== Snip ===
>
> On Wed, 2008-07-23 at 14:21 -0500, Mike McGrath wrote:
> > I've got a RHEL5.2 host with rsyslog-2.0.0-11 installed as a central
> > logging server.  When running tcpdump I'm seeing all the udp packets
> > coming in but many of them are not getting logged.  And we're talking
> > like 10% or so getting logged (maybe less) and the rest are just lost.
> > I've attached my config file.
> >
> > (side note, if I'm doing something stupid in the config please correct me)
> >
> > -Mike
> > ___ rsyslog mailing list 
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] rsyslog dropping logs

2008-07-24 Thread Rainer Gerhards 
(I am not commenting on v2 vs. v3 as this is already done)

First of all, we need to keep in mind that UDP is inherently lossy. Even
when a frame is seen received by the local stack, it does not mean that
it will eventually be forwarded to the application. If message bursts
come in very quickly and the OS scheduler does not schedule the app fast
enough to receive this messages (or the app is too slow in itself! ;))
new frames may overwrite frames inside the stack's receive buffers.

So it is always a good idea to avoid UDP if that's possible.

HOWEVER, I, too, find it somewhat unusual that around 90% of all
incoming frames are discarded before the rsyslog receiver could process
them. One explanation I have is that you have bursts (or volume in
general) that outperforms the configured actions. Having seen the config
file, and seeing it does not include any database writer, it is hard to
imagine this should happen, assuming reasonable hardware sizing is used.
A cause could be excessive synchronous writes. Many rules do not put a
dash in front of the file name and without it (in v2), every write is
immediately synced. This is very costly. But still, I have never seen
that this alone outperforms a system.

To dig deeper into what is happening, a debug log would be most useful,
together with the information which frames have been seen in tcpdump but
NOT in one of the log files. You can enable debug mode via -dn command
line switch and is recommended to run rsyslog interactively while doing
so. Then, you can simply capture its output via stdout redirection.
Please note that debug mode generates considerable output, and requires
considerable additional processing time. In any case, though, it should
show us where the bottleneck is. Please note that I need a consistent
excerpt from the debug log that shows how things began and how it worked
during the fault conditions. Usually, this means I need everything ;)
Debug logs may also reveal sensitive information, even passwords, so you
should be careful in what you do. I am used to log files around the size
of 1GB. With reasonable compression, the transfer is usually not a
problem (but I suggest you place them on a server for me to download).
Download links and/or smaller logs you can email me privately at
[EMAIL PROTECTED] (please NOT at my primary, adiscon, email address).

I hope this helps and I am looking forward for the additional
information.

Rainer


On Wed, 2008-07-23 at 14:21 -0500, Mike McGrath wrote:
> I've got a RHEL5.2 host with rsyslog-2.0.0-11 installed as a central
> logging server.  When running tcpdump I'm seeing all the udp packets
> coming in but many of them are not getting logged.  And we're talking
> like 10% or so getting logged (maybe less) and the rest are just lost.
> I've attached my config file.
> 
> (side note, if I'm doing something stupid in the config please correct me)
> 
>   -Mike
> ___ rsyslog mailing list 
> http://lists.adiscon.net/mailman/listinfo/rsyslog

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] rsyslog dropping logs

2008-07-23 Thread Mike McGrath 
On Wed, 23 Jul 2008, Kielek, Samuel wrote:

> Mike,
>
> You're using some v3 features that will not work with v2 such as the
> conditional filtering. I'll email you off list with a copy of my config
> (also using RHEL 5.2 here).
>

Thanks, I upgraded to a v3 version with my old config, same issue.  Used
the config provided off list and it works great.  even in v3.

-Mike
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] rsyslog dropping logs

2008-07-23 Thread Kielek, Samuel 
Mike,

You're using some v3 features that will not work with v2 such as the
conditional filtering. I'll email you off list with a copy of my config
(also using RHEL 5.2 here).

-Sam 

-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Mike McGrath
Sent: Wednesday, July 23, 2008 3:22 PM
To: rsyslog@lists.adiscon.com
Subject: [rsyslog] rsyslog dropping logs


I've got a RHEL5.2 host with rsyslog-2.0.0-11 installed as a central
logging server.  When running tcpdump I'm seeing all the udp packets
coming in but many of them are not getting logged.  And we're talking
like 10% or so getting logged (maybe less) and the rest are just lost.
I've attached my config file.

(side note, if I'm doing something stupid in the config please correct
me)

-Mike
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog