[rsyslog] imjournal: How can we avoid this module?

2017-07-21 Thread Mike Schleif 
CentOS Linux release 7.3.1611 (Core)
rsyslog.x86_64  8.28.0-1.el7
@rsyslog_v8
rsyslog-mysql.x86_648.28.0-1.el7
@rsyslog_v8

This week, after upgrading from 8.24 to 8.28, we noticed errors related to:
$OmitLocalLogging on

and:
$SystemLogSocketName /run/systemd/journal/syslog

We have removed both of them, and errors are gone.

Is the following configuration no longer supported?
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/s1-interaction_of_rsyslog_and_journal.html

How can we avoid imjournal on our systems?

~ Mike
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] ommysql: How to eliminate egregious DB write delays?

2017-07-21 Thread Mike Schleif 
After lunch, the delay between host events and DB write is > 15 minutes.

There are zero queue files under /var/lib/rsyslog. Where are my missing
events?

I'm studying impstats log and I find the following.

### grep ^2017-07-21T /var/log/rsyslog-stats | grep enqueued | grep -v
enqueued=0

2017-07-21T00:09:23.415219-05:00 hermes rsyslogd-pstats: action 10 queue:
origin=core.queue size=0 enqueued=38811 full=0 discarded.full=0 discarded.nf=0
maxqsize=65
2017-07-21T00:09:23.415228-05:00 hermes rsyslogd-pstats: action 11 queue:
origin=core.queue size=0 enqueued=212 full=0 discarded.full=0 discarded.nf=0
maxqsize=6
2017-07-21T00:09:23.415238-05:00 hermes rsyslogd-pstats: action 12 queue:
origin=core.queue size=0 enqueued=1912 full=0 discarded.full=0 discarded.nf=0
maxqsize=882
2017-07-21T00:09:23.415243-05:00 hermes rsyslogd-pstats: main Q:
origin=core.queue size=24 enqueued=38835 full=0 discarded.full=0
discarded.nf=0 maxqsize=65
2017-07-21T00:19:23.428644-05:00 hermes rsyslogd-pstats: action 10 queue:
origin=core.queue size=0 enqueued=39291 full=0 discarded.full=0 discarded.nf=0
maxqsize=65
2017-07-21T00:19:23.428653-05:00 hermes rsyslogd-pstats: action 11 queue:
origin=core.queue size=0 enqueued=212 full=0 discarded.full=0 discarded.nf=0
maxqsize=6
2017-07-21T00:19:23.428663-05:00 hermes rsyslogd-pstats: action 12 queue:
origin=core.queue size=0 enqueued=1912 full=0 discarded.full=0 discarded.nf=0
maxqsize=882
2017-07-21T00:19:23.428668-05:00 hermes rsyslogd-pstats: main Q:
origin=core.queue size=24 enqueued=39315 full=0 discarded.full=0
discarded.nf=0 maxqsize=65
2017-07-21T00:29:23.528895-05:00 hermes rsyslogd-pstats: action 10 queue:
origin=core.queue size=0 enqueued=39773 full=0 discarded.full=0 discarded.nf=0
maxqsize=65
2017-07-21T00:29:23.528904-05:00 hermes rsyslogd-pstats: action 11 queue:
origin=core.queue size=0 enqueued=212 full=0 discarded.full=0 discarded.nf=0
maxqsize=6
2017-07-21T00:29:23.528913-05:00 hermes rsyslogd-pstats: action 12 queue:
origin=core.queue size=0 enqueued=1912 full=0 discarded.full=0 discarded.nf=0
maxqsize=882
2017-07-21T00:29:23.528918-05:00 hermes rsyslogd-pstats: main Q:
origin=core.queue size=24 enqueued=39797 full=0 discarded.full=0
discarded.nf=0 maxqsize=65
2017-07-21T00:39:23.559652-05:00 hermes rsyslogd-pstats: action 10 queue:
origin=core.queue size=0 enqueued=40287 full=0 discarded.full=0 discarded.nf=0
maxqsize=65
2017-07-21T00:39:23.559663-05:00 hermes rsyslogd-pstats: action 11 queue:
origin=core.queue size=0 enqueued=212 full=0 discarded.full=0 discarded.nf=0
maxqsize=6
2017-07-21T00:39:23.559673-05:00 hermes rsyslogd-pstats: action 12 queue:
origin=core.queue size=0 enqueued=1912 full=0 discarded.full=0 discarded.nf=0
maxqsize=882
2017-07-21T00:39:23.559678-05:00 hermes rsyslogd-pstats: main Q:
origin=core.queue size=24 enqueued=40311 full=0 discarded.full=0
discarded.nf=0 maxqsize=65
2017-07-21T00:49:23.606583-05:00 hermes rsyslogd-pstats: action 10 queue:
origin=core.queue size=0 enqueued=40855 full=0 discarded.full=0 discarded.nf=0
maxqsize=65
2017-07-21T00:49:23.606591-05:00 hermes rsyslogd-pstats: action 11 queue:
origin=core.queue size=0 enqueued=212 full=0 discarded.full=0 discarded.nf=0
maxqsize=6
2017-07-21T00:49:23.606602-05:00 hermes rsyslogd-pstats: action 12 queue:
origin=core.queue size=0 enqueued=1912 full=0 discarded.full=0 discarded.nf=0
maxqsize=882
2017-07-21T00:49:23.606607-05:00 hermes rsyslogd-pstats: main Q:
origin=core.queue size=24 enqueued=40879 full=0 discarded.full=0
discarded.nf=0 maxqsize=65
2017-07-21T00:59:23.656640-05:00 hermes rsyslogd-pstats: action 10 queue:
origin=core.queue size=0 enqueued=41409 full=0 discarded.full=0 discarded.nf=0
maxqsize=65
2017-07-21T00:59:23.656648-05:00 hermes rsyslogd-pstats: action 11 queue:
origin=core.queue size=0 enqueued=212 full=0 discarded.full=0 discarded.nf=0
maxqsize=6
2017-07-21T00:59:23.656658-05:00 hermes rsyslogd-pstats: action 12 queue:
origin=core.queue size=0 enqueued=1912 full=0 discarded.full=0 discarded.nf=0
maxqsize=882
2017-07-21T00:59:23.656663-05:00 hermes rsyslogd-pstats: main Q:
origin=core.queue size=24 enqueued=41433 full=0 discarded.full=0
discarded.nf=0 maxqsize=65
2017-07-21T01:09:23.660590-05:00 hermes rsyslogd-pstats: action 10 queue:
origin=core.queue size=0 enqueued=42123 full=0 discarded.full=0 discarded.nf=0
maxqsize=65
2017-07-21T01:09:23.660598-05:00 hermes rsyslogd-pstats: action 11 queue:
origin=core.queue size=0 enqueued=212 full=0 discarded.full=0 discarded.nf=0
maxqsize=6
2017-07-21T01:09:23.660608-05:00 hermes rsyslogd-pstats: action 12 queue:
origin=core.queue size=44 enqueued=1956 full=0 discarded.full=0 discarded.nf=0
maxqsize=882
2017-07-21T01:09:23.660613-05:00 hermes rsyslogd-pstats: main Q:
origin=core.queue size=24 enqueued=42147 full=0 discarded.full=0
discarded.nf=0 maxqsize=65
2017-07-21T01:19:23.662644-05:00 hermes rsyslogd-pstats: action 10 queue:
origin=core.queue size=0 enqueued=42677 full=0 discarded.full=0 discarded.nf=0

Re: [rsyslog] ommysql: How to eliminate egregious DB write delays?

2017-07-21 Thread Mike Schleif 
So I noticed this entry pop up in the journalctl stream:

Jul 21 10:45:38 hermes.provell.com rsyslogd[11456]: queue 'strm
0x7f610c688810', file '/var/lib/rsyslog/dbSftpQueue.0001' opened for
non-append write, but already contains 25231 bytes  [v8.28.0.master try
http://www.rsyslog.com/e/0 ]

I knew that I was missing several SSH events in the DB; so, I ran this:
# ls -lrt /var/lib/rsyslog
total 4
-rw--- 1 root root 125 Jul 21 10:45 imjournal.state

No queue found - so, I restarted:

# /bin/systemctl restart rsyslog

Now, there are queue files:

# ls -lrt /var/lib/rsyslog
total 36
-rw--- 1 root root   125 Jul 21 10:45 imjournal.state
-rw--- 1 root root 25231 Jul 21 10:45 dbSftpQueue.0001
-rw--- 1 root root   532 Jul 21 10:45 dbSftpQueue.qi

Restart again:

# /bin/systemctl restart rsyslog

The queue files are gone AND I have have the missing events in DB:

# ls -lrt /var/lib/rsyslog
total 4
-rw--- 1 root root 125 Jul 21 10:46 imjournal.state

What am I missing?

~ Mike
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Imfile module parameters not known

2017-07-21 Thread Luv via rsyslog 
Corrected all these

Here is all the config file, 

template(name="all-json"
type="list"){
property(name="$!all-json")
 }

input(type="imfile"
File="/var/log/nginx/infotrack_access.log.5.gz"  
StateFile="/var/spool/rsyslog/statefile1" 
Tag="nginx"  
Severity="info" 
Facility="local7"
ruleset="nginxoldruleset"
)


ruleset(
 name="nginxoldruleset"
 queue.type="FixedArray"   
 queue.highwatermark="500" 
 queue.spoolDirectory="/var/run/rsyslog/queues" 
 queue.filename="stats_ruleset"
 queue.lowwatermark="2"
 queue.maxdiskspace="100m"
 queue.size="5000"   
 queue.dequeuebatchsize="1000"  
 queue.saveonshutdown="on" 
 )
{


action(
  type="mmnormalize"  
  rulebase="/opt/rsyslog/nginx-old-logs.rb"
)

action(
  name="oldlogs"
  type="omelasticsearch"
  server="xx.xx.xx.xx"
  serverport="9200"
  template="all-json"
  searchIndex="alpha-nginx-olg-logs"
  searchType="nginx"
  bulkmode="on"
  action.resumeretrycount="-1"
)

}





--
View this message in context: 
http://rsyslog-users.1305293.n2.nabble.com/Imfile-module-parameters-not-known-tp7592623p7592631.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Changes to CONF result in ZERO SSH logging?

2017-07-21 Thread Rainer Gerhards 
Sent from phone, thus brief.

Am 21.07.2017 16:18 schrieb "David Lang" :

On Fri, 21 Jul 2017, Mike Schleif wrote:

 GLOBAL DIRECTIVES 
> # Use default timestamp format
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> How ought I do that under globale directives?
>

There is no way to set defaults for the new format, you must specify things
in each action() call.


You can set defaults with the module statement. The doc has it all:

http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfile.html

Rainer



You should have a complaint at startup because of this, but rsyslog cannot
>> write any log messages as a result of this error.
>>
>
>
> No, there is no complaint I see at startup.
>
> Why will this suppress SSH logging (authpriv), but, not suppress any other
> logging?
>

no idea off the top of my head, I would expect all logging to stop.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Changes to CONF result in ZERO SSH logging?

2017-07-21 Thread David Lang 

On Fri, 21 Jul 2017, Mike Schleif wrote:


 GLOBAL DIRECTIVES 
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

How ought I do that under globale directives?


There is no way to set defaults for the new format, you must specify things in 
each action() call.



You should have a complaint at startup because of this, but rsyslog cannot
write any log messages as a result of this error.



No, there is no complaint I see at startup.

Why will this suppress SSH logging (authpriv), but, not suppress any other
logging?


no idea off the top of my head, I would expect all logging to stop.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Changes to CONF result in ZERO SSH logging?

2017-07-21 Thread Mike Schleif 
On Thu, Jul 20, 2017 at 7:06 PM, David Lang  wrote:

> On Thu, 20 Jul 2017, Mike Schleif wrote:
>
> action(type="omprog" template="RSYSLOG_TraditionalFileFormat")
>>
>
> If I am reading you correctly, you are telling rsyslog to output the log
> message to a program, but don't specify what program to send it to.
>

OK, I understand what you are saying. I will remove that line.

The intent of that line was to replace this legacy action:

 GLOBAL DIRECTIVES 
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

How ought I do that under globale directives?


> You should have a complaint at startup because of this, but rsyslog cannot
> write any log messages as a result of this error.


No, there is no complaint I see at startup.

Why will this suppress SSH logging (authpriv), but, not suppress any other
logging?

David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Imfile module parameters not known

2017-07-21 Thread mostolog--- via rsyslog 

you have two ")" at the end.. also 2 StateFile...review your config


On 21/07/17 15:06, Luv via rsyslog wrote:

This is the way I am reading logs from file,

input(type="imfile" File="/var/log/nginx/infotrack_access.log.5.gz"
StateFile="/var/spool/rsyslog/statefile1"
ruleset="nginxoldruleset"
Tag="nginx"
StateFile="/var/spool/rsyslog/statefile1"
Severity="info"
Facility="local7")
)



But I am getting these errors,

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'ruleset' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'Facility' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'Severity' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'StateFile' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'Tag' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'File' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]



It is saying that none of these parameters are known. Why so ?

Here are the 2 links I read,

http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html

http://www.rsyslog.com/using-the-text-file-input-module/



--
View this message in context: 
http://rsyslog-users.1305293.n2.nabble.com/Imfile-module-parameters-not-known-tp7592623.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Imfile module parameters not known

2017-07-21 Thread Luv via rsyslog 
This is the way I am reading logs from file, 

input(type="imfile" File="/var/log/nginx/infotrack_access.log.5.gz"
StateFile="/var/spool/rsyslog/statefile1"
ruleset="nginxoldruleset"
Tag="nginx"
StateFile="/var/spool/rsyslog/statefile1"
Severity="info" 
Facility="local7")
)



But I am getting these errors, 

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'ruleset' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'Facility' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'Severity' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'StateFile' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'Tag' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'File' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]



It is saying that none of these parameters are known. Why so ?

Here are the 2 links I read, 

http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html

http://www.rsyslog.com/using-the-text-file-input-module/



--
View this message in context: 
http://rsyslog-users.1305293.n2.nabble.com/Imfile-module-parameters-not-known-tp7592623.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.