date:20161221

Alright! Almost out the door for the holiday's and swore i would come back to
this before I left in the event anyone stumbled across it. Note that this is
intended to primary provide information on using the current iptables data
type in version 2 of liblognorm.

For reference tests were conducted on CentOS Linux Release 7.2.1511 with
liblognorm5-2.0.2-1 and liblognorm5-utils-2.0.2-1 installed from the Addison
repository.

The documentation for liblognorm is significantly lacking for its current
version and the iptables section demonstrates that severely. In version 2.x
the data type "iptables" no longer exists, instead it has been replaced by
its predecessor "v2-iptables". As per the documentation "v2-iptables" works
exactly as stated:

"Name=value pairs, separated by spaces, as in Netfilter log messages. Name
of the selector is not used; names from the line are used instead. This
selector always matches everything till end of the line. Cannot match zero
characters."

However there are some additional restrictions that are not mentioned. These
can be seen when checking out the test files under
"https://github.com/rsyslog/liblognorm/blob/master/tests; and they are as
follows:

1. Key names can only be uppercase
2. Key name can only contain alphabetic characters (no special characters or
numbers)
3. All Key-value pairs must be separated by a single space
4. A minimum of two pairs are required; a single key-value pair will not
work
5. value will contain all characters until the next space, this includes any
notation like brackets that are not necessarily part of said value.

A quick working test:
test.rb:
version=2
rule=:%field:v2-iptables%

lognormalizer:
echo 'KA=test1 KB="test2"' | lognormalizer -e json -r test.rb

It appears that iptables was built for a specific use case, aka Netfilter,
and then never adapted to be more applicable to key-value's. Other data
types exist like "cee" and "cef" who more accurately act as expected, though
when i started i simple assumed that they in themselves made use of the
iptables data type.

The problem now is that other formats that utilize key-value pairs cant be
parsed, especially as most of the time they do not come in any particular
order. One could possible use the "alternative" and "repeat" types to solve
the problem but this A) seems incredibly inefficient and B) is limited to
know key names. The more arbitrary the key-value output the harder it
becomes. CEF is very good example of this, though, lucky we have a parser
for it. The problem is when we start to look similar formats like MEF, LEEF,
or Fortinet logs.

CEF and CEE data types are locked down to a very specfic header that can not
be altered. In terms of CEF this also problematic because the RFC3164 parser
will often extract the "CEF:" part of the message as the syslog tag thus
making it difficult (not impossible) to use the "CEF" data type.

*Note: a new CEF version is expected sometime next year

IMO it seems that the CEE and CEF specific data types shouldn't even exist,
or if they do, then only as an abstraction; something like:

type=@CEF:CEF:0|%vendor:chart-to:|%|%product:chart-to:|%|%version:chart-to:|%|%signature:chart-to:|%|%name:chart-to:|%|%severity:chart-to:|%|%extensions:iptables%

It would also be advantages to assume that the key-value pair might not
always be separated by a single space, or have values encased in standard
brackets (ie. ", '). Having advanced options to alter those would increase
its flexibility.

I also do not necessarily agree with the the whole "until end of line"
portion. I see that as something that should only ever apply when using the
data type "rest" as it is feasible that a portion contain key-values (or
other type) but does is proceed by something else.

At present I do not have any examples but could probably dig up some vendor
sources/docs that might. I have seen in the past custom client applications
do awkward things as well as seeing key-value pair submitted encased in
other formats; tho i cant really supply those as examples :(

Thanks

~Regards

-
~Regards

Matthew Gaetano
--
View this message in context:
http://rsyslog-users.1305293.n2.nabble.com/Liblognorm-Usage-of-field-type-iptables-tp7591441p7592021.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.

Re: [rsyslog] Assert Failure in imtcp.c?


On Wed, 21 Dec 2016, Andrew Griffin via rsyslog wrote:

Attached is a full debug log (hostnames and dir names scrubbed).  But the 
basic setup is this - rsyslog listens on UDP, plain TCP, TLS and RELP ports, 
parses the syslogtag and outputs to a Kafka topic of the same name via 
omkafka. UDP, TCP and RELP work as intended, and the issue occurs on both 8.22 
and 8.23.  In my opinion it’s likely a configuration issue


we have a known issue (fixed in the git master, so it will be in 8.24) where you 
can have problems if you don't have imuxsock configured.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Deleting already "processed" files


On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:


To contextualize:

A server is generating multiple log files and sending them using RELP to a 
Rsyslog relay. I would love to automatically delete older logs files if they 
are already sent/acknowledged.



Is there a way to make Rsyslog delete a file after successfully sending it 
via RELP? (and not modified in the last...day?)


Is there an easy way to know, outside from Rsyslog, if a file has already 
been processed/sent, in order to delete the local file?


Maybe an script which gets "reading offset" from state file and compares with 
file size...


Rsyslog can't really know when a file isn't going to be written to any longer. 
It doesn't even know if another program has the file open to write to now.


In your situation, I would use a script with logger to write the messages to 
/dev/log for rsyslog to process. By spec, the write to /dev/log will not 
complete until rsyslog has the message in it's queue. So your script can feed 
the log to rsyslog and remove it immediatly after it's completed.


I'd put in an enhancement request for an option to imfile to delete files when 
it's finished reading them. It was created with the assumption that it's reading 
the file as it's being written, but I can see useful cases where the files are 
made available to rsyslog (mv into a watched directory) after they are complete.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog is recording more (~ 40 extra) log messages than expected from remote client


On Wed, 21 Dec 2016, rsyslog-users-lists.adiscon@whyaskwhy.org wrote:


On 12/21/2016 10:43 AM, David Lang wrote:

On Wed, 21 Dec 2016, rsyslog-users-lists.adiscon@whyaskwhy.org wrote:

you can also set the retries so that it will only attempt to deliver a 
message X times before it gives up on that message.


Thanks for mentioning that. Is there a setting or command-line option that 
you know of to dump all stock settings and current settings to the 
screen/file? Something like 'postconf -d' or 'postconf' ?


I don't think so (there may be something buried in the debug options, but 
that would be the only place for it)


There are a LOT of possible options, and the vast majority are not touched 
by 99.99%+ of people.


Understood. Aside from explicitly setting a value, is there any other way to 
know what a value is set to? For example, the watermark values that you 
mentioned earlier. I guess the safest approach is to be explicit with all of 
the values.


The documentation always lists the default value of an option.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Assert Failure in imtcp.c?

2016-12-21 Thread Andrew Griffin via rsyslog

Issue opened:

https://github.com/rsyslog/rsyslog/issues/1343 


Andrew Griffin
  ETS / Integration Services
☏ 408-783-8348

> On Dec 21, 2016, at 9:04 AM, Andrew Griffin via rsyslog 
>  wrote:
> 
> Attached is a full debug log (hostnames and dir names scrubbed).  But the 
> basic setup is this - rsyslog listens on UDP, plain TCP, TLS and RELP ports, 
> parses the syslogtag and outputs to a Kafka topic of the same name via 
> omkafka. UDP, TCP and RELP work as intended, and the issue occurs on both 
> 8.22 and 8.23.  In my opinion it’s likely a configuration issue
> 
> My configuration is setup in a straight forward include structure, my main 
> rsyslog.conf file looks like:
> 
> $IncludeConfig 
> //managed_by_ansible/rsyslog/etc/rsyslog.d/globals/*.conf
> $IncludeConfig 
> //managed_by_ansible/rsyslog/etc/rsyslog.d/modules/*.conf
> $IncludeConfig 
> //managed_by_ansible/rsyslog/etc/rsyslog.d/templates/*.conf
> $IncludeConfig 
> //managed_by_ansible/rsyslog/etc/rsyslog.d/inputs/*.conf
> $IncludeConfig 
> //managed_by_ansible/rsyslog/etc/rsyslog.d/rules/*.conf
> $IncludeConfig //managed_by_ansible/rsyslog/etc/rsyslog.d/*.conf
> 
> I have a number of small conf files from there:
> 
> ./rsyslog.conf
> ./rsyslog.d
> ./rsyslog.d/globals
> ./rsyslog.d/globals/01_common.conf
> ./rsyslog.d/globals/02_main_queue.conf
> ./rsyslog.d/inputs
> ./rsyslog.d/inputs/01_udp.conf
> ./rsyslog.d/inputs/02_tcp.conf
> ./rsyslog.d/inputs/03_tls.conf
> ./rsyslog.d/inputs/04_relp.conf
> ./rsyslog.d/modules
> ./rsyslog.d/modules/01_imudp.conf
> ./rsyslog.d/modules/02_imtcp.conf
> ./rsyslog.d/modules/03_imtls.conf
> ./rsyslog.d/modules/04_imrelp.conf
> ./rsyslog.d/modules/05_imfile.conf
> ./rsyslog.d/modules/09_impstats.conf
> ./rsyslog.d/modules/20_omkafka.conf
> ./rsyslog.d/rules
> ./rsyslog.d/rules/00_rsyslogd_internal.conf
> ./rsyslog.d/rules/01_udp2kafka.conf
> ./rsyslog.d/rules/02_tcp2kafka.conf
> ./rsyslog.d/rules/03_tls2kafka.conf
> ./rsyslog.d/rules/04_relp2kafka.conf
> ./rsyslog.d/rules/50_sawmill_stats.conf
> ./rsyslog.d/templates
> ./rsyslog.d/templates/kakfa_json.conf
> ./rsyslog.d/templates/local_files.conf
> 
> The dump occurs during the loading of the TLS portion (imtcp), here’s the 
> conf files for the TLS config (scrubbed):
> 
> 03_tls.conf:
> 
> input(
>type= "imtcp"
>port= "10448"
>ruleset = "tls_10448"
> )
> 
> 03_imtls.conf:
> 
> module(
>load  = "imtcp"
>maxSessions   = "300"
>StreamDriver.Name = "gtls"
>StreamDriver.Mode = "1"
>StreamDriver.AuthMode = "x509/name"
>PermittedPeer = [ "*.apple.com", “*..apple.com", 
> “*..apple.com" ]
> )
> 
> 03_tls2kafka.conf:
> 
> ruleset(name = "tls_10448") {
>if $syslogtag startswith "sawmill_test_topic_tls" then {
>action(
>name  = "tls10448_sawmill_test_topic_tls_to_kafka"
>type  = "omkafka"
># Queue configuration - in-memory up to 976.5625M, then to disk up 
> to 1024M
>queue.spoolDirectory  = 
> “//managed_by_ansible/rsyslog/var/spool"
>queue.type= "LinkedList"
>queue.size= "100"
>queue.filename= "tls10448_sawmill_test_topic_tls_to_kafka"
>queue.maxdiskspace= "1024M"
>queue.saveonshutdown  = "off"
>queue.highwatermark   = "90"
>queue.lowwatermark= "40"
>queue.discardseverity = "5"
># Kafka configuration
>template  = "KAFKA_JsonLineTemplate"
>broker= [ “broker1:9092", “broker2:9092", 
> “broker3:9092", “broker4:9092", “broker5.corp.apple.com:9092" ]
>confParam = ["queue.buffering.max.messages=1"]
>topic = "sawmill_test_topic_tls"
>partitions.auto   = "on"
>)
>stop
>}
> }
> 
> 
> 
> 
> 
> Andrew Griffin
>   ETS / Integration Services
> ☏ 408-783-8348
> 
>> On Dec 21, 2016, at 12:05 AM, mostolog--- via rsyslog 
>>  wrote:
>> 
>> It will be great if you provide a replication use case:
>> 
>> * rsyslog version (may be 8.23)
>> * configuration file
>> * log example to replicate the issue.
>> 
>> If you are able to replicate the issue with a "not too old" syslog version, 
>> please file an issue on github
>> 
>> 
>> El 20/12/16 a las 21:14, Andrew Griffin via rsyslog escribió:
>>> I’m working setting up a syslog pipeline with rsyslog at the front, and I’m 
>>> running in to an issue with rsyslogd core dumping when I try to start it.  
>>> Debug log shows everything loading fine, then this:
>>> 
>>> 4473.240773513:imtcp.c: nspoll.c:147 ISOBJ assert failure: invalid 
>>> object type, expected 'netstrms' actual 'nspoll', cookie: BADEFEE
>>> rsyslogd: nspoll.c:147: SetDrvrName: Assertion `0'

[rsyslog] Deleting already "processed" files


Hi


To contextualize:

A server is generating multiple log files and sending them using RELP to 
a Rsyslog relay. I would love to automatically delete older logs files 
if they are already sent/acknowledged.



Is there a way to make Rsyslog delete a file after successfully sending 
it via RELP? (and not modified in the last...day?)


Is there an easy way to know, outside from Rsyslog, if a file has 
already been processed/sent, in order to delete the local file?


Maybe an script which gets "reading offset" from state file and compares 
with file size...



Regards




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog is recording more (~ 40 extra) log messages than expected from remote client

2016-12-21 Thread rsyslog-users-lists . adiscon . net


On 12/21/2016 10:43 AM, David Lang wrote:

On Wed, 21 Dec 2016, rsyslog-users-lists.adiscon@whyaskwhy.org wrote:

you can also set the retries so that it will only attempt to deliver 
a message X times before it gives up on that message.


Thanks for mentioning that. Is there a setting or command-line option 
that you know of to dump all stock settings and current settings to 
the screen/file? Something like 'postconf -d' or 'postconf' ?


I don't think so (there may be something buried in the debug options, 
but that would be the only place for it)


There are a LOT of possible options, and the vast majority are not 
touched by 99.99%+ of people.


Understood. Aside from explicitly setting a value, is there any other 
way to know what a value is set to? For example, the watermark values 
that you mentioned earlier. I guess the safest approach is to be 
explicit with all of the values.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] About reloading configuration


Someone called Rainer is going to have a busy January :P

David/Rainer review that docs I pulled so I can write more!


El 21/12/16 a las 17:48, Rainer Gerhards escribió:

I don't have the time to check now, but it may be easier than it looks. I
think we have pdag handles, which means we can easily load in the
background and apply when ready. However, we need a rwlock to guard this
operation (action level).

If it works that way, we could do it on HUP (for starters).

Rainer

2016-12-21 17:37 GMT+01:00 David Lang :


I think the headache with doing this in liblognorm is the overhead that
can be there in reading/parsing/optimizing the ruleset.

if you have multiple worker threads, having two of them hit the
mmnormalize action while one is rebuilding the ruleset leads to problems :-)

what would be needed is for all existing parsing to continue to use the
old ruleset until the new ruleset is fully parsed, optimized, and ready for
use.

We have a similar issue with the reload of a lookup_table and should be
able to use a similar solution.

One complicating factor is that while a lookup_table has a global
existance and is something the rsyslog core knows about, mmnormalize
rulesets are handled only by the mmnormalize module code, and there's no
clean way to refer to a specific ruleset outside of that code.

David Lang


On Wed, 21 Dec 2016, Benoit DOLEZ wrote:

Yes, I think you are right about reloading only liblognorm. It's probably

my first step.

In most case I think we need a simple rainer-script if/then/else using a
liblognorm app specific rule file. Then adding a new parser implies
reloading rsyslog...

Thank you for responses. I'm continuing to analyse source...

Regards

Benoit

Le 21/12/2016 à 10:33, mostolog--- via rsyslog a écrit :


You may also find interesting to be able to reload liblognorm rules,
which might be an smaller change.

The question here is the needed amount of work.


El 21/12/16 a las 10:29, Rainer Gerhards escribió:


Possible of course, but considerable work (gut feeling: 6+ strong
weeks).

Rainer

Sent from phone, thus brief.

Am 21.12.2016 09:42 schrieb "Benoit DOLEZ" :

Hi

I saw with previous posts that it seems to be complicated to reload
configuration; this is well explained here :
http://www.rsyslog.com/doc/v8-stable/compatibility/v4compati
bility.html#hup-processing.

I wonder if the architecture of rulesets/queues could be flexible
enough
to implement a reload operation limited to rulesets. Do you think it is
possible to create new queues/rulesets/actions from a new config and
mark
the old ones for removal once idle/empty ?

Regards

Benoit
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

___

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.



___

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond

Re: [rsyslog] saving liblognorm literals


El 21/12/16 a las 17:46, Rainer Gerhards escribió:

see https://github.com/rsyslog/liblognorm/pull/238

wow! that was fast!




2016-12-21 16:54 GMT+01:00 David Lang :

Can you explain your ruleset where you need to store literal as a 
value in the json?


I think the original thinking was that since this is a fixed value, 
storing it as a variable doesn't help.


slapd [1] messages were processed in grok with the following expression:

   MYEXPR ^(.*?)conn=(?\d+) (fd=(?\d+)|op=(?\d+))
   ((?ACCEPT) from IP=%{IPORHOST:ip}|(%{WORD} )?RESULT
   (?.*)|%{WORD} (?attr=.*)|%{WORD:cmd}( (?.*))?)

The relevant part of that is that messages can have 5 formats:

   ACCEPT from IP=...
   FOO RESULT...  # FOO can be different words
   RESULT...
   BAR attr=... # BAR can be different words
   FOOBAR anything # FOOBAR can be different words
   # anything is the rest (not one of the above)

And with that grok we were indexing them "properly"...

[1] http://www.openldap.org/software/man.cgi?query=slapd
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omelasticsearch index warning

Thanks for your answers.

I'll also try to have a look on this.

El 21/12/16 a las 17:48, matthew.gaetano escribió:

Though the examples given are specific modifications for aspects of
Elasticsearch 5.X i believe the modification method would be valid for any
version of Elasticsearch supported by omelasticsearch. To be honest I am not
familiar with how omelasticsearch interfaces with ES (which is using a json
body), where as I am more familiar with the direct head requests to ES.

It is through assumptions and trial/error that this works. And I assume any
index related settings could be used.

ES version is based on node, so the cluster doesn't have an overall ES
version listed (as far as I can tell), so one would have to query the node
that the request is being sent to. Using curl it would be simply:

curl -XGET 'localhost:9200' # where localhost is the ip or hostname of the
target ES node.

the output looks like:
{
"name" : "node_name",
"cluster_name" : "cluster_name",
"cluster_uuid" : "dhB2-TUeRN2LyWmhhN1WJQ",
"version" : {
"number" : "5.0.2",
"build_hash" : "f6b4951",
"build_date" : "2016-11-24T10:07:18.101Z",
"build_snapshot" : false,
"lucene_version" : "6.2.1"
},
"tagline" : "You Know, for Search"
}

Too filter out the version number use:
curl -XGET 'localhost:9200/?filter_path=version.number'

output:
{
"version" : {
"number" : "5.0.2"
}
}

Another way to possible check would be to use the node api.
curl -XGET
'localhost:9200/_nodes/_local/info?filter_path=nodes.*.version'

output:
{
"nodes" : {
"F6UMqjbmQCO5-IqFeKzfDQ" : {
"version" : "5.0.2"
}
}
}

Note that "_local" equates to the node the request was sent to. Though there
are other ways to lookup the nodes based on other attributes this seems
easiest. Since its restricted to local we wont need to know the node id and
can just use a wildcard. Note that the "" at the end is to make the
output readable and is not required.

Thanks

~Regards

-
~Regards

Matthew Gaetano
--
View this message in context:
http://rsyslog-users.1305293.n2.nabble.com/omelasticsearch-index-warning-tp7591961p7592008.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.

Re: [rsyslog] saving liblognorm literals

Storing a literal value would be dependent on the desired output (which would
also then be dependent on what the output is being used by). As an example i
have lots of rules that represent different types of action events from
Cisco. Just looking at the built and teardown sessions shows a wide range of
variances in meaning and format, often complicated by optional segments. So
when i write the rule i used the literal Build or Teardown to distinguish
the differences; and while i could use prefixes to do this, its gets harder
to do as prefix only a single layer and I am already using them to denote
the difference in overarching types of sources/logs. In my output i want my
users to be able to search using a field called action and that action is
denoted by "Built" or "Teardown". With out the ability to capture literals
the only way to accommodate is to use annotations.

Since annotations require tags things can quickly becomes convoluted and
messy the larger they become; more so when the literal might be longer,
possibly presenting conflicts and undesired results. Repetition is also
annoying, why do something twice?

Side note: Prefix's should support the tags function and apply that tag to
any rule that applies to said prefix. At the moment i have to manually added
product/model/version tags to every applicable rule.

Thanks

~Regards

-
~Regards

Matthew Gaetano
--
View this message in context:
http://rsyslog-users.1305293.n2.nabble.com/saving-liblognorm-literals-tp7591994p7592011.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.

Re: [rsyslog] Assert Failure in imtcp.c?

2016-12-21 Thread Andrew Griffin via rsyslog

Attached is a full debug log (hostnames and dir names scrubbed).  But the basic 
setup is this - rsyslog listens on UDP, plain TCP, TLS and RELP ports, parses 
the syslogtag and outputs to a Kafka topic of the same name via omkafka. UDP, 
TCP and RELP work as intended, and the issue occurs on both 8.22 and 8.23.  In 
my opinion it’s likely a configuration issue

My configuration is setup in a straight forward include structure, my main 
rsyslog.conf file looks like:

$IncludeConfig 
//managed_by_ansible/rsyslog/etc/rsyslog.d/globals/*.conf
$IncludeConfig 
//managed_by_ansible/rsyslog/etc/rsyslog.d/modules/*.conf
$IncludeConfig 
//managed_by_ansible/rsyslog/etc/rsyslog.d/templates/*.conf
$IncludeConfig //managed_by_ansible/rsyslog/etc/rsyslog.d/inputs/*.conf
$IncludeConfig //managed_by_ansible/rsyslog/etc/rsyslog.d/rules/*.conf
$IncludeConfig //managed_by_ansible/rsyslog/etc/rsyslog.d/*.conf

I have a number of small conf files from there:

./rsyslog.conf
./rsyslog.d
./rsyslog.d/globals
./rsyslog.d/globals/01_common.conf
./rsyslog.d/globals/02_main_queue.conf
./rsyslog.d/inputs
./rsyslog.d/inputs/01_udp.conf
./rsyslog.d/inputs/02_tcp.conf
./rsyslog.d/inputs/03_tls.conf
./rsyslog.d/inputs/04_relp.conf
./rsyslog.d/modules
./rsyslog.d/modules/01_imudp.conf
./rsyslog.d/modules/02_imtcp.conf
./rsyslog.d/modules/03_imtls.conf
./rsyslog.d/modules/04_imrelp.conf
./rsyslog.d/modules/05_imfile.conf
./rsyslog.d/modules/09_impstats.conf
./rsyslog.d/modules/20_omkafka.conf
./rsyslog.d/rules
./rsyslog.d/rules/00_rsyslogd_internal.conf
./rsyslog.d/rules/01_udp2kafka.conf
./rsyslog.d/rules/02_tcp2kafka.conf
./rsyslog.d/rules/03_tls2kafka.conf
./rsyslog.d/rules/04_relp2kafka.conf
./rsyslog.d/rules/50_sawmill_stats.conf
./rsyslog.d/templates
./rsyslog.d/templates/kakfa_json.conf
./rsyslog.d/templates/local_files.conf

The dump occurs during the loading of the TLS portion (imtcp), here’s the conf 
files for the TLS config (scrubbed):

03_tls.conf:

input(
type= "imtcp"
port= "10448"
ruleset = "tls_10448"
)

03_imtls.conf:

module(
load  = "imtcp"
maxSessions   = "300"
StreamDriver.Name = "gtls"
StreamDriver.Mode = "1"
StreamDriver.AuthMode = "x509/name"
PermittedPeer = [ "*.apple.com", “*..apple.com", 
“*..apple.com" ]
)

03_tls2kafka.conf:

ruleset(name = "tls_10448") {
if $syslogtag startswith "sawmill_test_topic_tls" then {
action(
name  = "tls10448_sawmill_test_topic_tls_to_kafka"
type  = "omkafka"
# Queue configuration - in-memory up to 976.5625M, then to disk up 
to 1024M
queue.spoolDirectory  = 
“//managed_by_ansible/rsyslog/var/spool"
queue.type= "LinkedList"
queue.size= "100"
queue.filename= "tls10448_sawmill_test_topic_tls_to_kafka"
queue.maxdiskspace= "1024M"
queue.saveonshutdown  = "off"
queue.highwatermark   = "90"
queue.lowwatermark= "40"
queue.discardseverity = "5"
# Kafka configuration
template  = "KAFKA_JsonLineTemplate"
broker= [ “broker1:9092", “broker2:9092", 
“broker3:9092", “broker4:9092", “broker5.corp.apple.com:9092" ]
confParam = ["queue.buffering.max.messages=1"]
topic = "sawmill_test_topic_tls"
partitions.auto   = "on"
)
stop
}
}





Andrew Griffin
  ETS / Integration Services
☏ 408-783-8348

> On Dec 21, 2016, at 12:05 AM, mostolog--- via rsyslog 
>  wrote:
> 
> It will be great if you provide a replication use case:
> 
> * rsyslog version (may be 8.23)
> * configuration file
> * log example to replicate the issue.
> 
> If you are able to replicate the issue with a "not too old" syslog version, 
> please file an issue on github
> 
> 
> El 20/12/16 a las 21:14, Andrew Griffin via rsyslog escribió:
>> I’m working setting up a syslog pipeline with rsyslog at the front, and I’m 
>> running in to an issue with rsyslogd core dumping when I try to start it.  
>> Debug log shows everything loading fine, then this:
>> 
>> 4473.240773513:imtcp.c: nspoll.c:147 ISOBJ assert failure: invalid 
>> object type, expected 'netstrms' actual 'nspoll', cookie: BADEFEE
>> rsyslogd: nspoll.c:147: SetDrvrName: Assertion `0' failed.
>> 
>> Then it core dumps.  Is this a known issue?  I can provide a full debug log 
>> if necessary
>> 
>> Andrew Griffin
>>   ETS / Integration Services
>> ☏ 408-783-8348
>> 
>> 
>> 
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are

Re: [rsyslog] About reloading configuration

I don't have the time to check now, but it may be easier than it looks. I
think we have pdag handles, which means we can easily load in the
background and apply when ready. However, we need a rwlock to guard this
operation (action level).

If it works that way, we could do it on HUP (for starters).

Rainer

2016-12-21 17:37 GMT+01:00 David Lang :

> I think the headache with doing this in liblognorm is the overhead that
> can be there in reading/parsing/optimizing the ruleset.
>
> if you have multiple worker threads, having two of them hit the
> mmnormalize action while one is rebuilding the ruleset leads to problems :-)
>
> what would be needed is for all existing parsing to continue to use the
> old ruleset until the new ruleset is fully parsed, optimized, and ready for
> use.
>
> We have a similar issue with the reload of a lookup_table and should be
> able to use a similar solution.
>
> One complicating factor is that while a lookup_table has a global
> existance and is something the rsyslog core knows about, mmnormalize
> rulesets are handled only by the mmnormalize module code, and there's no
> clean way to refer to a specific ruleset outside of that code.
>
> David Lang
>
>
> On Wed, 21 Dec 2016, Benoit DOLEZ wrote:
>
> Yes, I think you are right about reloading only liblognorm. It's probably
>> my first step.
>>
>> In most case I think we need a simple rainer-script if/then/else using a
>> liblognorm app specific rule file. Then adding a new parser implies
>> reloading rsyslog...
>>
>> Thank you for responses. I'm continuing to analyse source...
>>
>> Regards
>>
>> Benoit
>>
>> Le 21/12/2016 à 10:33, mostolog--- via rsyslog a écrit :
>>
>>> You may also find interesting to be able to reload liblognorm rules,
>>> which might be an smaller change.
>>>
>>> The question here is the needed amount of work.
>>>
>>>
>>> El 21/12/16 a las 10:29, Rainer Gerhards escribió:
>>>
 Possible of course, but considerable work (gut feeling: 6+ strong
 weeks).

 Rainer

 Sent from phone, thus brief.

 Am 21.12.2016 09:42 schrieb "Benoit DOLEZ" :

 Hi
>
> I saw with previous posts that it seems to be complicated to reload
> configuration; this is well explained here :
> http://www.rsyslog.com/doc/v8-stable/compatibility/v4compati
> bility.html#hup-processing.
>
> I wonder if the architecture of rulesets/queues could be flexible
> enough
> to implement a reload operation limited to rulesets. Do you think it is
> possible to create new queues/rulesets/actions from a new config and
> mark
> the old ones for removal once idle/empty ?
>
> Regards
>
> Benoit
> --
> Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
> ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
 myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
 if you DON'T LIKE THAT.

>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>
>>
>> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omelasticsearch index warning

It is through assumptions and trial/error that this works. And I assume any
index related settings could be used.

curl -XGET 'localhost:9200' # where localhost is the ip or hostname of the
target ES node.

Too filter out the version number use:
curl -XGET 'localhost:9200/?filter_path=version.number'

output:
{
"version" : {
"number" : "5.0.2"
}
}

Another way to possible check would be to use the node api.
curl -XGET
'localhost:9200/_nodes/_local/info?filter_path=nodes.*.version'

output:
{
"nodes" : {
"F6UMqjbmQCO5-IqFeKzfDQ" : {
"version" : "5.0.2"
}
}
}

Thanks

~Regards

-
~Regards

Re: [rsyslog] rsyslog is recording more (~ 40 extra) log messages than expected from remote client

2016-12-21 Thread rsyslog-users-lists . adiscon . net


On 12/20/2016 11:09 PM, David Lang wrote:

On Tue, 20 Dec 2016, rsyslog-users-lists.adiscon@whyaskwhy.org wrote:


On 12/20/2016 1:38 AM, Benoit DOLEZ wrote:

Hi,

The ~40 messages are those of the window size : for performance 
reason, a train of messages are sent and acked globally. To ack each 
message independently you can try windowsize=1 with omrelp, but 
sending message synchronously is a very bad idea. For omfwd, a tcp 
window size is applied. With UDP, there is no ack, no window.


Regards

Benoit


Thanks for the reply. Am I correct in assuming then that I can expect 
duplicate messages each time I restart the remote rsyslog receiver 
instance?


If you are using RELP, any messages that the sender has not received a 
confirmation of recipt for will be re-sent.


Fantastic, sounds like I'm good on that point then (I'm using RELP).



If you are using plain TCP, messages in flight will be lost

If you happened to have looked over the configs, did you see anything 
that would result in the sending systems tossing old messages if the 
remote rsyslog receiver stays down longer than "X" seconds?


Rsyslog has nothing that will throw away messages based on time. You can 
set it to throw away messages if the queues get too full (look at the 
high watermark settings)


you can also set the retries so that it will only attempt to deliver a 
message X times before it gives up on that message.


Thanks for mentioning that. Is there a setting or command-line option 
that you know of to dump all stock settings and current settings to the 
screen/file? Something like 'postconf -d' or 'postconf' ?


Thanks for your help.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] saving liblognorm literals

see https://github.com/rsyslog/liblognorm/pull/238

Rainer

2016-12-21 17:22 GMT+01:00 Rainer Gerhards :

> There is something fishy. I did a quick test, and it looks like the
> optimizer combines literal parser where it should not do...
>
> Rainer
>
> 2016-12-21 17:05 GMT+01:00 Rainer Gerhards :
>
>> IMHO storing a literal should work, as long as a name is assigned to the
>> literal parser.
>>
>> Rainer
>>
>> 2016-12-21 16:54 GMT+01:00 David Lang :
>>
>>> On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:
>>>
>>> Does this means we shouldn't store the literal as variable? What If we
 need to?

>>>
>>> Can you explain your ruleset where you need to store literal as a value
>>> in the json?
>>>
>>> I think the original thinking was that since this is a fixed value,
>>> storing it as a variable doesn't help.
>>>
>>> David Lang
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>
>>
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] About reloading configuration


we do have an item that we think is doable (effort to be determined)

https://github.com/rsyslog/rsyslog/issues/1319

this puts some config data in a lookup table, it can't use all config 
statements, but would cover a lot of cases.


David Lang

On Wed, 21 Dec 2016, Rainer Gerhards wrote:


Date: Wed, 21 Dec 2016 10:29:50 +0100
From: Rainer Gerhards 
Reply-To: rsyslog-users 
To: rsyslog-users 
Subject: Re: [rsyslog] About reloading configuration

Possible of course, but considerable work (gut feeling: 6+ strong weeks).

Rainer

Sent from phone, thus brief.

Am 21.12.2016 09:42 schrieb "Benoit DOLEZ" :


Hi

I saw with previous posts that it seems to be complicated to reload
configuration; this is well explained here :
http://www.rsyslog.com/doc/v8-stable/compatibility/v4compati
bility.html#hup-processing.

I wonder if the architecture of rulesets/queues could be flexible enough
to implement a reload operation limited to rulesets. Do you think it is
possible to create new queues/rulesets/actions from a new config and mark
the old ones for removal once idle/empty ?

Regards

Benoit
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] saving liblognorm literals

There is something fishy. I did a quick test, and it looks like the
optimizer combines literal parser where it should not do...

Rainer

2016-12-21 17:05 GMT+01:00 Rainer Gerhards :

> IMHO storing a literal should work, as long as a name is assigned to the
> literal parser.
>
> Rainer
>
> 2016-12-21 16:54 GMT+01:00 David Lang :
>
>> On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:
>>
>> Does this means we shouldn't store the literal as variable? What If we
>>> need to?
>>>
>>
>> Can you explain your ruleset where you need to store literal as a value
>> in the json?
>>
>> I think the original thinking was that since this is a fixed value,
>> storing it as a variable doesn't help.
>>
>> David Lang
>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] saving liblognorm literals

IMHO storing a literal should work, as long as a name is assigned to the
literal parser.

Rainer

2016-12-21 16:54 GMT+01:00 David Lang :

> On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:
>
> Does this means we shouldn't store the literal as variable? What If we
>> need to?
>>
>
> Can you explain your ruleset where you need to store literal as a value in
> the json?
>
> I think the original thinking was that since this is a fixed value,
> storing it as a variable doesn't help.
>
> David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omelasticsearch index warning

Ah, OK, so this is in the JSON body and not in an actual HEAD request. This
could be modified whereever META_TYPE is used. Do you know if this works
for older versions of ES as well? If not, is it somehow possible to detect
the ES version?

Rainer

2016-12-21 16:31 GMT+01:00 matthew.gaetano :

> I have the same warning messages in my Elasticsearch deprecation log,
> though
> i have yet to sort it out, and likely wont until early January.
> Elasticsearch 5.X is still relatively new, thus omelasticsearch likely has
> not caught up to some of the recent API changes. I have myself yet to
> submit
> a request in github.
>
> In lieu of knowledge in modifying the omelasticsearch code the following
> can
> be done as a work around, or in general to modify the API request sent to
> elasticsearch.
>
> Using TCPDUMP you can see the whats submitted to the cluster and pick out
> the API header at the start of omelasticsearch submission; looks like:
>
> {"index":{"_index": "myindex-2016.12.21","type":"events"}}{}
>
> Because omelasticsearch allows for dynamic templates we can be sneaky and
> modify the request by adding additional json to as a constant. The
> following
> is an example of adding a pipeline using the index template
> (dynSearchIndex).
>
> template(
>  name="myindex"
>  type="list"
> ){
>  constant(value="myindex")
>  constant(value="-")
>  property(name="timegenerated" dateFormat="rfc3339"
> position.from="1" position.to="4")
>  constant(value=".")
>  property(name="timegenerated" dateFormat="rfc3339"
> position.from="6" position.to="7")
>  constant(value=".")
>  property(name="timegenerated" dateFormat="rfc3339"
> position.from="9" position.to="10")
>  constant(value="\",\"pipeline\":\"mypipeline")
> }
>
> Now when we check the tcpdump output for the request we see the following:
> {"index":{"_index":
> "myindex-2016.12.21","pipeline":"mypipline","type":"events"}}{}
>
> This is what I currently do to provide geoip information (as i have yet to
> sort out the rsyslog version) as well as re-map field names when there are
> conflicts. A good example is when using the regular json output from
> impstats.
>
> The same could possible be applied to the type template (dynSearchType) to
> modify the API. Though i have yet to try; Im not super optimastic that it
> will. In theroy it might look something like the following (Note that this
> also assumes this fixes the deprecation concerns.):
>
> template(
>  name="mytype"
>  type="list"
> ){
>  constant(value="_mapping\":{")
>  constant(value="mytype")
>  constant(value="}")
> }
>
>
> Thanks
>
> ~Regards
>
>
>
> -
> ~Regards
>
> Matthew Gaetano
> --
> View this message in context: http://rsyslog-users.1305293.
> n2.nabble.com/omelasticsearch-index-warning-tp7591961p7591997.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] saving liblognorm literals


On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:


Hi


(Moving from grok to liblognorm) We are trying to store a literal as a 
variable, as we were doing using grok. eg: (?ACCEPT)

We aren't sure if this is correct:

   rule=:%{"type":"literal", "text":"a", "name":"var"}%

As stated in 
http://www.liblognorm.com/files/manual/configuration.html#full-json-format


   /*the literal text shall not be stored inside an output variable*;
   for this reason no name attribute is given (we could also have used
   "name":"-" which achives the same effect but is more verbose)./

Does this means we shouldn't store the literal as variable? What If we 
need to?


you can set a variable to a fixed value as an ammend, or try to use word instead 
of literal for the item (if this doesn't make the match ambiguous)


or, depending on what you are needing the literal for, you may be able to set it 
as a tag.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] saving liblognorm literals


On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:

Does this means we shouldn't store the literal as variable? What If we 
need to?


Can you explain your ruleset where you need to store literal as a value in the 
json?


I think the original thinking was that since this is a fixed value, storing it 
as a variable doesn't help.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omelasticsearch index warning

I have the same warning messages in my Elasticsearch deprecation log, though
i have yet to sort it out, and likely wont until early January.
Elasticsearch 5.X is still relatively new, thus omelasticsearch likely has
not caught up to some of the recent API changes. I have myself yet to submit
a request in github.

In lieu of knowledge in modifying the omelasticsearch code the following can
be done as a work around, or in general to modify the API request sent to
elasticsearch.

Using TCPDUMP you can see the whats submitted to the cluster and pick out
the API header at the start of omelasticsearch submission; looks like: 

{"index":{"_index": "myindex-2016.12.21","type":"events"}}{}

Because omelasticsearch allows for dynamic templates we can be sneaky and
modify the request by adding additional json to as a constant. The following
is an example of adding a pipeline using the index template
(dynSearchIndex).

template(
 name="myindex"
 type="list"
){
 constant(value="myindex")
 constant(value="-")
 property(name="timegenerated" dateFormat="rfc3339"
position.from="1" position.to="4")
 constant(value=".")
 property(name="timegenerated" dateFormat="rfc3339"
position.from="6" position.to="7")
 constant(value=".")
 property(name="timegenerated" dateFormat="rfc3339"
position.from="9" position.to="10")
 constant(value="\",\"pipeline\":\"mypipeline")
}

Now when we check the tcpdump output for the request we see the following:
{"index":{"_index":
"myindex-2016.12.21","pipeline":"mypipline","type":"events"}}{}

This is what I currently do to provide geoip information (as i have yet to
sort out the rsyslog version) as well as re-map field names when there are
conflicts. A good example is when using the regular json output from
impstats.

The same could possible be applied to the type template (dynSearchType) to
modify the API. Though i have yet to try; Im not super optimastic that it
will. In theroy it might look something like the following (Note that this
also assumes this fixes the deprecation concerns.):

template(
 name="mytype"
 type="list"
){
 constant(value="_mapping\":{")
 constant(value="mytype")
 constant(value="}")
}


Thanks

~Regards



-
~Regards

Matthew Gaetano
--
View this message in context: 
http://rsyslog-users.1305293.n2.nabble.com/omelasticsearch-index-warning-tp7591961p7591997.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omelasticsearch index warning


ping!


El 15/12/16 a las 12:37, mosto...@gmail.com escribió:

Hi

Indexing on elasticsearch 5.1.1 we're getting:

[2016-12-15T12:35:27,844][WARN ][o.e.d.r.a.a.i.RestTypesExistsAction] 
[HEAD /{index}/{type}] is deprecated! Use [HEAD /{index}/_mapping/{type}]


Is that a omelasticsearch issue or are we missing something?



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] About reloading configuration

2016-12-21 Thread Benoit DOLEZ

Yes, I think you are right about reloading only liblognorm. It's 
probably my first step.


In most case I think we need a simple rainer-script if/then/else using a 
liblognorm app specific rule file. Then adding a new parser implies 
reloading rsyslog...


Thank you for responses. I'm continuing to analyse source...

Regards

Benoit

Le 21/12/2016 à 10:33, mostolog--- via rsyslog a écrit :

You may also find interesting to be able to reload liblognorm rules,
which might be an smaller change.

The question here is the needed amount of work.


El 21/12/16 a las 10:29, Rainer Gerhards escribió:

Possible of course, but considerable work (gut feeling: 6+ strong weeks).

Rainer

Sent from phone, thus brief.

Am 21.12.2016 09:42 schrieb "Benoit DOLEZ" :


Hi

I saw with previous posts that it seems to be complicated to reload
configuration; this is well explained here :
http://www.rsyslog.com/doc/v8-stable/compatibility/v4compati
bility.html#hup-processing.

I wonder if the architecture of rulesets/queues could be flexible enough
to implement a reload operation limited to rulesets. Do you think it is
possible to create new queues/rulesets/actions from a new config and
mark
the old ones for removal once idle/empty ?

Regards

Benoit
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] saving liblognorm literals


Hi


(Moving from grok to liblognorm) We are trying to store a literal as a 
variable, as we were doing using grok. eg: (?ACCEPT)

We aren't sure if this is correct:

   rule=:%{"type":"literal", "text":"a", "name":"var"}%

As stated in 
http://www.liblognorm.com/files/manual/configuration.html#full-json-format


   /*the literal text shall not be stored inside an output variable*;
   for this reason no name attribute is given (we could also have used
   "name":"-" which achives the same effect but is more verbose)./

Does this means we shouldn't store the literal as variable? What If we 
need to?


   /*using the “literal” parser in JSON should be avoided currently*;
   the experimental version does have some rough edges where conflicts
   in literal processing will not be properly handled. This should not
   be an issue in “closed environments”, like “repeat”, where no such
   conflict can occur./

Is that still recommended? How could I "parse and store" a literal using 
condensed format?


Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] About reloading configuration

You may also find interesting to be able to reload liblognorm rules, 
which might be an smaller change.


The question here is the needed amount of work.


El 21/12/16 a las 10:29, Rainer Gerhards escribió:

Possible of course, but considerable work (gut feeling: 6+ strong weeks).

Rainer

Sent from phone, thus brief.

Am 21.12.2016 09:42 schrieb "Benoit DOLEZ" :


Hi

I saw with previous posts that it seems to be complicated to reload
configuration; this is well explained here :
http://www.rsyslog.com/doc/v8-stable/compatibility/v4compati
bility.html#hup-processing.

I wonder if the architecture of rulesets/queues could be flexible enough
to implement a reload operation limited to rulesets. Do you think it is
possible to create new queues/rulesets/actions from a new config and mark
the old ones for removal once idle/empty ?

Regards

Benoit
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] About reloading configuration

Possible of course, but considerable work (gut feeling: 6+ strong weeks).

Rainer

Sent from phone, thus brief.

Am 21.12.2016 09:42 schrieb "Benoit DOLEZ" :

> Hi
>
> I saw with previous posts that it seems to be complicated to reload
> configuration; this is well explained here :
> http://www.rsyslog.com/doc/v8-stable/compatibility/v4compati
> bility.html#hup-processing.
>
> I wonder if the architecture of rulesets/queues could be flexible enough
> to implement a reload operation limited to rulesets. Do you think it is
> possible to create new queues/rulesets/actions from a new config and mark
> the old ones for removal once idle/empty ?
>
> Regards
>
> Benoit
> --
> Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] About reloading configuration

2016-12-21 Thread Benoit DOLEZ


Hi

I saw with previous posts that it seems to be complicated to reload 
configuration; this is well explained here : 
http://www.rsyslog.com/doc/v8-stable/compatibility/v4compatibility.html#hup-processing.


I wonder if the architecture of rulesets/queues could be flexible enough 
to implement a reload operation limited to rulesets. Do you think it is 
possible to create new queues/rulesets/actions from a new config and 
mark the old ones for removal once idle/empty ?


Regards

Benoit
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Assert Failure in imtcp.c?