Re: [rsyslog] Sending rsyslog messages to a MS SQL Server instance is failing
On Fri, 12 Jun 2015, Patrick Slattery wrote:
Hi, I'm trying to get log data from rsyslogd to a MS SQL Server Express
2014 instance but I'm having a hell of a time getting it to work.
I'm on CentOS v6.6 and I recompiled rsyslog from source with
--enable-libdbi so that the omlibdbi option would work.
# rsyslogd -v
rsyslogd 8.11.0.master, compiled with:
PLATFORM: x86_64-unknown-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code):No
uuid support: Yes
Number of Bits in RainerScript integers: 64
Contents of /etc/myapp-rsyslog.cfg are:
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
module (load="omlibdbi")
template (name="MSSQLformat" type="string" option.sql="on" string="insert
into dbo.myapp (host, msg) values ('%HOSTNAME%', '%msg%')")
*.* action (type="omlibdbi" server="server.company.com" db="Syslog"
uid="sa" pwd="password" template="MSSQLformat" driver="freetds")
Running:
# /usr/local/sbin/rsyslogd -N1 -f /etc/my_rsyslog-log.cfg
gives:
rsyslogd: version 8.11.0.master, config validation run (level 1), master
config /etc/myapp-rsyslog.cfg
rsyslogd: End of config validation run. Bye.
So it would appear that the conf file is valid.
I can definitely connect to the SQL DB from the rsyslog server:
# TDSVER=7.1 tsql -H server.company.com -p 1433 -U sa -P "password"
1> EXEC sp_databases
2> go
DATABASE_NAME DATABASE_SIZE REMARKS
master 6400NULL
model 2752NULL
msdb14976 NULL
Syslog 204800 NULL
tempdb 2560NULL
(5 rows affected)
(return status = 0)
When I run rsyslogd interactively I get:
# /sbin/rsyslogd -f /etc/myapp-rsyslog.cfg -dn
6817.343677791:main Q:Reg/w0 : action 'action 0': called, logging to
omlibdbi (susp 0/0, direct q 1)
6817.343681180:main Q:Reg/w0 : action 'action 0': is transactional -
executing in commit phase
6817.343684163:main Q:Reg/w0 : actionTryResume: action 0x7f3217e60490
state: susp, next retry (if applicable): 1434086847 [now 1434086817]
6817.343686762:main Q:Reg/w0 : action 'action 0': set suspended state to 1
6817.343689186:main Q:Reg/w0 : processBATCH: next msg 1: can not
initialize libdbi connection [v8.10.0.ad1 try http://www.rsyslog.com/e/2007
]
so this is saying that it can't initialize the connection to MSSQL, it would be
worth looking in the logs of MSSQL to see what it has to say. Is is even seeing
the connection attempt? if it is, is it complaining about a login failure? or is
the login succeeding and it's failing to select a database? or doesn't have
permission to insert into the database? or is sending a malformed command when
it's trying to insert.
David Lang
6817.343691543:main Q:Reg/w0 : ACTION 0
[omlibdbi:action(type="omlibdbi" ...)]
6817.343696392:main Q:Reg/w0 : executing action 0
6817.343698956:main Q:Reg/w0 : action 'action 0': called, logging to
omlibdbi (susp 0/1, direct q 1)
6817.343702170:main Q:Reg/w0 : action 'action 0': is transactional -
executing in commit phase
6817.343705075:main Q:Reg/w0 : actionTryResume: action 0x7f3217e60490
state: susp, next retry (if applicable): 1434086847 [now 1434086817]
6817.343707610:main Q:Reg/w0 : action 'action 0': set suspended state to 1
and nothing shows up in the DB.
Any ideas on what I'm doing wrong here? (Other than the obvious of using MS
SQL ;-)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
[rsyslog] Sending rsyslog messages to a MS SQL Server instance is failing
Hi, I'm trying to get log data from rsyslogd to a MS SQL Server Express
2014 instance but I'm having a hell of a time getting it to work.
I'm on CentOS v6.6 and I recompiled rsyslog from source with
--enable-libdbi so that the omlibdbi option would work.
# rsyslogd -v
rsyslogd 8.11.0.master, compiled with:
PLATFORM: x86_64-unknown-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code):No
uuid support: Yes
Number of Bits in RainerScript integers: 64
Contents of /etc/myapp-rsyslog.cfg are:
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
module (load="omlibdbi")
template (name="MSSQLformat" type="string" option.sql="on" string="insert
into dbo.myapp (host, msg) values ('%HOSTNAME%', '%msg%')")
*.* action (type="omlibdbi" server="server.company.com" db="Syslog"
uid="sa" pwd="password" template="MSSQLformat" driver="freetds")
Running:
# /usr/local/sbin/rsyslogd -N1 -f /etc/my_rsyslog-log.cfg
gives:
rsyslogd: version 8.11.0.master, config validation run (level 1), master
config /etc/myapp-rsyslog.cfg
rsyslogd: End of config validation run. Bye.
So it would appear that the conf file is valid.
I can definitely connect to the SQL DB from the rsyslog server:
# TDSVER=7.1 tsql -H server.company.com -p 1433 -U sa -P "password"
1> EXEC sp_databases
2> go
DATABASE_NAME DATABASE_SIZE REMARKS
master 6400NULL
model 2752NULL
msdb14976 NULL
Syslog 204800 NULL
tempdb 2560NULL
(5 rows affected)
(return status = 0)
When I run rsyslogd interactively I get:
# /sbin/rsyslogd -f /etc/myapp-rsyslog.cfg -dn
6817.343677791:main Q:Reg/w0 : action 'action 0': called, logging to
omlibdbi (susp 0/0, direct q 1)
6817.343681180:main Q:Reg/w0 : action 'action 0': is transactional -
executing in commit phase
6817.343684163:main Q:Reg/w0 : actionTryResume: action 0x7f3217e60490
state: susp, next retry (if applicable): 1434086847 [now 1434086817]
6817.343686762:main Q:Reg/w0 : action 'action 0': set suspended state to 1
6817.343689186:main Q:Reg/w0 : processBATCH: next msg 1: can not
initialize libdbi connection [v8.10.0.ad1 try http://www.rsyslog.com/e/2007
]
6817.343691543:main Q:Reg/w0 : ACTION 0
[omlibdbi:action(type="omlibdbi" ...)]
6817.343696392:main Q:Reg/w0 : executing action 0
6817.343698956:main Q:Reg/w0 : action 'action 0': called, logging to
omlibdbi (susp 0/1, direct q 1)
6817.343702170:main Q:Reg/w0 : action 'action 0': is transactional -
executing in commit phase
6817.343705075:main Q:Reg/w0 : actionTryResume: action 0x7f3217e60490
state: susp, next retry (if applicable): 1434086847 [now 1434086817]
6817.343707610:main Q:Reg/w0 : action 'action 0': set suspended state to 1
and nothing shows up in the DB.
Any ideas on what I'm doing wrong here? (Other than the obvious of using MS
SQL ;-)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Template Mystery for me
On Thu, 11 Jun 2015, Muhammad Asif wrote:
I want to use elasticsearch. I have already taken your so much time.
Apology for that. Actually my goal is just to send the following log in ES.
Actually whole message is saved in msg field but I want it different fields.
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP","dns":{"type":"query","id":19589,"rrname":"
daisy.ubuntu.com","rrtype":"A"}}
I followed many tutorials but get more confuse. Please elaborate these
templates.
template(name="CEETemplate" type="string" string="%TIMESTAMP% %HOSTNAME%
%syslogtag% @cee: %$!all-json%\n")
This template is converting simple log into json format. Adding these
three things "%TIMESTAMP% %HOSTNAME% %syslogtag% and convert it in json.
This is not my requirement. I just want my log fields. Why it add slashes
in message like this.
log
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
output
Jun 11 11:52:31 new-sr @cee: {
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\","
}
well, in this case, you are adding an extra space between the : of @cee: and the
{ that will start the beginning of the json data. That will cause parsing
errors. If you write the log entry out with the format RSYSLOG_DebugFormat, you
can see the raw log, and the result of parsing. Eliminate that space and look at
the result again.
with the space there, you will see that $! includes one property, "msg" that
containst a bunch of text that would be JSON if it wasn't escaped.
Afterwords, you will see not only the msg property, but also the various things
that are parsed from it.
--
What is happening in below template. I used it and take output in file.
It is writing {}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}.
template(name="messageToES" type="list") { property(name="$!all-json") }
try doing
template(name="messageToES" type="subtree" subtree="$!all-json")
I think it will e a lot closer to what you want.
Why we use constant. Is it field name.
template(name="customTemplate"
type="list") {
constant(value="{\"timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"syslogtag\":\"")
property(name="syslogtag" format="json")
constant(value="\",")
property(name="$!all-json" position.from="2")
}
try
template(name="customTemplate" type="list") {
property(name="timereported" dateFormat="rfc3339" format="jsonf"
outname="timestamp")
property(name="syslogtag" format="jsonf" outname="syslogtag")
property(name="$!all-json" format=jsonf)
}
or just
set $!timestamp = $timereported;
set $!syslgtag = $syslogtag
$template customTemplate,"$!"
Ok, if this is what you want to output
{
"message":
{
"timestamp":"2014-12-29T21:01:13.600536",
"event_type":"dns",
"src_ip":"172.20.16.93",
"src_port":49112,
"dest_ip":"4.2.2.2",
"dest_port":53,
"proto":"UDP"
}
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\"}",
"fromhost": "test-host",
"facility": "user",
"priority": "info",
"timereported": "2013-03-12T18:05:01.344864+02:00",
"timegenerated": "2013-03-12T18:05:01.344864+02:00"
"src_ip":"172.20.16.93"
"src_port":"49112"
"dest_ip":"4.2.2.2"
"dest_port":"53"
"proto":"UDP"
}
whem you receive the log message
Jun 11 11:52:31 new-sr something[pid]:
@cee:{"timestamp":"2014-12-29T21:01:13.586962","event_type":"dns","src_ip":"8.8.8.8","src_port":53,"dest_ip":"172.20.16.93","dest_port":49112,"proto":"UDP"}
then what I would do is:
action(type="mmjsonparse" name="jsonparse")
(note 'pri', 'something', 'pid' are placeholders for this example)
This will populate variables (as shown by RSYSLOG_DebugFormat
$FROMHOST: 'new-sr',
$fromhost-ip: 'ipaddress',
$HOSTNAME: 'new-sr',
$PRI: pri,
$syslogtag 'something[pid]:',
$programname: 'something',
$APP-NAME: 'something',
$PROCID: 'pid',
$MSGID: '-',
$TIMESTAMP: 'Jun 11 11:52:31'
$STRUCTURED-DATA: '-',
$msg: '
@cee:{"timestamp":"2014-12-29T21:01:13.586962","event_type":"dns","src_ip":"8.8.8.8","src_port":53,"dest_ip":"172.20.16.93","dest_port":49112,"proto":"UDP"}',
escaped msg: '
@cee:{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\"}',
$inputnam
Re: [rsyslog] rsyslog packaging project - was: rsyslog adiscon packages
Florian - plan will probably depend on what we decide we're deploying. Andre has set up an account already ( Rainer may have the details as well ). We gave them access to the beta for the team account features - so it should already be set up as a team account. We should make sure any infrastructure we set up is provisioned from that account since we (DO) have applied quite a bit of credit to it. If we're just working together on how we'd like to automate package building I'd suggest a small server to start with and if we grow out of it we can bump it up.. maybe the 2GB ram / 2 core / 40GB ssd option to prototype on? Hard to know until we nail down exactly what we're trying to stand up. Cheers! Brian On Thu, Jun 11, 2015 at 12:17 PM, Florian Riedl wrote: > Hi, > > I am very happy, that this project already received some initial drive and > I am looking forward to working with you all. > > +1 for the mailing list as communication device > > I guess we could establish some prefix for the subject, so emails regarding > the packaging project can be easily distinguished from the regular stuff. > Something like [rpp]? > > +1 for the Digital Ocean hosting > > I guess, using the cloud hosting solution by DO might be the best way to > set this up, because it is probably the most accessible solution. No need > to buy Hardware, no need to set up complicated access clutter to internal > networks. Though, I am not sure where to start in that matter or what plan > to choose from the available options. Brian, do you have a suggestion what > could be a good starting point? > > Another question that needs to be discussed is the use of SSH certificates > for the signing of packages. This is not ideally solved for the current > packages. I guess there are two possibilities: > 1. All packages are built and signed with a common project key. > 2. All packages are built and signed with a individual key from each > person. > > Option 1 has some charme, but it is impossible to determine who actually > built the packages, whereas this is possible with option 2, but there every > "builder" needs to have a key. This even applies to logins. > > Opinions and comments anyone? > > Florian > > 2015-06-09 19:32 GMT+02:00 Brian Knox : > > > Rsyslog is very useful and appreciated :) > > > > On Tue, Jun 9, 2015 at 1:28 PM, Rainer Gerhards < > rgerha...@hq.adiscon.com> > > wrote: > > > > > Sent from phone, thus brief. > > > Am 09.06.2015 19:01 schrieb "Brian Knox" : > > > > > > > > Coordinating on the mailing list is fine with me. My employer > > > > (DigitalOcean) has given the rsyslog project a fairly sizeable free > > > hosting > > > > budget so I'd be remiss to not advocate for us ;). > > > > > > I just realize that I slipped something (the Digital Ocean sponsoring) > > that > > > I wanted to announce more formally ... Well that will follow but let me > > say > > > now that it is very useful and appreciated :) > > > > > > Rainer > > > > If another option works > > > > out to be better I certainly understand! > > > > > > > > I totally agree that converging on an agreed upon problem statement > is > > > the > > > > right place to start. > > > > > > > > Cheers, > > > > Brian > > > > > > > > On Tue, Jun 9, 2015 at 12:53 PM, David Lang wrote: > > > > > > > > > On Tue, 9 Jun 2015, Rainer Gerhards wrote: > > > > > > > > > > Hi all, > > > > >> > > > > >> chances are extremely well to get to better packaging projects. We > > had > > > > >> some discussions internally in Adiscon, and I was able to secure > the > > > > >> help of Florian Riedl for getting this in the best possible shape. > > > > >> > > > > >> Our goal is to get > > > > >> > > > > >> - better packages > > > > >> - more timely support for new distro releases > > > > >> - support for a broader set of distros (e.g. Fedora, often > > requested) > > > > >> - more ability for the community to steer this previous > all-Adiscon > > > > >> project > > > > >> > > > > >> The 0mq discussion that started this thread is a good example of > > what > > > > >> this means. > > > > >> > > > > >> With the help of more community involvment we can reach the goals. > > And > > > > >> in order to make it easier to contribute, we need to streamline > the > > > > >> process of how we build, release, test, and announce packages. > > > > >> > > > > >> Thankfully, Brian has offered to become an active team member. It > > > > >> would be great if others would also join in. > > > > >> > > > > >> I currently think that the right path to success is to start with > > > > >> small but sufficiently large project part. So I would propose that > > we > > > > >> focus on Ubuntu initially, get that part organized, learn a couple > > of > > > > >> things and apply the gained experience later to a "final" project > > that > > > > >> covers other distros as well (as mentioned by darix, the use of > OBS > > > > >> sounds very appealing to me). > > > > >> > > > > >> In order to get going, I would like to see some id
Re: [rsyslog] Unable to use foreach
Just a note that there is definantly something wrong with the variable handling right now. I've got a hard to reproduce bug that I've been fighting for the last month where messages from just a couple sources seem to trigger corruption when sending messages out with an action that has it's own queue and a custom template (output prior to the point is fine, but the message send and output after shows corrupt data, occasionally a double free or other corruption message) Some of the cleanup in the clang support branch seems to help some of it, but the one box that I tested it on hasn't let me reproduce the problem since, even with older versions :-( David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Unable to use foreach
I believe that Janmejay suggests below (far below) that in the "second-last
branch in msgSetPropViaJSON" a ref count bump should be added before the call
to msgAddJSON.
I started with a freshly unpacked 8.10.0 tree and inserted the bump at that
location. That results in an rsyslogd that segfaults with my setup, although in
a different spot than before. This happens in both 8.9 and 8.10. My suggested
patch, which is in the body of msgAddJSON, only does the bump when name[1] is
'\0' (and from debugging this appears to be the case for $!).
If I am correct about what Janmejay is suggesting as a patch, then my patch
will bump the ref count for a subset of the cases in which his patch would bump
it. That is not to say that my patch is correct, just that further debugging is
needed by someone who is familiar with the guts of rsyslogd. (If I've
misinterpreted Janmejay's suggested fix, my apologies).
The backtrace for the new segfault is:
#0 0x77192592 in json_object_object_to_json_string
(jso=0x7fffe8003150, pb=0x7fffe8006fd0, level=0, flags=1) at json_object.c:310
#1 0x7719248c in json_object_to_json_string_ext (jso=0x7fffe8003150,
flags=1) at json_object.c:276
#2 0x0041f005 in getJSONPropVal (pMsg=0x70004030, pProp=0x6c73b0,
pRes=0x74bdc8a0, buflen=0x74bdc89c,
pbMustBeFreed=0x74bdca56) at msg.c:2802
#3 0x00420075 in MsgGetProp (pMsg=0x70004030, pTpe=0x6c7390,
pProp=0x6c73b0, pPropLen=0x74bdca44,
pbMustBeFreed=0x74bdca56, ttNow=0x74bdcb30) at msg.c:3285
#4 0x00454a5a in tplToString (pTpl=0x6c50b0, pMsg=0x70004030,
iparam=0x7fffe8002d90, ttNow=0x74bdcb30)
at ../template.c:194
#5 0x0044e5c9 in prepareDoActionParams (pAction=0x6d7c30,
pWti=0x6ce940, pMsg=0x70004030, ttNow=0x74bdcb30)
at ../action.c:933
#6 0x0044f332 in processMsgMain (pAction=0x6d7c30, pWti=0x6ce940,
pMsg=0x70004030, ttNow=0x74bdcb30) at ../action.c:1300
#7 0x0044f979 in doSubmitToActionQ (pAction=0x6d7c30, pWti=0x6ce940,
pMsg=0x70004030) at ../action.c:1449
#8 0x00445568 in execAct (stmt=0x6d7ba0, pMsg=0x70004030,
pWti=0x6ce940) at ruleset.c:202
#9 0x00445f52 in scriptExec (root=0x6d41b0, pMsg=0x70004030,
pWti=0x6ce940) at ruleset.c:454
#10 0x0044619a in processBatch (pBatch=0x6ce978, pWti=0x6ce940) at
ruleset.c:508
#11 0x0045be5f in msgConsumer (notNeeded=0x0, pBatch=0x6ce978,
pWti=0x6ce940) at rsyslogd.c:575
...
At the site of the break we have a macro, json_object_object_foreachC. That
macro pulls the lh_table out of the json_object as json->o.c_object. For the
json_object being traversed in my rsyslogd, json->o.c_object is
0x21.Interestingly, most of the fields are 0x21, which is '!'. The macro treats
this as a pointer value and bombs.
I looked at the test code area and it looks to have a steep learning curve. The
code (rsyslogd conf, test client, and mmexternal target) I posted on github
(https://github.com/davidboles/snowflake.git) reproduces the problem quite
readily. This patch:
--- ../pristine/rsyslog-8.10.0/runtime/msg.c2015-05-19 03:53:40.0
-0500
+++ runtime/msg.c 2015-06-09 14:47:51.066158522 -0500
@@ -4343,6 +4343,7 @@
}
if(name[1] == '\0') { /* full tree? */
+ json_object_get(json);
if(*pjroot == NULL)
*pjroot = json;
else
removes the issue and doesn't leak memory for my test case even when run
hundreds of thousand times.
I enabled testbench in my patched 8.10 tree and a make check ends with:
make check-TESTS
make[2]: Entering directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
make[3]: Entering directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
PASS: stop-localvar.sh
PASS: stop-msgvar.sh
PASS: fac_authpriv.sh
PASS: fac_local0.sh
PASS: fac_local7.sh
PASS: fac_mail.sh
PASS: fac_news.sh
PASS: fac_ftp.sh
PASS: fac_ntp.sh
PASS: fac_uucp.sh
PASS: fac_invld1.sh
PASS: fac_invld2.sh
PASS: fac_invld3.sh
PASS: fac_invld4_rfc5424.sh
PASS: rfc5424parser.sh
PASS: arrayqueue.sh
PASS: global_vars.sh
PASS: da-mainmsg-q.sh
PASS: validation-run.sh
PASS: empty-ruleset.sh
PASS: imtcp-multiport.sh
PASS: daqueue-persist.sh
PASS: diskqueue.sh
PASS: diskqueue-fsync.sh
make[3]: *** No rule to make target `rulesetmultiqueue.sh', needed by
`rulesetmultiqueue.sh.log'. Stop.
make[3]: Leaving directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
make[2]: *** [check-TESTS] Error 2
make[2]: Leaving directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
make: *** [check-recursive] Error 1
I'd be happy to assist the rsyslog team in any way or answer any questions. For
now, I'll just proceed with my patched vers
Re: [rsyslog] rsyslog packaging project - was: rsyslog adiscon packages
On Thu, 11 Jun 2015, Florian Riedl wrote: Hi, I am very happy, that this project already received some initial drive and I am looking forward to working with you all. +1 for the mailing list as communication device I guess we could establish some prefix for the subject, so emails regarding the packaging project can be easily distinguished from the regular stuff. Something like [rpp]? +1 for the Digital Ocean hosting I guess, using the cloud hosting solution by DO might be the best way to set this up, because it is probably the most accessible solution. No need to buy Hardware, no need to set up complicated access clutter to internal networks. Though, I am not sure where to start in that matter or what plan to choose from the available options. Brian, do you have a suggestion what could be a good starting point? Another question that needs to be discussed is the use of SSH certificates for the signing of packages. This is not ideally solved for the current packages. I guess there are two possibilities: 1. All packages are built and signed with a common project key. 2. All packages are built and signed with a individual key from each person. Option 1 has some charme, but it is impossible to determine who actually built the packages, whereas this is possible with option 2, but there every "builder" needs to have a key. This even applies to logins. Opinions and comments anyone? If the package is built from the master git tree, it should be signed by a common project key. If it's built from someone's personal branch, it should be signed by that person. David Lang Florian 2015-06-09 19:32 GMT+02:00 Brian Knox : Rsyslog is very useful and appreciated :) On Tue, Jun 9, 2015 at 1:28 PM, Rainer Gerhards wrote: Sent from phone, thus brief. Am 09.06.2015 19:01 schrieb "Brian Knox" : Coordinating on the mailing list is fine with me. My employer (DigitalOcean) has given the rsyslog project a fairly sizeable free hosting budget so I'd be remiss to not advocate for us ;). I just realize that I slipped something (the Digital Ocean sponsoring) that I wanted to announce more formally ... Well that will follow but let me say now that it is very useful and appreciated :) Rainer If another option works out to be better I certainly understand! I totally agree that converging on an agreed upon problem statement is the right place to start. Cheers, Brian On Tue, Jun 9, 2015 at 12:53 PM, David Lang wrote: On Tue, 9 Jun 2015, Rainer Gerhards wrote: Hi all, chances are extremely well to get to better packaging projects. We had some discussions internally in Adiscon, and I was able to secure the help of Florian Riedl for getting this in the best possible shape. Our goal is to get - better packages - more timely support for new distro releases - support for a broader set of distros (e.g. Fedora, often requested) - more ability for the community to steer this previous all-Adiscon project The 0mq discussion that started this thread is a good example of what this means. With the help of more community involvment we can reach the goals. And in order to make it easier to contribute, we need to streamline the process of how we build, release, test, and announce packages. Thankfully, Brian has offered to become an active team member. It would be great if others would also join in. I currently think that the right path to success is to start with small but sufficiently large project part. So I would propose that we focus on Ubuntu initially, get that part organized, learn a couple of things and apply the gained experience later to a "final" project that covers other distros as well (as mentioned by darix, the use of OBS sounds very appealing to me). In order to get going, I would like to see some ideas float on: - how should we communicate? (rsyslog mailing list, dedicated mailing list, github issue trackers, IRC, ...) I'd say the rsyslog mailing list, failing that a dedicated mailing list. But I think that the issues we will be working through are useful to people who need to roll their own version (to test something from git, or to enable specific features). - where do we track issues? (I have a strong preferrence for the github issue trackers) - what about doc? - where should we place the build platform (cloud I would guess, could we use Digital Ocean sponsorship for this)? I don't know the details of using it, but the Suse Open Build Platform is already setup to support a whole bunch of target distros. How close does it come to covering everything we need? Can it be used for all the different uses we want from this (distro release builds, nightly builds, other) If we have to roll our own infrastructure, some sort of cloud system is right. Google donates time on their cloud system to opensource projects, I don't know if it would be enough or not. Sponsorship from whoever is good :-) - when do we start ;) Clarification of the prob
Re: [rsyslog] rsyslog packaging project - was: rsyslog adiscon packages
Hi, I am very happy, that this project already received some initial drive and I am looking forward to working with you all. +1 for the mailing list as communication device I guess we could establish some prefix for the subject, so emails regarding the packaging project can be easily distinguished from the regular stuff. Something like [rpp]? +1 for the Digital Ocean hosting I guess, using the cloud hosting solution by DO might be the best way to set this up, because it is probably the most accessible solution. No need to buy Hardware, no need to set up complicated access clutter to internal networks. Though, I am not sure where to start in that matter or what plan to choose from the available options. Brian, do you have a suggestion what could be a good starting point? Another question that needs to be discussed is the use of SSH certificates for the signing of packages. This is not ideally solved for the current packages. I guess there are two possibilities: 1. All packages are built and signed with a common project key. 2. All packages are built and signed with a individual key from each person. Option 1 has some charme, but it is impossible to determine who actually built the packages, whereas this is possible with option 2, but there every "builder" needs to have a key. This even applies to logins. Opinions and comments anyone? Florian 2015-06-09 19:32 GMT+02:00 Brian Knox : > Rsyslog is very useful and appreciated :) > > On Tue, Jun 9, 2015 at 1:28 PM, Rainer Gerhards > wrote: > > > Sent from phone, thus brief. > > Am 09.06.2015 19:01 schrieb "Brian Knox" : > > > > > > Coordinating on the mailing list is fine with me. My employer > > > (DigitalOcean) has given the rsyslog project a fairly sizeable free > > hosting > > > budget so I'd be remiss to not advocate for us ;). > > > > I just realize that I slipped something (the Digital Ocean sponsoring) > that > > I wanted to announce more formally ... Well that will follow but let me > say > > now that it is very useful and appreciated :) > > > > Rainer > > > If another option works > > > out to be better I certainly understand! > > > > > > I totally agree that converging on an agreed upon problem statement is > > the > > > right place to start. > > > > > > Cheers, > > > Brian > > > > > > On Tue, Jun 9, 2015 at 12:53 PM, David Lang wrote: > > > > > > > On Tue, 9 Jun 2015, Rainer Gerhards wrote: > > > > > > > > Hi all, > > > >> > > > >> chances are extremely well to get to better packaging projects. We > had > > > >> some discussions internally in Adiscon, and I was able to secure the > > > >> help of Florian Riedl for getting this in the best possible shape. > > > >> > > > >> Our goal is to get > > > >> > > > >> - better packages > > > >> - more timely support for new distro releases > > > >> - support for a broader set of distros (e.g. Fedora, often > requested) > > > >> - more ability for the community to steer this previous all-Adiscon > > > >> project > > > >> > > > >> The 0mq discussion that started this thread is a good example of > what > > > >> this means. > > > >> > > > >> With the help of more community involvment we can reach the goals. > And > > > >> in order to make it easier to contribute, we need to streamline the > > > >> process of how we build, release, test, and announce packages. > > > >> > > > >> Thankfully, Brian has offered to become an active team member. It > > > >> would be great if others would also join in. > > > >> > > > >> I currently think that the right path to success is to start with > > > >> small but sufficiently large project part. So I would propose that > we > > > >> focus on Ubuntu initially, get that part organized, learn a couple > of > > > >> things and apply the gained experience later to a "final" project > that > > > >> covers other distros as well (as mentioned by darix, the use of OBS > > > >> sounds very appealing to me). > > > >> > > > >> In order to get going, I would like to see some ideas float on: > > > >> > > > >> - how should we communicate? > > > >> (rsyslog mailing list, dedicated mailing list, github issue > > > >> trackers, IRC, ...) > > > >> > > > > > > > > I'd say the rsyslog mailing list, failing that a dedicated mailing > > list. > > > > But I think that the issues we will be working through are useful to > > people > > > > who need to roll their own version (to test something from git, or to > > > > enable specific features). > > > > > > > > - where do we track issues? > > > >> (I have a strong preferrence for the github issue trackers) > > > >> - what about doc? > > > >> - where should we place the build platform > > > >> (cloud I would guess, could we use Digital Ocean sponsorship for > > this)? > > > >> > > > > > > > > I don't know the details of using it, but the Suse Open Build > Platform > > is > > > > already setup to support a whole bunch of target distros. How close > > does it > > > > come to covering everything we need? > > > > > > > > Can it be used for all th
Re: [rsyslog] Strange MySQL behaviour
2015-06-10 22:29 GMT+02:00 Thomas D. : > Hi, > > Mh, do I really need to get RHEL7 to trigger the problem? I tried on > Gentoo but was unable to reproduce (openssl-1.0.2a, gnutls-3.3.15, > libmysqlclient from mariadb-10.0.19). > I only produced the error only under RHEL 7 and currently we don't know any other platform to produce this error. > > Are you using official MariaDB and GnuTLS packages from RHEL or did you > build these packages from source, too? > I used the official packages form RHEL > > - gnutls-3.3.8-12.el7? > - mariadb-5.5.41-2.el7_0? > And currently I have the same versions installed > > I am currently wondering why ommysql.so is linked against libgnutls.so > and libssl.so: > > > # ldd /usr/lib64/rsyslog/ommysql.so > > linux-vdso.so.1 (0x7ffc7fb7) > > libmysqlclient.so.18 => /usr/lib64/libmysqlclient.so.18 > (0x7f96d3bb) > > libpthread.so.0 => /lib64/libpthread.so.0 (0x7f96d3995000) > > libz.so.1 => /lib64/libz.so.1 (0x7f96d377f000) > > libm.so.6 => /lib64/libm.so.6 (0x7f96d3483000) > > libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0 > (0x7f96d320d000) > > libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0 > (0x7f96d2dcf000) > > libdl.so.2 => /lib64/libdl.so.2 (0x7f96d2bcb000) > > libgnutls.so.28 => /usr/lib64/libgnutls.so.28 > (0x7f96d28b6000) > > libc.so.6 => /lib64/libc.so.6 (0x7f96d2513000) > > /lib64/ld-linux-x86-64.so.2 (0x7f96d4314000) > > libtasn1.so.6 => /usr/lib64/libtasn1.so.6 (0x7f96d230) > > libnettle.so.4 => /usr/lib64/libnettle.so.4 (0x7f96d20cd000) > > libhogweed.so.2 => /usr/lib64/libhogweed.so.2 > (0x7f96d1e9d000) > > libgmp.so.10 => /usr/lib64/libgmp.so.10 (0x7f96d1c24000) > > > -Thomas > > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Template Mystery for me
I want to use elasticsearch. I have already taken your so much time.
Apology for that. Actually my goal is just to send the following log in ES.
Actually whole message is saved in msg field but I want it different fields.
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP","dns":{"type":"query","id":19589,"rrname":"
daisy.ubuntu.com","rrtype":"A"}}
I followed many tutorials but get more confuse. Please elaborate these
templates.
template(name="CEETemplate" type="string" string="%TIMESTAMP% %HOSTNAME%
%syslogtag% @cee: %$!all-json%\n")
This template is converting simple log into json format. Adding these
three things "%TIMESTAMP% %HOSTNAME% %syslogtag% and convert it in json.
This is not my requirement. I just want my log fields. Why it add slashes
in message like this.
log
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
output
Jun 11 11:52:31 new-sr @cee: {
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\","
}
--
What is happening in below template. I used it and take output in file.
It is writing {}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}.
template(name="messageToES" type="list") { property(name="$!all-json") }
Why we use constant. Is it field name.
template(name="customTemplate"
type="list") {
constant(value="{\"timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"syslogtag\":\"")
property(name="syslogtag" format="json")
constant(value="\",")
property(name="$!all-json" position.from="2")
}
$template JSONDefault,
"{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}"
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
template(name="custom"
type="list") {
constant(value="{\"timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"syslogtag\":\"")
property(name="syslogtag" format="json")
constant(value="\",")
property(name="$!all-json" position.from="2")
}
All these templates are add some rsyslog properties like %msg:::json%
%syslogtag% etc. how can I add my own fields which are in log. Which
template is near to my requirement. I want below form of output in ES.
log:
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
Jun 11 11:52:31 new-sr @cee: {
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\","
}
output
{
"message":
"{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
Jun 11 11:52:31 new-sr @cee: {
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\","
}",
"fromhost": "test-host",
"facility": "user",
"priority": "info",
"timereported": "2013-03-12T18:05:01.344864+02:00",
"timegenerated": "2013-03-12T18:05:01.344864+02:00"
"src_ip":"172.20.16.93"
"src_port":"49112"
"dest_ip":"4.2.2.2"
"dest_port":"53"
"proto":"UDP"
}
action(
type="omelasticsearch"
template="mytemplate"
server="127.0.0.1"
serverport="9200"
searchType="syslogapp"
searchIndex="myindex"
bulkmode="on"
queue.dequeuebatchsize="1000"
action.resumeretrycount="-1"
)
On Thu, Jun 11, 2015 at 12:08 PM, David Lang wrote:
> On Thu, 11 Jun 2015, Muhammad Asif wrote:
>
> 1- I have installed mmjasonparse, mmnormalize, omelasticsearch by
>> following
>> command.
>> sudo apt-get install rsyslog-mmjsonparse,
>> rsyslog-omelasticsearch,rsyslog-mmnormalize
>> Still I need to compile rsyslog as ./configure --prefix=/usr
>> --enable-imtcp --enable-mmjsonparse --enable-ommongodb ? or not.
>>
>
> you shouldn't mix a self-compiled version with distro packages, do one or
> the other.
>
> do you need mongodb? you don't mention it anywhere else? If you need
> mmnormalize, omelasticsearch, and other things, you
Re: [rsyslog] mapping action numbers in impstats to config file lines
Its easy, I remember missing it too :-) -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Jun 11, 2015 3:40 AM, "Micah Yoder" wrote: > On 6/9/15 2:05 PM, David Lang wrote: > > > This is already available. Just add name="name" in the action() statement > > > > I name the things that it makes sense to put in action() statements, and > > put up with numbers for the items between them. > > sweet, not sure how I missed that! > Thanks > > > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Template Mystery for me
On Thu, 11 Jun 2015, Muhammad Asif wrote:
1- I have installed mmjasonparse, mmnormalize, omelasticsearch by following
command.
sudo apt-get install rsyslog-mmjsonparse,
rsyslog-omelasticsearch,rsyslog-mmnormalize
Still I need to compile rsyslog as ./configure --prefix=/usr
--enable-imtcp --enable-mmjsonparse --enable-ommongodb ? or not.
you shouldn't mix a self-compiled version with distro packages, do one or the
other.
do you need mongodb? you don't mention it anywhere else? If you need
mmnormalize, omelasticsearch, and other things, you would need to include them
as well
2- You mean the following set of instructions help me. Things on left side
are items in log and things on right side are fields in elasticsearch
against which these values will be saved. Am I on right track?
the variable name is what will show up as the fieldname in elasticsearch, you
are setting the variable equal to the value on the right.
But you are saying elasticsearch, but invoking mongodb, which are you trying to
do?
you also don't show us the definition of the template that you are using.
David Lang
ruleset(name="mongodb") {
action(type="mmjsonparse")
if $parsesuccess == "OK" then { # How this condition will
be OK, mean what will be checked.
set $!time = $timestamp;
set $!sys = $hostname;
set $!procid = $syslogtag;
set $!syslog_fac = $syslogfacility;
set $!syslog_sever = $syslogpriority;
set $!pid = $procid;
action(type="ommongodb" server="127.0.0.1" db="logs"
collection="syslog" template="mongodball")
}
}
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
