Thank you for the the advice! I re-upload it and it should work now.
Tim
2013/7/18 Todd Mortensen <tmorten...@gmail.com>
> An another side note, I wonder if my liblognorm may be too old. I was not
> able to find what version rsyslog 7.4.2 requires.
>
> Here is the version I am using, the one supplied in the ubuntu repo's.
>
> ii liblognorm-dev
> 0.3.2-1 Log normalizing library
> ii liblognorm0
> 0.3.2-1 Log normalizing library
>
>
> On Thu, Jul 18, 2013 at 7:48 AM, Todd Mortensen <tmorten...@gmail.com
> >wrote:
>
> > This is the sample line.
> >
> > [1084540.211910] Denied: IN=eth0 OUT= MAC=00:30:48:90:cc:a6:00:30:
> > 48:da:48:e8:08:00 SRC=10.10.10.10 DST=10.10.10.11 LEN=60 TOS=0x10
> > PREC=0x00 TTL=64 ID=28843 DF PROTO=TCP SPT=44075 DPT=444 WINDOW=14600
> > RES=0x00 SYN URGP=0
> >
> > I will try to get a run going with valgrind and send the output.
> >
> > Todd
> >
> >
> >
> > On Thu, Jul 18, 2013 at 7:24 AM, Rainer Gerhards <
> rgerha...@hq.adiscon.com
> > > wrote:
> >
> >> I am not sure if I manage to look at it before vacation, but could you
> >> send
> >> me a log sample that generates the garbagge? Could you run it under
> >> valgrind control? I guess there is some mis-access taking place...
> >>
> >> Rainer
> >>
> >>
> >> On Wed, Jul 17, 2013 at 4:03 AM, Todd Mortensen <tmorten...@gmail.com
> >> >wrote:
> >>
> >> > I have made some progress in getting it to work, I removed the
> leading
> >> > space in the rule and it now matches. I totally missed that on log
> >> output.
> >> >
> >> > rule=:%kerntime:word% Denied: IN=%in:word% OUT=%out:word%
> MAC=%mac:word%
> >> > SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS=%tos:word%
> >> > PREC=%prec:word% TTL=%ttl:number% ID=%id:number% %DF:word%
> >> > PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number%
> >> > WINDOW=%window:number% RES=%res:word% %pkt-type:word%
> URGP=%urgp:number%
> >> >
> >> > I did find how to use the iptables type but my results where not very
> >> good.
> >> >
> >> > using this rule my output was has a lot of garbage output in the log,
> >> looks
> >> > like it wrote out binary data:
> >> >
> >> > rule=:%kerntime:word% Denied: %iptables:iptables%
> >> >
> >> > { "IN": "eth0", "OUT": "", "MAC":
> >> > "00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00", "SRC": "10.10.10.10",
> >> "DST":
> >> > "10.10.10.11", "LEN": "60", "TOS": "0x10", "PREC": "0x00", "TTL":
> "64",
> >> > "ID": "10181", "DF": "[*PRESENT*]", "PROTO": "TCP", "SPT": "51765",
> >> "DPT":
> >> > "444", "WINDOW": "14600", "RES": "0x00", "SYN": "[*PRESENT*]", "URGP":
> >> "0",
> >> >
> >> >
> >>
> "U)@00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00ü«^?xü«^?ÿÿÿÿÿÿÿÿ%\u0003\u0010LENü«^?<80>¸ü«^?%\u0002\u001060ü«^?¼ü«^?%LEN«^?xü«^?@µü«^?%60ü«^?xü«^?¹ü«^?%\býý\u0001p¾ü«^?<80>ºü«^?%\u0003\bLENü«^?<80>»ü«^?%`´ü«^?ð¿ü«^?ðbü«^?%\u0003\u0010TOSü«^?ÿÿÿÿÿÿÿÿ%\u0004\u00100x10«^?ÿÿÿÿÿÿÿÿ%TOS«^?xü«^?¸ü«^?%0x10^?xü«^?ÿÿÿÿÿÿÿÿ%\býý\u0001P¿ü«^?ÿÿÿÿÿÿÿÿ%\u0003\bTOSü«^?»ü«^?%":
> >> > "[*PRESENT*]", "kerntime": "[1151929.993763]" }
> >> >
> >> > Using the full rule I get results like this, nice and clean.
> >> >
> >> > { "urgp": "0", "pkt-type": "SYN", "res": "0x00", "window": "14600",
> >> > "dst-port": "444", "src-port": "51954", "proto": "TCP", "DF": "DF",
> >> "id":
> >> > "31890", "ttl": "64", "prec": "0x00", "tos": "0x10", "len": "60",
> >> "dst-ip":
> >> > "10.10.10.10", "src-ip": "10.10.10.11", "mac":
> >> > "00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00", "in": "eth0", "kerntime":
> >> > "[1152127.460873]" }
> >> >
> >> > I will continue my journey using the other method for now.
> >> >
> >> >
> >> > On Tue, Jul 16, 2013 at 3:19 PM, Todd Mortensen <tmorten...@gmail.com
> >> > >wrote:
> >> >
> >> > > I am not sure, I honestly stole the examples from the sagan
> project.
> >> > >
> >> > > I just tried using the following to test.
> >> > >
> >> > > rule=: %kerntime:word% Denied: %iptables%
> >> > > rule=: [%kerntime:word%] Denied: %iptables%
> >> > >
> >> > > I also tried adding [] to the above full line as well. Still
> doesn't
> >> > > return any parsed values.
> >> > >
> >> > > I forgot to mention this is version 7.4.2 on ubuntu 12.04 from my
> ppa
> >> > >
> >> > > ii rsyslog-mmnormalize 7.4.2-0ubuntu1ppa2
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Tue, Jul 16, 2013 at 3:12 PM, David Lang <da...@lang.hm> wrote:
> >> > >
> >> > >> it looks to me like your ruleset is not matching the rule
> >> > >>
> >> > >> are you sure that %kerntime:word% includes the square brackets and
> it
> >> > >> shouldn't be [%kerntime:word%] instead?
> >> > >>
> >> > >> there is a special type 'iptables' that was created explicitly for
> >> > >> matching the name=value format of iptables rules.
> >> > >>
> >> > >> I think your rule can be simplified to something like:
> >> > >>
> >> > >> rule=: [%kerntime:word%] Denied: %iptables%
> >> > >>
> >> > >> David Lang
> >> > >>
> >> > >> On Tue, 16 Jul 2013, Todd Mortensen wrote:
> >> > >>
> >> > >> Date: Tue, 16 Jul 2013 14:53:36 -0700
> >> > >>> From: Todd Mortensen <tmorten...@gmail.com>
> >> > >>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
> >> > >>> To: rsyslog-users <rsyslog@lists.adiscon.com>
> >> > >>> Subject: [rsyslog] mmnormalize and iptable logs
> >> > >>>
> >> > >>>
> >> > >>> I am trying my first attempts at using mmnormalize.
> >> > >>>
> >> > >>> My goal is to parse out my iptable firewall logs into a cee format
> >> so I
> >> > >>> can
> >> > >>> then send them to elasticsearch.
> >> > >>>
> >> > >>> But so far my logs just show unparsed items. Can anyone shed any
> >> light
> >> > >>> to
> >> > >>> where I am going wrong here.
> >> > >>>
> >> > >>> Relevant sections from rsyslog.conf
> >> > >>>
> >> > >>> module(load="mmnormalize")
> >> > >>> action(type="mmnormalize" UseRawMsg="off"
> >> > >>> ruleBase="/etc/rsyslog.d/**normalize.rb")
> >> > >>>
> >> > >>> module(load="mmjsonparse")
> >> > >>>
> >> > >>>
> >> > >>> #template(name="cee" type="string" string="%$!all-json%\n")
> >> > >>> template(name="cee" type="string" string="%$!%\n")
> >> > >>>
> >> > >>> *.* { action (name="cee" type="omfile"
> >> > >>> Template="cee" file="/var/log/cee.log") }
> >> > >>>
> >> > >>> I have also tried a couple variations of this but they still don't
> >> > match
> >> > >>>
> >> > >>> normalize.rb
> >> > >>> rule=: %kerntime:word% Denied: IN=%in:word% OUT= MAC=%mac:word%
> >> > >>> SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number%
> TOS=%tos:word%
> >> > >>> PREC=%prec:word% TTL=%ttl:number% ID=%id:number% %DF:word%
> >> > >>> PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number%
> >> > >>> WINDOW=%window:number% RES=%res:word% %pkt-type:word%
> >> > URGP=%urgp:number%
> >> > >>>
> >> > >>> I also tried to use the iptales type, but I am not able to find
> many
> >> > docs
> >> > >>> on its usage.
> >> > >>>
> >> > >>> rule=: %kerntime:word% Denied: %iptables:iptables%
> >> > >>>
> >> > >>>
> >> > >>> Sample log line:
> >> > >>>
> >> > >>> [1084540.211910] Denied: IN=eth0 OUT=
> >> > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10
> >> > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF
> >> PROTO=TCP
> >> > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0
> >> > >>>
> >> > >>>
> >> > >>> This is what I see in my cee logfile and it shows unpared-data.
> >> > >>>
> >> > >>> # { "originalmsg": "[1084540.211910] Denied: IN=eth0 OUT=
> >> > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10
> >> > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF
> >> PROTO=TCP
> >> > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 ",
> >> "unparsed-data":
> >> > >>> "[1084540.211910] Denied: IN=eth0 OUT=
> >> > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10
> >> > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF
> >> PROTO=TCP
> >> > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 " }
> >> > >>> ______________________________**_________________
> >> > >>> rsyslog mailing list
> >> > >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog>
> >> > >>> http://www.rsyslog.com/**professional-services/<
> >> > http://www.rsyslog.com/professional-services/>
> >> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> > myriad
> >> > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> >> you
> >> > >>> DON'T LIKE THAT.
> >> > >>>
> >> > >>> ______________________________**_________________
> >> > >> rsyslog mailing list
> >> > >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog>
> >> > >> http://www.rsyslog.com/**professional-services/<
> >> > http://www.rsyslog.com/professional-services/>
> >> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad
> >> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> >> you
> >> > >> DON'T LIKE THAT.
> >> > >>
> >> > >
> >> > >
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com/professional-services/
> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> > DON'T LIKE THAT.
> >> >
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
