Re: [RADIATOR] Request for enhancement: Log Handler InfluxDB or at least UDP

[Barry Ard]

I like this, very simple. Please ensure that this ends up in goodies.

Thanks,
Barry

On Fri, Feb 5, 2016 at 5:47 AM, Heikki Vatiainen  wrote:

> On 2.2.2016 13.14, Karl Gaissmaier wrote:
>
> > yes, like heka http://hekad.readthedocs.org as forwarding agent and/or
> > anomaly processor.
>
> Interesting, thanks for sharing this.
>
> > Heka has also a sandboxed Lua interpreter to decode unusual log formats,
> > maybe I'll not implement the hook in RADIATOR.
> >
> > Maybe it's really enough to create normal logs and use heka (or similar
> > tools)
> > to process anomaly detection and forward it to graphite/influxdb.
>
> Meanwhile, I did a basic Influxdb and Grafana installation to test it a
> little. Below is a simple AuthLog FILE format hook that creates an entry
> in Influxdb line protocol format and sends it before logging it to a
> file. It simply removes some of the characters that need to be quoted in
> the line protocol format and creates a new socket for each call. It's
> very primitive but, it will do basic logging and is a quick way to
> experiment and get something stored in Influxdb and visible in Grafana.
>
> The entry that gets logged in authlog file is useful to see how the line
> that was sent to Influxdb was formatted.
>
> # AuthLog in InfluxDB format
> sub
> {
>  my ($s, $reason, $p) = @_;
>
>  my $ap = $p->get_attr('NAS-Identifier');
>  my $client_mac = $p->get_attr('Calling-Station-Id');
>  my $username = $p->get_attr('User-Name');
>
>  my ($sec, $usec) = Radius::Util::getTimeHires();
>  my $influxtime = "$sec$usec"."000";
>
>  # Strip space, \ and "
>  # See Influxdb docs for what/how to quote
>  $username =~ s/[ \\"]//g;
>  $reason =~ s/[ \\"]//g;
>
>  my $dp; # InfluxDB line protocol data point
>  if ($s == $main::ACCEPT)
>  {
>  my $key =
> "radius,type=accept,ap=$ap,special=$username,special_type=username";
>
>  my $fields = "value=\"$username\"";
>  $dp = "$key $fields $influxtime";
>  }
>  elsif ($s == $main::REJECT)
>  {
>  my $key =
> "radius,type=rejected,ap=$ap,special=$reason,special_type=reason";
>
>  my $fields = "value=\"$username\",special_val=\"$reason\"";
>  $dp = "$key $fields $influxtime";
>  }
>
>  use IO::Socket::INET;
>  my $socket = IO::Socket::INET->new(PeerAddr => '127.0.0.1',
>   PeerPort => '8090',
>   Proto=> 'udp');
>  $socket->send($dp . "\n");
>  return $dp;
> }
>
> Here's the config I used.
>
> Foreground
> LogStdout
> LogDir  .
> DbDir   .
> Trace   4
>
> 
> Secret  mysecret
> 
>
> 
> Identifier myauthlogger-influxdb
> Filename %L/authlog-influx.txt
> LogFormatHook file:"%D/format-influx.pl"
> LogSuccess 1
> LogFailure 1
> 
>
> 
> 
> Filename %D/users
> 
>
> AuthLog myauthlogger-influxdb
> 
>
>
> --
> Heikki Vatiainen 
>
> Radiator: the most portable, flexible and configurable RADIUS server
> anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
> Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
> TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
> DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS,
> NetWare etc.
> ___
> radiator mailing list
> radiator@open.com.au
> http://www.open.com.au/mailman/listinfo/radiator
>



-- 

Barry Ard   barry@ualberta.ca
IST
University of Alberta
Edmonton, Alberta   Canada
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator

Re: [RADIATOR] Request for enhancement: Log Handler InfluxDB or at least UDP

[Heikki Vatiainen]

On 2.2.2016 13.14, Karl Gaissmaier wrote:

> yes, like heka http://hekad.readthedocs.org as forwarding agent and/or
> anomaly processor.

Interesting, thanks for sharing this.

> Heka has also a sandboxed Lua interpreter to decode unusual log formats,
> maybe I'll not implement the hook in RADIATOR.
>
> Maybe it's really enough to create normal logs and use heka (or similar
> tools)
> to process anomaly detection and forward it to graphite/influxdb.

Meanwhile, I did a basic Influxdb and Grafana installation to test it a 
little. Below is a simple AuthLog FILE format hook that creates an entry 
in Influxdb line protocol format and sends it before logging it to a 
file. It simply removes some of the characters that need to be quoted in 
the line protocol format and creates a new socket for each call. It's 
very primitive but, it will do basic logging and is a quick way to 
experiment and get something stored in Influxdb and visible in Grafana.

The entry that gets logged in authlog file is useful to see how the line 
that was sent to Influxdb was formatted.

# AuthLog in InfluxDB format
sub
{
 my ($s, $reason, $p) = @_;

 my $ap = $p->get_attr('NAS-Identifier');
 my $client_mac = $p->get_attr('Calling-Station-Id');
 my $username = $p->get_attr('User-Name');

 my ($sec, $usec) = Radius::Util::getTimeHires();
 my $influxtime = "$sec$usec"."000";

 # Strip space, \ and "
 # See Influxdb docs for what/how to quote
 $username =~ s/[ \\"]//g;
 $reason =~ s/[ \\"]//g;

 my $dp; # InfluxDB line protocol data point
 if ($s == $main::ACCEPT)
 {
 my $key = 
"radius,type=accept,ap=$ap,special=$username,special_type=username";

 my $fields = "value=\"$username\"";
 $dp = "$key $fields $influxtime";
 }
 elsif ($s == $main::REJECT)
 {
 my $key = 
"radius,type=rejected,ap=$ap,special=$reason,special_type=reason";

 my $fields = "value=\"$username\",special_val=\"$reason\"";
 $dp = "$key $fields $influxtime";
 }

 use IO::Socket::INET;
 my $socket = IO::Socket::INET->new(PeerAddr => '127.0.0.1',
  PeerPort => '8090',
  Proto=> 'udp');
 $socket->send($dp . "\n");
 return $dp;
}

Here's the config I used.

Foreground
LogStdout
LogDir  .
DbDir   .
Trace   4


Secret  mysecret



Identifier myauthlogger-influxdb
Filename %L/authlog-influx.txt
LogFormatHook file:"%D/format-influx.pl"
LogSuccess 1
LogFailure 1




Filename %D/users


AuthLog myauthlogger-influxdb



-- 
Heikki Vatiainen 

Radiator: the most portable, flexible and configurable RADIUS server
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, 
NetWare etc.
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator

Re: [RADIATOR] Request for enhancement: Log Handler InfluxDB or at least UDP

[Karl Gaissmaier]

Hi Heikki,

thanks for fast reply and interest!

Sorry, I was ill, therefore I couldn't answer til yet.


Am 29.01.2016 um 18:31 schrieb Heikki Vatiainen:
> ...
> Yes, this is very interesting. I looked at the line protocol
> specification and it should be easy to implement with a formatting hook
> for authentication. Accounting should be fairly easy too.
>
> It might be worth considering a seprate log agent to forward the logs to
> InfluxDB (or in genral to other logging, graphing, etc. systems). This
> would separate the duties: radiator would create formatted logs and the
> agent could handle the actual log forwarding.


yes, like heka http://hekad.readthedocs.org as forwarding agent and/or 
anomaly processor.

Heka has also a sandboxed Lua interpreter to decode unusual log formats, 
maybe I'll not implement the hook in RADIATOR.

Maybe it's really enough to create normal logs and use heka (or similar 
tools)
to process anomaly detection and forward it to graphite/influxdb.

I'll rethink my request for enhancements.

> This would also make it easier to add accounting and debug log
> forwarding too since they can already be formatted when written to files.
>
> If you need help with logformat hook, just let me know. I am interested
> in helping you with this.
>
>

If I do it as a RADIATOR hook, I'll come back to your offer.

Thank you very much, the RADIATOR team is great!


Best Regards
Charly

-- 
Karl Gaissmaier
Universität Ulm
kiz, Kommunikations und Informationszentrum
89069 Ulm
Tel.: 49(0)731/50-22499
Fax : 49(0)731/50-12-22499

___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator

Re: [RADIATOR] Request for enhancement: Log Handler InfluxDB or at least UDP

[Heikki Vatiainen]

On 26.1.2016 17.31, Karl Gaissmaier wrote:

> I'm in the process to feed an InfluxDB from RADIATOR logfiles. Much
> nicer would it be if RADIATOR team would implement:
>
>  with the very simple but effective line protocol over
> HTTP or at least an generic
>  with a proper logformat hook done by the users and shipped as
> goodies.

How about starting with a logformat hook to generate the datapoints in 
the line protocol format and then using, for example, curl to send the 
files to InfluxDB? I'm think about this:

https://docs.influxdata.com/influxdb/v0.9/guides/writing_data/

and 'Writing points from a file' described therein.

> Interested? Have a look at https://blog.haschek.at/post/fc060

Yes, this is very interesting. I looked at the line protocol 
specification and it should be easy to implement with a formatting hook 
for authentication. Accounting should be fairly easy too.

It might be worth considering a seprate log agent to forward the logs to 
InfluxDB (or in genral to other logging, graphing, etc. systems). This 
would separate the duties: radiator would create formatted logs and the 
agent could handle the actual log forwarding.

This would also make it easier to add accounting and debug log 
forwarding too since they can already be formatted when written to files.

If you need help with logformat hook, just let me know. I am interested 
in helping you with this.

Thanks,
Heikki

-- 
Heikki Vatiainen 

Radiator: the most portable, flexible and configurable RADIUS server
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, 
NetWare etc.
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator

Re: [RADIATOR] Request for enhancement: Log Handler InfluxDB or at least UDP

[Hugh Irvine]

Hi Heikki, Hi Karl -

Two thoughts on this:

1. you can use the “|” pipe character in the “Filename …” parameter of the  clause to pipe the log messages to another program directly, together 
with LogFormat

2. one can easily imagine a new  clause with a hook as a parameter 
to do whatever one might wish, being mindful to limit overhead of course

regards

Hugh


> On 30 Jan 2016, at 04:31, Heikki Vatiainen  wrote:
> 
> On 26.1.2016 17.31, Karl Gaissmaier wrote:
> 
>> I'm in the process to feed an InfluxDB from RADIATOR logfiles. Much
>> nicer would it be if RADIATOR team would implement:
>> 
>>  with the very simple but effective line protocol over
>> HTTP or at least an generic
>>  with a proper logformat hook done by the users and shipped as
>> goodies.
> 
> How about starting with a logformat hook to generate the datapoints in 
> the line protocol format and then using, for example, curl to send the 
> files to InfluxDB? I'm think about this:
> 
> https://docs.influxdata.com/influxdb/v0.9/guides/writing_data/
> 
> and 'Writing points from a file' described therein.
> 
>> Interested? Have a look at https://blog.haschek.at/post/fc060
> 
> Yes, this is very interesting. I looked at the line protocol 
> specification and it should be easy to implement with a formatting hook 
> for authentication. Accounting should be fairly easy too.
> 
> It might be worth considering a seprate log agent to forward the logs to 
> InfluxDB (or in genral to other logging, graphing, etc. systems). This 
> would separate the duties: radiator would create formatted logs and the 
> agent could handle the actual log forwarding.
> 
> This would also make it easier to add accounting and debug log 
> forwarding too since they can already be formatted when written to files.
> 
> If you need help with logformat hook, just let me know. I am interested 
> in helping you with this.
> 
> Thanks,
> Heikki
> 
> -- 
> Heikki Vatiainen 
> 
> Radiator: the most portable, flexible and configurable RADIUS server
> anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
> Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
> TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
> DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, 
> NetWare etc.
> ___
> radiator mailing list
> radiator@open.com.au
> http://www.open.com.au/mailman/listinfo/radiator


--

Hugh Irvine
h...@open.com.au

Radiator: the most portable, flexible and configurable RADIUS server 
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald, 
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS, 
TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
DIAMETER, SIM, etc. 
Full source on Unix, Linux, Windows, MacOSX, Solaris, VMS, NetWare etc.

___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator

[RADIATOR] Request for enhancement: Log Handler InfluxDB or at least UDP

[Karl Gaissmaier]

Hi RADIATOR team,

you know, most of us want to 'Measure Anything, Measure Everything'

https://codeascraft.com/2011/02/15/measure-anything-measure-everything/

I'm in the process to feed an InfluxDB from RADIATOR logfiles. Much 
nicer would it be if RADIATOR team would implement:

 with the very simple but effective line protocol over 
HTTP or at least an generic
 with a proper logformat hook done by the users and shipped as 
goodies.

Interested? Have a look at https://blog.haschek.at/post/fc060

Best Regards
Charly

-- 
Karl Gaissmaier
Universität Ulm
kiz, Kommunikations und Informationszentrum
89069 Ulm
Tel.: 49(0)731/50-22499
Fax : 49(0)731/50-12-22499

___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator