Re: [RADIATOR] Log file encoding

[Heikki Vatiainen]

On 17.6.2015 11.10, Tim Jones wrote:

> I've been asked to ensure that the log files created by our Radiator
> instance are using a consistant particular character encoding. As when
> doing some analysis, it has been discovered that some logs are latin1,
> some us-ascii, and some utf-8.

Mostly they should be in us-ascii, but some log messages include binary 
attribute values, output from libraries and other non-Radiator Perl 
modules, Radius messages, etc. and this can cause non-ascii characters 
in logs.

> I cannot find any reference to character encoding within the Radiator
> documentation, but of course all string-type attributes in the RADIUS
> protocol are defined as UTF-8 encoded so I would have expected that of
> the Radiator logs as well.

The RFCs specify UTF-8, but it is not uncommon to see other encodings 
too. For example, a web captive portal may have its login page 
configured to use UTF-8 or ISO-8859-15. When a user logs in, their 
username may be passed unmodified to RADIUS authentication by the 
controller.

> Is there an option supporting this within Radiator, or is it more do
> with with the Perl environment as a whole?

I recommend considering a separate utility to process the logs to your 
log storage system. Especially when you use debug logging, the logs will 
contain non-ascii characters. There's currently no option to make sure 
all characters in log messages are from e.g., us-ascii.

Thanks,
Heikki

-- 
Heikki Vatiainen 

Radiator: the most portable, flexible and configurable RADIUS server
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, 
NetWare etc.
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator

Re: [RADIATOR] Log file encoding

[Tim Jones]

Thanks for the response.

I recommend considering a separate utility to process the logs to your log
> storage system.


Essentially what I have been specced to build, but it's a batch-processing
after-the-fact, so we only have the log files to go on. I will suggest the
idea of logging to something else in addition to files (Syslog\SQL) but I
have been rebuffed in the past on this same point. Though since this also
fixes other problems (users leaving carriage-returns in user-name fields
splitting the log line when printed to a file), I may get a better
reception this time around.

There's currently no option to make sure all characters in log messages are
> from e.g., us-ascii.


Are there plans to add support for this? Looking into it it seems that, for
Perl, simply having a call to open(FILE, ">>$filename") replaced with
open(FILE,
'>>:encoding(UTF-8)', $filename) should do the trick. Since it looks as if
all log file writing is done via Radius::Util::append, would it be a hard
feature to add?

Many thanks for your time,


[image: Fon] Tim JonesSoftware Development+34 612345678
C/ Quintanavides 15, Edificio 2, Planta 1ª
Parque Empresarial Vía Norte de Metrovacesa
Las Tablas
28050 MadridSkype: tim.jones.fonAll information in this email is
confidential 

On 18 June 2015 at 09:20, Heikki Vatiainen  wrote:

> On 17.6.2015 11.10, Tim Jones wrote:
>
> > I've been asked to ensure that the log files created by our Radiator
> > instance are using a consistant particular character encoding. As when
> > doing some analysis, it has been discovered that some logs are latin1,
> > some us-ascii, and some utf-8.
>
> Mostly they should be in us-ascii, but some log messages include binary
> attribute values, output from libraries and other non-Radiator Perl
> modules, Radius messages, etc. and this can cause non-ascii characters
> in logs.
>
> > I cannot find any reference to character encoding within the Radiator
> > documentation, but of course all string-type attributes in the RADIUS
> > protocol are defined as UTF-8 encoded so I would have expected that of
> > the Radiator logs as well.
>
> The RFCs specify UTF-8, but it is not uncommon to see other encodings
> too. For example, a web captive portal may have its login page
> configured to use UTF-8 or ISO-8859-15. When a user logs in, their
> username may be passed unmodified to RADIUS authentication by the
> controller.
>
> > Is there an option supporting this within Radiator, or is it more do
> > with with the Perl environment as a whole?
>
> I recommend considering a separate utility to process the logs to your
> log storage system. Especially when you use debug logging, the logs will
> contain non-ascii characters. There's currently no option to make sure
> all characters in log messages are from e.g., us-ascii.
>
> Thanks,
> Heikki
>
> --
> Heikki Vatiainen 
>
> Radiator: the most portable, flexible and configurable RADIUS server
> anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
> Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
> TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
> DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS,
> NetWare etc.
> ___
> radiator mailing list
> radiator@open.com.au
> http://www.open.com.au/mailman/listinfo/radiator
>
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator

Re: [RADIATOR] Log file encoding

[Heikki Vatiainen]

On 18.6.2015 10.46, Tim Jones wrote:

> Essentially what I have been specced to build, but it's a
> batch-processing after-the-fact, so we only have the log files to go on.
> I will suggest the idea of logging to something else in addition to
> files (Syslog\SQL) but I have been rebuffed in the past on this same
> point.

What if you use something like Logstash? It can do similar to 'tail -f' 
to read the log file and forward the new entries as they are written 
into the log file.

> Are there plans to add support for this? Looking into it it seems that,
> for Perl, simply having a call to open(FILE, ">>$filename") replaced
> with open(FILE, '>>:encoding(UTF-8)', $filename) should do the trick.
> Since it looks as if all log file writing is done via
> Radius::Util::append, would it be a hard feature to add?

There are no plans yet, but I'll make a note about this.

Meanwhile, check the 4.14 patches and goodies/logformat.cfg. The 
LogFormatHook is already present in 4.14 and the patches now have more 
examples about how to use it. You might be able to use the hooks to make 
sure the logs are formatted as required.

Thanks,
Heikki

-- 
Heikki Vatiainen 

Radiator: the most portable, flexible and configurable RADIUS server
anywhere. SQL, proxy, DBM, files, LDAP, NIS+, password, NT, Emerald,
Platypus, Freeside, TACACS+, PAM, external, Active Directory, EAP, TLS,
TTLS, PEAP, TNC, WiMAX, RSA, Vasco, Yubikey, MOTP, HOTP, TOTP,
DIAMETER etc. Full source on Unix, Windows, MacOSX, Solaris, VMS, 
NetWare etc.
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator

Re: [RADIATOR] Log file encoding

[Hartmaier Alexander]

@Heikki: please read Tom Christensens great article about encoding:
http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default/

Short summary: each input value has to be decoded, each output encoded,
everything else is useless!

BR Alex

On 2015-06-18 10:36, Heikki Vatiainen wrote:
> On 18.6.2015 10.46, Tim Jones wrote:
>
>> Essentially what I have been specced to build, but it's a
>> batch-processing after-the-fact, so we only have the log files to go on.
>> I will suggest the idea of logging to something else in addition to
>> files (Syslog\SQL) but I have been rebuffed in the past on this same
>> point.
> What if you use something like Logstash? It can do similar to 'tail -f'
> to read the log file and forward the new entries as they are written
> into the log file.
>
>> Are there plans to add support for this? Looking into it it seems that,
>> for Perl, simply having a call to open(FILE, ">>$filename") replaced
>> with open(FILE, '>>:encoding(UTF-8)', $filename) should do the trick.
>> Since it looks as if all log file writing is done via
>> Radius::Util::append, would it be a hard feature to add?
> There are no plans yet, but I'll make a note about this.
>
> Meanwhile, check the 4.14 patches and goodies/logformat.cfg. The
> LogFormatHook is already present in 4.14 and the patches now have more
> examples about how to use it. You might be able to use the hooks to make
> sure the logs are formatted as required.
>
> Thanks,
> Heikki
>



*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien
Handelsgericht Wien, FN 79340b
*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
Notice: This e-mail contains information that is confidential and may be 
privileged.
If you are not the intended recipient, please notify the sender and then
delete this e-mail immediately.
*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
___
radiator mailing list
radiator@open.com.au
http://www.open.com.au/mailman/listinfo/radiator