Re: Framed-IP-Address accounted in Hex
On 11 Jul 2006, at 17:00, Alan DeKok wrote: Graeme Hinchliffe [EMAIL PROTECTED] wrote: Is there a chance that on a fast loaded box that threads are accessing the dictionary index which is being dynamically modified (it would appear) and using non-valid memory for their lookup ? resulting in the value being kept as octet and the symptoms we are seeing? Once the dictionaries are loaded and installed, the *lookups* should be thread-safe. The non-thread-safe portion of the hash table is only for insertions or deletes. I'll send you a patch privately that *may* address it, but it's just a guess at this point. Thanks for that, I shall apply it to the 1.1.2 code and install it on the server. I will let you know what we find (positive or not) in a few days Thanks Graeme - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
On 7 Jul 2006, at 17:46, Alan DeKok wrote: Are dictionaries loaded each time a child is started? or just once and then kept in memory? The server doesn't start any children. The dictionaries are loaded once, and cached as long as it's running. Hi, Have been digging through the source for FreeRADIUS 1.1.2. As I am understanding it from my quick dig through. value_pair defaults to octets as the datatype for a given attribute, this is then looked up a bit further on and set to the correct type via the dictionary. Dict.c refers to a function in lib/hash.c and at the top of this source file is the line : Non-thread-safe split-ordered hash table Is there a chance that on a fast loaded box that threads are accessing the dictionary index which is being dynamically modified (it would appear) and using non-valid memory for their lookup ? resulting in the value being kept as octet and the symptoms we are seeing? The most common under heavy load was session ID but we are still seeing the IP address being set as a hex value also, these cases have only been spotted due to the error logging of postgres or the errors it has generated in sub-systems working on the accounting data. I have reduced the number of updates per hour and these problems have almost completely vanished. Thanks Graeme - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
Graeme Hinchliffe [EMAIL PROTECTED] wrote: Is there a chance that on a fast loaded box that threads are accessing the dictionary index which is being dynamically modified (it would appear) and using non-valid memory for their lookup ? resulting in the value being kept as octet and the symptoms we are seeing? Once the dictionaries are loaded and installed, the *lookups* should be thread-safe. The non-thread-safe portion of the hash table is only for insertions or deletes. I'll send you a patch privately that *may* address it, but it's just a guess at this point. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
On 7 Jul 2006, at 17:46, Alan DeKok wrote: Graeme Hinchliffe [EMAIL PROTECTED] wrote: these are hex values, not octal, and it seems to be an intermittent thing. Dang. Those bugs are hard to track down. Yup, looking high and low for correlation Are dictionaries loaded each time a child is started? or just once and then kept in memory? The server doesn't start any children. The dictionaries are loaded once, and cached as long as it's running. damn, thats that theory out then :) Looking through the data we have, as it's easy to spot the errored data with the session ID we are logging when it is occuring. We have noticed this occurs mainly when the accounting server is busy. This would also explain why this has suddenly occured in such large numbers as we have recently increased the number of updates from the NASes. This has obvioulsy increased the load on the server. Looking at the backend DB (postgres) there are very few free postgress handles availible (we are using 50 connections from the server to the DB) if any. When FreeRADIUS has no free connection with which to account the packet what does it do? Does the packet sit in RAM and try again? or is it simply discarded? I am suspecting that this may be a cause for what we are seeing as the problem only seems to occur during server backups and high load. Thanks Graeme - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
On 6 Jul 2006, at 22:20, Alan DeKok wrote: Graeme Hinchliffe [EMAIL PROTECTED] wrote: What would cause FreeRADIUS to output in this manner, we have summized that if it sees a none ASCII byte in the field it would convert the whole field into a hex representation to stop trying to write binary to the db. No, it should print out non-ASCII bytes as octal in that case. It will create octal attributes if it can't find the attribute in the dictionaries. these are hex values, not octal, and it seems to be an intermittent thing. example session ID 0x464631304646464635383230333045322d3434363938363135 and the dictionaries are installed in /usr/share/freeradius where they have been since initial install on this system. The dictionary includes the rfc dictionaries at the start which includes the Acct- Session-Id attribute. Are dictionaries loaded each time a child is started? or just once and then kept in memory? Graeme - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
Graeme Hinchliffe [EMAIL PROTECTED] wrote: these are hex values, not octal, and it seems to be an intermittent thing. Dang. Those bugs are hard to track down. Are dictionaries loaded each time a child is started? or just once and then kept in memory? The server doesn't start any children. The dictionaries are loaded once, and cached as long as it's running. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
On 4 Jul 2006, at 17:01, Alan DeKok wrote: Graeme Hinchliffe [EMAIL PROTECTED] wrote: Further to this, I have just noticed that this doesn't seem to just be restricted to the IP address, but also the Session ID field. Instead of displaying the session ID as say 020268001A6C-44A618FF I am seeing: 0x303230324646464636383030314136432d3434413631384646 Which is the hex dump of the ASCII string. 0x30 == '0'. indeed. I think the dictionaries from 1.1.2 didn't get installed, so many of the attributes default to type octets. I have found entries which indicate this has happened prior to my upgrade to 1.1.2, and also checking I can see that all the dictionary files are the versions that came with the 1.1.2 code. What would cause FreeRADIUS to output in this manner, we have summized that if it sees a none ASCII byte in the field it would convert the whole field into a hex representation to stop trying to write binary to the db. If this is the case then it could well be a bug with the hardware and we can chase the vendor and apply a patch to the code in freeradius until we get it fixed. Also, when I ran the code in debug mode (-X switch) I never saw one occurence of this problem in the screen output or the postgres logs. Of course computers being what they are it's most likley that it was just a fluke, but at what point would FreeRADIUS decide to convert the output to a hex representation? our NASes are all Redbacks, SE400's and SMS10K's Thanks Graeme - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
On 6 Jul 2006, at 09:58, Graeme Hinchliffe wrote: On 4 Jul 2006, at 17:01, Alan DeKok wrote: Graeme Hinchliffe [EMAIL PROTECTED] wrote: Further to this, I have just noticed that this doesn't seem to just be restricted to the IP address, but also the Session ID field. Instead of displaying the session ID as say 020268001A6C-44A618FF I am seeing: 0x303230324646464636383030314136432d3434413631384646 Which is the hex dump of the ASCII string. 0x30 == '0'. indeed. I think the dictionaries from 1.1.2 didn't get installed, so many of the attributes default to type octets. I have found entries which indicate this has happened prior to my upgrade to 1.1.2, and also checking I can see that all the dictionary files are the versions that came with the 1.1.2 code. What would cause FreeRADIUS to output in this manner, we have summized that if it sees a none ASCII byte in the field it would convert the whole field into a hex representation to stop trying to write binary to the db. If this is the case then it could well be a bug with the hardware and we can chase the vendor and apply a patch to the code in freeradius until we get it fixed. Also, when I ran the code in debug mode (-X switch) I never saw one occurence of this problem in the screen output or the postgres logs. Of course computers being what they are it's most likley that it was just a fluke, but at what point would FreeRADIUS decide to convert the output to a hex representation? our NASes are all Redbacks, SE400's and SMS10K's I just checked a bit further, and noticed the problem with on eof our Dialup NASes which are Cisco AS5300's so I don't think it is a hardware vendor issue. Not as many sessions go through the Dialin racks anymore so not seeing as much data from them. Graeme - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
Graeme Hinchliffe [EMAIL PROTECTED] wrote: What would cause FreeRADIUS to output in this manner, we have summized that if it sees a none ASCII byte in the field it would convert the whole field into a hex representation to stop trying to write binary to the db. No, it should print out non-ASCII bytes as octal in that case. It will create octal attributes if it can't find the attribute in the dictionaries. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Framed-IP-Address accounted in Hex
Hi, I have just (as of yesterday) upgraded FreeRADIUS from 1.0.3 to 1.1.2 keeping all the same config files. I am running it on a Debian Woody system and using Postgres 7.4.7 as the database. This particular box is solely for RADACCT and only uses RADIUS as a test to test the state of the service/db for health checks. I have postgres configured to dump any SQL lines which throw up an error and I have seen a large number being logged failing to insert the Framed-IP-Address attribute, in the dumped SQL string the IP Address is a hex value instead of a dotted quad which fails the insert. The RADACCT packets causing this are not just coming from one NAS either, and accounting still seems to be happening on the most part.any ideas? or have i found a bug ?thanks - Graeme Hinchliffe (BSc) Core Systems Designer Zen Internet (http://www.zen.co.uk/) Direct: 0845 058 9074 Main : 0845 058 9000 Fax : 0845 058 9005 - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
On 4 Jul 2006, at 12:05, Graeme Hinchliffe wrote:Hi, I have just (as of yesterday) upgraded FreeRADIUS from 1.0.3 to 1.1.2 keeping all the same config files. I am running it on a Debian Woody system and using Postgres 7.4.7 as the database. This particular box is solely for RADACCT and only uses RADIUS as a test to test the state of the service/db for health checks. I have postgres configured to dump any SQL lines which throw up an error and I have seen a large number being logged failing to insert the Framed-IP-Address attribute, in the dumped SQL string the IP Address is a hex value instead of a dotted quad which fails the insert. The RADACCT packets causing this are not just coming from one NAS either, and accounting still seems to be happening on the most part.any ideas? or have i found a bug ?Further to this, I have just noticed that this doesn't seem to just be restricted to the IP address, but also the Session ID field. Instead of displaying the session ID as say 020268001A6C-44A618FFI am seeing:0x303230324646464636383030314136432d3434413631384646 - Graeme Hinchliffe (BSc) Core Systems Designer Zen Internet (http://www.zen.co.uk/) Direct: 0845 058 9074 Main : 0845 058 9000 Fax : 0845 058 9005 - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Framed-IP-Address accounted in Hex
Graeme Hinchliffe [EMAIL PROTECTED] wrote: Further to this, I have just noticed that this doesn't seem to just be restricted to the IP address, but also the Session ID field. Instead of displaying the session ID as say 020268001A6C-44A618FF I am seeing: 0x303230324646464636383030314136432d3434413631384646 Which is the hex dump of the ASCII string. 0x30 == '0'. I think the dictionaries from 1.1.2 didn't get installed, so many of the attributes default to type octets. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
