Re: radrelay transmission rate (Kostas Kalevras)
I 've been working on a few changes to radrelay, mainly regarding making
the
sleep time configurable and adding a few more things. The changes have
been
made
in radsqlrelay initialy but they 'll go in radrelay also. That won't
change
your
numbers but at least make a few things configurable.
good, that's a useful intermediate step - will allow to tune without
changing code.
I also tried various values of NR_SLOTS, but it doesn't change the
overall
time it takes to transfer a large backlog of accounting requests.
Well, NR_SLOTS does not really matter if your accounting is quick enhough.
Try commenting out the ms_sleep() between the do_send() calls.
I had tried this too, and just retried again, and the rate goes down to ~30
packets/sec.
Looks like the trafic becomes quite bursty, and the retransmission pattern
causes intermittent silent periods.
For example, packet 434 is a response from RS2 to RS1, and packet 435 goes
out 14.52
seconds after it (request from RS1 to RS2). I saw 37 periods of 1-sec or
more silence,
listed in this below:
No.TimeSource Destination Info
651.900217192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=616)
1292.741964192.168.12.80 192.168.12.34 Accounting Response(5)
(id=7, l=20)
2881.857164192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=618)
2905.989420192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=616)
35611.695879 192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=616)
43514.520559 192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=618)
7361.346075192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=618)
9112.044088192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=618)
10792.088782192.168.12.34 192.168.12.80 Accounting Request(4)
(id=1, l=618)
12641.970677192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=618)
14631.826821192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=618)
16601.881720192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
19161.593158192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
20651.727769192.168.12.34 192.168.12.80 Accounting Request(4)
(id=1, l=620)
27621.740835192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
28314.966427192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
28858.717565192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
32302.007186192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
33762.199381192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
35971.820478192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
37311.794183192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
41672.125003192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
47072.043159192.168.12.34 192.168.12.80 Accounting Request(4)
(id=1, l=620)
49651.586214192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
51312.111933192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
52982.088369192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
55211.750188192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
58921.934175192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
64591.809148192.168.12.34 192.168.12.80 Accounting Request(4)
(id=1, l=620)
68551.856238192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
70292.015883192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
73901.962640192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
77641.936479192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
82461.374217192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
84921.877480192.168.12.34 192.168.12.80 Accounting Request(4)
(id=1, l=620)
90222.055465192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
94911.375418192.168.12.34 192.168.12.80 Accounting Request(4)
(id=0, l=620)
Note that the CPU on RS1 (running the radrelay instance that is
processing
the backlog) still takes less than 1% of CPU, with or without the above
changes.
That's normal it's an I/O bound application.
well, CPU is ~0%, and so is disk usage, and bandwidth usage. For the moment,
no resource is pushed to its limit.
threading sound like an idea yes. Another idea is to get load-balancing
code
inside freeradius. Then you could do something like the following:
accounting {
loadbalance {
relay_detail1
relay_detail2
Re: radrelay transmission rate (Kostas Kalevras)
On Mon, 6 Dec 2004, Bruno Lague wrote:
Well, NR_SLOTS does not really matter if your accounting is quick enhough.
Try commenting out the ms_sleep() between the do_send() calls.
I had tried this too, and just retried again, and the rate goes down to ~30
packets/sec.
Looks like the trafic becomes quite bursty, and the retransmission pattern
causes intermittent silent periods.
For example, packet 434 is a response from RS2 to RS1, and packet 435 goes
out 14.52
seconds after it (request from RS1 to RS2). I saw 37 periods of 1-sec or
more silence,
The silence can be attributed to the backoff mechanism of radrelay.
Maybe we could sleep after we 've sent a few packets instead of sleeping after
every packet.You can try to put in ms_sleep() after a few packets have been
sent (say every 4-5 packets). That's a nice thing to know, thanks. I 'll put it
in radrelay anyway. I also need to take a look at radclient and see if it's
worth to use the same mechanism.
Note that the CPU on RS1 (running the radrelay instance that is
processing
the backlog) still takes less than 1% of CPU, with or without the above
changes.
That's normal it's an I/O bound application.
well, CPU is ~0%, and so is disk usage, and bandwidth usage. For the moment,
no resource is pushed to its limit.
Well, even for 200 reqs/sec, that's ~200KB/sec for the disk and much less for
the network. As for CPU, i don't think that rate can be a problem. So 0-1% is
normal.
threading sound like an idea yes. Another idea is to get load-balancing
code
inside freeradius. Then you could do something like the following:
accounting {
loadbalance {
relay_detail1
relay_detail2
relay_detail3
}
}
radrelay relay_detail1
radrelay relay_detail2
radrelay relay_detail3
That way you don't need to change much (apart from a few changes to the
server
core) and you increase the overall performance by parallelizing radrelay
and
the detail module.
Agree that would work too. I'd be glad to give it a try as soon as it's
available.
Well, Alan was quite nice to provide that feature in a flash :-D, so it's
available in CVS.
thanks,
Bruno
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
--
Kostas Kalevras Network Operations Center
[EMAIL PROTECTED] National Technical University of Athens, Greece
Work Phone: +30 210 7721861
'Go back to the shadow' Gandalf
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: radrelay transmission rate
threading sound like an idea yes. Another idea is to get load-balancing
code
inside freeradius. Then you could do something like the following:
accounting {
loadbalance {
relay_detail1
relay_detail2
relay_detail3
}
}
radrelay relay_detail1
radrelay relay_detail2
radrelay relay_detail3
That way you don't need to change much (apart from a few changes to the
server
core) and you increase the overall performance by parallelizing radrelay
and
the detail module.
Agree that would work too. I'd be glad to give it a try as soon as it's
available.
Well, Alan was quite nice to provide that feature in a flash :-D, so it's
available in CVS.
Trying to try it, but can't get it to work. I must be doing something
obviously
wrong, just can't see what.
- I downloaded freeradius-snapshot-20041205.tar.gz + modcall.c v1.26.
- configure/make/make install'ed
- added the above loadbalance section in accounting section of
radiusd.conf,
- added a detail item for each entry of that loadbalance section (also
tried
without it, same result).
- start radiusd, and get this:
radiusd.conf[1808] Unknown module rcode 'relay_detail1'.
any help appreciated...
thanks,
Bruno
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: radrelay transmission rate
Bruno Lague [EMAIL PROTECTED] wrote: - added the above loadbalance section in accounting section of radiusd.conf, The name was implemented as load-balance. - start radiusd, and get this: radiusd.conf[1808] Unknown module rcode 'relay_detail1'. It's trying to load a module called loadbalance. Change the name, and it should work. I'll write some more docs in a few days... Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: radrelay transmission rate
Kostas Kalevras [EMAIL PROTECTED] wrote: That value should be configurable at some point. It was selected somewhat large so that radrelay didn't create problems to slow radius servers when it started sending a large detail file. It should instead send packets as fast as the server responds. You can safely make it as small as you want. radrelay is single threaded and serialized so there 's not much to worry about (although the single threading should change also). I don't see any benefit in having it threaded. See the updated radclient for code which keeps track of many packets simultaneously, and does it for a controlled maximum rate. That code could probably be re-used in radrelay. Alan DeKok. Hi, a few notes on tests I did to transfer a large backlog of accounting request from server RS1 to RS2: 1. Using radrelay, the best I could achieve is about 50 packets/sec, with the following two changes: # sdiff -s radrelay.c radrelay.c.backupOriginal tv.tv_usec = 1000; | tv.tv_usec = 25000; ms_sleep(1);| ms_sleep(140); I also tried various values of NR_SLOTS, but it doesn't change the overall time it takes to transfer a large backlog of accounting requests. Note that the CPU on RS1 (running the radrelay instance that is processing the backlog) still takes less than 1% of CPU, with or without the above changes. 2. With radclient, I got the following results, using it unchanged: #radclient -c 4600 -f afew -r 5 -t 1 192.168.12.80 acct pwtest - v1.72 (the one in freeradius 1.0.1): 87.9 pkts/sec - v1.76 (from Dec 3 CVS snapshot): 86.7 pkts/sec == about the same for v72 v76, but definitely faster than radrelay. btw, when using the -n option in 1.76, the best rate goes down to 50 pq/sec, just like radrelay (even when using a large number, say -n 100, or -n 200). It seems to be due to the fact that select(0, NULL, NULL, NULL, tv) sleeps for a minimal time. This time appears to be 10 ms on the two platforms I used for testing (one Linux/I386, one SUN Solaris/UltraSPARC-II 296MHz) 3. Results in (1) and (2) above are way below my target of 200 packets/sec, and way below what radiusd can take from a NAS. That 200 p/s rate is needed to keep up with my NAS (which can sustain that 200 p/s for a few hours), and avoid accumulating a backlog when both RS1 and RS2 are up. So, threading of radrelay sounds like a must. Agree, or are there other configuration parameters I can try? thanks, -Bruno - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: radrelay transmission rate
On Sun, 5 Dec 2004, Bruno Lague wrote:
Kostas Kalevras [EMAIL PROTECTED] wrote:
That value should be configurable at some point. It was selected
somewhat large so that radrelay didn't create problems to slow
radius servers when it started sending a large detail file.
It should instead send packets as fast as the server responds.
You can safely make it as small as you want. radrelay is single
threaded and serialized so there 's not much to worry about
(although the single threading should change also).
I don't see any benefit in having it threaded. See the updated
radclient for code which keeps track of many packets simultaneously,
and does it for a controlled maximum rate. That code could probably be
re-used in radrelay.
Alan DeKok.
Hi,
a few notes on tests I did to transfer a large backlog of accounting
request from server RS1 to RS2:
I 've been working on a few changes to radrelay, mainly regarding making the
sleep time configurable and adding a few more things. The changes have been made
in radsqlrelay initialy but they 'll go in radrelay also. That won't change your
numbers but at least make a few things configurable.
1. Using radrelay, the best I could achieve is about 50 packets/sec,
with the following two changes:
# sdiff -s radrelay.c radrelay.c.backupOriginal
tv.tv_usec = 1000; | tv.tv_usec = 25000;
ms_sleep(1);| ms_sleep(140);
I also tried various values of NR_SLOTS, but it doesn't change the overall
time it takes to transfer a large backlog of accounting requests.
Well, NR_SLOTS does not really matter if your accounting is quick enhough. Try
commenting out the ms_sleep() between the do_send() calls.
Note that the CPU on RS1 (running the radrelay instance that is processing
the backlog) still takes less than 1% of CPU, with or without the above
changes.
That's normal it's an I/O bound application.
2. With radclient, I got the following results, using it unchanged:
#radclient -c 4600 -f afew -r 5 -t 1 192.168.12.80 acct pwtest
- v1.72 (the one in freeradius 1.0.1): 87.9 pkts/sec
- v1.76 (from Dec 3 CVS snapshot): 86.7 pkts/sec
== about the same for v72 v76, but definitely faster than radrelay.
btw, when using the -n option in 1.76, the best rate goes down to 50 pq/sec,
just like radrelay (even when using a large number, say -n 100, or -n 200).
It seems to be due to the fact that select(0, NULL, NULL, NULL, tv)
sleeps
for a minimal time. This time appears to be 10 ms on the two platforms I
used
for testing (one Linux/I386, one SUN Solaris/UltraSPARC-II 296MHz)
3. Results in (1) and (2) above are way below my target of 200 packets/sec,
and way below what radiusd can take from a NAS.
That 200 p/s rate is needed to keep up with my NAS (which can sustain
that 200 p/s for a few hours), and avoid accumulating a backlog when
both RS1 and RS2 are up.
So, threading of radrelay sounds like a must. Agree, or are there other
configuration parameters I can try?
threading sound like an idea yes. Another idea is to get load-balancing code
inside freeradius. Then you could do something like the following:
accounting {
loadbalance {
relay_detail1
relay_detail2
relay_detail3
}
}
radrelay relay_detail1
radrelay relay_detail2
radrelay relay_detail3
That way you don't need to change much (apart from a few changes to the server
core) and you increase the overall performance by parallelizing radrelay and the
detail module.
thanks,
-Bruno
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
--
Kostas Kalevras Network Operations Center
[EMAIL PROTECTED] National Technical University of Athens, Greece
Work Phone: +30 210 7721861
'Go back to the shadow' Gandalf
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: radrelay transmission rate
Kostas Kalevras [EMAIL PROTECTED] wrote: threading sound like an idea yes. Another idea is to get load-balancing code inside freeradius. Then you could do something like the following: ... ~ 40 lines of code in src/main/mod*.c It's pretty trivial. Copy the pick a random element from a list code from src/main/proxy.c, and apply it to the module sections. That way we can have load balancing among LDAP and SQL servers, too. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: radrelay transmission rate
Kostas Kalevras [EMAIL PROTECTED] wrote: That value should be configurable at some point. It was selected somewhat large so that radrelay didn't create problems to slow radius servers when it started sending a large detail file. It should instead send packets as fast as the server responds. You can safely make it as small as you want. radrelay is single threaded and serialized so there 's not much to worry about (although the single threading should change also). I don't see any benefit in having it threaded. See the updated radclient for code which keeps track of many packets simultaneously, and does it for a controlled maximum rate. That code could probably be re-used in radrelay. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: radrelay transmission rate
On Mon, 29 Nov 2004, Bruno Lague wrote:
Hi,
I want to use radrelay as described in the doc for accounting replication.
My testing so far shows that, functionally, it works well. I'm experiencing
a
glitch in speed though. Here is a short description of the problem:
- Normally, both radiusd and radrelay are running on two servers, RS1 and
RS2.
- RS2 is shutdown.
- The detail file dedicated to replication on RS1 accumulates records at the
same rate the NAS is sending, that is, about 200 accounting requests/sec
(a rate that can be sustained for hours, and with which radiusd has no
problems).
- the NAS is stopped after about 25 seconds - there are about 5000 records
in the replication detail file.
- RS2 is brought back up.
From there, radrelay sends the packets from RS1 to RS2 at about 5
packets/sec.
Snooping shows that RS2 sends a response back after 2-3 ms after it gets
a request, and that RS1 takes about 180-220ms before sending another request
once it receives a response to the last sent request.
Looking at radrelay.c, I saw that there is a 140ms sleep in the middle of
the main processing loop:
for (i = 0; i NR_SLOTS; i++) {
if (slots[i].state == STATE_FULL) {
n += do_send(slots[i], r_args-secret);
ms_sleep(140); /** ---here
*/
if (n NR_SLOTS / 2)
break;
}
}
Not sure why this sleep time has been introduced. For the sake of seeing the
effect, I changed that sleep time to 1ms, and then the transmission rate
reaches about 50 packets/sec, but I didn't check if this introduced any sort
of
race conditions.
That rate is better, but still too slow to cope with the NAS rate.
Can radrelay be configured to be as fast as radiusd? If yes, I didn't find
it
and any help would be appreciated. If not, is this an already planned
improvement?
That value should be configurable at some point. It was selected somewhat large
so that radrelay didn't create problems to slow radius servers when it started
sending a large detail file. You can safely make it as small as you want.
radrelay is single threaded and serialized so there 's not much to worry about
(although the single threading should change also).
thanks in advance,
Bruno
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
--
Kostas Kalevras Network Operations Center
[EMAIL PROTECTED] National Technical University of Athens, Greece
Work Phone: +30 210 7721861
'Go back to the shadow' Gandalf
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
