Send kea-dev mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/kea-dev
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of kea-dev digest..."
Today's Topics:
1. Re: Design for Kea Host Reservation (Marcin Siodelski)
2. Re: Design for Kea Host Reservation (Marcin Siodelski)
3. Re: Design for Kea Host Reservation (Tomek Mrugalski)
4. Re: Design for Kea Host Reservation (Tomek Mrugalski)
5. Re: Design for Kea Host Reservation (Marcin Siodelski)
----------------------------------------------------------------------
Message: 1
Date: Mon, 06 Oct 2014 15:46:20 +0200
From: Marcin Siodelski <[email protected]>
To: Thomas Markwalder <[email protected]>
Cc: [email protected]
Subject: Re: [kea-dev] Design for Kea Host Reservation
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
On 03/10/14 12:47, Thomas Markwalder wrote:
> On 9/30/14, 7:20 AM, Marcin Siodelski wrote:
> > All,
> >
> > I have created a new design document on Kea wiki:
> > http://kea.isc.org/wiki/HostReservationDesign for Host Reservation. It
> > summarizes the discussions that we conducted on a mailing list and
> > during the Kea hackathon.
> >
> > The document introduces detailed layout of the database for host
> > reservations. It also presents relations between the old and new C++
> > classes and some other details.
> >
> > At present, the document doesn't cover the design of a management tool
> > for updating and adding new host reservations to the database. I think
> > this tool is out of scope for now.
> >
> > The Trac ticket for creating a HR design is #3559. This ticket is now in
> > the review queue and the regular review should be conducted by one of
> > ISC engineers. I also encourage mail list subscribers to have a look and
> > comment.
> >
> > Marcin
> > _______________________________________________
> > kea-dev mailing list
> > [email protected]
> > https://lists.isc.org/mailman/listinfo/kea-dev
> Marcin:
>
> I'm sending this out to the email list to share our jabber-chat
> yesterday. I applaud your effort, as a stand-alone component your
> document is coherent and the design is complete enough to implement.
> Your design describes host reservation "persistence" but not how this
> will be put to use within Kea. Without reviewing it within the larger
> context of a how it will integrate into Kea, reaching a full opinion on
> its viability is impossible.
>
> I believe what is needed is a high level design as a prerequisite to
> your design so one can see how host reservation use cases will flow
> through Kea. It should cover configuration use cases as well as DHCP
> client use cases.
>
> It is my impression that there was a lot of discussion and perhaps
> emails regarding the overall design, and if so codifying should not be
> too grate an effort, however email chains do-not-a-design make ;)
> Without an overall design document I think there is too great a risk for
> stumbling into unaccounted for situations or needs.
>
>
Tom,
You have a valid point. The Host Reservation indeed introduces some more
complexity to the message processing logic, as well as to the allocation
engine and this additional complexity deserves documentation. Thanks for
pointing this out.
I updated the http://kea.isc.org/wiki/HostReservationDesign document by
adding a new section "Integration of HR into DHCP Messsage Processing"
to address this. It comes with simplified flow diagrams which present
how the server will process DHCPv6 Renew/Rebind, DHCPv6 Request and
DHCPREQUEST messages with the Host Reservations in use.
Note that while diagrams sometimes dive into the details of message
processing (especially a Renew/Rebind case which seems to be more
complex), they are not meant to cover message processing end-to-end
because we do not try to design the server from the scratch. Having said
that, we may need to go over the current logic to check if we are always
doing correct things according to RFCs.
On the diagrams I tried to highlight the new elements in the message
processing. These new elements will obviously have "some" impact on the
server's performance but I think this impact may be minimized. One
notable factor is a use of SQL database, which will result in additional
queries to obtain host reservations. There are many ways to mitigate
this problem:
- SQL database tuning
- caching query results (e.g. client sends DISCOVER, get client's
reservation and keep in memory until DHCPREQUEST(s) so as additional
queries are not needed)
- dump the database to the in-memory container and don't do any
subsequent queries, expect for new reservations
- do not use database at all (and simply hold reservations in the
configuration file)
There is a great chance that if we wanted to document the ways to
mitigate the host reservation impact on the performance we would end up
having 50 page long document and many of us would still be unhappy about
some of the solutions, or lack of thereof. So, I got away from trying to
address the caching problem in the HR design doc as I want to first get
it running functionally. At the same time, I am confident that the
design, as it is now, doesn't preclude any future optimizations in this
respect.
For the Host Reservation, there is an assumption that the server will
always try to use reserved resources for a host, if any. But, if the
reserved resource is unavailable for some reason (e.g. is in use) the
server should still be able to provision the client by allocating some
other resource. We may obviously speculate whether it is always
appropriate for the server to allocate a different address than the one
that the administrator wanted a client to get and whether the client
should rather not be provisioned in such case. But, I think it is not a
problem to make this configurable at the later time once the whole logic
is in place. So, this discussion is out of scope in the doc.
Marcin
------------------------------
Message: 2
Date: Mon, 06 Oct 2014 15:55:31 +0200
From: Marcin Siodelski <[email protected]>
To: Tomek Mrugalski <[email protected]>, [email protected]
Subject: Re: [kea-dev] Design for Kea Host Reservation
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
Tomek,
Thanks for the review. I addressed most of the things you pointed out in
the review.
For those things that I haven't addressed I provide short responses below.
On 01/10/14 20:52, Tomek Mrugalski wrote:
> On 30.09.2014 13:20, Marcin Siodelski wrote:
>> I have created a new design document on Kea wiki:
>> http://kea.isc.org/wiki/HostReservationDesign for Host Reservation.
> Here are my comments. I like the design a lot. I think it will address
> our needs. We will probably tweak it a little bit over time, but it a
> very solid proposal. Here are my comments:
>
> Information stored for HR section
> IP address(es) - should mention that list is for IPv6 only, v4 need a
> single address only. There's no need to reserve more than one IPv4
> address. Also, please explicitly say that IP means IPv4 and IPv6.
>
> Temporary addresses. We do not support temporary addresses at all and
> in a sane environment, there is no requirement to reserve them in any
> way. On the contrary, it should be an explicit non-requirement. We
> should probably state that somewhere in the design (and hope that odd
> users wanting their temporaries to be static will never appear :)
>
> We should add that currently there is no way to define what options
> can be sent for a given class. This capability will be added in the
> future.
>
> I'd love to see DUID reservation example for DHCPv4.
>
> I like the paragraph explaining what the prefixes (dhcp_, dhcp4_ etc.)
> mean. This is sort of intuitive, but it's nice to have it explicitly
> stated.
>
> Shouldn't host_id and reservation_id be of type primary key? The diagram
> in MySQL database schema suggests it is a simple int. I'm not a
> MySQL expert, so this is an honest question. I think that setting a
> field to primary key type introduces three things: first, it is being
> assigned automatically during inserts. Second, the next value is being
> stored and applied automatically (autoincrement). Third, a unique
> restriction is added.
Yes, they are supposed to be primary key. I don't know why the SQL has
been generated without them being primary keys.
>
> I like the BEFORE DELETE trigger, but I think we need something
> similar for the other way. Something like BEFORE INSERT that checks
> that the host_id exists when adding entries into ipv6_reserverations.
> Is there a BEFORE UPDATE trigger? It could be useful, too.
No, this is handled by the fact that the ipv6_reservations table has an
"Identifying Relation" with a hosts table. The entry in the
ipv6_reservations table can't exist without an appropriate entry in the
hosts table. So, this is already handled.
>
> Comment for dhcp_identifier in the table does not mention that it must
> not be null. I think we won't support a case when no identifier is
> given at all.
>
> Description for ipv4_address is strange: "This value must not be set
> to NOT-NULL if dhcp4_subnet_id is not specified." It would be better
> to simply say "must be NULL if ..."
>
> I'm thinking that we may specify hostname a bit more precisely. How
> about this: it may contain a single label, e.g. "host-foo" or a FQDN,
> e.g. "host-foo.example.org.". If only hostname is specified, it will
> be appended with domain name specified for a given subnet. I can
> imagine that sysadmins would hate to write down their own domain
> thousands of times.
>
> Why do we need reservation_id in the ipv6_reservations at all? To
> refer to specific option when updating it manually? That's fine, but
> if that is the rationale for it, we should say in the description that
> the field is not going to be actively used by the server and can be
> used by third party during database maintenance.
>
> prefix_len description should mention that allowed length is 0..128.
>
> Thanks for explaining the rationale for splitting v4 and v6 options
> into separate tables. "Why?" was my immediate question when I saw the
> tables layout. Your explanation answers my doubt.
>
> I'm a total MySQL noob, so forgive my ignorance, but why do we need
> ALLOW_INVALID_DATES? MySQL does not simply insert invalid dates, and
> sort of sanitizes it to zeros. MySQL doc
> (http://dev.mysql.com/doc/refman/5.5/en/sql-mode.html, also applies to
> older versions) says that invalid date of 2014-04-31 will be sanitized
> to 0000-00-00.
I don't have strong opinion here. In fact, the database schema doesn't
use any time values at the moment. So, having this one way or another
doesn't hurt. The reason why it is set to this value is that the MySQL
workbench tends to inject this into the generated SQL. How about we
revisit this if we introduce some time/date columns to the database?
>
> Space description: It may be useful to add a reference to Kea ARM,
> section 6.2.9 (v4) and 7.2.11 (v6) for people who are unfamiliar with
> the concept of option spaces. It is pretty Kea and ISC DHCP specific.
>
> The section about how we interpret dhcp_client_class, host_id and
> subnet_id need to be more precise. People may read it, think that
> we'll support something and the be disappointed if we don't. It think
> the following approach will give us the most flexibility:
>
> SELECT ... FROM dhcp4_options WHERE subnet_id=X AND (host_id=Y OR
> dhcp_client_class in (class1,class2,...,classN))
>
> I'm not sure if the way how you organized the relation between
> SrvConfig and CfgHost and HostContainer are optimal. I would think
> that since the host reservations are defined per subnet, it should be
> SrvConfig keeps Subnet4 and Subnet6 collections (as it does now) and
> each Subnet{4,6} can contain zero or more CfgHost objects. The
> difference is actually important from performance perspective.
> Imagine 100 subnets with 1000 hosts in each. In the solution you
> propose you'd have to pick the appropriate reservation out of 100.000
> for every request. It will be much slower than picking it out of 1000
> reservations. You don't waste any additional time for picking subnet
> as that is done in the code already and is unavoidable.
Marcin
------------------------------
Message: 3
Date: Mon, 06 Oct 2014 18:29:41 +0200
From: Tomek Mrugalski <[email protected]>
To: Marcin Siodelski <[email protected]>, [email protected]
Subject: Re: [kea-dev] Design for Kea Host Reservation
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
On 06.10.2014 15:55, Marcin Siodelski wrote:
> On 01/10/14 20:52, Tomek Mrugalski wrote:
>> On 30.09.2014 13:20, Marcin Siodelski wrote:
>>> I have created a new design document on Kea wiki:
>>> http://kea.isc.org/wiki/HostReservationDesign for Host Reservation.
>> Here are my comments. I like the design a lot. I think it will address
>> our needs. We will probably tweak it a little bit over time, but it a
>> very solid proposal. Here are my comments:
>>
>> Information stored for HR section
>> IP address(es) - should mention that list is for IPv6 only, v4 need a
>> single address only. There's no need to reserve more than one IPv4
>> address. Also, please explicitly say that IP means IPv4 and IPv6.
>>
>> Temporary addresses. We do not support temporary addresses at all and
>> in a sane environment, there is no requirement to reserve them in any
>> way. On the contrary, it should be an explicit non-requirement. We
>> should probably state that somewhere in the design (and hope that odd
>> users wanting their temporaries to be static will never appear :)
>>
>> We should add that currently there is no way to define what options
>> can be sent for a given class. This capability will be added in the
>> future.
>>
>> I'd love to see DUID reservation example for DHCPv4.
>>
>> I like the paragraph explaining what the prefixes (dhcp_, dhcp4_ etc.)
>> mean. This is sort of intuitive, but it's nice to have it explicitly
>> stated.
>>
>> Shouldn't host_id and reservation_id be of type primary key? The diagram
>> in MySQL database schema suggests it is a simple int. I'm not a
>> MySQL expert, so this is an honest question. I think that setting a
>> field to primary key type introduces three things: first, it is being
>> assigned automatically during inserts. Second, the next value is being
>> stored and applied automatically (autoincrement). Third, a unique
>> restriction is added.
>
> Yes, they are supposed to be primary key. I don't know why the SQL has
> been generated without them being primary keys.
Host_id, option_id (in 2 tables) and reservation_id fields still may
contains non-unique values. NOT NULL and AUTO_INCREMENT are fine, just
something about values being unique is missing.
>> I'm a total MySQL noob, so forgive my ignorance, but why do we need
>> ALLOW_INVALID_DATES? MySQL does not simply insert invalid dates, and
>> sort of sanitizes it to zeros. MySQL doc
>> (http://dev.mysql.com/doc/refman/5.5/en/sql-mode.html, also applies to
>> older versions) says that invalid date of 2014-04-31 will be sanitized
>> to 0000-00-00.
>
> I don't have strong opinion here. In fact, the database schema doesn't
> use any time values at the moment. So, having this one way or another
Huh? What about expire in lease4 and lease6? If a user decides to keep
both hosts and leases (a common assumption, I suppose), then it will
affect the leases data.
> doesn't hurt. The reason why it is set to this value is that the MySQL
> workbench tends to inject this into the generated SQL. How about we
> revisit this if we introduce some time/date columns to the database?
No, let's not add something we don't need. It's cleaner approach.
Tomek
------------------------------
Message: 4
Date: Mon, 06 Oct 2014 18:36:30 +0200
From: Tomek Mrugalski <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] Design for Kea Host Reservation
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
On 30.09.2014 13:20, Marcin Siodelski wrote:
> I have created a new design document on Kea wiki:
> http://kea.isc.org/wiki/HostReservationDesign for Host Reservation. It
> summarizes the discussions that we conducted on a mailing list and
> during the Kea hackathon.
There was a recent update to the host reservation design page. Here are
my comments:
"If the allocation of the new lease fails for the reserved address or
prefix, the allocation engine retries using the client's hint. If that
fails, it proceeds with a normal allocation process."
That's completely wrong, I'm afraid. Host reservation is not a
guideline or a suggestion, it's a strict rule the server must follow. HR
can be used to grant or special service, but also also to confine users
in various ways. We can't simply give them a regular address instead.
Reservation means that the address or prefix is reserved for a given
host and nobody else can get it. It also works the other way
around. That host can get that specific, reserved address or
prefix. The server is not supposed to assign anything else instead.
Users will immediately say that the reservation didn't work.
If a less than clueful admin inserts a host reservation for a lease
that is currently assigned to someone else, we should log an error and
be very vocal about it. It's a configuration error. The client should
not get any other address. He should pick up the phone and call his
support to tell his ISP or his IT that something is broken.
This is quite important. Otherwise we'll get a nasty negative comments
from users. "Hey, I did a host reservation, but the server didn't like
it for some reason and assigned something else. Your HR is broken."
Here's how it should work in my option:
1. there are no reservations
2. client A gets address X
3. admin add reservation for address X to client B
4. client B requests an address, the server discovers that there is a
reservation, but also a lease for client A. It logs an error and
the client B is not assigned any address.
5. Client B repeats discovery process in loop, with exponential backoff.
6. Client A eventually renews, the server discovers that the address
it has is reserved to someone else, send X with 0 lifetime back to A
and assigns other address Y.
7. Client B does another discovery attempt and get reserved address X.
Obviously, 3. is a misconfiguration, but we can't completely prevent
that from happening.
This is the right way to recover from a misconfiguration in my
opinion. If we implement the way you propose, then users will start
asking questions: why didn't the host reservation work? How long till
the server starts using the host reservation I specified? And there
would be no easy answers, because it would depend on T1 and lease
lifetmes (think about clients that get an address and disappear: waiting
till T1 wouldn't be enough, you'd have to wait till valid-lifetime).
You may argue that the plan I described about generates more
traffic. That is true, but it's a weak argument. First, such
misconfiguration is expected to be a rare event. Second, it gives much
better recovery time. The usual exponential backoff counts to 120
seconds (or 3600 seconds if a client supports RFC 7083). That's still
much better recovery time than some of the real networks we heard about
(e.g. 7 days lifetime in cable networks).
In time, when we get reconfigure support, we will trigger it after
step 4 to make the recovery much faster.
-----
My second comment is about something I tried to point out previously.
This design as presented allows in-pool reservations. That is more
convenient for the administrators, but it less performant.
Allowing that is a mistake in my opinion. The algorithm should be much
simpler:
Is there HR for this client? If yes, use whatever is reserved and be
done with it. If not, use dynamic allocation as it is defined now,
without performing any HR queries at all. It's faster and the code is
simpler.
This is a design decision. It will have a non-trivial performance
penalty. I'm not trying to overrule your decision here. If you
understand the implications and still decide to go as planned, so be
it. In any case I'll insist on running performance tests before and
after HR is implemented.
Other changes are fine.
That's it. I don't have any other comments.
Tomek
------------------------------
Message: 5
Date: Mon, 06 Oct 2014 19:28:25 +0200
From: Marcin Siodelski <[email protected]>
To: Tomek Mrugalski <[email protected]>, [email protected]
Subject: Re: [kea-dev] Design for Kea Host Reservation
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
On 06/10/14 18:36, Tomek Mrugalski wrote:
> On 30.09.2014 13:20, Marcin Siodelski wrote:
>> I have created a new design document on Kea wiki:
>> http://kea.isc.org/wiki/HostReservationDesign for Host Reservation. It
>> summarizes the discussions that we conducted on a mailing list and
>> during the Kea hackathon.
> There was a recent update to the host reservation design page. Here are
> my comments:
>
> "If the allocation of the new lease fails for the reserved address or
> prefix, the allocation engine retries using the client's hint. If that
> fails, it proceeds with a normal allocation process."
>
> That's completely wrong, I'm afraid. Host reservation is not a
> guideline or a suggestion, it's a strict rule the server must follow. HR
> can be used to grant or special service, but also also to confine users
> in various ways. We can't simply give them a regular address instead.
>
I take this point. But, you might have seen my email sent earlier today
to Thomas where I state this:
"For the Host Reservation, there is an assumption that the server will
always try to use reserved resources for a host, if any. But, if the
reserved resource is unavailable for some reason (e.g. is in use) the
server should still be able to provision the client by allocating some
other resource. We may obviously speculate whether it is always
appropriate for the server to allocate a different address than the one
that the administrator wanted a client to get and whether the client
should rather not be provisioned in such case. But, I think it is not a
problem to make this configurable at the later time once the whole logic
is in place. So, this discussion is out of scope in the doc."
In my opinion we should not make too strict assumptions because I can
imagine customers having some use cases in which this would be allowed.
And, I state this again: it is much easier to restrict something (with a
configuration knob) than extend the mechanism if the use case appears.
I tend to agree that this is going to be a rare case. But for this
reason I don't see a massive escalation of issues that someone has
received different address than reserved.
As you seem to be pretty confident here, I would like you to make a
final statement on this: "that we will never, ever need a configuration
knob which would allow for dynamic allocation if reserved address is
unavailable". If you can make this statement I can remove this from the
design.
> Reservation means that the address or prefix is reserved for a given
> host and nobody else can get it. It also works the other way
> around. That host can get that specific, reserved address or
> prefix. The server is not supposed to assign anything else instead.
> Users will immediately say that the reservation didn't work.
Ok, so I am an admin I created a pool of addresses and have no HRs. Then
I decided that I want to use some addresses from this pool for HRs and I
reconfigured my server appropriately. But, I didn't send Reconfigure or
anything of this kind. One of my clients is still using an address
dedicated for someone else. The other client comes in and
wants an address but it is in use. What can I do to help my client?
One of the possible approaches would be to wait for the first client to
renew his address and once the server sees the renewing client it may
send 0 lifetimes to this client to say: "don't use this address anymore,
because I have reservation for it. Instead I am giving you this
dynamically allocated address". The reserved address gets back to the
server and waits for the second client to renew in which case the second
client gets the reserved address and the previously allocated address is
de-allocated. So, over time there is a transition and both clients
remain in service and they finally get their addresses as appropriate.
What is wrong with this?
I'd be careful expressing desire of users, especially if they have a
configuration knob to switch between one or another. But, I agree that
this may have been included in the document.
>
> If a less than clueful admin inserts a host reservation for a lease
> that is currently assigned to someone else, we should log an error and
> be very vocal about it. It's a configuration error. The client should
> not get any other address. He should pick up the phone and call his
> support to tell his ISP or his IT that something is broken.
>
> This is quite important. Otherwise we'll get a nasty negative comments
> from users. "Hey, I did a host reservation, but the server didn't like
> it for some reason and assigned something else. Your HR is broken."
>
I don't want to get panicked by this. Maybe let's ask users? Maybe let's
disable this by default and display warnings when enabled?
> Here's how it should work in my option:
>
> 1. there are no reservations
> 2. client A gets address X
> 3. admin add reservation for address X to client B
> 4. client B requests an address, the server discovers that there is a
> reservation, but also a lease for client A. It logs an error and
> the client B is not assigned any address.
> 5. Client B repeats discovery process in loop, with exponential backoff.
> 6. Client A eventually renews, the server discovers that the address
> it has is reserved to someone else, send X with 0 lifetime back to A
> and assigns other address Y.
> 7. Client B does another discovery attempt and get reserved address X.
>
> Obviously, 3. is a misconfiguration, but we can't completely prevent
> that from happening.
>
> This is the right way to recover from a misconfiguration in my
> opinion. If we implement the way you propose, then users will start
> asking questions: why didn't the host reservation work? How long till
> the server starts using the host reservation I specified? And there
> would be no easy answers, because it would depend on T1 and lease
> lifetmes (think about clients that get an address and disappear: waiting
> till T1 wouldn't be enough, you'd have to wait till valid-lifetime).
>
> You may argue that the plan I described about generates more
> traffic. That is true, but it's a weak argument. First, such
> misconfiguration is expected to be a rare event. Second, it gives much
> better recovery time. The usual exponential backoff counts to 120
> seconds (or 3600 seconds if a client supports RFC 7083). That's still
> much better recovery time than some of the real networks we heard about
> (e.g. 7 days lifetime in cable networks).
Ok, so this is an exponential backoff for the Client B. But Client B
still needs to wait for Client A to renew and so as the server can
replace the address it is using with a new (not reserved address). So,
the Client's B retransmission period doesn't mean anything on its own.
If the Client A waits for 7 days before it Renews, Client B is out of
service for 7 days. Whereas, with the approach I described it could use
some address during the transition period.
>
> In time, when we get reconfigure support, we will trigger it after
> step 4 to make the recovery much faster.
Obviously not for DHCPv4.
>
> -----
>
> My second comment is about something I tried to point out previously.
> This design as presented allows in-pool reservations. That is more
> convenient for the administrators, but it less performant.
> Allowing that is a mistake in my opinion. The algorithm should be much
> simpler:
I don't see a reason whye the reservation can't be out of the pool. So
you're proposing that when I define a reservation, the configuration
mechanism checks if this reservation happens to be in one of the pools
defined for a subnet? And if it is, reject the reservation? How would I
guarantee this for the HR configuration in the database? What about the
cases that someone reconfigured the server as we were discussing above?
Sorry if I completely misunderstood your point.
>
> Is there HR for this client? If yes, use whatever is reserved and be
> done with it. If not, use dynamic allocation as it is defined now,
> without performing any HR queries at all. It's faster and the code is
> simpler.
So you're proposing that the server doesn't check if the lease exists
for the particular address in the lease database when it has HR?
I also don't understand "without performing any HR queries at all". The
query to the HR database has to be made to obtain information if the
reservation is specified for the host.
>
> This is a design decision. It will have a non-trivial performance
> penalty. I'm not trying to overrule your decision here. If you
> understand the implications and still decide to go as planned, so be
> it. In any case I'll insist on running performance tests before and
> after HR is implemented.
I have no issue with performance tests. In my opinion we should run them
as soon as possible for all changes we make.
Marcin
------------------------------
_______________________________________________
kea-dev mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/kea-dev
End of kea-dev Digest, Vol 7, Issue 4
*************************************
