Send kea-dev mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/kea-dev
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of kea-dev digest..."
Today's Topics:
1. Re: Fixing Kea package (Marcin Siodelski)
2. KEA performance and hook framework (Angelo Failla)
----------------------------------------------------------------------
Message: 1
Date: Mon, 14 Mar 2016 11:18:59 +0100
From: Marcin Siodelski <[email protected]>
To: Jiri Popelka <[email protected]>, [email protected]
Subject: Re: [kea-dev] Fixing Kea package
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
On 11.03.2016 18:44, Jiri Popelka wrote:
> On 11.3.2016 16:00, Zdenek Dohnal wrote:
>> I'm writing this email to inform you about I made several changes to your
>> package because of standard updating to C++11. Changes are in these files:
>
> Background story is that we were unable to build kea with new gcc-6.0.0
> (forthcoming Fedora 24), which defaults to -std=gnu++14.
> We worked-around some of the failures by using -std=gnu++11 and
> -Wno-deprecated-declarations and the rest has been fixed by the Zdenek's
> patch. Let us know if you need more details.
>
> with regards,
> --
> Jiri
>
Zdenek, Jiri,
A while ago we have started making necessary changes to allow for
building Kea with C++11 enabled. Refer to the following Kea ticket:
http://kea.isc.org/ticket/3845 for details. This work has been put on
hold because we had other more urgent work to do.
I'll bring the topic of C++1X on our internal call this week and we'll
figure out what we want to do about it in the near future.
Thanks,
Marcin Siodelski
DHCP Software Engineer
ISC
------------------------------
Message: 2
Date: Mon, 14 Mar 2016 10:20:30 +0000
From: Angelo Failla <[email protected]>
To: "[email protected]" <[email protected]>
Subject: [kea-dev] KEA performance and hook framework
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"
Hello KEA developers!
I am writing to discuss about hook framework performance in light of a recent
incident we had in our production network.
I am hoping this would be interesting/actionable feedback for you guys.
In short what happened is that a lot of clients (good portion of a cluster of
~10k nodes) went offline and lost their lease; then they came up all at once
asking for DHCP leases and our KEA instances were struggling trying to cope
with traffic.
Initial symptoms were maxed out RecvQ buffers as seen by `netstat --tunap`
command. That was a symptom that the user space application (KEA) was not able
to pop packets from that kernel queue fast enough.
We use KEA via an homemade hook library that we use to resolve host information
via thrift calls to an internal backend which is relatively slow and has a p99
response time of 80ms.
It seems KEA is a single threaded UDP server using blocking sockets, that means
that the next request can?t be processed if the current in-flight one is not
done. Some back of the napkin math I did told me that we were doing approx.
12.5 RPS (1000ms / 80ms) as perfdhcp confirmed (see below).
We tried lowering sysctl options (net.core.rmem_default or net.core.rem_max) in
the hope that that would at least have the server respond to some request in a
reasonable amount of time but it didn?t help.
Anyway, as I said, after the incident was resolved (we had to blackhole traffic
and slowly bring up dhcp clients back online), I went running some perfdhcp
tests against our KEA instances:
??????? start of perfdhcp ?????
[[email protected] ~]# ~pallotron/bin/perfdhcp -4 -r 1500 -p 10 -t 1 -b
mac=90:e2:ba:95:b9:28 10.88.77.95
Running: perfdhcp -4 -r 1500 -p 10 -t 1 -b mac=90:e2:ba:95:b9:28 10.88.77.95
sent: 1368/14; received: 14/0; drops: 1354/14
sent: 2736/28; received: 28/0; drops: 2708/28
sent: 4101/40; received: 40/0; drops: 4061/40
sent: 5466/56; received: 56/0; drops: 5410/56
sent: 6834/70; received: 70/0; drops: 6764/70
sent: 8198/83; received: 83/0; drops: 8115/83
sent: 9564/97; received: 97/0; drops: 9467/97
sent: 10928/113; received: 113/0; drops: 10815/113
sent: 12290/128; received: 128/0; drops: 12162/128
***Rate statistics***
Rate: 14.3996 exchanges/second, expected rate: 1500 exchanges/second
***Statistics for: DISCOVER-OFFER***
sent packets: 13647
received packets: 144
drops: 13503
min delay: 125.167 ms
avg delay: 5072.366 ms
max delay: 9876.990 ms
std deviation: 2869.381 ms
collected packets: 0
***Statistics for: REQUEST-ACK***
sent packets: 144
received packets: 0
drops: 144
Delay summary unavailable! No packets received.
??????? end of perfdhcp ?????
We are addressing this problem by modifying our hook library by adding a
background thread that makes async backend calls and caches results in memory.
This cache would use a low ttl for entries (few seconds), and likely have a
fixed size. Hooks would try to fetch from this cache. If there's an entry the
hooks pass a result back. If there's not cache entry they queue a backend call
and error (basically not respond to the clients). Clients will retry after the
cache has been seeded by the background thread and succeed.
However I am wondering if you are considering improving the server performance
and improve concurrency for example using some async model for the hook
framework.
Happy to discuss, keep up the good work :)
--
Angelo Failla
Cluster Infrastructure - Dublin
[email protected]
------------------------------
_______________________________________________
kea-dev mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/kea-dev
End of kea-dev Digest, Vol 24, Issue 4
**************************************
