Send kea-dev mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/kea-dev
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of kea-dev digest..."
Today's Topics:
1. Re: Requirements for the Logging and Diagnostics in Kea
(Shawn Routhier)
2. Re: Requirements for statistics module in Kea (Shawn Routhier)
3. Re: Requirements for the Logging and Diagnostics in Kea
(Chaigneau, Nicolas)
4. Re: Requirements for the Logging and Diagnostics in Kea
(Stephen Morris)
5. Re: Requirements for the Logging and Diagnostics in Kea
(Stephen Morris)
----------------------------------------------------------------------
Message: 1
Date: Wed, 8 Apr 2015 22:48:20 -0700
From: Shawn Routhier <[email protected]>
To: Marcin Siodelski <[email protected]>
Cc: Kea Dev List <[email protected]>
Subject: Re: [kea-dev] Requirements for the Logging and Diagnostics in
Kea
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
> On Apr 7, 2015, at 4:10 AM, Marcin Siodelski <[email protected]> wrote:
>
> Hi All,
>
> One of the goals for the Kea 0.9.2. release is "Logging and Diagnostics
> Improvements". This includes a number of little improvements and tweaks to
> the existing logging system, but some of the possible enhancements are much
> more significant than this.
>
> I have created the requirements document
> http://kea.isc.org/wiki/DiagnosticsRequirements for "Logging and Diagnostics
> Improvements", which is meant to summarize the requirements mentioned in
> different discussions we've had so far.
>
> This document is still a "draft" because there is the outstanding requirement
> which we're discussing internally and I am not sure if there is a reliable
> way to implement it. That is:
>
> "Kea MUST provide an utility to automatically locate a core file after server
> crash to prevent future server runs from overriding the core file."
>
> Note that not only does this requirement provide a way to prevent override of
> the core file but it also allows for having a convenient way to put all
> useful debugging information in a single place (The same utility could copy
> the current configuration file, lease file, log file etc).
>
> However, I am not familiar with any standard and reliable way to locate the
> core files. On Linux I know that there is a core_pattern file but it may
> redirect the core file to a program, rather than save it in the specific
> location on the disk. In that case, you don't really know where the core dump
> goes.
>
>
> Please review all other requirements in the document and provide your
> feedback.
>
> Thanks,
>
> Marcin Siodelski
>
> DHCP SW Engineer
> ISC
> _______________________________________________
> kea-dev mailing list
> [email protected]
> https://lists.isc.org/mailman/listinfo/kea-dev
I have one high level concern which is to make sure that the logging doesn?t
end up
impacting the performance of the server in general. Using different log levels
should
help some there but we may want to do some testing to see if anything more needs
to be done (allow to enable / disable some of the logging at run time or at
compile time
etc).
1-8 - this is written such that it implies a single tool. Do you want to mean
that? Or would
several tools be acceptable (for example one tool per backend each with
something to read
that backend and that uses a common method to output to the CSV file).
Note that in the CSV case we may have several files that represent the ?lease
file?. As they
are in the proper format we would probably want to simply copy all of them as
part of the
collection process.
9, 10, 11, 12 - I?d like to clarify that this is happening for each individual
callout. So if
there are two callouts at the same hook we would get
info about calling callout 1
info about returning from callout 1
info about calling callout 2
info about returning from callout 2
If so we might also want a log message to indicate the start and end of the
whole process
and might be able to move some of the duplicated information there so:
info about entering hook
same as above - except no hook information as that is already available.
info about leaving hook
We may want to include some sort of internal transaction id on each packet and
use that
to connect the various log messages together.
In 12 we have the callout execution time, I?m wondering if there are other
times we might want
as well - potentially the total execution time and the time waiting for
responses from the backend
DB might be useful.
In Tomek?s mail he mentioned logging information similar to that from ISC DHCP.
There
are three useful notes to consider.
1) There may be a couple of items to add in some of the
messages. Sadly I don?t recall them offhand but there were a couple of them
where you
couldn?t tell quite what had happened from the message.
2) we may wish to make some of
them more or less visible (possibly by use of log level). With the current
code we can get
rather large log files fairly quickly on a busy system, if Kea does get much
better performance
this will be much worse. Trying to notice ?interesting? items within a large
log file can be
difficult. Yes you can search for things such as DHCPINFORM if you know that
your
enterprise doesn?t use that but if we can provide a good way to stress odd
items people
would probably like that.
3) With ISC DHCP many people use scripts to review their log
files. We try hard not to change the log entries once we have started using
them to avoid
breaking existing scripts. It would be best if we decide on the structure and
format of
these log entries for Kea by 1.0 and don?t change them thereafter. Also
providing a good
description of them in the docs would probably be appreciated.
Shawn
------------------------------
Message: 2
Date: Wed, 8 Apr 2015 22:48:32 -0700
From: Shawn Routhier <[email protected]>
To: Tomek Mrugalski <[email protected]>
Cc: "[email protected]" <[email protected]>
Subject: Re: [kea-dev] Requirements for statistics module in Kea
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
> On Apr 3, 2015, at 8:20 AM, Tomek Mrugalski <[email protected]> wrote:
>
> Folks,
> One of the major features in upcoming 0.9.2 release are statistics. I
> just wrote an initial set of requirements for this piece of code:
>
> http://kea.isc.org/wiki/StatsRequirements
>
> I'd love to hear your comments. I plan to work on the design next week.
> There's no strict deadline for your feedback, but the sooner you provide
> it the better.
>
> Thanks,
> Tomek
> _______________________________________________
> kea-dev mailing list
> [email protected]
> https://lists.isc.org/mailman/listinfo/kea-dev
In reading through the requirements and Marcin?s comments I have
concerns about overloading the servers with work to collect and
maintain the statistics. I would move all the requirements to process
the data out of the server and into something else. This could be
a process that is supplied with Kea or a process that a user creates.
(It isn?t clear from the requirements if you are already thinking along
these lines or were considering having the server do all the work itself.)
This would avoid the sever processes spending time trying to
keep any derived information updated but would still allow a user
to derive that information themselves if and when they wanted it.
If we feel that having the derived information is an early requirement
we could include work on that process as well as the data collection,
but it?s also something we might be able to leave to the users for now.
In my model this process would also be where any decisions about
saving the statistics over time would be made as well as how often
to poll the server processes. This suggests that this process might
need to be able to poll for objects in some sort of grouped fashion
(for example get global stats every 10 seconds, get pool usage every
60 seconds, get DDNS usage every 10 minutes).
This would also require two externally visible items 1) something to
describe the various objects that are available (essentially a MIB from
the SNMP days) 2) a path for the stats process to get the information
from the servers (in the short term this could be the CSV file somebody
else mentioned).
**
One issue that should be addressed, especially when exporting the data,
is the atomicity of the objects. If I do a poll of the global values for v4
will
they all represent the same time? (In some cases it is useful to have them
be atomic with respect to each other but I think in general that isn?t required
in DHCP. However we do need to be clear on what the rules for the objects are.)
**
I would also consider if it is worthwhile having an easy method to disable
the actual gathering of the statistics. Optimally this could be enabled /
disabled
without a restart of the system. I may not normally care about the stats
but want to be able to enable them if something seems to be broken.
(This would work better for some counters than others - starting to count
the number of DISCOVERS is pretty easy to understand, starting to count
the number of addresses in use would be a lot less useful.)
This may make more sense if some of the counters are specified as
primarily for debugging. For example one might want to have a set of
different counters for why a packet didn?t generate a response (such as
malformed packet, malformed options, unknown options, bad options,
bad checksum, etc) and not count them most of the time,
**
For 1 we then may not need the floating values - in general I would
probably try to limit the types of values to those that we actually need
and use but try to make the code extensible.
For 4 we may want a specific incrementCounter(?foo?) if that provides a
useful performance improvement. I imagine that the great bulk of the calls
will be a simple increment and so I think it is probably worth optimizing that
path.
I would move 5, 12, and 16 into the stats process I described above.
Can you elaborate a bit on 6? I can see that meaning something like
the number of DISCOVERS received or the total number of addresses in
use (or both). In the first case it is probably a direct counter. In the
second
that might either be an item that is directly counted or something that is
derived, perhaps by adding up all the addresses in use in all of the pools.
In some cases we may wish to include such a derived value while in others
we may simply have the consumer do the derivation themselves.
For 9 and 11 we may wish to be able to get some grouped subset of the
statistics (by server, by pool?)
For the specific requirements we may want to look at a Case diagram to
track the packets through processing and see where we might want to put
counters. It can be useful to have a counter for all the places a packet could
go.
------------------------------
Message: 3
Date: Thu, 9 Apr 2015 07:03:36 +0000
From: "Chaigneau, Nicolas" <[email protected]>
To: Shawn Routhier <[email protected]>, Marcin Siodelski <[email protected]>
Cc: Kea Dev List <[email protected]>
Subject: Re: [kea-dev] Requirements for the Logging and Diagnostics in
Kea
Message-ID:
<ab94b0b675bdf14189cd5a861db36c841951a...@de-cm-mbx26.corp.capgemini.com>
Content-Type: text/plain; charset="utf-8"
> I have one high level concern which is to make sure that the logging doesn?t
> end up impacting the performance of the server in general. Using different
> log levels should help some there but we may want to do some testing to see
> if anything more needs to be done (allow to enable / disable some of the
> logging at run time or at compile time etc).
This is a very valid concern.
I added (within a hook) a log of each received and sent packet as described by
Tomek, and noticed a very significant drop in performances.
(by performances here I mean "how many DORA/s can I handle at most")
I tried to disable the automatic flush (see http://kea.isc.org/ticket/3752
about that), which resulted in improved results.
Then I tried to bypass log4cplus completely (using a fstream with flush
disabled).
In this case the results are much better.
So I guess the convenience of logc4plus comes at a (big) cost for large volumes
of logs.
Maybe there are parameters (besides the automatic flush) that can be tuned to
improve performance ?
This message contains information that may be privileged or confidential and is
the property of the Capgemini Group. It is intended only for the person to whom
it is addressed. If you are not the intended recipient, you are not authorized
to read, print, retain, copy, disseminate, distribute, or use this message or
any part thereof. If you receive this message in error, please notify the
sender immediately and delete all copies of this message.
------------------------------
Message: 4
Date: Thu, 09 Apr 2015 11:41:17 +0100
From: Stephen Morris <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] Requirements for the Logging and Diagnostics in
Kea
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
On 09/04/15 08:03, Chaigneau, Nicolas wrote:
>
> This is a very valid concern.
> I added (within a hook) a log of each received and sent packet as described
> by Tomek, and noticed a very significant drop in performances.
> (by performances here I mean "how many DORA/s can I handle at most")
>
> I tried to disable the automatic flush (see http://kea.isc.org/ticket/3752
> about that), which resulted in improved results.
>
> Then I tried to bypass log4cplus completely (using a fstream with flush
> disabled).
> In this case the results are much better.
>
> So I guess the convenience of logc4plus comes at a (big) cost for large
> volumes of logs.
Yes, there is an overhead in using the logging framework.
In your particular case, with the logging of every packet, are you doing
format conversions (e.g. converting a 32-bit IP address to
dotted-decimal format) before writing to disk? If so, I'm wondering
whether writing the messages in a binary format (and viewing the log
off-line using a custom-written program) would give you a significant
increase in performance.
> Maybe there are parameters (besides the automatic flush) that can be tuned to
> improve performance ?
I've located this article:
http://stackoverflow.com/questions/7338439/is-log4cplus-really-so-slow
Admittedly it is somewhat old (2011), but the comments on that page
suggest that it is the FileAppender backend that is the bottleneck,
possibly due to the use of C++ streams to write the output. It suggests
that tuning the log4cplus buffer size (which at present will have to be
done by modifying the code) may improve performance.
Stephen
------------------------------
Message: 5
Date: Thu, 09 Apr 2015 12:16:45 +0100
From: Stephen Morris <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] Requirements for the Logging and Diagnostics in
Kea
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
Comments on the logging requirements:
1) Requirement 6: I suggest that MUST should be a SHOULD. Knowing the
number of leases is nice, but not essential.
2) Requirements 9, 11: suggest that something be included to note that
the logging of messages is controlled by the configuration. As it is
currently worded, it appears to be mandatory for Kea to output log
messages in all cases.
3) Requirement 23: I'm not sure this is needed. If the code is written
to as not do something in an RFC-compliant way, that should be
documented in the administrator's manual. If the reason for
non-compliance is due to a configuration setting, that should be
documented as an effect of setting the configuration item.
4) I suggest that we include requirements to dump the contents of
incoming and outgoing packets (if requested) to a file in binary format.
This will be outside the normal logging mechanism, but a packet trace
could be useful in some diagnoses.
Stephen
------------------------------
_______________________________________________
kea-dev mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/kea-dev
End of kea-dev Digest, Vol 13, Issue 4
**************************************
