Send kea-dev mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/kea-dev
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of kea-dev digest..."
Today's Topics:
1. Various pull requests (Andrei Pavel)
2. Re: Various pull requests (Andrei Pavel)
----------------------------------------------------------------------
Message: 1
Date: Thu, 15 Dec 2016 18:25:16 +0200
From: Andrei Pavel <[email protected]>
To: [email protected]
Subject: [kea-dev] Various pull requests
Message-ID:
<CAENSVGUdANOj=nop9sq1s9fklfo0urq1mng2mtvtc3we1hy...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
We've done a few changes and we've submitted them via pull requests 35 to
47.
There are two dependencies between those that are related to Cassandra.
Cassandra Host Data Source (CHDS) PR contains everything that the Cassandra
update PR has and the stress test PR contains everything that the CHDS PR
contains. In both cases, it might be easier to merge the contained PR first
and then compare the containing PR with it if you so choose.
A bit about each:
*Cassandra update*
We've added some wrapper functions for communicating with Cassandra:
executeWrite for inserts, updates and deletes as they behave similarly,
executeRead for selects. This PR contains other minor changes like coding
style that are really hard to separate. We apologize for the hardship the
code review might entail. It could be minimized by using a formatter on
both codes and then comparing.
*Cassandra host data source*
The database schema is radically different than the MySQL and the
PostgreSQL schemas. Rather than creating a different table for hosts,
reservations, DHCPv4 options and DHCPv6 options respectively, we've
denormalized them all into a single table to benefit from Cassandra's
non-relational nature. To make up for the lack of relations, on insertion,
the reservations and options are matched against hosts on the server and
merged into database entries. When retrieving, each database row is split
into the corresponding host, reservation and options. There can be an
inconsistency in the database due to the order of the changes e.g. if you
insert a host with no reservations and no options followed by the same host
with one reservation will result in 2 entries versus inserting the host
with reservation from the beginning which will result in a single entry. In
spite of this, retrieving the host will give you the attached reservation
in both cases.
Unit tests are exactly like the other HDS with minor adjustments There are
two possible issues that we've found during unit testing.
1. In some unit tests, a few database operations are expected to return the
rows in the order they have been inserted. This cannot be the case with
Cassandra. Since it doesn't support an ORDER BY type of clause on SELECT
statements, results are always ordered by it's internal row UUID or by the
CLUSTERING ORDER BY clause on the INSERT statement. So the order is
consistent, but it cannot be changed at SELECT. We've modified the generic
HDS unittest to sort the results manually if Cassandra HDS passed through
there. We know his is unprecedented in Kea and bad design, but we can't
think of another solution other than removing the sort check altogether.
2. MySQL and PostgreSQL have unique indexes or constraints on the HDS
tables which Cassandra doesn't support. They are half-simulated only
through the primary key that we've chosen for our single table.
So the unique indexes are:
hosts: dhcp_identifier - dhcp_identifier_type - dhcp4_subnet_id
hosts: dhcp_identifier - dhcp_identifier_type - dhcp6_subnet_id
hosts: ipv4_address - dhcp4_subnet_id
ipv6_reservations: address - prefix_len
And our primary key is a hash of concatenated values of the following
columns:
dhcp4_subnet_id - dhcp6_subnet_id - ipv4_address - ipv6_prefix_address -
ipv6_prefix_length - option_code - option_space
So this is like one big loose constraint. It happens to pass the unit tests
that check for duplicate entries, but they aren't thorough enough. So in
one unit test, dhcp_identifier is changed, and constraint #3 is expected to
be triggered. This works for us because we've expressly excluded
dhcp_identifier and type from our key even though it normally should be
part of the row identifier. If constraint #1 was to be checked, you would
change ipv4_address and constraint #3 would be satisfied, but constraint #1
wouldn't, so the INSERT would fail. This doesn't happen in our case. One
costly solution is to do 4 SELECTs (maybe 3) before every insert to check
if the values to be inserted are constrained by existing values in the
database. We'd like to hear your thoughts on this.
*Cassandra host data source stress test*
Some test that we used for benchmarking included as part of the generic HDS
unittest. It does a bunch of inserts followed by select statements, all
timed and repeated on a increasingly number of hosts and. DISABLED_ by
default.
*clang-format*
Added a Clang format style file and a script to apply it to C++ source
files. Very handy if used from inside an IDE since you can apply it to
blocks of code (e.g. your contribution).
*Changes for uniform compilation*
Added #include <config.h> to all .cc files.
Added KEA_CXXFLAGS to Makefiles in order to uniformly compile all libraries.
*docbook-upgrade*
Upgraded Docbook from v4.2 to v5.0
Used script http://docbook.org/xml/5.0/tools/db4-upgrade.xsl as specified
in the installation documentation at http://tdg.docbook.org/tdg/5.0
/appa.html with command:
find . -type f | grep -F ".xml" | xargs -I{} bash -c "xsltproc -o {}
db4-upgrade.xsl {}"
*Doxyfile*
Updated Doxyfile from 1.8.6 to 1.8.11
This is better generated manually with doxygen -g Doxyfile with doxygen
version 1.8.11.
*makefile-am*
report-cpp-coverage wasn't working because of a syntax error
Added coverage target to .NOTPARALLEL to run unit tests and only then build
reports.
Fixed cppcheck.
*googletest*
Corrected gtest library name. Otherwise it doesn't compile --with-gtest.
*initialize-logging-before-reading-conf*
Started logging before reading the "Dhcp4" and "Dhcp6" configuration, but
after reading the logfile location of course, to log errors thrown because
of any configuration loading issues.
*schema-version*
Changed PostgreSQL version variable names to be inline with MySQL and
Cassandra's
*typos*
The important stuff, right? :)
We've run a spell checker and "indentified" some spelling mistakes.
If you'll review from the command line, we recommend one of the following
commands:
*git diff --color-words*
git diff --color-words | sed -n '/31m/,/m/p'
git diff --color-words --unified=0
*with-dhcp*
with-dhcp-mysql -> with-mysql
with-dhcp-pgsql -> with-pgsql
Regards,
Qualitance
--
*Andrei Pavel* | junior c++ developer | QUALITANCE?
*RO:* +40 (748) 220 135 <+40%20748%20220%20135> | *office:* +40 (372) 944
741 <+40%20372%20944%20741>
Opera Center bldg. 2, fl. 7-8, 2 Dr. Nicolae D. Staicovici, Bucharest 050556
<https://www.google.ro/maps/place/QUALITANCE/@44.433653,26.079464,17z/data=!4m6!1m3!3m2!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede!2sQUALITANCE!3m1!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede?hl=ro>
[email protected] | www.qualitance.com
a Deloitte Technology Fast 50
<http://www2.deloitte.com/content/dam/Deloitte/global/Documents/About-Deloitte/central-europe/CE_Fast_50_2015.pdf>
company | ranked in the Inc. 5000 Europe
<http://www.inc.com/inc5000eu/list/2016/> for 2016
--
The information contained in this email is confidential. It is intended
only for the stated addressee and access to it by any other person is
unauthorised. If you are not an addressee, you must not disclose, copy,
circulate or in another way use or reply to the information contained in
this email. Such unauthorised use may be unlawful. If you have received
this email in error, please notify the sender immediately and delete all
copies from your computer and network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://lists.isc.org/pipermail/kea-dev/attachments/20161215/330a930e/attachment-0001.html>
------------------------------
Message: 2
Date: Fri, 16 Dec 2016 10:46:16 +0200
From: Andrei Pavel <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] Various pull requests
Message-ID:
<caensvgwedm-k_j81hzuwirwcgpt_tcctb1huu6+82s0gh4v...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Forgot to mention that the Cassandra update (PR #35) is meant to replace
"refactored cassandra backend and fixed issues" (PR #26) and was created on
top of it. So only PR #35 should be taken into account and PR #26 should be
closed. If this cannot be easily done e.g. you started working on PR #26
already, let us know how to proceed.
On Thu, Dec 15, 2016 at 6:25 PM, Andrei Pavel <[email protected]>
wrote:
> Hi,
>
> We've done a few changes and we've submitted them via pull requests 35 to
> 47.
>
> There are two dependencies between those that are related to Cassandra.
> Cassandra Host Data Source (CHDS) PR contains everything that the Cassandra
> update PR has and the stress test PR contains everything that the CHDS PR
> contains. In both cases, it might be easier to merge the contained PR first
> and then compare the containing PR with it if you so choose.
> A bit about each:
>
> *Cassandra update*
> We've added some wrapper functions for communicating with Cassandra:
> executeWrite for inserts, updates and deletes as they behave similarly,
> executeRead for selects. This PR contains other minor changes like coding
> style that are really hard to separate. We apologize for the hardship the
> code review might entail. It could be minimized by using a formatter on
> both codes and then comparing.
>
> *Cassandra host data source*
> The database schema is radically different than the MySQL and the
> PostgreSQL schemas. Rather than creating a different table for hosts,
> reservations, DHCPv4 options and DHCPv6 options respectively, we've
> denormalized them all into a single table to benefit from Cassandra's
> non-relational nature. To make up for the lack of relations, on insertion,
> the reservations and options are matched against hosts on the server and
> merged into database entries. When retrieving, each database row is split
> into the corresponding host, reservation and options. There can be an
> inconsistency in the database due to the order of the changes e.g. if you
> insert a host with no reservations and no options followed by the same host
> with one reservation will result in 2 entries versus inserting the host
> with reservation from the beginning which will result in a single entry. In
> spite of this, retrieving the host will give you the attached reservation
> in both cases.
>
> Unit tests are exactly like the other HDS with minor adjustments There are
> two possible issues that we've found during unit testing.
> 1. In some unit tests, a few database operations are expected to return
> the rows in the order they have been inserted. This cannot be the case with
> Cassandra. Since it doesn't support an ORDER BY type of clause on SELECT
> statements, results are always ordered by it's internal row UUID or by the
> CLUSTERING ORDER BY clause on the INSERT statement. So the order is
> consistent, but it cannot be changed at SELECT. We've modified the generic
> HDS unittest to sort the results manually if Cassandra HDS passed through
> there. We know his is unprecedented in Kea and bad design, but we can't
> think of another solution other than removing the sort check altogether.
> 2. MySQL and PostgreSQL have unique indexes or constraints on the HDS
> tables which Cassandra doesn't support. They are half-simulated only
> through the primary key that we've chosen for our single table.
> So the unique indexes are:
>
> hosts: dhcp_identifier - dhcp_identifier_type - dhcp4_subnet_id
> hosts: dhcp_identifier - dhcp_identifier_type - dhcp6_subnet_id
> hosts: ipv4_address - dhcp4_subnet_id
> ipv6_reservations: address - prefix_len
>
> And our primary key is a hash of concatenated values of the following
> columns:
>
> dhcp4_subnet_id - dhcp6_subnet_id - ipv4_address - ipv6_prefix_address -
> ipv6_prefix_length - option_code - option_space
>
> So this is like one big loose constraint. It happens to pass the unit
> tests that check for duplicate entries, but they aren't thorough enough. So
> in one unit test, dhcp_identifier is changed, and constraint #3 is expected
> to be triggered. This works for us because we've expressly excluded
> dhcp_identifier and type from our key even though it normally should be
> part of the row identifier. If constraint #1 was to be checked, you would
> change ipv4_address and constraint #3 would be satisfied, but constraint #1
> wouldn't, so the INSERT would fail. This doesn't happen in our case. One
> costly solution is to do 4 SELECTs (maybe 3) before every insert to check
> if the values to be inserted are constrained by existing values in the
> database. We'd like to hear your thoughts on this.
>
> *Cassandra host data source stress test*
> Some test that we used for benchmarking included as part of the generic
> HDS unittest. It does a bunch of inserts followed by select statements, all
> timed and repeated on a increasingly number of hosts and. DISABLED_ by
> default.
>
> *clang-format*
> Added a Clang format style file and a script to apply it to C++ source
> files. Very handy if used from inside an IDE since you can apply it to
> blocks of code (e.g. your contribution).
>
> *Changes for uniform compilation*
> Added #include <config.h> to all .cc files.
> Added KEA_CXXFLAGS to Makefiles in order to uniformly compile all
> libraries.
>
> *docbook-upgrade*
> Upgraded Docbook from v4.2 to v5.0
> Used script http://docbook.org/xml/5.0/tools/db4-upgrade.xsl as specified
> in the installation documentation at http://tdg.docbook.org/tdg/
> 5.0/appa.html with command:
> find . -type f | grep -F ".xml" | xargs -I{} bash -c "xsltproc -o {}
> db4-upgrade.xsl {}"
>
> *Doxyfile*
> Updated Doxyfile from 1.8.6 to 1.8.11
> This is better generated manually with doxygen -g Doxyfile with doxygen
> version 1.8.11.
>
> *makefile-am*
> report-cpp-coverage wasn't working because of a syntax error
> Added coverage target to .NOTPARALLEL to run unit tests and only then
> build reports.
> Fixed cppcheck.
>
> *googletest*
> Corrected gtest library name. Otherwise it doesn't compile --with-gtest.
>
> *initialize-logging-before-reading-conf*
> Started logging before reading the "Dhcp4" and "Dhcp6" configuration, but
> after reading the logfile location of course, to log errors thrown because
> of any configuration loading issues.
>
> *schema-version*
> Changed PostgreSQL version variable names to be inline with MySQL and
> Cassandra's
>
> *typos*
> The important stuff, right? :)
> We've run a spell checker and "indentified" some spelling mistakes.
> If you'll review from the command line, we recommend one of the following
> commands:
> *git diff --color-words*
> git diff --color-words | sed -n '/31m/,/m/p'
> git diff --color-words --unified=0
>
> *with-dhcp*
> with-dhcp-mysql -> with-mysql
>
> with-dhcp-pgsql -> with-pgsql
>
> Regards,
> Qualitance
> --
> *Andrei Pavel* | junior c++ developer | QUALITANCE?
> *RO:* +40 (748) 220 135 <+40%20748%20220%20135> | *office:* +40 (372)
> 944 741 <+40%20372%20944%20741>
> Opera Center bldg. 2, fl. 7-8, 2 Dr. Nicolae D. Staicovici, Bucharest
> 050556
> <https://www.google.ro/maps/place/QUALITANCE/@44.433653,26.079464,17z/data=!4m6!1m3!3m2!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede!2sQUALITANCE!3m1!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede?hl=ro>
> [email protected] | www.qualitance.com
>
> a Deloitte Technology Fast 50
> <http://www2.deloitte.com/content/dam/Deloitte/global/Documents/About-Deloitte/central-europe/CE_Fast_50_2015.pdf>
> company | ranked in the Inc. 5000 Europe
> <http://www.inc.com/inc5000eu/list/2016/> for 2016
> --
> The information contained in this email is confidential. It is intended
> only for the stated addressee and access to it by any other person is
> unauthorised. If you are not an addressee, you must not disclose, copy,
> circulate or in another way use or reply to the information contained in
> this email. Such unauthorised use may be unlawful. If you have received
> this email in error, please notify the sender immediately and delete all
> copies from your computer and network.
>
--
*Andrei Pavel* | junior c++ developer | QUALITANCE?
*RO:* +40 (748) 220 135 | *office:* +40 (372) 944 741
Opera Center bldg. 2, fl. 7-8, 2 Dr. Nicolae D. Staicovici, Bucharest 050556
<https://www.google.ro/maps/place/QUALITANCE/@44.433653,26.079464,17z/data=!4m6!1m3!3m2!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede!2sQUALITANCE!3m1!1s0x40b1ff173c5bc0b9:0x7226343f810c7ede?hl=ro>
[email protected] | www.qualitance.com
a Deloitte Technology Fast 50
<http://www2.deloitte.com/content/dam/Deloitte/global/Documents/About-Deloitte/central-europe/CE_Fast_50_2015.pdf>
company | ranked in the Inc. 5000 Europe
<http://www.inc.com/inc5000eu/list/2016/> for 2016
--
The information contained in this email is confidential. It is intended
only for the stated addressee and access to it by any other person is
unauthorised. If you are not an addressee, you must not disclose, copy,
circulate or in another way use or reply to the information contained in
this email. Such unauthorised use may be unlawful. If you have received
this email in error, please notify the sender immediately and delete all
copies from your computer and network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://lists.isc.org/pipermail/kea-dev/attachments/20161216/0726d730/attachment.html>
------------------------------
Subject: Digest Footer
_______________________________________________
kea-dev mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/kea-dev
------------------------------
End of kea-dev Digest, Vol 33, Issue 1
**************************************
