Okay, I sorted it out. The reason the city DB was corrupted is because we were putting so many locations into the location file that we were overflowing the addressable places to put them. The Maxmind format uses 3 bytes to store offsets, so when we exceed offset 0xFFFFFF our offsets wrap around to 0 and nothing good happens after that. I added a check for the overflow condition, so the code that creates city DBs errors out instead of silently making a corrupted database. Coincidentally, we were just barely over the size limit for Maxmind's format, which is why a fairly small number of entries were affected. That left the issue of why we were overflowing in the first place. It turns out that a large fraction of the locations in the CSV data aren't actually needed (aren't associated with any IP address). In order to fit the data into a .dat file in Maxmind's format, we have to scan for those and remove them. There are a huge number of them, as I said -- removing them shrinks the .dat file from 31 MB to 20 MB. I've attached a patch which does both of those things. I've built gdnsd successfully with this patch applied and the DBs rebuilt. I've also tested it by running geoiplookup against the database for several addresses. The patch also fixes the area code and metro code for US addresses, which I noticed were backwards (switched with one another). Hope this helps, let me know if you see any issues. -Andrew
On Fri, Jan 23, 2015 at 11:52 AM, Andrew Moise <andrew.mo...@gmail.com> wrote: > Hi Patrick. > I tracked it down to a few broken entries in the city DB. It looks > like the DB creation code makes almost all the entries correctly, but > there are a couple of them that come out corrupted: > > (jessie)moise@localhost:~$ geoiplookup 1.120.146.170 > GeoIP Country Edition: AU, Australia > GeoIP City Edition, Rev 1: GP, > ���-��-��%��%��%��%��- �- �-Ɣ-��%��%"�-��-x�-v�-, N/A, N/A, N/A, > -180.000000, -179.993500, 0, 0 > GeoIP ASNum Edition: AS30722 Vodafone Omnitel B.V. > > Note that's with a local build of the database -- that particular IP > address may not be corrupted in the actual jessie database. In that > local DB, as in the actual jessie database, almost all the entries are > correct, just there are a handful that look like that one. > I've been busy the last couple days, but I expect that I'll get > enough time today to actually track down what's going wrong and fix > it. > -Andrew > > On Jan 23, 2015 9:43 AM, "Patrick Matthäi" <pmatth...@debian.org> wrote: >> >> Hi Andrew, >> >> do you have got any news? :) >> >> Am 20.01.2015 um 18:43 schrieb Andrew Moise: >> >> Thanks Patrick. One note - just like the v6 issue, the gdnsd test suite is >> detecting a genuine problem in the city DB. It is something wrong with the >> city DB creation tools that's causing it (i.e. not just an issue with the >> gdnsd tests). I just haven't finished tracking down exactly what the issue >> is yet. >> -Andrew >> >> On Jan 20, 2015 12:22 PM, "Patrick Matthäi" <pmatth...@debian.org> wrote: >>> >>> tag #775638 + confirmed >>> clone #775638 -1 >>> reassign -1 geoip-bin >>> retitle -1 geoip-generator produces faulty v6/city database >>> severity -1 grave >>> found -1 1.6.2-3 >>> thanks >>> >>> Hi >>> >>> Am 18.01.2015 um 05:21 schrieb Debian Bug Tracking System: >>> > Processing commands for cont...@bugs.debian.org: >>> > >>> >> reassign 775638 geoip-database 20141027-1 >>> > Bug #775638 [src:gdnsd] gdnsd: FTBFS in jessie: dh_auto_test: make -j1 >>> > test returned exit code 2 >>> > Bug reassigned from package 'src:gdnsd' to 'geoip-database'. >>> > No longer marked as found in versions gdnsd/2.1.0-1. >>> > Ignoring request to alter fixed versions of bug #775638 to the same >>> > values previously set >>> > Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make >>> > -j1 test returned exit code 2 >>> > Marked as found in versions geoip-database/20141027-1. >>> >> retitle 775638 IPv6 database is corrupt >>> > Bug #775638 [geoip-database] gdnsd: FTBFS in jessie: dh_auto_test: make >>> > -j1 test returned exit code 2 >>> > Changed Bug title to 'IPv6 database is corrupt' from 'gdnsd: FTBFS in >>> > jessie: dh_auto_test: make -j1 test returned exit code 2' >>> >> severity 775638 grave >>> > Bug #775638 [geoip-database] IPv6 database is corrupt >>> > Severity set to 'grave' from 'serious' >>> >> thanks >>> >>> thanks for spotting it. Curious that no one (also myself!) spotted it. >>> The patch for the v6 database is just: >>> >>> --- geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-19 >>> 18:50:04 UTC (rev 5693) >>> +++ geoip/branches/jessie/debian/src/geoip-csv-to-dat.cpp 2015-01-20 >>> 08:31:03 UTC (rev 5694) >>> @@ -959,6 +959,7 @@ >>> address_family = AF_INET; >>> break; >>> case '6': >>> + database_type = GEOIP_COUNTRY_EDITION_V6; >>> address_family = AF_INET6; >>> break; >>> case 'i': >>> >>> But Andrew found out that there is also an issue with the city database, >>> which is working in general, but the gdnsd tests also fail. >>> >>> -- >>> /* >>> Mit freundlichem Gruß / With kind regards, >>> Patrick Matthäi >>> GNU/Linux Debian Developer >>> >>> Blog: http://www.linux-dev.org/ >>> E-Mail: pmatth...@debian.org >>> patr...@linux-dev.org >>> */ >>> >> >> -- >> /* >> Mit freundlichem Gruß / With kind regards, >> Patrick Matthäi >> GNU/Linux Debian Developer >> >> Blog: http://www.linux-dev.org/ >> E-Mail: pmatth...@debian.org >> patr...@linux-dev.org >> */
fix-city-corruption.patch
Description: Binary data
