Bug#934147: geoip-database: Please ship GeoLite2 databases in MMDB format
On Thu, Aug 29, 2019 at 12:21:11PM +0200, Patrick Matthäi wrote: > So why not leaving src:geoip-database for the legacy databases and > introduce a new src package for the new formats? Well, that's certainly doable, but it just feels like duplicated effort, considering that the CSV/MMDB files are paired/part of the same distribution by upstream, as well as the fact that these are volatile files that need frequent updates (and even in stable). I was hoping to collaborate here, but either way is OK with me. Faidon
Bug#934147: geoip-database: Please ship GeoLite2 databases in MMDB format
Am 29.08.2019 um 12:09 schrieb Faidon Liambotis:
> On Wed, Aug 28, 2019 at 03:05:07PM +0200, Patrick Matthäi wrote:
>>> I'd be happy to help with that. Is the package in git somewhere? I don't
>>> see Vcs-* headers - perhaps you could import it to salsa?
>> I have got my own subversion system for my packages. If you want to
>> co-maintain geoip{-database} I could grant you access to it
> I haven't used Subversion for about a decade, and I'd rather not do
> collaborative development on someone's private server. Could we just
> move this into salsa?
>
>> For me building them from source was a requirement all the time, so that
>> the package could be in main and not non-free or contrib. So it would be
>> great if the MMDBs also could be in main.
> We don't have any reason to believe CSV is the source and MMDB is not,
> so I don't think this should affect inclusion in main.
>
> Faidon
So why not leaving src:geoip-database for the legacy databases and
introduce a new src package for the new formats?
--
/*
Mit freundlichem Gruß / With kind regards,
Patrick Matthäi
GNU/Linux Debian Developer
Blog: http://www.linux-dev.org/
E-Mail: pmatth...@debian.org
patr...@linux-dev.org
*/
Bug#934147: geoip-database: Please ship GeoLite2 databases in MMDB format
On Wed, Aug 28, 2019 at 03:05:07PM +0200, Patrick Matthäi wrote:
> > I'd be happy to help with that. Is the package in git somewhere? I don't
> > see Vcs-* headers - perhaps you could import it to salsa?
>
> I have got my own subversion system for my packages. If you want to
> co-maintain geoip{-database} I could grant you access to it
I haven't used Subversion for about a decade, and I'd rather not do
collaborative development on someone's private server. Could we just
move this into salsa?
> For me building them from source was a requirement all the time, so that
> the package could be in main and not non-free or contrib. So it would be
> great if the MMDBs also could be in main.
We don't have any reason to believe CSV is the source and MMDB is not,
so I don't think this should affect inclusion in main.
Faidon
Bug#934147: geoip-database: Please ship GeoLite2 databases in MMDB format
Am 27.08.2019 um 16:30 schrieb Faidon Liambotis:
> Hi there,
>
> Thanks Colin for re-raising this! My intention for #885442 was to
> include the GeoLite2 *databases* (i.e. MMDB), rather than their data
> converted to the legacy GeoIP format. I'll avoid making a mess out of
> the BTS though, given we have this bug now :)
>
> On Thu, Aug 08, 2019 at 09:33:27AM +0200, Patrick Matthäi wrote:
>> Dont know. Patches would be welcome :)
> I'd be happy to help with that. Is the package in git somewhere? I don't
> see Vcs-* headers - perhaps you could import it to salsa?
I have got my own subversion system for my packages. If you want to
co-maintain geoip{-database} I could grant you access to it
>> Then also everything which is required to build the MMDB format has to
>> be in buster-backports and stretch-backports-sloppy. It was enough work
>> now to get everything to work again on both releases
> Could you elaborate a little bit more on why do you think that's a
> requirement? As I mentioned repeatedly in #885442, I don't share that
> view; I think we should be shipping MMDBs as-is and not building them
> out of CSVs. We have no reason to believe that MMDBs are generated out
> of CSVs and the most likely scenario is that the opposite holds true.
> Moreover, MMDB is openly and freely documented and with reader and
> writer implementations in all kinds of languages -- more than what we
> can say for most file formats out there.
For me building them from source was a requirement all the time, so that
the package could be in main and not non-free or contrib. So it would be
great if the MMDBs also could be in main.
--
/*
Mit freundlichem Gruß / With kind regards,
Patrick Matthäi
GNU/Linux Debian Developer
Blog: http://www.linux-dev.org/
E-Mail: pmatth...@debian.org
patr...@linux-dev.org
*/
Bug#934147: geoip-database: Please ship GeoLite2 databases in MMDB format
Hi there, Thanks Colin for re-raising this! My intention for #885442 was to include the GeoLite2 *databases* (i.e. MMDB), rather than their data converted to the legacy GeoIP format. I'll avoid making a mess out of the BTS though, given we have this bug now :) On Thu, Aug 08, 2019 at 09:33:27AM +0200, Patrick Matthäi wrote: > Dont know. Patches would be welcome :) I'd be happy to help with that. Is the package in git somewhere? I don't see Vcs-* headers - perhaps you could import it to salsa? > Then also everything which is required to build the MMDB format has to > be in buster-backports and stretch-backports-sloppy. It was enough work > now to get everything to work again on both releases Could you elaborate a little bit more on why do you think that's a requirement? As I mentioned repeatedly in #885442, I don't share that view; I think we should be shipping MMDBs as-is and not building them out of CSVs. We have no reason to believe that MMDBs are generated out of CSVs and the most likely scenario is that the opposite holds true. Moreover, MMDB is openly and freely documented and with reader and writer implementations in all kinds of languages -- more than what we can say for most file formats out there. Regards, Faidon
Bug#934147: geoip-database: Please ship GeoLite2 databases in MMDB format
>> Also if I have to use the new sources now (since geoip v1 sources are >> not available anymore) it does not mean that src:geoip-database >> fullfils the MMDB format. > I don't understand that. Please could you rephrase? As far as I can > see you're already using the new sources. Is the difficulty just in > building the MMDB files from the CSV files? Dont know. Patches would be welcome :) Then also everything which is required to build the MMDB format has to be in buster-backports and stretch-backports-sloppy. It was enough work now to get everything to work again on both releases -- /* Mit freundlichem Gruß / With kind regards, Patrick Matthäi GNU/Linux Debian Developer Blog: http://www.linux-dev.org/ E-Mail: pmatth...@debian.org patr...@linux-dev.org */
Bug#934147: geoip-database: Please ship GeoLite2 databases in MMDB format
On Wed, Aug 07, 2019 at 03:51:53PM +0200, Patrick Matthäi wrote: > Am 07.08.2019 um 15:41 schrieb Colin Watson: > > Could you please ship the MMDB files as well? I don't mind whether > > they're in different binary packages, although long-term I would expect > > most people to want to use the modern format. > > Geoip2 with the MMDB format is a different package. As I said, "I don't mind whether they're in different binary packages". > Also if I have to use the new sources now (since geoip v1 sources are > not available anymore) it does not mean that src:geoip-database > fullfils the MMDB format. I don't understand that. Please could you rephrase? As far as I can see you're already using the new sources. Is the difficulty just in building the MMDB files from the CSV files? > IMO someone should start a new source package and provide all the new > MMDB databases. I also had to drop now the city and asn edition here. I guess I'm not completely opposed to attempting that, but it does seem like a certain amount of duplicated effort. -- Colin Watson [cjwat...@debian.org]
Bug#934147: geoip-database: Please ship GeoLite2 databases in MMDB format
Am 07.08.2019 um 15:41 schrieb Colin Watson: > Package: geoip-database > Version: 20181108-1 > Severity: wishlist > > Hi, > > Thanks for fixing #885442. I see that you did so by converting the new > databases into the old format. Presumably this was in order to maximise > compatibility, and I don't object to that. However, it has a few > problems: > > * Only the country-level databases are shipped. Regarding the others >that used to be in geoip-database-extra, in your changelog you said >"the sources are dropped from the homepage", which I don't quite >understand because there seem to be CSV files for all of City, >Country, and ASN on https://dev.maxmind.com/geoip/geoip2/geolite2/ >(notwithstanding Faidon's comment that the MMDB files may in fact be >the preferred form for modification anyway). But it means that some >users may be out of luck. > > * We're stuck using old client code to query them. Ideally I'd much >prefer to be using versions of client libraries that are still >maintained upstream, and that generally means using the ones that >expect MMDB input. > > * As I understand it, the old format requires shipping separate files >for IPv4 and IPv6 (at least that's how they're shipped in >geoip-database at the moment). It would be much more convenient to >just open a single database and get results for both IPv4 and IPv6 >addresses. > > Could you please ship the MMDB files as well? I don't mind whether > they're in different binary packages, although long-term I would expect > most people to want to use the modern format. > > Thanks, Geoip2 with the MMDB format is a different package. Also if I have to use the new sources now (since geoip v1 sources are not available anymore) it does not mean that src:geoip-database fullfils the MMDB format. IMO someone should start a new source package and provide all the new MMDB databases. I also had to drop now the city and asn edition here. -- /* Mit freundlichem Gruß / With kind regards, Patrick Matthäi GNU/Linux Debian Developer Blog: http://www.linux-dev.org/ E-Mail: pmatth...@debian.org patr...@linux-dev.org */
Bug#934147: geoip-database: Please ship GeoLite2 databases in MMDB format
Package: geoip-database Version: 20181108-1 Severity: wishlist Hi, Thanks for fixing #885442. I see that you did so by converting the new databases into the old format. Presumably this was in order to maximise compatibility, and I don't object to that. However, it has a few problems: * Only the country-level databases are shipped. Regarding the others that used to be in geoip-database-extra, in your changelog you said "the sources are dropped from the homepage", which I don't quite understand because there seem to be CSV files for all of City, Country, and ASN on https://dev.maxmind.com/geoip/geoip2/geolite2/ (notwithstanding Faidon's comment that the MMDB files may in fact be the preferred form for modification anyway). But it means that some users may be out of luck. * We're stuck using old client code to query them. Ideally I'd much prefer to be using versions of client libraries that are still maintained upstream, and that generally means using the ones that expect MMDB input. * As I understand it, the old format requires shipping separate files for IPv4 and IPv6 (at least that's how they're shipped in geoip-database at the moment). It would be much more convenient to just open a single database and get results for both IPv4 and IPv6 addresses. Could you please ship the MMDB files as well? I don't mind whether they're in different binary packages, although long-term I would expect most people to want to use the modern format. Thanks, -- Colin Watson [cjwat...@debian.org]
