Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package geolite2legacy for openSUSE:Factory checked in at 2022-08-27 11:49:51 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/geolite2legacy (Old) and /work/SRC/openSUSE:Factory/.geolite2legacy.new.2083 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "geolite2legacy" Sat Aug 27 11:49:51 2022 rev:3 rq:999372 version:0+git20210326.05733c4 Changes: -------- --- /work/SRC/openSUSE:Factory/geolite2legacy/geolite2legacy.changes 2020-03-17 13:10:00.657792776 +0100 +++ /work/SRC/openSUSE:Factory/.geolite2legacy.new.2083/geolite2legacy.changes 2022-08-27 11:49:53.661817846 +0200 @@ -1,0 +2,21 @@ +Mon Aug 22 14:29:24 UTC 2022 - valentin.lefeb...@suse.com + +- Update to version 0+git20210326.05733c4: + * cleanup + * conditional ipaddr only for python < 3.3 + * Use correct docker image name on README + * fix ASN + * add python3 compatibility using built-in module ipaddress + * use custom comment in country file + * geolite2legacy: pep8 + * geoname2fips: added tipical filenames to help + * geoname2fips: some country names changed after 20200505 + * geoname2fips: fixed Vlaams-Brabant spelling + * Dockerfile to run the converter inside Docker container + * Use realpath to determine fips file location + * Automatically determine output file name +- remove patches fixed in upstream + * 0001-Automatically-determine-output-file-name.patch + * 0001-Use-realpath-to-determine-fips-file-location.patch + +------------------------------------------------------------------- Old: ---- 0001-Automatically-determine-output-file-name.patch 0001-Use-realpath-to-determine-fips-file-location.patch geolite2legacy-0+git20200101.56d8a4f.tar.xz New: ---- geolite2legacy-0+git20210326.05733c4.tar.xz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ geolite2legacy.spec ++++++ --- /var/tmp/diff_new_pack.OimqsH/_old 2022-08-27 11:49:54.201819006 +0200 +++ /var/tmp/diff_new_pack.OimqsH/_new 2022-08-27 11:49:54.205819014 +0200 @@ -1,7 +1,7 @@ # # spec file for package geolite2legacy # -# Copyright (c) 2020 SUSE LLC +# Copyright (c) 2022 SUSE LLC # # All modifications and additions to the file contributed by third parties # remain the property of their copyright owners, unless otherwise agreed @@ -17,14 +17,12 @@ Name: geolite2legacy -Version: 0+git20200101.56d8a4f +Version: 0+git20210326.05733c4 Release: 0 Summary: GeoLite2 (CSV) to Legacy format converter License: MIT URL: https://github.com/sherpya/geolite2legacy Source: %{name}-%{version}.tar.xz -Patch0: 0001-Automatically-determine-output-file-name.patch -Patch1: 0001-Use-realpath-to-determine-fips-file-location.patch Requires: python3-ipaddr BuildArch: noarch @@ -46,7 +44,8 @@ ln -s %{_datadir}/geolite2legacy/geolite2legacy.py %{buildroot}%{_bindir}/geolite2legacy %files -%doc README.md LICENSE +%license LICENSE +%doc README.md %{_bindir}/geolite2legacy %{_datadir}/geolite2legacy ++++++ _servicedata ++++++ --- /var/tmp/diff_new_pack.OimqsH/_old 2022-08-27 11:49:54.249819109 +0200 +++ /var/tmp/diff_new_pack.OimqsH/_new 2022-08-27 11:49:54.253819117 +0200 @@ -1,6 +1,6 @@ <servicedata> <service name="tar_scm"> <param name="url">https://github.com/sherpya/geolite2legacy.git</param> - <param name="changesrevision">56d8a4fe2491cf46ccd25c4ef89f4c93c09f3633</param></service></servicedata> + <param name="changesrevision">05733c435f0faa3e375bfa1c386695216e1b9e6a</param></service></servicedata> (No newline at EOF) ++++++ geolite2legacy-0+git20200101.56d8a4f.tar.xz -> geolite2legacy-0+git20210326.05733c4.tar.xz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/geolite2legacy-0+git20200101.56d8a4f/.gitignore new/geolite2legacy-0+git20210326.05733c4/.gitignore --- old/geolite2legacy-0+git20200101.56d8a4f/.gitignore 1970-01-01 01:00:00.000000000 +0100 +++ new/geolite2legacy-0+git20210326.05733c4/.gitignore 2021-03-26 14:51:40.000000000 +0100 @@ -0,0 +1,6 @@ +/venv +/.idea +*.dat +*.zip +*.pyc +__pycache__ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/geolite2legacy-0+git20200101.56d8a4f/Dockerfile new/geolite2legacy-0+git20210326.05733c4/Dockerfile --- old/geolite2legacy-0+git20200101.56d8a4f/Dockerfile 1970-01-01 01:00:00.000000000 +0100 +++ new/geolite2legacy-0+git20210326.05733c4/Dockerfile 2021-03-26 14:51:40.000000000 +0100 @@ -0,0 +1,10 @@ +FROM python:3.5-slim + +WORKDIR /geoip +COPY requirements.txt requirements.txt +RUN pip install -U pip && \ + pip install -r requirements.txt + +COPY *.py geoname2fips.csv ./ + +ENTRYPOINT ["./geolite2legacy.py"] diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/geolite2legacy-0+git20200101.56d8a4f/README.md new/geolite2legacy-0+git20210326.05733c4/README.md --- old/geolite2legacy-0+git20200101.56d8a4f/README.md 2020-01-01 20:34:55.000000000 +0100 +++ new/geolite2legacy-0+git20210326.05733c4/README.md 2021-03-26 14:51:40.000000000 +0100 @@ -14,7 +14,7 @@ - Processing may be slow, expecially for City blocks, consider using pypy, it is a lot faster - Some software may expect iso-8859-1 encoded names, but the script will output utf-8, - you can force a different encoding e.g. using `-e iso-8859-1` but some name may result wrong + you can force a different encoding e.g. using `-e iso-8859-1` but some name may result wrong ## Examples @@ -23,7 +23,7 @@ Database type Country - Blocks IPv4 wrote 306385-node trie with 300679 networks (251 distinct labels) in 8 seconds -# ./geolite2legacy.py -i GeoLite2-ASN-CSV.zip -o GeoIPASNum.dat +# ./geolite2legacy.py -i GeoLite2-ASN-CSV.zip -o GeoIPASNum.dat Database type ASN - Blocks IPv4 wrote 518484-node trie with 417952 networks (62896 distinct labels) in 15 seconds ``` @@ -48,6 +48,20 @@ -6, --ipv6 use ipv6 database ``` +## Run inside Docker container + +1. Build the Docker image: + +```bash +docker build -t geolite2legacy . +``` + +2. This command assmes that you have downloaded the GeoLite2 database to the current directory. + +```bash +docker run -it -v $(pwd):/src geolite2legacy:latest -i /src/GeoLite2-Country-CSV.zip -o /src/GeoIP.dat +``` + ```text The MIT License (MIT) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/geolite2legacy-0+git20200101.56d8a4f/geolite2legacy.py new/geolite2legacy-0+git20210326.05733c4/geolite2legacy.py --- old/geolite2legacy-0+git20200101.56d8a4f/geolite2legacy.py 2020-01-01 20:34:55.000000000 +0100 +++ new/geolite2legacy-0+git20210326.05733c4/geolite2legacy.py 2021-03-26 14:51:40.000000000 +0100 @@ -30,10 +30,16 @@ import csv import struct import codecs -import ipaddr import logging import argparse +# python 2 ipaddr +try: + from ipaddr import IPNetwork +# python3 built-in ipaddress +except: + from ipaddress import ip_network as IPNetwork + from time import time from zipfile import ZipFile from collections import defaultdict @@ -106,7 +112,12 @@ def __setitem__(self, net, data): self.netcount += 1 - inet = int(net) + # python2 ipaddr + try: + inet = int(net) + # python3 built-in ipaddress + except: + inet = int(net.network_address) node = self.segments[0] for depth in range(self.seek_depth, self.seek_depth - (net.prefixlen - 1), -1): if inet & (1 << depth): @@ -214,14 +225,14 @@ def gen_nets(self, locations, infile): for row in csv.DictReader(infile): - nets = [ipaddr.IPNetwork(row['network'])] + nets = [IPNetwork(row['network'])] org = decode_text(row['autonomous_system_organization']) asn = row['autonomous_system_number'] entry = u'AS{} {}'.format(asn, org) yield nets, (serialize_text(entry),) def encode(self, data): - return data + b'\0' + return data + b'\0\0\0' class ASNv6RadixTree(ASNRadixTree): @@ -243,7 +254,7 @@ if location is None: continue - nets = [ipaddr.IPNetwork(row['network'])] + nets = [IPNetwork(row['network'])] country_iso_code = location['country_iso_code'] or location['continent_code'] fips_code = geoname2fips.get(location['geoname_id']) if fips_code is None: @@ -305,7 +316,7 @@ if location is None: continue - nets = [ipaddr.IPNetwork(row['network'])] + nets = [IPNetwork(row['network'])] country_iso_code = location['country_iso_code'] or location['continent_code'] yield nets, (country_iso_code,) @@ -339,7 +350,7 @@ f.write(self.serialize_node(node.rhs)) f.write(struct.pack('B', 0x00) * 3) - f.write(b'geolite2legacy.py') # .dat file comment - can be anything + f.write(datfilecomment.encode('ascii')) # .dat file comment - can be anything f.write(struct.pack('B', 0xff) * 3) f.write(struct.pack('B', self.edition)) f.write(self.encode_rec(len(self.segments), self.segreclen)) @@ -358,6 +369,12 @@ 'ASN': {'IPv4': ASNRadixTree, 'IPv6': ASNv6RadixTree} } +Filenames = { + 'Country': {'IPv4': "GeoIP.dat", 'IPv6': "GeoIPv6.dat"}, + 'City': {'IPv4': "GeoIPCity.dat", 'IPv6': "GeoIPCityv6.dat"}, + 'ASN': {'IPv4': "GeoIPASNum.dat", 'IPv6': "GeoIPASNumv6.dat"} +} + def parse_fips(fipsfile): with open(fipsfile) as f: @@ -371,7 +388,7 @@ parser = argparse.ArgumentParser() parser.add_argument('-i', '--input-file', required=True, help='input zip file containings csv databases') - parser.add_argument('-o', '--output-file', required=True, help='output GeoIP dat file') + parser.add_argument('-o', '--output-file', help='output GeoIP dat file') parser.add_argument('-f', '--fips-file', help='geonameid to fips code mappings') parser.add_argument('-e', '--encoding', help='encoding to use for the output rather than utf-8') parser.add_argument('-d', '--debug', action='store_true', default=False, help='debug mode') @@ -433,7 +450,7 @@ sys.exit(1) if dbtype != 'ASN': - fips_file = opts.fips_file or os.path.join(os.path.dirname(__file__), 'geoname2fips.csv') + fips_file = opts.fips_file or os.path.join(os.path.dirname(os.path.realpath(__file__)), 'geoname2fips.csv') parse_fips(fips_file) tstart = time() @@ -441,6 +458,10 @@ r.load(locs, TextIOWrapper(ziparchive.open(blocks, 'r'), encoding='utf-8')) + if not opts.output_file: + opts.output_file = Filenames[dbtype][opts.ipv6] + print('Output file {}'.format(opts.output_file)) + with open(opts.output_file, 'wb') as output: r.serialize(output) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/geolite2legacy-0+git20200101.56d8a4f/geoname2fips.py new/geolite2legacy-0+git20210326.05733c4/geoname2fips.py --- old/geolite2legacy-0+git20200101.56d8a4f/geoname2fips.py 2020-01-01 20:34:55.000000000 +0100 +++ new/geolite2legacy-0+git20210326.05733c4/geoname2fips.py 2021-03-26 14:51:40.000000000 +0100 @@ -91,7 +91,7 @@ 'COCOS ISLANDS': 'CK', 'COLOMBIA': 'CO', 'COMOROS': 'CN', - 'CONGO': 'CG', + 'DR CONGO': 'CG', # renamed from 'CONGO' in GeoLite2-City-CSV_20200505 'COOK ISLANDS': 'CW', 'CORAL SEA ISLANDS': 'CR', 'COSTA RICA': 'CS', @@ -100,7 +100,7 @@ 'CUBA': 'CU', 'CYPRUS': 'CY', 'CZECHIA': 'EZ', - 'DEMOCRATIC REPUBLIC OF TIMOR LESTE': 'TT', + 'EAST TIMOR': 'TT', # renamed from 'DEMOCRATIC REPUBLIC OF TIMOR LESTE' in GeoLite2-City-CSV_20200505 'DENMARK': 'DA', 'DHEKELIA SOVEREIGN BASE AREA': 'DX', 'DJIBOUTI': 'DJ', @@ -180,7 +180,7 @@ 'LIECHTENSTEIN': 'LS', 'LUXEMBOURG': 'LU', 'MACAO': 'MC', - 'MACEDONIA': 'MK', + 'NORTH MACEDONIA': 'MK', # before 2019 'MACEDONIA' 'MADAGASCAR': 'MA', 'MALAWI': 'MI', 'MALAYSIA': 'MY', @@ -232,10 +232,10 @@ 'PORTUGAL': 'PO', 'PUERTO RICO': 'RQ', 'QATAR': 'QA', - 'REPUBLIC OF KOREA': 'KS', + 'SOUTH KOREA': 'KS', # renamed from 'REPUBLIC OF KOREA' in GeoLite2-City-CSV_20200505 'REPUBLIC OF LITHUANIA': 'LH', 'REPUBLIC OF MOLDOVA': 'MD', - 'REPUBLIC OF CONGO': 'CF', + 'CONGO REPUBLIC': 'CF', # renamed from 'REPUBLIC OF CONGO' in GeoLite2-City-CSV_20200505 'ROMANIA': 'RO', 'RUSSIA': 'RS', 'RWANDA': 'RW', @@ -320,7 +320,7 @@ # empty in fips csv 'AMERICAN SAMOA', 'COOK ISLANDS', - 'DEMOCRATIC REPUBLIC OF TIMOR LESTE', + 'EAST TIMOR', 'FRENCH POLYNESIA', 'FRENCH SOUTHERN TERRITORIES', 'HONG KONG', @@ -355,7 +355,7 @@ 'LIBERIA', 'LIBYA', 'LUXEMBOURG', - 'MACEDONIA', + 'NORTH MACEDONIA', 'MALAWI', 'MAURITANIA', 'MEXICO', @@ -372,8 +372,8 @@ 'PHILIPPINES', 'POLAND', 'QATAR', - 'REPUBLIC OF CONGO', - 'REPUBLIC OF KOREA', + 'CONGO REPUBLIC', + 'SOUTH KOREA', 'REPUBLIC OF LITHUANIA', 'REPUBLIC OF MOLDOVA', 'RUSSIA', @@ -405,7 +405,7 @@ 'CZECHIA': ('CZECHIA',), # city? - 'CONGO': ('BOENDE',), + 'DR CONGO': ('BOENDE',), 'ESWATINI': ('ESWATINI',), # city? } @@ -566,7 +566,7 @@ 'BE': { 'BRABANT WALLON': 'WALLONIA', 'BRUSSELS HOOFDSTEDELIJK GEWEST/REGION DE BRUXELLES CAPITALE': 'BRUSSELS CAPITAL', - 'VLAMMS BRABANT': 'FLANDERS' + 'VLAAMS BRABANT': 'FLANDERS' }, 'BF': { # 'FREEPORT': 'CITY OF FREEPORT', @@ -1300,9 +1300,9 @@ if __name__ == '__main__': parser = argparse.ArgumentParser() - parser.add_argument('-i', '--input-file', required=True, help='input csv fips 10-4 data file') - parser.add_argument('-o', '--output-file', required=True, help='output csv file') - parser.add_argument('-l', '--location-file', required=True, help='location file csv') + parser.add_argument('-i', '--input-file', required=True, help='input csv fips 10-4 data file (data.csv)') + parser.add_argument('-o', '--output-file', required=True, help='output csv file (geoname2fips.csv)') + parser.add_argument('-l', '--location-file', required=True, help='location file csv (GeoLite2-City-Locations-en.csv)') opts = parser.parse_args() geoid2fips = {} diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/geolite2legacy-0+git20200101.56d8a4f/requirements.txt new/geolite2legacy-0+git20210326.05733c4/requirements.txt --- old/geolite2legacy-0+git20200101.56d8a4f/requirements.txt 2020-01-01 20:34:55.000000000 +0100 +++ new/geolite2legacy-0+git20210326.05733c4/requirements.txt 2021-03-26 14:51:40.000000000 +0100 @@ -1 +1 @@ -ipaddr +ipaddr==2.2.0; python_version < '3.3'
