On Wed, Jun 10, 2015 at 01:51:06PM +0200, Peter Pöml wrote: > On Fri, May 08, 2015 at 08:43:13 +0200, Michael Meier wrote: > > > bzcat rib.20150508.0000.bz2 | ./zebra-dump-parser.pl | > > > ./filteruniqueprefixes.pl > /tmp/v6routedump.dat > > > asn_get_routeviews > /tmp/v4routedump.dat > > > cat /tmp/v6routedump.dat /tmp/v4routedump.dat | asn_import.py [...] > That's all great. I'm only struggling to package it in "digestible" > form, so the existing users don't run into problems. Unfortunately, the > old mod_asn SQL query doesn't work with the new database schema. > I think I might add the new column alongside the old one first and > duplicate the data, so that mod_asn can continue running until it's updated > as well.
Trying to simplify importing the new IPv6 data source, by using the existing scripts, I now provide http://mirrorbrain.org/routeviews/ipv6-rib-snapshot-latest.dat.bz2 (nightly updated snapshot from routeviews /route-views6/bgpdata/%Y.%m/RIBS/) and also http://mirrorbrain.org/routeviews/ipv6-rib-snapshot-latest.txt.bz2 ^^^ which is the same, as textfile, parsed by zebra-dump-parser.pl by Marco d'Itri. The zebra dump parser can be switched to output the data in a format including neighbouring AS numbers; I chose that format because IPv4 data comes that way, and the neighbour AS could be imported into the database one day, which is easy to do and I think could be quite interesting. I extended asn_get_routeviews to download the new data source as well, and to filter it similar to filteruniqueprefixes.pl. (The result verified to be identical.) I merged your asn_import changes regarding the AS number validity check. I didn't merge the whole pull request though, since I solved few things differently in the end. Hope you don't mind. The result is, for the system administrator, that the scripts can be used as before, and no changes are necessary. Hence, after you guys did the clever work, I have hopefully completed the dump job as well now; or at least one of them ;-) > The helper tools used in this and the slightly modified version of > asn_import.py that I used can be found in > https://github.com/poempelfox/mod_asn/tree/ipv6support > While this works, it could need some more streamlining, e.g.: > - providing a download-tool like for the IPv4 data check > - providing a mirror of the data used on mirrorbrain.org/routeviews/ check > - install the new helper tool(s) in packaging check > - perhaps merging it all together into asn_get_routeviews.py check > It would really be nice if this could be integrated into mirrorbrain. Missing: logic for upgrading the database, possibly without disruption. Peter
signature.asc
Description: Digital signature