On Thu, Apr 16, 2015 at 4:53 PM, Karsten Loesing <kars...@torproject.org> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 15/04/15 21:18, nusenu wrote: >> Hi, >> >> I'm planing to store relay data in a database for analysis. I >> assume others have done so as well, so before going ahead and >> designing a db schema I'd like to make sure I didn't miss >> pre-existing db schemas one could build on. >> >> Data to be stored: - (most) descriptor fields - everything that >> onionoo provides in a details record (geoip, asn, rdns, tordnsel, >> cw, ...) - historic records >> >> I didn't find something matching so far, so I'll go ahead, but if >> you know of other existing relay db schemas I'd like to hear about >> it. >> >> thanks, nusenu >> >> >> >> >> "GSoC2013: Searchable Tor descriptor archive" (Kostas Jakeliunas) >> https://www.google-melange.com/gsoc/project/details/google/gsoc2013/wfn/ >> >> > 5866452879933440 >> https://lists.torproject.org/pipermail/tor-dev/2013-May/004923.html >> >> > https://lists.torproject.org/pipermail/tor-dev/2013-September/005357.htm >> l https://github.com/wfn/torsearch (btw, someone knows the license >> of this?) > > Cc'ing Kostas for this question.
Hi nusenu, I've been going through old mail, and on 2015-04-16 you asked about about a license (see above). Just added a LICENSE file - can't hurt (standard BSD 3-clause). If you're still by any chance collating (ha) and/or want to talk about schema design for descriptors (I personally would not lose hope for RDBMSes for large datasets - not until one gets into *actually* big data - say, terabytes at least, or more - but of course it gets nuanced real fast). -- Kostas. 0x0e5dce45 @ pgp.mit.edu > >>> This is true: the summary/details documents (just like in Onionoo >>> proper) deal with the *last* known info about relays. >> >> >> ernie >> https://gitweb.torproject.org/metrics-db.git/plain/doc/manual.pdf >> (didn't find db/tordir.sql mentioned in the pdf) > > That file lives here now: > > https://gitweb.torproject.org/metrics-web.git/tree/modules/legacy/db/tordir.sql > > A better schema might be the following one though. It's smaller, but > it's better documented: > > https://gitweb.torproject.org/exonerator.git/tree/db/exonerator.sql > >> "Instructions for setting up relay descriptor database" >> https://lists.torproject.org/pipermail/tor-dev/2010-March/001783.html > > That's >> > five years old. I'd say ignore that one. > >> "Set up descriptor database for other researchers" >> https://trac.torproject.org/projects/tor/ticket/1643 > > Also five years old. Better ignore. > > Hope that helps. > > All the best, > Karsten > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > Comment: GPGTools - http://gpgtools.org > > iQEcBAEBAgAGBQJVL9rcAAoJEJD5dJfVqbCrFZgIAIEv/Yi4sNoa8clYVAxuk0Sh > FFbRDT0kLs19t/DgTwUtB6jD4Lh0akMc806AaIFgfCdL+QwcG0llBfZnSsrbszoH > Xoi226PRx9lPITrA7KYds4PUZfqIqg3ECpNsKNa4PLB7SlQdNfJQ1wDngcwu2CrF > Hk+zHbu0gfSkfZRBqxt5aJLTFXR0aBYybF4d6sPJ4OW5Al2U8r9DYysXc0xALvwq > bvEDFctV1wkDgA3mP3guRrXImXYT1AQPFFlz0TR1eBruuSJBiPKIv7Fs/ocns4aR > OhxIEaKBaAO+HkvyxDcZ1ukXldR13s3MUPD0XvvZ8xQRCBZpNMygqTMi6pIjTN4= > =a0Nb > -----END PGP SIGNATURE----- _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev