"David Allen" <[EMAIL PROTECTED]> writes: > I heard through the grapevine (IRC) that all watchme logs are now being > piped into a postgresql database. It also seems that the regular log files > aren't available through hawk, and that the nightly log archives are gone as > well. > This is all true.
> Did somebody just modify the daemon to put the log data into the database > rather than spit them out in ip-address-as-filename format? > I did. The problem is that parsing all the logs just to show the data on one request is jooky. I would have preferred to do a one-file-per-request format and just re-arrange the logs like that, and even went and coded that all up, but I quickly found out that the ext2/ext3 filesystem isn't efficient enough at storing small files to handle tens of thousands, even when I put them in directories. So I made up a database schema and moves everything over to that. > I had been mirroring the log archives at > http://www.oldhat.org/freenet/watchme-logs/ but I've stopped now. I was > doing that because hawk wasn't keeping copies of all of the old logs - have > you guys put more disk space out there to hold onto everything, or is the > database just keeping one day's worth of information at a time, or what's > going on? > I'm planning on doing consistent dumps of the entire contents of the database right before resetting it each night; that way the database stays efficient (tens of thousands of rows is a bit much for even postgresql), and we have nice dump files for people to download. > Is there any way that some people may get access to that database? I was > rather hoping that all of the watchme test data would remain public. > > M. David Allen I have to be careful with anything involving deleting data out of the database, so excuse me if I don't just rush to make this happen. I've still got more work to see if I can make things more efficient; the watchme nodes are pouring a lot of data on hawk. I may end up ignoring data involving Void messages (which are basically useless to log) and FCP client handshaking. I've considered moving to a different structure for processing messages involving multiple tables that a single message would move between as more information on the request it was involved in arrives. But that's for the future. Thelema -- E-mail: [EMAIL PROTECTED] Raabu and Piisu GPG 1024D/36352AAB fpr:756D F615 B4F3 BFFC 02C7 84B7 D8D7 6ECE 3635 2AAB _______________________________________________ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
