Hill, Brett wrote: > Not sure how much more overhead it would require, but his > other idea also sounds pretty nifty (regarding > non-duplication of items in the spam/notspam folders based on > checksums, but I wouldn't include the errors folders for > obvious reasons). I can definitely see how this would be > beneficial for those times that you get 100-200 of the same > spam email at a time, for preventing corpus corruption.
I just had an idea of how this could work. Instead of overwriting existing content with the new content right away, you could create an interim folder where all new incoming spams go to. Then, when rebuildspamdb.pl runs, it can go through the emails in just that one folder calculating hashes, writing the hashes to a file (if needed), then deleting the duplicate files based on the hashes (you could even add the option to keep more than one if you desire, but not sure why you would). Next, the remaining files are then integrated into the corpus overwriting some of the existing content(so what if there's a few duplicates out there, it's the 100-200 that we want to keep out). Then do your maxfile subtractions . Finally, the database rebuild portion runs. I guess I don't see this being much of a problem with the notspam folder, but if deemed so, the same process could be done there too adding a second interim folder for ham). I guess this is all based on the feasibility of getting accurate hashes as described by GrayHat. Kind Regards, Brett ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Assp-test mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/assp-test
