Jaco Breitenbach <jjbreitenb...@gmail.com> wrote: > I am building an application that filters out duplicate input data by > generating an MD5 hash of each input, and implicitly comparing that against > a set of keys already stored in the SQLite database by doing an insert into > a unique-indexed table. If the insert fails, a duplicate is assumed, > otherwise the new unique key is stored, and the input processed. > > The problem that I'm facing, is that I would ultimately need to process > 1,000,000,000 records a day, with history to be kept for up to 128 days. I > am currently creating a new data file per day, with hourly tables.
Doesn't that defeat the point? Wouldn't that only guarantee uniqueness within the last hour? -- Igor Tandetnik _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
