Have you ever used SQLite before? 10M records is not a big deal.
With experience it should take you less than hour to prove this one way or the other. But that depends on you having experience with databases (indexes) and SQLite in particular (WAL mode, exclusive locking, etc). Without experience it will take you a day or two. Your XML files will benefit from disk cache. SQLite will benefit from disk cache plus it's own cache. You'll benefit from less overhead due to XML verbosity which will improve everything. If you're not writing to these files then you could put your data in a memory DB which would be faster yet. But are you sure you need a database at all? What kind of queries are you doing? What does your data look like? Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate ________________________________ From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Sumit Gupta [gamersu...@gmail.com] Sent: Thursday, August 11, 2011 7:14 AM To: sqlite-users@sqlite.org Subject: EXT :[sqlite] SQLite with 10M record Hello, I am developing an application using .NET 4.0, and we need to store about 10M entries to let our system work on them. Currently we are using XML files to store those value [in multiple XMLFile tokeep XML size small] and later load one of XML File based on parameter. It took our software about 30 minute to write those entries in XML and then software works Okay when retrieving information. Now my Question is, does it make sense to use SQLite instead of XML? Can Sqlite is able to hold 10M records and parse them in better speed ? If not in One table, maybe I should make multiple Table [with same logic as in XML] if it reduce storage or processing time? I never use SQlite and now sure if it is good to use or not. Any suggestion is welcome. Other ground information, we are going to use this software on PC having Windows XP or 7 with 4 GB RAM etc. So no hardware issue. Our XML method take about 1 GB Storage on harddisk for data, mainly due to overhead of XML itself. I am eying to have Harddisk usage of about 300-400MB with use of SQLite if possible. With Regards, Sumit Gupta _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
