Hello, I have Experience with SQL Server and MySql, but not with SQLite. Since this application will run on Desktop Machine and Data it store is kind a Temporary, so I think Locking is not needed at all [as this reduce processing time for application].
The data is actually Reading on 48 different parameter for every second of day for 30 days. So it is like 10-10-2011 18:48:42, 10,20, 30, 40, 50, 60, 80 .. <48 such entries> And we need to query on Time Only, i.e. when we create say PDF we find average or Sum of all entry [rows and column] and find values etc. That part is already working in great speed so I am not worried there. Just that I need to organize application data better. With Regards, Sumit Gupta -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Black, Michael (IS) Sent: 11 August 2011 17:54 To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite with 10M record Have you ever used SQLite before? 10M records is not a big deal. With experience it should take you less than hour to prove this one way or the other. But that depends on you having experience with databases (indexes) and SQLite in particular (WAL mode, exclusive locking, etc). Without experience it will take you a day or two. Your XML files will benefit from disk cache. SQLite will benefit from disk cache plus it's own cache. You'll benefit from less overhead due to XML verbosity which will improve everything. If you're not writing to these files then you could put your data in a memory DB which would be faster yet. But are you sure you need a database at all? What kind of queries are you doing? What does your data look like? Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate ________________________________ From: [email protected] [[email protected]] on behalf of Sumit Gupta [[email protected]] Sent: Thursday, August 11, 2011 7:14 AM To: [email protected] Subject: EXT :[sqlite] SQLite with 10M record Hello, I am developing an application using .NET 4.0, and we need to store about 10M entries to let our system work on them. Currently we are using XML files to store those value [in multiple XMLFile tokeep XML size small] and later load one of XML File based on parameter. It took our software about 30 minute to write those entries in XML and then software works Okay when retrieving information. Now my Question is, does it make sense to use SQLite instead of XML? Can Sqlite is able to hold 10M records and parse them in better speed ? If not in One table, maybe I should make multiple Table [with same logic as in XML] if it reduce storage or processing time? I never use SQlite and now sure if it is good to use or not. Any suggestion is welcome. Other ground information, we are going to use this software on PC having Windows XP or 7 with 4 GB RAM etc. So no hardware issue. Our XML method take about 1 GB Storage on harddisk for data, mainly due to overhead of XML itself. I am eying to have Harddisk usage of about 300-400MB with use of SQLite if possible. With Regards, Sumit Gupta _______________________________________________ sqlite-users mailing list [email protected] http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list [email protected] http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list [email protected] http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
