Although I think you'll pick up speed going to a database my philosophy is "if it ain't broke don't fix it".
If you're speed is OK now why worry about the organization? Putting it in a database only gives you more chance for corruption which could hose your entire database. If I were you I'd just memory map the whole thing rather than using XML or a database since you have a fixed record size. That's also the most efficient storage. Then you just do a binary search on the time. You can use a C++ boost memory map class to do that. http://www.boost.org/doc/libs/1_38_0/doc/html/interprocess/sharedmemorybetweenprocesses.html This would be blazingly fast. Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate ________________________________ From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Sumit Gupta [gamersu...@gmail.com] Sent: Thursday, August 11, 2011 7:28 AM To: 'General Discussion of SQLite Database' Subject: EXT :Re: [sqlite] SQLite with 10M record Hello, I have Experience with SQL Server and MySql, but not with SQLite. Since this application will run on Desktop Machine and Data it store is kind a Temporary, so I think Locking is not needed at all [as this reduce processing time for application]. The data is actually Reading on 48 different parameter for every second of day for 30 days. So it is like 10-10-2011 18:48:42, 10,20, 30, 40, 50, 60, 80 .. <48 such entries> And we need to query on Time Only, i.e. when we create say PDF we find average or Sum of all entry [rows and column] and find values etc. That part is already working in great speed so I am not worried there. Just that I need to organize application data better. With Regards, Sumit Gupta -----Original Message----- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Black, Michael (IS) Sent: 11 August 2011 17:54 To: General Discussion of SQLite Database Subject: Re: [sqlite] SQLite with 10M record Have you ever used SQLite before? 10M records is not a big deal. With experience it should take you less than hour to prove this one way or the other. But that depends on you having experience with databases (indexes) and SQLite in particular (WAL mode, exclusive locking, etc). Without experience it will take you a day or two. Your XML files will benefit from disk cache. SQLite will benefit from disk cache plus it's own cache. You'll benefit from less overhead due to XML verbosity which will improve everything. If you're not writing to these files then you could put your data in a memory DB which would be faster yet. But are you sure you need a database at all? What kind of queries are you doing? What does your data look like? Michael D. Black Senior Scientist NG Information Systems Advanced Analytics Directorate ________________________________ From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Sumit Gupta [gamersu...@gmail.com] Sent: Thursday, August 11, 2011 7:14 AM To: sqlite-users@sqlite.org Subject: EXT :[sqlite] SQLite with 10M record Hello, I am developing an application using .NET 4.0, and we need to store about 10M entries to let our system work on them. Currently we are using XML files to store those value [in multiple XMLFile tokeep XML size small] and later load one of XML File based on parameter. It took our software about 30 minute to write those entries in XML and then software works Okay when retrieving information. Now my Question is, does it make sense to use SQLite instead of XML? Can Sqlite is able to hold 10M records and parse them in better speed ? If not in One table, maybe I should make multiple Table [with same logic as in XML] if it reduce storage or processing time? I never use SQlite and now sure if it is good to use or not. Any suggestion is welcome. Other ground information, we are going to use this software on PC having Windows XP or 7 with 4 GB RAM etc. So no hardware issue. Our XML method take about 1 GB Storage on harddisk for data, mainly due to overhead of XML itself. I am eying to have Harddisk usage of about 300-400MB with use of SQLite if possible. With Regards, Sumit Gupta _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
