I am currently using SQLite to process @ 400 million records (and climbing) a day by reading files, importing them into SQLite, and summarizing. The summed data goes into Oracle. This is a production application that is very stable. Holding the data in SQLite in memory as opposed to a C struct reduced development time and makes it easy to change the summarization output by just modifying the SQL statements.
If you think the data layouts may be changed in the future or even dynamic / "defined on the fly" then SQLite would be a good tool. ----- Original Message ----- From: "Lorenzo Jorquera" <[EMAIL PROTECTED]> To: <sqlite-users@sqlite.org> Sent: Monday, June 27, 2005 4:55 PM Subject: [sqlite] using sqlite as a temporary database to process lots of data Hi, My company is starting a project in wich we will have to process large amounts of data and save the results to a db2 database. Because this data process needs to be very fast, the idea is to load the data in memory and apply the transformations in C++. I think that sqlite inmemory databases could be a good alternative to this, but when I proposed this solution to my boss, he asked if someone has already used sqlite for this purpose. I found that this use of sqlite is cited in the wiki, but I can't find any concrete examples. If any of you can share some success histories with me, I will be very grateful. Thanks in advance, Lorenzo. PD: please, forgive my english, it is a little rusty...
