Allan, Thanks for your reply. I'm new to SQLite, but have used a similar list of databases (with an emphasis on server vs. client side databases) professionally for years. My background is designing and building enterprise BI, ETL, and data warehouse systems using databases like Oracle, DB2, SQL Server and ETL tools like Informatica, Ab Initio, BO Data Integrator, etc.
My goal is to be able to offer our customers cost effective, high performance alternatives to traditional commercial ETL solutions. We recently re-implemented a mid-sized Oracle/Informatica based ETL system in Python using in memory data structures (dicts) and improved the performance by a factor of 20x. We're now looking to expand this approach to SQLite for environments where the volume of data is greater than available memory. > When you talk about performance comparisons your question really needs to be > squared up to what the database is used for. Are you on a server, > workstation, or embedded? How large in the database? How are the internal > algos of the database engine lined up? What is the memory footprint? How did > you configure what is configurable in the database? I'm interested in exploring whether or not SQLite can be used as an ETL tool with large data sets (~80+ Gb). In this context, SQLite would be run on 64-bit Intel servers with lots of RAM (16-64 Gb). The data would be stored/processed on server local SCSI drives vs. located on a SAN. File access would be via a single process per SQLite database. The interface language would most likely be Python. > The reality is I BET and I would love to know myself Sqlite is just as fast > on inserts to the same hard drive as oracle. I would love to see such a benchmark as well. > Are you some kind of political Oracle covering biggot? LOL. No. I try to be as database agnostic as possible. > Or, do you want another chance to post something that states you are trying > to find the best data storage solution to deliver sincere value to your > client in terms of a database choice? Subtle point here ... I'm not looking for a long term data storage solution - I'm exploring the possibility of using SQLite as a way to validate, transform, and pre-aggregate raw data that would in turn be exported to raw text files and imported (in final form) to a customer's data warehouse. > Sometimes that choice is Sqlite, but in some cases it is Oracle instead. Agreed. Regards, Malcolm _______________________________________________ sqlite-users mailing list [email protected] http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
